Predicting TB treatment outcomes using baseline risk and treatment response markers: developing the PredictTB early treatment completion criteria

Background

Multiple clinical trials over the last 40 years attempting to shorten therapy for pulmonary, drug-sensitive tuberculosis from six to four months have failed^1–4. Despite this, the various four-month treatment arms consistently cured about 80–85% of patients. The premise of the ongoing PredictTB clinical trial is that this sub-population of lower risk patients who are cured at four months can be identified prospectively⁵. Two subsequent secondary analyses of the three fluoroquinolone treatment shortening trials^2–4 support this premise by finding that a subset of lower risk participants, those without cavity on baseline chest-x-ray, acid-fast bacilli (AFB) smear less than 2+ at baseline, or AFB smear negative at month 2 of treatment were associated with successful four-month treatment outcomes^6,7. Another trial prospectively attempted a similar risk stratification by shortening treatment for those without cavity on baseline chest x-ray, a measure of baseline risk, and those with negative sputum cultures at two months of treatment, a measure of treatment response⁸. Although this trial also failed, the treatment success rate in the four-month arm achieved 93%, higher than other four-month treatment trials that did not risk stratify participants. The PredictTB trial was developed on the hypothesis that more precise methods of evaluating baseline risk and treatment response will successfully identify lower risk participants cured with four months of treatment.

The value of early chest x-ray changes to predict TB treatment outcomes has been recognized for over 60 years⁹. Cavities on baseline chest x-ray^10–12 and residual cavity at cure¹³ have been associated with poor treatment outcomes. Computed tomography scans are more sensitive than x-ray. Further, changes on 2-deoxy-2-[¹⁸F]fluoro-D-glucose (FDG)-positron emission tomography/computed tomography (PET/CT) scans have been associated with treatment outcomes in nonhuman primates^14,15. Results in patients have been limited to small numbers due to the difficulty of collecting PET/CT imaging in this setting. However, a study of patients with pulmonary, multidrug-resistant tuberculosis (MDR-TB) showed a relationship with long-term treatment outcomes¹⁶. In drug sensitive TB (DS-TB), distinct response patterns on PET/CT scan from baseline to month 6 on treatment correlated with treatment outcomes¹⁷.

Month 2 sputum culture conversion has traditionally been considered the best biomarker of treatment outcome available¹⁸ despite a meta-analysis demonstrating poor sensitivity and specificity in predicting outcomes¹⁹. This has been confirmed by additional analyses of the REMoxTB treatment shortening trial², where month 2 culture conversion status poorly predicted final outcomes²⁰. Time-delays associated with culture conversion tests can be as long as 6-8 weeks, resulting in delayed assessments of the patient’s bacillary status. Immediately available test results are likely to predict outcomes better than delayed results because the results reflect the patient’s current status rather than a historical status from 1-2 months ago and thus poor results can be acted upon immediately. The value of contemporaneous results has been shown with CD4 cell counts in HIV²¹, as well as in TB where a month 6 culture conversion status predicted final outcomes significantly better than month 2 culture conversion status²². A point-of-care test that measures TB bacterial load has the potential to predict outcomes better than a delayed culture conversion result.

The aim of this analysis is to describe how the early treatment completion criteria used in the ongoing PredictTB clinical trial (NIH IRB #16IN133; NCT02821832) were developed. In this analysis, we assess the relationship between measures of baseline risk and treatment response with poor treatment outcomes in a cohort of pulmonary DS-TB patients treated programmatically in South Africa. Baseline risk is assessed using quantitative markers from FDG-PET/CT scans at baseline. Early treatment response is assessed by changes in these markers on FDG-PET/CT scan at week 4 of treatment. Adherence is also assessed, as is a quantitative measure of residual bacterial load using sputum Xpert MTB/RIF cycle threshold at week 16.

Methods

Results

Among the 92 patients that we analyzed from the Catalysis cohort, 73 were cured (asymptomatic two years after the end of treatment), eight failed treatment, and 11 programmatically restarted TB treatment during follow-up. For the PET/CT imaging analysis, the baseline and week 4 PET/CT scans were each read by a single reader, with overall summary statistics presented in Table 1. At baseline, cured patients were significantly different from treatment failure patients in CT cavity air volume, with CT hard volume and PET TLG differences being borderline significant (P=0.059 for both). At week 4, only the difference in total cavity air remained significantly different. In contrast, the results for patients who were cures and retreatments were not significantly different from each other in any parameter at baseline or at week 4, making it very difficult to differentiate these two cohorts using these parameters. The comparison of treatment failures with retreatments was similar to that of treatment failures with cures. We therefore developed our criteria based primarily on differences between the cured and failure cohorts.

To identify specific thresholds that predicted cure vs treatment failure, ROC curves were drawn for each variable (Figure 1). When using the optimal ROC thresholds of all baseline and week 4 PET/CT criteria into a single algorithm, the combined criteria predicted cure with 100% sensitivity, capturing all eight failures and 11 retreatments as high risk. However, specificity was very poor at 19.2% and only 14/81 (17.3%) of subjects with both baseline and week 4 PET/CT scans were classified as low risk (Table 2). This is well below our target of 50%, classifying too many as false positive high risk, resulting in an algorithm that is neither practical nor scalable. We therefore adjusted the thresholds to be more specific at the cost of sensitivity to approach our 50% target.

Figure 1. ROC curves of radiological biomarkers in predicting failure vs. cure.

Panel A–D: baseline; Panel E–H: %change of week 4 from baseline; AUCs along with 95% CIs were added as blue text at the bottom of each plot; optimal threshold and the corresponding sensitivity and specificity were labeled on the curve. ROC, receiver operating characteristic; AUC, area-under-the-curve; CI, confidence intervals; TLG, total lesion glycolysis.

Because prior clinical trials data already demonstrated that baseline cavity was a risk factor for poor outcomes^8,10, and because cavity size was the strongest predictor of poor outcome in our ROC curves (cavity air area-under-the-curve (AUC) > CT hard volume and PET TLG AUCs), we built the algorithm around this parameter first. We defined cavity as largest single cavity size rather than total cavity size because we posited that a single large cavity induced higher risk and may take longer to heal than multiple smaller cavities (i.e. one 30 mL cavity had a higher risk for poor outcome than two 15 mL cavities). In examining the baseline cavity size threshold, a 10.5 mL threshold captured 6/8 (75%) of treatment failures as high risk but only 46/73 (63.0%) cures as low risk. By increasing the threshold to 30 mL, the algorithm would miss one additional failure (now only 5/8 [62.5%] as high risk; increasing beyond 30 mL would lose more than one additional failure) but specificity would increase to 86.3% (63/73 cures now classified as low risk; Table 3). Overall, using 30 mL instead of 10.5 mL as the baseline cavity threshold increases the proportion of cured (N=73) and failure (N=8) patients defined as low risk from 48/81 (59.3%) to 66/81 (81.5%). Changing the week 4 cavity volume reduction threshold from 64% to 20% results in a similar sensitivity/specificity tradeoff. After applying both baseline and week 4 cavity change thresholds, only 29/81 (35.8%) would have been classified as low risk (before adding any other imaging criteria) using the 64% threshold but 58/81 (71.6%) were low risk with the 20% cavity reduction threshold (Table 3). In contrast to prior data on the risk from baseline cavities, quantitation of CT disease volumes and PET TLG has not previously been validated. Therefore, weighing cavity size to threshold about half (28.4% of total) of the target 50% as high risk seemed appropriate, allowing the remaining criteria (CT hard volume, PET TLG, Xpert cycle threshold, and adherence) to threshold the other half.

Similar to the cavity size thresholds, the cutoffs for hard CT volume and PET TLG were adjusted by decreasing sensitivity but increasing specificity from the optimal ROC parameters (Figure 1, Table 2) to stratify about 50% of the total cohort as low risk, aiming to capture as many treatment failure and retreatment patients as possible within the 50%. Figure 2 demonstrates the patient stratifications when baseline CT hard volume <200 mL and TLG <1500 units were used as low risk criteria at baseline. When combined with largest cavity air <30 mL, 60/81 (74.1%) patients were classified as low risk at baseline (Table 4). When applied to the entire Catalysis cohort, including the retreatment patients, 6/8 (75%) failures but only 1/11 retreatments were classified as high risk at baseline, suggesting that treatment failure patients may be more correlated with severity of baseline disease compared to retreatment TB patients. Week 4 change criteria for CT hard volume and PET TLG were similarly adjusted and allowed for slight increases to account for potential paradoxical treatment responses that were ultimately still favorable. The final week 4 criteria selected allowed up to a 10% increase in hard volume and 30% increase in TLG at week 4 to remain low risk, resulting in 46/81 (56.8%) classified as low risk after applying both baseline and week 4 PET/CT criteria (Table 4). The week 4 criteria captured only one additional failure but two additional retreatments, suggesting that retreatment TB patients may be more correlated with poor treatment response at week 4 rather than severity of disease at baseline.

Figure 2. Hard volume vs. total lesion glycolysis (TLG).

Left: baseline; Right: % change at four weeks from baseline. Left plot: Six failures and one retreated were caught by baseline criteria (five failures and one retreated have cavity air >=30 and two have TLG>1500); Right plot: Two retreated subjects were selected by week 4 cavity air criteria (decrease of cavity air < 20%). Two additional retreated cases and one failure were caught by Week 16 Xpert <30.

Finally, we included a measure of residual TB bacterial load in sputum in the early treatment completion criteria based on an analysis that Xpert cycle threshold around 30 at weeks 8 and 24 correlated with culture negativity and patient treatment outcomes²⁹. We incorporated this measure at week 16 as a safety mechanism to ensure that participants with higher sputum bacterial load (cycle threshold <30) did not stop treatment early. Combining the baseline and week 4 criteria stratified 41/81 (50.6%) of all cured and failure patients as low risk (Table 4). When applied to the entire Catalysis cohort, including retreatment patients, 1/8 (12.5%) failures and 6/11 (54.5%) retreatments were captured as low risk. The PredictTB early treatment completion criteria when the trial started is shown in Table 5a.

The initial criteria were established as described above, acknowledging that early changes may be needed once the trial began if the actual proportion of PredictTB study participants stratified to the low- and high-risk arms were not close to the 50:50 target. Indeed, after about nine months of enrollment, only 23.4% of participants to reach week 16 were stratified as low risk (Arms B and C), with the remainder stratified to the high-risk arm (Arm A). This was less than half of the estimated 50% we expected to be low risk, which had major implications for the cost and duration of the study (total sample size, study duration, and cost would need to be increased to achieve required sample size in Arms B and C) as well as the scientific relevance of the study (if trial successful, it would only apply to less than 25% of TB patients, diminishing relevance). After discussion with our study Data and Safety Monitoring Board (DSMB), we revised our study early treatment completion criteria.

Revising the early treatment completion criteria

We considered how to change both the Xpert cycle threshold cutoff and the PET/CT radiology thresholds. For the Xpert cycle threshold, the original cutoff at week 16 was based on a cohort study in South Africa with MGIT culture results, the only data available to us at the time. We adopted a stringent cycle threshold value of 30 based upon analysis of these data for subjects to be randomized to Arms B and C. Xpert detects bacterial DNA but does not determine the viability of detected DNA (i.e., detected bacteria may be dead). For PredictTB, however, LJ culture is used to determine primary study outcomes. For this re-analysis, we received unpublished results from TBTC study 29, which collected cycle threshold values and LJ culture results (Rada Savic, personal communication). In evaluating the change, we considered the chance of missing an LJ+ result, as well as the sensitivity and specificity of various cycle threshold cutoffs. In contrast to positive and negative predictive values, sensitivity and specificity do not depend on the underlying proportion of culture positive results, which varies over time and from study to study. That said, patient safety was a driving factor so we considered how many positive cultures might be missed for various cutoffs. This was defined as the probability of being LJ+ given a Xpert cycle threshold value less than the cutoff, i.e., P(LJ+ | Ct-). We assumed what we considered were high proportions of LJ+ cultures (i.e., 10% and 5% at week 16 of treatment in the lower risk cohort of arm B/C) when making this decision. In contrast to TBTC study 29, which randomized all-comers and did not stratify participants by risk, the PredictTB study further excludes poorly adherent participants, those with too severe disease at baseline, or those not responding appropriately to treatment at one month. As a result, the expected LJ+ rates of 10% and 5% were considered to be very high. Table 6 describes these proportions for the sensitivity and specificity estimates from TBTC study 29. Based on these estimates, a cycle threshold of 30 was expected to miss 2.1% of LJ+ results, while a threshold of 28 would miss 2.5%, assuming a 10% LJ+ rate. This translated to an increase in less than one participant being missed among those randomized to arm C. That is, if the background LJ+ rate was 10%, 3.3 (of 155 randomized to stop treatment at week 16) true LJ+ participants may be missed with a cycle threshold of 30, and 3.9 may be missed with a cycle threshold of 28. If the underlying LJ+ rate was 5%, this becomes 1.6 missed LJ+ participants with cycle threshold 30 and 1.9 missed LJ+ participants with cycle threshold 28. If the true underlying LJ+ rate was even lower (as we would expect it to be), the difference between the two cycle threshold values becomes even smaller. Of the 12 participants already enrolled in the PredictTB study with week 16 Xpert cycle threshold results at the time of this analysis, eight had negative results, two had cycle thresholds below 28 (18.2 and 25.2) and two had cycle thresholds above 28 (28.4 and 28.5). Thus changing the Xpert cycle threshold cutoff from 30 to 28 would potentially (depending on radiology criteria) have retained an additional two participants in arms B and C.

To further correct the arm imbalance, we also changed the baseline and week 4 radiologic criteria. Prior studies validated that cavity on baseline CXR is a risk factor for treatment relapse. In our analyses of prior data, cavity size was also the strongest factor in predicting poor treatment outcome so we did not adjust this criterion. The data for CT hard volume and PET TLG as risk factors for poor treatment outcomes, however, were weak. Figure 3a shows the distribution of participants stratified to Arm A at baseline by the original radiology criteria. The numbers in the circles represent the number of participants that fell into arm A according to the defined criteria. The hard volume and total activity criteria were relatively well correlated in capturing participants, with only five participants moved to Arm A based on a single criterion, hard volume or PET TLG. Therefore, instead of arbitrarily increasing the hard volume and PET TLG cutoffs, we changed the criteria from requiring both hard volume AND total activity to be below the thresholds to be considered low risk, to only requiring one criterion. That is, participants with either hard volume OR total activity below the threshold at both baseline and week 4 would be considered low risk. The thresholds themselves did not change. Applying this change to the PET/CT criteria results in the revised Venn diagram in Figure 3b, which is the same as Figure 3a except for the five participants moved to Arm A based on hard volume or PET activity alone are no longer considered high risk. The revised early treatment completion criteria incorporating both Xpert cycle threshold and radiologic criteria changes are shown in Table 5b.

Figure 3a. Venn diagram of the original baseline PET/CT criteria by which participants were stratified to Arm A.

Figure 3b. Venn diagram of the revised baseline PET/CT criteria by which participants are stratified to Arm A.

Table 5b. Amended Predict TB early treatment completion criteria to correct arm imbalance between Arm A and Arms B/C.

Changes are highlighted in yellow.

Early completion criteria:	Determined at Week 16 – unless known to have failed a radiologic criterion at baseline or week 4.
Radiologic criteria	Baseline PET/CT:    •  No total lung collapse of a single side, AND    •  No pleural effusion, AND    •  No single cavity air volume on CT scan >30 mL, AND    •  CT scan hard volume (-100 to +100 HU density) <200 mL OR PET total lesion glycolysis <1500 units Week 4 PET/CT:    •  All individual cavities decrease by >20% (unless cavity <2 mL), AND    •  CT scan hard volume does not increase by >10% unless the increase is <5 mL OR PET total lesion glycolysis does not increase by >30% unless the increase is <50 units
Bacterial load criterion	Week 16 Xpert cycle threshold ≥28*
Adherence criterion	Minimum of 100 doses received by week 16

*If the week 16 solid medium sputum culture is subsequently found to be positive for Mtb in a participant randomized to Arm B or C, this participant will be called in for evaluation and to provide sputum for a repeat culture. If the initial positive culture is confirmed by a second culture positive for Mtb, this participant will be considered to have met the study endpoint as a treatment failure and will be referred for continued treatment.

TB, tuberculosis; PET, positron emission tomography; CT, computed tomography; HU, Hounsfield unit.

Table 6. Sensitivity and specificity estimates from TBTC study 29 for various Xpert cycle threshold cutoffs, along with estimates of missed LJ+ and missed LJ- results for assumed (week 16) culture-positivity rates of 10% and 5%.

Xpert cycle threshold Cutoff	Sensitivity: P(Ct<c|LJ+)	Specificity: P(Ct>c|LJ-)	Chance of missed LJ+ P(LJ+|Ct-) with 10% LJ+ rate	Chance of missed LJ+ P(LJ+|Ct-) with 5% LJ+ rate	Chance of missed LJ+ P(LJ+|Ct-) with 2.5% LJ+ rate
31	0.93	0.43	0.018	0.008	0.004
30	0.91	0.46	0.021	0.010	0.005
29	0.89	0.49	0.024	0.012	0.006
28	0.88	0.52	0.025	0.012	0.006
27	0.86	0.55	0.028	0.013	0.006
26	0.84	0.60	0.029	0.014	0.007
25	0.79	0.66	0.034	0.016	0.008

The revised early treatment completion criteria were accepted by the NIAID DSMB on March 16, 2018 and implemented after local regulatory approvals at the Henan, China sites on May 19, 2018 and at the Western Cape, South Africa sites on June 15, 2018. Only 12 (3.9%) of the total sample size of 310 to the low risk arms were recruited under the original early treatment completion criteria. The revised criteria re-balanced the arm proportions very nicely, approaching 50:50 in Arms A and B/C. The data used to develop both the original and revised early treatment completion criteria are deposited on Harvard Dataverse (see Data availability).

Discussion

Previously conducted treatment shortening studies for DS-TB suggested that approximately 80-85% of patients are cured with four months of treatment^1–4. Shortening treatment only in lower risk participants who had no cavity on baseline chest x-ray and had sputum culture converted to negative by two months of treatment resulted in the four-month treatment success proportion increasing to 93% in one trial but this was still significantly worse than six months of treatment⁸. The PredictTB trial tests an alternate risk stratification criteria based on FDG-PET/CT disease burden at baseline, the change in PET/CT disease burden at week 4 of treatment, and a marker of residual bacterial load and adherence dose count at the end of treatment, hypothesizing that this combination will identify patients with tuberculosis who are cured with four months of standard treatment⁵. Risk signatures based on transcriptomics have recently been shown to correlate with treatment outcomes^33,34.

The development of the PredictTB early treatment completion criteria was based on a cohort of 92 DS-TB patients programmatically treated in Cape Town, South Africa on whom we had PET/CT scans at baseline and week 4 of treatment, Xpert cycle threshold data, and programmatic treatment outcomes (Catalysis cohort). Because these patients were treated programmatically, treatment was not directly observed and we were thus unable to determine the proportion of treatment failures due to poor adherence or differentiate true relapsed disease patients from those re-infected. The lack of these data confounded our attempts to develop early treatment completion criteria that captured treatment failure and true relapse patients with meaningful sensitivity and specificity. We resorted to developing criteria that stratified about 50% of patients as high risk, trusting that the most severely diseased patients at baseline, those with poor treatment responses at week 4, and those under the Xpert cycle threshold cutoff at week 16 were captured as higher risk and therefore not eligible for treatment shortening.

A major limitation of developing our algorithm was the lack of sufficient relapse data to validate our early treatment completion criteria. This limitation is challenging to overcome, given the small numbers of available patient data anywhere with microbiological, strain-type confirmed relapses and the even smaller numbers of these with FDG-PET/CT scan data. We acknowledge the risk of overfitting our data (i.e., producing a risk model that may not be generalizable because it was fit only to the data on which it was developed) and in fact, after the study started, it became clear that our criteria were too conservative, stratifying >75% of participants as high risk and therefore not eligible for treatment shortening. Without immediate correction, we would likely run out of funding before the end of the trial due to the increased total sample size needed to achieve the required lower risk cohort sample size. Even worse, we would end up with a trial result that was applicable only to the 20–25% of patients stratified to the lower risk arms and therefore not relevant to the majority TB patients. Our amended early treatment completion criteria, however, have been stratifying participants at roughly 50:50 to the high- vs. low-risk arms.

The PredictTB trial early treatment completion criteria were developed to identify those with the most severe disease at baseline (potentially at higher risk for treatment failure) and with a poor week 4 treatment response (potentially at higher risk of relapse), along with a marker of residual bacterial load and an adherence dose count at treatment completion. These criteria are currently stratifying about 50% of patients to the higher risk arm and 50% of patients to the two lower risk arms, which is the target goal. Whether or not this will successfully identify a lower risk cohort that can be successfully cured with four months of standard therapy awaits the results of the trial, expected in 2022.

Data availability