Funding global health product R&D: the Portfolio-To-Impact Model (P2I), a new tool for modelling the impact of different research portfolios

a) Determination of archetypes

Medical products were classified into eleven different types of intervention. Table 1 shows the basis for this classification and examples of each archetype. Products were first classified into five major categories: new vaccine, new chemical entity (NCE), repurposed drug, new biologic, or diagnostic. Each of these categories was then further sub-divided; for example, new vaccines were sub-divided into simple versus complex. The main differentiation point for archetypes within each of these five broad categories (e.g., within the new vaccine category) is whether the approach or mechanism of action is novel or has already been validated. For each of the 11 archetypes, separate assumptions on costs, attrition rates, and cycle times per phase were developed to recognize the variation in R&D characteristics and more accurately estimate R&D costs.

b) Development of assumptions

The assumptions on development costs at each phase were initially based on a bottom-up analysis of clinical trial costs from Parexel’s R&D cost sourcebook, which is the leading resource for statistics, trends, and proprietary market intelligence and analyses within the biopharmaceutical industry⁶. These were then further refined and validated following interviews, with a wide variety of stakeholders from PDPs, biopharmaceutical and diagnostic companies, and major funders of global health R&D.

Interviews were semi-structured held over the phone between the researchers creating the models and senior R&D stakeholders from identified entities (question used in the interviews are presented in Supplementary File 1). Conversations were confidential to protect private research data of entities involved, hence no digital recordings were created. We used these interviews to review and validate the cost assumptions from the R&D cost sourcebook.

The assumptions on attrition rates and cycle times at each phase were initially based on a review of over 25,000 development candidates for attrition rates and cycle time⁷. These assumptions were further refined and validated based on academic literature^8,9, industry publications^6,7, and stakeholder interviews. Stakeholders were initially identified from across the networks of the authors and stakeholders were invited to recommend further interviewees as appropriate.

For the stakeholder interviews, a total of 228 stakeholders representing a cross-section of the global R&D landscape were contacted to request an interview and 133 agreed to be interviewed, a response rate of 58%. Of these 113 stakeholders who agreed, 29 were representatives of low- and middle-income countries (LMICs). The breakdown of these stakeholders by WHO region and type of organization is shown in Figure 2. Supplementary File 2 shows the full list of stakeholder organizations contacted and which organizations agreed to be interviewed. The interviews were conducted to gather external perspectives, to help validate emerging assumptions and findings, and to gain stakeholder feedback on the seven different pooled fund scenarios we generated using the P2I model (described further below).

As a final validation step, the P2I model and its assumptions were reviewed by TDR’s Scientific and Technical Advisory Committee, who provided an additional round of expert inputs.

Figure 2. Stakeholder interviews—response rate and stakeholder categories.

Development of assumptions on costs per phase. The R&D scope of the model begins at the preclinical phase (after lead optimization) and ends at phase III clinical trials. In the P2I model, the estimated R&D costs include direct expenses, such as investigator grants and clinical supplies, internal headcount costs (full time equivalents [FTEs]), and in-kind (non-monetary) contributions that partnership-based organizations, such as PDPs, may receive from industry partners (e.g., lab costs, some internal headcount). The included costs are shown in Table 2.

Several costs are not included in the P2I model as they were deemed out of scope. These include: all costs related to basic research through lead optimization; chemistry, manufacturing, and controls (CMC); good manufacturing practice (GMP); manufacturing build up and scale-up costs; regulatory or registration fees (post-phase III); and all post-market commitments (e.g., phase IV pharmacovigilance studies).

R&D costs estimates were developed for preclinical through phase III and include lower bound, upper bound, and point estimates (Table 3). As mentioned, the cost estimates were initially derived from a bottom-up analysis using Parexel’s R&D cost sourcebook⁶ and subsequently validated by interviews with stakeholders from the pharmaceutical industry and PDPs and through triangulation with academic and industry literature^6–9. These development cost estimates reflect an R&D system that would be performing in a highly cost efficient manner. The cost estimates do not include capability building or infrastructure development expenses and thus reflect an idealized model in which R&D is performed in an established and streamlined organization, facility or partnership. The detailed cost assumptions separately for vaccines, NCEs, repurposed drugs, and biologics is shown in Supplementary File 3.

Development of assumptions on attrition rates and cycle times per phase. Probability of success assumptions and cycle time per phase assumptions (Table 4) were similarly developed for each of the eleven archetypes from preclinical through phase III (or the relevant phases for diagnostics). For both sets of assumptions data was sourced from PDP stakeholder interviews, experts in the pharmaceutical industry, published literature, and industry databases^6,7.

The assumptions were based on the Pharmaprojects database, a subscription database that tracks R&D filings for over 60,000 individual assets⁷. Specific sub-selections of the assets were made to differentiate between archetypes. Candidates that reported R&D activity from 2007–2014 and met specific criteria for each archetype were included in the analysis. After review with relevant stakeholders and other R&D experts, the assumptions were adjusted (primarily for the preclinical phase) using academic literature. The methodology used to identify appropriate candidates and the sample size is shown in Table 5, and the model assumptions on development costs, attrition rates, and cycle times for each phase were based on data on over 25,000 product candidates.

Due to the lack of academic literature available on development characteristics for diagnostics, the archetype definitions and assumptions were based on expert interviews with a leading PDP specializing in diagnostics and with diagnostics developers from large pharmaceutical companies.

c) Health impact methodology

The P2I model allows users to estimate the impact of a launched product on both disability, measured in disability-adjusted life years (DALYs) averted, and mortality, measured in deaths averted. The economic value of the DALYs averted is also calculated in terms of US dollars. The user is expected to make an informed estimate of the expected reduction in disease burden and expected reduction in mortality based on project-specific characteristics and estimates. These characteristics/estimates include, but are not limited to, the impact of improved efficacy over standard of care on mortality and morbidity, coverage rates, and disease prevalence and incidence.

The health impact calculation determines the DALYs averted, associated economic value, and deaths averted for a single launched intervention, irrespective of product archetype. The 2012 disease burden (DALYs) and mortality data for each Type III and II diseases were based on the WHO Global Health Estimates¹⁰. The DALYs averted metric is calculated by multiplying the 2012 LMIC disease burden by the expected reduction in disease burden (determined on a case-by-case basis by the model user). The same methodology is used for the deaths averted metric based on the users’ input.

The associated economic value of the health impact is based on a valuation of $500 per DALY averted, as used in estimates by the International Finance Facility for Immunization (IFFIm)¹¹. The IFFIm recognizes that the true economic value of a single DALY averted is “likely much higher,” implying that the economic value is a relatively conservative estimate of the actual economic impact. The intention is not to arrive at a ‘true’ economic estimate. The use of such an estimate is to inform priority setting through the comparison of the impact of one product versus another. Further details of the health impact methodology are shown in Supplementary File 4.

d) Construction of the Microsoft Excel tool

The assumptions on costs, attrition rates, and cycle time per phase for the 11 different archetypes and the health impact estimation formulae were used to create a P2I tool in Microsoft Excel 2016 (Supplementary File 5). A users’ guide to this tool is presented in Supplementary File 6. In brief, the tool allows users to input a portfolio of up to 150 product development projects. Users can select which diseases these projects will target and which product archetypes will be modelled. They can then choose a specific number of compounds that will reach a specific phase, which can either be outcome-based (e.g. “I want three compounds in Phase III” or “I want one launch”) or funding-based “(I want to fund two phase I projects”). Users select the year that each project for each archetype would start and the development phase at which the modelling should begin. The model also gives users the ability to set which phases their particular funding organization would fund, and to estimate health impacts of product launches. It is worth noting here that the model calculates the expected number of candidates moving from one phase to the next, including the expected launches. This means that on average, such a portfolio is expected to yield a certain number of launches; this is a floating-point number in the model. As an actual launch is binary the number of expected launches should then be rounded for presentation purposes.

e) Estimating capacity constraints – the number of projects that can be supported per annum.

There is a final parameter which allows the user to model the impact of the capacity of an R&D system to absorb new projects on an annual basis. For a pooled fund operating at a global level this is set to 100% for each project i.e. it assumes everything that could be funded could be moved through the pipeline. Within this model changes to constrain the annual capacity parameter need to be applied per project/row in the Excel model sheet in line with the User Guide. Within the TDR Report 2016 the range of capacity constraints and portfolio considerations (for example a strategy of “quick wins” focused on mainly re-purposed drugs) were developed by the team. This allowed us to explore the relationship between the total product pipeline at the start of the funding process, the number of new projects supported per year, the number of launches in 2030 and funding requirements. In order to derive final recommendations in the TDR Report 2016 a range of funding scenarios using different strategies (e.g. “quick wins” vs. a mixed model of some repurposed and some new chemical entities) were illustrated in the report.

Strengths and limitations

There are three major strengths of the P2I model. The first is that we developed the model using an established portfolio methodology approach that is widely used by industry. The model assumptions on development costs, attrition rates, and cycle times for each phase were based on a very large number of data points (including data on over 25,000 product candidates), and they were “reality tested” with stakeholders out of a very large and broad group of over 130 expert stakeholders from around 80 organizations. These organizations included foundations, bilateral and multilateral development agencies, ministries of health, non-profit NGOs, organizations conducting R&D (universities, PDPs, companies, government agencies), regulatory agencies, and inter-governmental organizations. TDR’s Scientific and Technical Advisory Committee provided an additional round of expert inputs. We believe that the use of a large dataset and the extensive consultation and validation process led to a model that is based on realistic assumptions.

Second, we deliberately kept the model simple by focusing on product launches, rather than trying to include market dimensions such as anticipated price, demand, sales volumes, or profits. Investment scenarios generated by the model are based on product development costs from preclinical to launch and not on profit considerations. This approach was chosen because of the model’s focus on type II and III diseases, which disproportionately or exclusively affect people in poor countries. These patients are unlikely to be able to afford to buy new medical products themselves; it will usually be governments and external funders who purchase them.

Third, the model is highly flexible, can be used for multiple purposes, and can be modified, adapted, or built upon by users. For example, the model can be used prospectively, i.e. the user inputs the number of candidates funded in each phase and the model then estimates future costs and launches. It can be used retrospectively, i.e. the user inputs the desired number of candidates at a chosen phase, then the model determines the number of candidates needed in prior phases and forecasts associated costs and outputs. Users can change the model’s input parameters (cost, attrition rate, and cycle time per phase) to see how these changes would affect overall costs and outputs. Other archetypes can be added, as shown in an accompanying study, in which we added archetypes such as vector control products and “unprecedented vaccines” (vaccines under development for HIV, TB, or malaria, which are likely to have higher attrition rates than complex vaccines). The model can be applied to other disease areas, such as developing new technologies to combat antimicrobial resistance or to control epidemics and pandemics (e.g., the WHO R&D Blueprint for Action to Prevent Epidemic used the model to estimate financing needs to develop medical countermeasures against epidemics)¹³.

Nevertheless, the P2I model also has a number of limitations. It is a deterministic model using single point estimates and it does not account for uncertainty or risk (it does not incorporate Monte Carlo simulations)¹⁴. It assumes that all projects involving a particular archetype will have the same averaged costs, attrition rates, and cycle times per phase (e.g. all simple vaccines would have the same overall development costs). The model outputs are highly dependent on the underlying assumptions (the input parameters), which were based on aggregate data. The assumptions on attrition rates and cycle times per phase for different archetypes were based on different volumes of data—for example, data from 247 candidates for simple vaccines versus data from 14,425 candidates for innovative NCEs. The health impact estimates in the model are entirely reliant on users estimating the future expected impact, which is subject to tremendous uncertainty. Finally, the model excludes several costs, such as the costs of basic research and early discovery until lead optimization; regulatory, registration, and post-market expenditures incl. phase 4; chemistry, manufacturing and controls; good manufacturing practice; purchase of equipment; development of tools, technologies, facilities, or infrastructure; manufacture, implementation, and delivery of products.

As a consequence the expected R&D costs per launch using the P2I model lie below $1 billion (depending on archetype chosen). This is lower than other published estimates for overall drug development costs within the private pharmaceutical sector¹⁵.

It is important to stress that individual product predictions are outside the scope of the model and that its real utility lies in its predictive value for modelling the impact of different funding strategies at the portfolio level.

Next steps

The supplementary files include a User Guide and the P2I tool embedded in an Excel file, and readers are encouraged to use the tool and publish the results. An example of this adaptation is described in an accompanying paper. In that paper Young et al. describe how they adapted the modelling tool to create a new version, P2I V.2, in order to undertake a whole portfolio analysis of the current pipeline for those neglected diseases included in the annual G-FINDER survey of R&D financing¹⁶. The model estimates it would cost about $16.3 billion to move the 538 applicable products through the pipeline, resulting in about 128 (89–160) expected product launches. Based on this analysis the model suggests there would be very few launches of complex new chemical entities and any launches of highly efficacious vaccines for HIV, TB, or malaria would be unlikely. One conclusion is that the current portfolio is not balanced across health needs.

Please note the P2I model published here in the supplementary files has been expanded to allow for modelling up to the year 2040 based on the updates described by Young et al.. We aim to continually refine and improve the modelling tool and ask users to provide us with their own inputs that can help us update key parameters and assumptions. We hope to catalyse users to adapt the model in ways that can increase its value, accuracy, and applications.