Funding global health product R&D: the Portfolio-To-Impact Model (P2I), a new tool for modelling the impact of different research portfolios

Background: The Portfolio-To-Impact (P2I) Model is a novel tool, developed to estimate minimum funding needs to accelerate health product development from late stage preclinical study to phase III clinical trials, and to visualize potential product launches over time. Methods: A mixed methods approach was used. Assumptions on development costs at each phase were based on clinical trial costs from Parexel’s R&D cost sourcebook. These were further refined and validated by interviews, with a wide variety of stakeholders from Product Development Partnerships, biopharmaceutical and diagnostic companies, and major funders of global health R&D. Results: the tool was used to create scenarios describing the impact, in terms of products developed, of different product portfolios with funding ranging from $1 million per annum through to $500 million per annum. These scenarios for a new global financing mechanism have been previously presented in a report setting out the potential for a new fund for research and development which would assist in accelerating product development for the diseases of poverty. Conclusion: The P2I tool does enable a user to model different scenarios in terms of cost and number of health products launched when applied to a portfolio of health products. The model is published as open access accompanied with a user guide. The design allows it to be adapted and used for other health R&D portfolio analysis as described in an accompanying publication focussing on the pipeline for neglected diseases in 2017. We aim to continually refine and improve the model and we ask users to provide us with their own inputs that can help us update key parameters and assumptions. We hope to catalyse users to adapt the model in ways that can increase its value, accuracy, and applications.


Introduction
In 2012 the World Health Organization (WHO) considered the findings of a report from an expert working group into research and development to meet the health needs of developing countries. Among a number of recommendations this highlighted the need for better financial incentive mechanisms for health product research and development (R&D) for the neglected diseases. These diseases often have the greatest burden among the poorest populations and this absence of purchasing power-described as market failure -means there is often a need for R&D to be financed from public or philanthropic sources, rather than relying on conventional market forces 1 .
As a consequence, in 2014, the Special Programme for Research and Training in Tropical Diseases (TDR) was asked by the World Health Organization to explore potential financial mechanisms -in particular a new fund for R&D -which would assist in accelerating product development for the diseases of poverty, including Type III and II diseases as well as the specific R&D needs of developing countries in relation to Type I diseases 2 .
TDR, the Special Programme for Research and Training in Tropical Diseases, is a global programme of scientific collaboration that helps facilitate, support and influence efforts to combat diseases of poverty. It is hosted at the World Health Organization (WHO), and is sponsored by the United Nations Children's Fund (UNICEF), the United Nations Development Programme (UNDP), the World Bank and WHO. Established in 1975 TDR has codeveloped 12 new drugs for tropical parasitic diseases, building research capacity in three generations of public health leaders throughout the developing world and pioneered the role of communities and community health workers in delivering health interventions in many low-income countries 3 .
The disease Types I, II and III were first introduced by the Commission on Macroeconomics and Health and elaborated in the report of the Commission on Intellectual Property Rights, Innovation and Public Health. The definition of diseases into Types mixes a number of concepts together including the wealth of a country between rich and poor; the state of its development between developed and developing, and most importantly a measure of the burden of diseases by the incidence of the disease within the population. The definitions themselves are combined such that 4 : Type I diseases: are incident in both rich and poor countries, with large numbers of vulnerable populations in each and the focus is on the R&D needs specific to developing countries.
Type II diseases: are incident in both rich and poor countries, but with a substantial proportion of the cases in poor countries.
Type III diseases: are those that are overwhelmingly or exclusively incident in developing countries.
In response to this request, TDR conducted a study to analyse current funding landscape and product pipelines of Type III and II diseases and to identify funding bottlenecks and operational issues. The study included the development of a financial model to create scenarios that could estimate the impact of different sized portfolios of R&D products. The findings of the study were published as a report entitled Health Product Research & Development Fund: A Proposal for Financing and Operation (2016) -referred to in this article as TDR Report 2016 5 .
Here, we describe in more detail the creation of the financial tool called the Portfolio To-Impact (P2I) model, a novel financial forecasting tool to estimate funding needs of pharmaceutical product development over the period 2017-2030. In this modelling outputs are relative to the number of projects that can be funded per year and subsequently year-on-year. The original 13 year period was the time period chosen at that time for the purpose of demonstrating to the Member States of WHO the different impact of global R&D funds of different sizes with a realistic potential for producing product launches.

Methods
The P2I model is a financial portfolio tool that enables users to estimate funding needs to move candidate health products through the pipeline from late stage preclinical to phase III clinical trials, as well as potential product launches over time. For the purposes of this paper we modelled outcomes from a starting point of 2017 up to 2030. The modelling tool, which is deterministic, can

Amendments from Version 1
The majority of the reviewer comments highlighted areas where greater clarity and explanation could be brought to the text. The changes made in version two reflect that. The oversight of mixing results into the methods section has been resolved with inclusion of a separate Results section. Figure 3 has been updated to clarify the content and the numbering of the scenarios. While 7 scenarios are described in the original TDR Report 2016 scenarios 1 & 2 describe mechanisms providing coordination and priority setting without any support for R&D projects. The remaining scenarios (3-7) describe funds of different sizes from $15 -500 million and it is these 5 scenarios which were modelled by P2I -hence the mix up -which we believe is now resolved.
The main purpose of publishing this paper was to highlight the Portfolio-to-Impact Model tool and enable its use by others by providing an open access version and users' manual. The original results of our analysis using P2I are shown in greater detail in the TDR Report 2016 [Special Programme for Research and Training in Tropical Diseases, "Health Product Research & Development Fund: A Proposal for Financing and Operation," World Health Organization, Geneva, 2016]. Readers are encouraged to read that report to understand the context more fully and see how any new financial mechanism for R&D needs to consider three elements: an advisory group, that balances a mix of expertise, geographical representation and gender, how funding decisions will be made transparent and how that is organized in a funding strategy related to the available resources. However, we have provided greater detail in this paper to enable a reader to understand the utility of the P2I modelling tool without necessarily returning to the original source material.

See referee reports
REVISED estimate the costs of delivering different portfolios of product interventions and the health impacts of these portfolios. It is based on assumptions for costs, attrition rates, and cycle times for four development phases (preclinical to phase III) for eleven different kinds of medical products, called archetypes. A conceptual overview of the model is given in Figure 1. This section describes: (a) how the archetypes were determined; (b) the development of the model's assumptions; (c) the health impact methodology; (d) the construction of the P2I Microsoft Excel tool; and (e) estimating capacity constraints -the number of projects that can be supported per annum. a) Determination of archetypes Medical products were classified into eleven different types of intervention. Table 1 shows the basis for this classification and examples of each archetype. Products were first classified into five major categories: new vaccine, new chemical entity (NCE), repurposed drug, new biologic, or diagnostic. Each of these categories was then further sub-divided; for example, new vaccines were sub-divided into simple versus complex. The main differentiation point for archetypes within each of these five broad categories (e.g., within the new vaccine category) is whether the approach or mechanism of action is novel or has already been validated. For each of the 11 archetypes, separate assumptions on costs, attrition rates, and cycle times per phase were developed to recognize the variation in R&D characteristics and more accurately estimate R&D costs.

b) Development of assumptions
The assumptions on development costs at each phase were initially based on a bottom-up analysis of clinical trial costs from Parexel's R&D cost sourcebook, which is the leading resource for statistics, trends, and proprietary market intelligence and analyses within the biopharmaceutical industry 6 . These were then further refined and validated following interviews, with a wide variety of stakeholders from PDPs, biopharmaceutical and diagnostic companies, and major funders of global health R&D.
Interviews were semi-structured held over the phone between the researchers creating the models and senior R&D stakeholders from identified entities (question used in the interviews are presented in Supplementary File 1). Conversations were confidential to protect private research data of entities involved, hence no digital recordings were created. We used these interviews to review and validate the cost assumptions from the R&D cost sourcebook.
The assumptions on attrition rates and cycle times at each phase were initially based on a review of over 25,000 development candidates for attrition rates and cycle time 7 . These assumptions were further refined and validated based on academic literature 8,9 , industry publications 6,7 , and stakeholder interviews. Stakeholders were initially identified from across the networks of the authors and stakeholders were invited to recommend further interviewees as appropriate.  For the stakeholder interviews, a total of 228 stakeholders representing a cross-section of the global R&D landscape were contacted to request an interview and 133 agreed to be interviewed, a response rate of 58%. Of these 113 stakeholders who agreed, 29 were representatives of low-and middle-income countries (LMICs). The breakdown of these stakeholders by WHO region and type of organization is shown in Figure 2. Supplementary File 2 shows the full list of stakeholder organizations contacted and which organizations agreed to be interviewed. The interviews were conducted to gather external perspectives, to help validate emerging assumptions and findings, and to gain stakeholder feedback on the seven different pooled fund scenarios we generated using the P2I model (described further below).
As a final validation step, the P2I model and its assumptions were reviewed by TDR's Scientific and Technical Advisory Committee, who provided an additional round of expert inputs.

Development of assumptions on costs per phase.
The R&D scope of the model begins at the preclinical phase (after lead optimization) and ends at phase III clinical trials. In the P2I model, the estimated R&D costs include direct expenses, such as investigator grants and clinical supplies, internal headcount costs (full time equivalents [FTEs]), and in-kind (non-monetary) contributions that partnership-based organizations, such as PDPs, may receive from industry partners (e.g., lab costs, some internal headcount). The included costs are shown in Table 2.
Several costs are not included in the P2I model as they were deemed out of scope. These include: all costs related to basic research through lead optimization; chemistry, manufacturing, and controls (CMC); good manufacturing practice (GMP); manufacturing build up and scale-up costs; regulatory or registration fees (post-phase III); and all post-market commitments (e.g., phase IV pharmacovigilance studies).
R&D costs estimates were developed for preclinical through phase III and include lower bound, upper bound, and point estimates ( Table 3). As mentioned, the cost estimates were initially derived from a bottom-up analysis using Parexel's R&D cost sourcebook 6 and subsequently validated by interviews with stakeholders from the pharmaceutical industry and PDPs and through triangulation with academic and industry literature 6-9 . These development cost estimates reflect an R&D system that would be performing in a highly cost efficient manner. The cost estimates do not include capability building or infrastructure development expenses and thus reflect an idealized model in which R&D is performed in an established and streamlined organization, facility or partnership. The detailed cost assumptions separately for vaccines, NCEs, repurposed drugs, and biologics is shown in Supplementary File 3.

Development of assumptions on attrition rates and cycle times per phase.
Probability of success assumptions and cycle time per phase assumptions (Table 4) were similarly developed for each of the eleven archetypes from preclinical through phase III (or the relevant phases for diagnostics). For both sets of assumptions data was sourced from PDP stakeholder interviews, experts in the pharmaceutical industry, published literature, and industry databases 6,7 .
The assumptions were based on the Pharmaprojects database, a subscription database that tracks R&D filings for over 60,000     individual assets 7 . Specific sub-selections of the assets were made to differentiate between archetypes. Candidates that reported R&D activity from 2007-2014 and met specific criteria for each archetype were included in the analysis. After review with relevant stakeholders and other R&D experts, the assumptions were adjusted (primarily for the preclinical phase) using academic literature. The methodology used to identify appropriate candidates and the sample size is shown in Table 5, and the model assumptions on development costs, attrition rates, and cycle times for each phase were based on data on over 25,000 product candidates.
Due to the lack of academic literature available on development characteristics for diagnostics, the archetype definitions and assumptions were based on expert interviews with a leading PDP specializing in diagnostics and with diagnostics developers from large pharmaceutical companies.

c) Health impact methodology
The P2I model allows users to estimate the impact of a launched product on both disability, measured in disability-adjusted life years (DALYs) averted, and mortality, measured in deaths averted. The economic value of the DALYs averted is also calculated in terms of US dollars. The user is expected to make an informed estimate of the expected reduction in disease burden and expected reduction in mortality based on project-specific characteristics and estimates. These characteristics/estimates include, but are not limited to, the impact of improved efficacy over standard of care on mortality and morbidity, coverage rates, and disease prevalence and incidence.
The health impact calculation determines the DALYs averted, associated economic value, and deaths averted for a single launched intervention, irrespective of product archetype. The 2012 disease burden (DALYs) and mortality data for each Type III and II diseases were based on the WHO Global Health Estimates 10 . The DALYs averted metric is calculated by multiplying the 2012 LMIC disease burden by the expected reduction in disease burden (determined on a case-by-case basis by the model user). The same methodology is used for the deaths averted metric based on the users' input.
The associated economic value of the health impact is based on a valuation of $500 per DALY averted, as used in estimates by the International Finance Facility for Immunization (IFFIm) 11 . The IFFIm recognizes that the true economic value of a single DALY averted is "likely much higher," implying that the economic value is a relatively conservative estimate of the actual economic impact. The intention is not to arrive at a 'true' economic estimate. The use of such an estimate is to inform priority setting through the comparison of the impact of one product versus another. Further details of the health impact methodology are shown in Supplementary File 4.

d) Construction of the Microsoft Excel tool
The assumptions on costs, attrition rates, and cycle time per phase for the 11 different archetypes and the health impact estimation formulae were used to create a P2I tool in Microsoft Excel 2016 (Supplementary File 5). A users' guide to this tool is presented in Supplementary File 6. In brief, the tool allows users to input a portfolio of up to 150 product development projects. Users can select which diseases these projects will target and which product archetypes will be modelled. They can then choose a specific number of compounds that will reach a specific phase, which can either be outcome-based (e.g. "I want three compounds in Phase III" or "I want one launch") or fundingbased "(I want to fund two phase I projects"). Users select the year that each project for each archetype would start and the development phase at which the modelling should begin. The model also gives users the ability to set which phases their particular funding organization would fund, and to estimate health impacts of product launches. It is worth noting here that the model calculates the expected number of candidates moving from one phase to the next, including the expected launches. This means that on average, such a portfolio is expected to yield a certain number of launches; this is a floating-point number in the model. As an actual launch is binary the number of expected launches should then be rounded for presentation purposes. e) Estimating capacity constraints -the number of projects that can be supported per annum There is a final parameter which allows the user to model the impact of the capacity of an R&D system to absorb new projects on an annual basis. For a pooled fund operating at a global level this is set to 100% for each project i.e. it assumes everything that could be funded could be moved through the pipeline. Within this model changes to constrain the annual capacity parameter need to be applied per project/row in the Excel model sheet in line with the User Guide. Within the TDR Report 2016 the range of capacity constraints and portfolio considerations (for example a strategy of "quick wins" focused on mainly re-purposed drugs) were developed by the team. This allowed us to explore the relationship between the total product pipeline at the start of the funding process, the number of new projects supported per year, the number of launches in 2030 and funding requirements. In order to derive final recommendations in the TDR Report 2016 a range of funding scenarios using different strategies (e.g. "quick wins" vs. a mixed model of some repurposed and some new chemical entities) were illustrated in the report.

Results
The P2I tool was used to develop a series of scenarios used to estimate the impact of launching a new fund for global health R&D, of varying fund sizes. The funds would support the launch of new health products in 2030 if they became operational in 2017. The results are previously reported in full in the TDR Report 2016.
A number of scenarios were modelled and discussed with R&D stakeholders with respect to how each fund could contribute to a new financial mechanisms to stimulate global health R&D (Supplementary File 2 lists the stakeholders). These interviews revealed a broad spectrum of options for the financial mechanism, ranging from a specialized group that sets and communicates priorities, to a large global fund with its own secretariat. The seven scenarios that we modelled were chosen based on their feasibility and stakeholders' willingness to implement them (Table 6).
Scenarios 1 and 2 describe the funds required for different degrees of priority setting and coordination of R&D at the global level. In scenarios 3 -7 we used the P2I model to evaluate each of the five fund options that would support R&D projects by estimating the expected impact of various investment portfolios. The analysis evaluates the full development costs from preclinical (after lead optimization phase) through to phase III, accounting also for in-kind contributions from industry partners. Recognizing the numerous permutations of potential funding focus areas, a spectrum of financing focus strategies was constructed and explored for each of the fund sizes ranging from a fund of $15 million per annum up to $500 million per annum. This assumed three strategies. One focussed on quick wins and repurposing of existing drugs, one focussing on new innovations and a mixed model where the portfolio is balanced equally between these two approaches. The number of new projects that would need to be initiated each year (assuming attrition in existing projects), what the number of projects would be in a steady state portfolio, how many health products might be expected to be launched are shown in Figure 3.

Discussion
We developed a flexible, portfolio-based financial model, the P2I model, which can analyse and estimate the financing needs of various R&D investment portfolios for poverty-related and neglected diseases (PRNDs). The model can estimate preclinical to launch costs of a portfolio of candidates of 11 different archetypes: new vaccines (simple, complex); NCEs (simple, innovative, and complex); repurposed drugs (simple, complex); new biologics (simple, complex); and diagnostics (assay development, simple technology platform development). It also estimates the outputs (i.e. number of launches) that can be expected from investing in a specific portfolio of candidates and the timeframe for these launches. Finally, a constraint parameter can be used to model the number of projects that can be undertaken within an R&D system. R&D funders can therefore use the P2I model to improve their understanding of which health products they could potentially launch for their specific investments. To the best of our knowledge, the P2I model is the first model that allows users to estimate the costs and outputs across a whole portfolio of candidates and, as such, it can help to guide overall portfolio management by informing priority setting.
The P2I model is not a tool for estimating the exact costs of developing a specific product or for estimating the timing or probability of success of individual projects; instead, it estimates costs of different portfolios selected by the user based on aggregates of historical data. The model can help to show R&D funders where across the portfolio of candidates they should focus their investments to maximize the chances of getting the most meaningful outputs (e.g., a new chemical entity or vaccine for a disease that is missing such tools). By showing the costs and outputs of different scenarios, it can help to guide R&D investments. By estimating the likely impacts of a product in reducing disability and mortality, funders can use the P2I model to forecast the different trade-offs from different investments. The tool can also be used for financial planning-for example, it can help to determine the number of projects requiring oversight per year to inform staffing needs.
We used the model to forecast the potential impacts of a novel pooled fund for R&D of various sizes with a focus on neglected diseases, these findings have been previously published in the TDR Report 2016 5 . Seven different scenarios were modelled, based on different annual fund sizes, ranging from US$1 million to help improve R&D coordination through to a global fund of over $500 million that could fund 140-160 projects across a range of priority areas. The study found that an annual disbursement of at least $100 million would be required (i.e. a medium sized fund) to fill significant gaps in the R&D pipeline. A fund of this size could support 25-40 projects. If it became operational in 2017, more than 10 simple repurposed drugs or three reformulation or repurposed drugs, one simple NCE, and one complex repurposed drug could potentially be launched by 2030, though the launch of an innovation-focused product would be unlikely.

Strengths and limitations
There are three major strengths of the P2I model. The first is that we developed the model using an established portfolio methodology approach that is widely used by industry. The model assumptions on development costs, attrition rates, and cycle times for each phase were based on a very large number of data points (including data on over 25,000 product candidates), and they were "reality tested" with stakeholders out of a very large and broad group of over 130 expert stakeholders from around 80 organizations. These organizations included foundations, bilateral and multilateral development agencies, ministries of health, non-profit NGOs, organizations conducting R&D (universities, PDPs, companies, government agencies), regulatory agencies, and inter-governmental organizations. TDR's Scientific and Technical Advisory Committee provided an additional round of expert inputs. We believe that the use of a large dataset and the extensive consultation and validation process led to a model that is based on realistic assumptions.
Second, we deliberately kept the model simple by focusing on product launches, rather than trying to include market dimensions such as anticipated price, demand, sales volumes, or profits. Investment scenarios generated by the model are based on product development costs from preclinical to launch and not on profit considerations. This approach was chosen because of the model's focus on type II and III diseases, which disproportionately or exclusively affect people in poor countries. These patients are unlikely to be able to afford to buy new medical products themselves; it will usually be governments and external funders who purchase them.  14 . It assumes that all projects involving a particular archetype will have the same averaged costs, attrition rates, and cycle times per phase (e.g. all simple vaccines would have the same overall development costs). The model outputs are highly dependent on the underlying assumptions (the input parameters), which were based on aggregate data. The assumptions on attrition rates and cycle times per phase for different archetypes were based on different volumes of data-for example, data from 247 candidates for simple vaccines versus data from 14,425 candidates for innovative NCEs. The health impact estimates in the model are entirely reliant on users estimating the future expected impact, which is subject to tremendous uncertainty. Finally, the model excludes several costs, such as the costs of basic research and early discovery until lead optimization; regulatory, registration, and post-market expenditures incl. phase 4; chemistry, manufacturing and controls; good manufacturing practice; purchase of equipment; development of tools, technologies, facilities, or infrastructure; manufacture, implementation, and delivery of products.
As a consequence the expected R&D costs per launch using the P2I model lie below $1 billion (depending on archetype chosen). This is lower than other published estimates for overall drug development costs within the private pharmaceutical sector 15 .
It is important to stress that individual product predictions are outside the scope of the model and that its real utility lies in its predictive value for modelling the impact of different funding strategies at the portfolio level.

Next steps
The supplementary files include a User Guide and the P2I tool embedded in an Excel file, and readers are encouraged to use the tool and publish the results. An example of this adaptation is described in an accompanying paper. In that paper Young et al. describe how they adapted the modelling tool to create a new version, P2I V.2, in order to undertake a whole portfolio analysis of the current pipeline for those neglected diseases included in the annual G-FINDER survey of R&D financing 16 . The model estimates it would cost about $16.3 billion to move the 538 applicable products through the pipeline, resulting in about 128 (89-160) expected product launches. Based on this analysis the model suggests there would be very few launches of complex new chemical entities and any launches of highly efficacious vaccines for HIV, TB, or malaria would be unlikely. One conclusion is that the current portfolio is not balanced across health needs.
Please note the P2I model published here in the supplementary files has been expanded to allow for modelling up to the year 2040 based on the updates described by Young et al. We aim to continually refine and improve the modelling and ask users to provide us with their own inputs that can help us update key parameters and assumptions. We hope to catalyse users to adapt the model in ways that can increase its value, accuracy, and applications. As declared in my first review, I co-wrote a 2018 report where Rob Terry (author Competing Interests: here) wrote the foreword to that report. The Portfolio-To-Impact (P2I) model was developed to address an important gap in the spectrum of research & development (R&D), product development, and innovation diffusion. The P2I methods and analysis by Terry et al. provide a crucial first step in efforts to more systematically evaluate product development portfolios -particularly for neglected diseases. This work should be refined by applying it "in real time" to specific diseases and to be pragmatic/salient for organizations such as the Coalition for Epidemic Preparedness Innovations (CEPI), which invest in novel vaccines to protect against three specific infections and broader platform technologies.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
A response number of 133 and response rate of 58% for stakeholder interviews is high and clearly helped provide valuable insights to enhance the P2I model. Could the authors clarify whether the 42% that declined were homogenous or relatively comparable in characteristics to those that agreed to contribute to the P2I consultation? The concern to mitigate here would be bias in the type of information provided by those shaping the construction of the new model. It would be helpful therefore to include the names and organizations of those that were unable to contribute to the consultations, for readers to assess salience of the model to those organizations. Looking at the supplementary figure 1, it seems a good breadth of engagement in terms of organizations was 1.

If applicable, is the statistical analysis and its interpretation appropriate? Not applicable
Are all the source data underlying the results available to ensure full reproducibility? Yes

Are the conclusions drawn adequately supported by the results? Partly
Between October 2016 and October 2018, I was an employee at the Bill & Competing Interests: Melinda Gates Foundation, which is listed as one of the funders of two of the P2I model authors. However, I did not have any engagement in this, or affiliated, bodies of work.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Author Response 10 Jul 2018 ,

Gavin Yamey
We agree the P2I modelling tool is only a first step in providing one approach to estimating the impact of an R&D portfolio. That is our main motivation in publishing it in full and completely open access. We encourage others to use, improve and update the tool and that is why we published two papers in parallel: one paper explaining the original P2I tool and its development and the second paper adapting the P2I to undertake the first full analysis of the neglected diseases pipeline. The original work in the TDR Report 2016 is cited by the WHO R&D Blueprint and therefore subsequently informed the thinking of the CEPI strategy. There has been some correspondence with CEPI as -at the time of writing -they are currently undertaking an analysis of the CEPI portfolio (and we are undertaking a new analysis, using the P2I tool, on diagnostics and therapeutics for the Blueprint diseases). However, it is worth noting their costing estimate is a broader end-to-end approach encompassing basic science through to manufacture and delivery. These elements are not covered in the P2I tool as there is much wider variability in these costs due to the choices that can be made with respect to manufacturing. In designing P2I we felt including data with such wide variables would be the equivalent of estimating the length of an unknown piece of string so we chose to exclude them as we felt this would significantly reduce the utility of what we were trying to achieve.
With respect to your numbered comments we have provided our responses below.
With regard to the consultation we were also very grateful to the wide range of stakeholders that contributed.This is due in large part to the consultants we worked with, McKinsey & Co., who has greater access to commercial pharma data and mediated the gap between public/philanthropic and private funding.The respondents are broadly representative particularly within the product development partnerships (PDPs) which are the main mechanism currently to address the neglected diseases. Our hope is that this approach as a first step can only be improved and will provide a framework to widen the stakeholder engagement in global health priority setting and decision making.
The relatively limited input from regulatory bodies probably has more impact on a specific Gates Open Research 2. 3.

5.
The relatively limited input from regulatory bodies probably has more impact on a specific desired product profile than on a portfolio as a whole which is our main focus. As the data used in the tool is historical data that has been aggregated there is a reasonable assumption -but not tested -that the impact of regulatory requirements is already averaged across the tool and subsumed into the data.We think there might be some interesting analysis of how personalized treatments for certain conditions that would not be available to the general population might be affected more of less by regulatory requirements but this is outside the scope of this work and again would be better suited to specific cases rather than the generic analysis we present here. With regard to how other costs affect decision making many of the stakeholders have commented that while this is a useful tool to gain an overview of a portfolio, the go/no-go decisions on individual projects would be very specific to the products under investigation.We highlight this in the caveats. Yes we completely agree the measure of impact is quite crude.However, we decided to go with established and accepted methods to assess impact recognising their limitations.The suggestions you make on improving this measure are exactly the type of refinement and improvement we would wish to see and hope this initial piece of research will stimulate. Again the point about maintaining the relevance of the data and updating the tool is completely accepted and we would welcome the opportunity to undertake this worksubject to identifying suitable resources. This is an interesting and potentially high-impact piece of work, looking at the important topic of how funders may invest in the future with regards product development for neglected diseases. The methods are rigorous, described in detail, and their strengths and limitations well explained. Supporting data via various supplementary documents are available. Noted that the interview transcripts are not available, this is perfectly reasonable.
There are a few areas that could do with being revised.
Primarily, there is no results section within the paper (though there is within the abstract). The article goes from methods straight to discussion. Whilst this is not a 'traditional' type of research, it is presented as an original research article, and there are results from the modeled scenarios that are not adequately described and would fit neatly into the results section.
The abstract also needs some rewriting, a few points below, but overall it could be a better shop window for the work. There will be potential readers who want to scan the abstract to get the thrust of the findings and "what's new", at the moment it's a little unclear.
For these reasons, I think the paper should certainly be indexed, but for now I've categorized as 'approved For these reasons, I think the paper should certainly be indexed, but for now I've categorized as 'approved with reservations'.
A few more specific points: Title -doesn't have a study descriptor in it. As it's written as an original research article, then ideally needs to clearly state e.g. 'a modelling exercise' or something that tells the reader the type of investigation contained therein. Given there are modelling and stakeholder interview components, then 'a mixed-methods approach' or similar might also be appropriate Methods -Page 6, column 1, 'for both cases data was sourced from PDP stakeholder...'. 'both cases' -is this referring to the assumptions of success probability and cycle times? Suggest rewording from 'cases' to 'scenarios' or 'assumptions' or similar Discussion --What did happen to the results section? -Page 10, column 2, "seven different scenarios were modelled..."; this and some ensuing text is results rather than discussion -Strengths and limitations, the list of organisations covers would be better in methods rather than a discussion section -If enthusiasm and word count allows, then there could be a fuller discussion of how the relevant stakeholders make priorities at the moment, the large number of stakeholders with limited communication between them even in the same sectors, what data is available to them to inform their strategy/intelligence (e.g. ResIn, gfinder etc). And then you have a natural point to move the discussion onto 'therefore, this tool can help in these specific areas...'

Is the work clearly and accurately presented and does it cite the current literature? Yes
Is the study design appropriate and is the work technically sound? Yes

If applicable, is the statistical analysis and its interpretation appropriate? Yes
Are all the source data underlying the results available to ensure full reproducibility? Partly Are the conclusions drawn adequately supported by the results? Yes In 2018, I co-authored a report called 'Sizing Up Pneumonia Research', published Competing Interests: from my research project, the Research Investments in Global Health study. One of the authors of this paper, Rob Terry, wrote the foreword to that report. There was no financial or contractual agreement for this collaboration, either with Rob Terry as an individual or via his institution.
I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Gavin Yamey
Again many thanks for the carefully considered comment and editorial suggestions. In response: The mix-up of results with methods has been resolved with a specific results section. Much of the detailed discussion and context for this study has already been published in the TDR Report 2016 so we accept that the results in this paper were probably underreported and we have attempted to redress that.
You highlight a number of areas in the text requiring changes and improvement and for the most part we have made those changes. The confusion over the numbers of financial mechanism scenarios we developed in the TDR Report 2016 (7) and the use of the P2I tool in generating the impact of different fund sizes (5) has been resolved. Figure 3 has also been updated and replaced.
With regard to a wider discussion about how the impact of product development for global health might be improved with greater communication and subsequent coordination between all the actors we agree but felt we wanted to keep this paper focused on the tool and its use rather than the implications for its use. Again there is more context in the TDR Report 2016 and the parallel paper by Young analysing the neglected diseases pipeline. The lead author, , has et al R F Terry published a number of papers on how greater transparency in R&D funding would be the first step towards improving coordination and this led in part to the creation of the WHO Global Health R&D Observatory (see reference below).
Again we would hope that by creating this tool, whilst recognising its limitations, we will stimulate and support greater efforts at reaching the goal of improved coordination between the wide range of stakeholders.