Estimating the impact of vaccination: lessons learned in the first phase of the Vaccine Impact Modelling Consortium

4.1 Defining vaccine impact

The main modelled outcomes for VIMC have been deaths averted, cases averted, and disability-adjusted life years (DALYs) averted. Whilst this does not capture all the benefits of vaccination, such as reduced strain on health systems, costs of illness averted, or economic disruptions due to outbreaks prevented, VIMC’s efforts have supported examination of the wider effect of vaccination through partnership with other research groups^5,6.

One of the roles of VIMC is to define and estimate appropriate metrics to evaluate the benefits of vaccination. Vaccine impact can be valued and estimated in multiple ways, but the methodological approach should be informed by the context and the question of interest. An area of continuing development is the methodology to estimate different values of vaccine impact; e.g., ways to attribute impact either over people’s lifetimes, over a calendar year, or due to one year’s worth of immunisation activities. VIMC has developed a ‘year of vaccination’ approach to valuing impact which considers the longer-term effects of vaccination attributed to the year in which vaccination activities took place^7,8. This approach links more directly with decision making where the outcome of a specific vaccine activity can be examined. The VIMC approach relies on the estimation of ‘impact ratios’: the burden averted per dose, or course of vaccination, over the lifetime of the cohort. The method is not without its limitations, with some of its strengths and drawbacks examined in Echeverria-Londono et al. 2021⁷; however, it allows for projection of the implications of minor changes in vaccination coverage on impact. This approach, and its efficiency compared to the computational burden of running multiple health impact models, has been invaluable in updating estimates given new data releases, such as the WHO and UNICEF estimates of immunisation coverage (WUENIC), on a short timescale. It is also an approach that has been adopted by Gavi, BMGF, and the IA2030 project team in their estimates⁹.

4.2 Delivery platform

The technical infrastructure required to deliver robust, coordinated, rigorous, and repeatable estimates at scale is considerable. Setting up the VIMC delivery platform infrastructure was more demanding than first anticipated and was a key focus of the early years of VIMC 1.0. VIMC developed an open-source delivery platform that hosts input data, impact estimates, and documentation centrally, with different interfaces for modellers and other stakeholders. The platform allows quality and consistency checks on modellers’ estimates, automated report execution, and includes data visualisation tools.

A robust and reproducible approach to analysis and reports has allowed us to incorporate lessons learnt and methodological improvements throughout the course of VIMC 1.0. For example, through repeatedly running our analyses on data and estimates, we have identified where problems often arise and incorporated diagnostics for issues as they appear. These include anything from automated checks (e.g. are deaths projected to be negative?) to more nuanced verification approaches implemented by the VIMC science team (e.g. are projections between modelling groups notably different?). As a result, each report drafted by the secretariat is not only developed in a collaborative atmosphere focused on best practices such as code review but incorporates learning and diagnostics from the previous six years of research. This is made possible by the robust technical infrastructure, specific software developments like orderly (described below), and support from a diverse range of domain experts including research software engineers. A cross-disciplinary approach that puts reproducibility and reusability as a priority alongside other scientific aims has benefited VIMC efficiency and accountability.

One particular software development that has evolved out of VIMC is orderly¹⁰, an open-source R package. Now in its second incarnation, orderly underpins all reports and outputs from the VIMC science team and has been used extensively by the Ebola and COVID-19 Imperial response teams¹¹. orderly versions not only included code but also the data that informed a particular analysis; this is critical when dealing with multiple data inputs and a changing landscape of analytic requests. This way of working is important to address an item on the GATHER checklist for responsible estimate reporting, given that the differences between different iterations of the same analysis can be tracked and fully explained along the entire analysis pipeline¹.

4.3 Vaccine impact estimates

Through the framework shown in Figure 2, the VIMC science team collates inputs from country and regional data sources as well as the published literature to supply standardised input data for modellers. These sources include reports on vaccination campaigns, such as described in the WHO weekly epidemiological record, estimates of routine immunisation coverage from WUENIC, or population size estimates from United Nations World Population Prospects. It is critical that inputs such as demography are standardised across modelling groups to ensure comparability of results; this is facilitated by the infrastructure and processes related to the delivery platform. These standardised inputs then inform the modelled estimates of disease burden under each vaccination coverage scenario. The burden estimates from each modelling group are provided to the science team to calculate impact by different views (see Section 4.1) and the results are reported, reviewed, and prepared for publication. Groups were also asked to provide 200 runs of their model for each vaccination scenario to capture parameter uncertainty. This set of ’stochastic runs’ were computationally intensive but vital for communicating the variation in results. This framework has led to some of the key results detailed in Figure 3 and below.

Figure 2. Flow of estimates and input data for the Vaccine Impact Modelling Consortium (VIMC), adapted from 12 to highlight the joining of data flows compared to either the UN agency or academic frameworks.

Figure 3. Summary of timing and scope of three consortium wide publications/preprints⁸,13,14.

Timelines indicate the last year of included vaccination data, year of model runs and year of publication/ preprint. The table summarises characteristics such as number of included countries or modelling groups included.

The first consortium-wide publication, presenting results from 16 modelling groups across 98 countries for 10 diseases, estimated that 69 million lives may be saved by vaccination between calendar years 2000 and 2030, 37 million of which were projected to be saved between calendar years 2000 and 2019¹³. This paper also presented estimates over the lifetime of individuals vaccinated, projecting a 72% reduction in vaccine-preventable disease (VPD) burden for those born in 2019. These results used available data up to and including 2016 and did not account for the effects of COVID-19 in later years as our analyses pre-dated the pandemic. As such, we may wish to focus on projections for these years of data only: 29.4 million deaths averted between calendar years 2000–2016 and 52.2 million deaths averted for birth years 2000–2016. COVID-19 disruption and the focus on technical development contributed to the time between model runs, in 2017, and publication date in 2021.

The second consortium-wide publication, published in 2021, presented a new vaccine impact view, impact by year of vaccination, and an expanded country scope⁸. The corresponding estimates were 97 million lives saved by vaccination activities taking place between 2000–2030 with 50 million lives saved by vaccination occurring between 2000–2019. This new vaccine impact view captured the long-term benefits of immunisation, as well as the later burden attributable to HPV or Hepatitis B. The estimates generally overlapped with the first VIMC study, the comparative estimates for 2000–2016 were 30.2 million deaths averted between calendar years 2000 and 2016, and 48.8 million deaths averted for birth years 2000–2016. The time from last year of data and publication date was also reduced, a product of the fully formed technical infrastructure and lessons learnt from the first publication.

The third VIMC paper shown here had a different focus, necessitated by the COVID-19 pandemic¹⁴. This work was more motivated by a key policy question at the time: how has the pandemic affected progress in preventing VPDs and the extent to which catch-up vaccination can mitigate losses. There was a much faster progression from data release to modelling to preprint showing the experience in the process and the urgency of the work. Coverage disruptions were found to have led to a 2.7% drop in projected impact for vaccination activities implemented between 2020 and 2030, assuming coverage recovers to pre-pandemic levels in 2025. This paper also examined catch-up activities and projected they could avert 78.9% of excess burden occurring between 2023 and 2030.

5.1 The impact of COVID-19 on vaccine-preventable diseases

We engaged in two major workstreams examining the impact of COVID-19 on vaccination: one in the early months of the pandemic speculatively examining the impact of coverage disruptions, and another after data had become available on what disruptions had occurred^14,15. Both workstreams were developed in collaboration with stakeholders, including WHO and UNICEF, who contributed to defining the scope of the question and communicating the results. The results have informed discussions on prioritisation of vaccination and in understanding the disruption itself¹⁶. Furthermore, VIMC estimates were utilised by Abbas et al. to make the case for continued routine immunisation during COVID¹⁷.

5.2 Immunization Agenda 2030

IA2030 is a global vision and strategy set out by the World Health Organisation in 2020 with support from countries and partners to ensure strong immunisation systems, and as part of that, the global immunisation committed to several goals by 2030, including 50m lives saved^9,18. VIMC estimates are a key input into the Immunization Agenda 2030 estimates of vaccine impact. They inform ‘impact goal 1: Prevent disease, indicator 1.1,’ which presents a target number of lives saved through routine vaccination by 2030. VIMC will also be supporting future updates to these indicators. Throughout this process, there has been a productive dialogue between the teams and with the wider vaccine community through the WHO Immunization and Vaccines related Implementation Research Advisory Committee (IVIR-AC). The 50 million deaths averted figure, estimated through the IA2030 project, is based on a ‘year of vaccination’ view of vaccine impact directly informed by VIMC methods. Through the close dialogue between VIMC and the IA2030 project team, and the adoption of VIMC methods by the IA2030 project team, a wider level of feedback and scrutiny has been sought, which enhances both the IA2030 and VIMC estimate generation.

5.3 Uses of estimates by BMGF and Gavi

Gavi has utilised VIMC estimates to develop health impact targets in their 5.0 investment opportunity, report on progress in annual reports, and compare impact and value for money of new vaccines being assessed as part of the Gavi Vaccine Investment Strategy to that of current portfolio vaccines^18–22. Gavi uses (a) the impact by year of vaccination method to reflect the full value /long-term impact of vaccination as well as to place vaccines that impact mortality later in life (e.g. HepB and HPV), on the same playing field as vaccines that have more immediate impact such as measles. (b) the average impact across multiple disease models for each antigen in order to account for uncertainty in estimated impact due to model differences. Both Gavi and IA2030 are using the same approach for key reporting on Gavi 5.0 and IA2030 estimates to ensure clarity and alignment. BMGF has used estimates for internal strategy development and reporting.

7.1 Vision by 2027

By the end of the five-year grant in 2027, VIMC expects to be:

7.2 Strategy

A major update of estimated vaccine impact is planned for 2023/4 to support Gavi 6.0 and IA2030, with annual updates to support the ongoing work of Gavi, BMGF, WHO, and other stakeholders. Prior methodological and technical investments in VIMC 1.0 deliver the infrastructure which allows standardised model outputs to be submitted, rigorously checked, and analysed across a broad range of vaccines. Fewer major updates will allow space for a more proactive and responsive research programme providing evidence for vaccine policy. Modelling groups with foundational support will be commissioned to respond to specific policy questions in their disease area. Further requests for proposals (RfPs) will be issued by the secretariat for vaccine impact modelling that falls outside the scope of the consortium’s existing model capability. For all RfPs (including those for new foundational models), we will particularly encourage applications from modelling groups in countries with a high disease burden. To ensure that our research is timely, relevant, and useful, we will strengthen our relationships with key stakeholders and policymakers and an active process of soliciting research questions will be led by the VIMC Director. These questions will be prioritised in line with VIMC resources and according to a systematic process, see Figure 4 for considerations for prioritisation.

Figure 4. Considerations for prioritisation of modelling and policy questions.

VIMC 2.0 is working with modelling groups in Africa and Asia who are supported by parallel initiatives. Additionally, we will invite individuals from across our network to become affiliate members of VIMC. Affiliate membership will facilitate access to VIMC resources, including regular webinars, training opportunities, and a buddy scheme for PhD students to support peer-to-peer learning and encourage the development of international links between students. Training activities will be co-created in partnership with LMIC modellers and a bespoke vaccine modelling course will run in alternate years complemented by visiting fellowships of longer duration.

One of the key strengths of VIMC is research capacity across multiple pathogens and antigens, enabling the consortium to examine cross-cutting issues in VPDs. Initial specific research questions will focus on optimising the use of current vaccines by examining the potential for improvement in schedules, campaign synergies, and efficiency in the use of stockpiles. Furthermore, inequity in vaccine access due to socioeconomic or geographic factors, and the differential benefits of vaccination in individuals with multiple deprivations, are key areas of research where VIMC can build on its existing expertise.