Gates Open Res Gates Open Research 2572-4754 F1000 Research Limited London, UK 10.12688/gatesopenres.13649.1 Research Note Articles Better pandemic preparedness does correlate with lower COVID-19 mortality

[version 1; peer review: 3 approved with reservations]

 Dowell Scott F Conceptualization Formal Analysis Methodology Visualization Writing – Original Draft Preparation https://orcid.org/0000-0002-3878-5056 a 1 Wang Haidong Formal Analysis Methodology Writing – Review & Editing 2 Blazes David Formal Analysis Writing – Review & Editing 1 Bill & Melinda Gates Foundation, Seattle, Washington, 98109, USA Institute for Health Metrics and Evaluation, Seattle, Washington, 98104, USA Scott.Dowell@gatesfoundation.org

No competing interests were disclosed.

 5 7 2022 2022 6 76 6 6 2022 Copyright: © 2022 Dowell SF et al. 2022 This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Paradoxically, many countries with strong pandemic preparedness have reported high coronavirus disease 2019 (COVID-19) mortality. After adjusting for country under-reporting by using total excess mortality estimates, and age distributions by using indirect standardization, we find the resulting comparative mortality ratios are predicted by pandemic preparedness. Countries with higher scores on the Global Health Security Index had significantly lower COVID-19 mortality (r(192) = -.32, p<.001). These findings can help inform and prioritize future pandemic preparedness work.

 COVID-19 Mortality Pandemic Preparedness Global Health Security Index Pandemic Preparedness and Response Gates Foundation INV-008166 This work was supported by the Gates Foundation [INV-008166]. Research note

An apparent paradox of the coronavirus disease 2019 (COVID-19) pandemic has been that many countries with the strongest public health systems have reported the highest numbers of COVID-19 cases and deaths. Early in the pandemic it became clear that national-level assessments using preparedness measures such as the World Health Organization (WHO) Joint External Evaluation (JEE) or the Global Health Security (GHS) Index showed poor correlation with countries’ reported COVID-19 mortality, which was much better predicted by simple measures such as the percentage of the population at 65 years and above 1, 2 . Analyses of US states similarly showed poor correlation with the National Health Security Preparedness Index and the reported rates of COVID-19 mortality in the first 6 months of the pandemic 3 . A more recent analysis showed that countries with higher scores on the GHS Index and the International Health Regulations (IHR) self-assessments did have fewer reported COVID-19 deaths, but only for the first 8 weeks 4 . A comparison of 12 pandemic preparedness indices with cumulative infection-fatality ratios showed that the age profile of the country, but not preparedness index scores predicted infection-fatality ratios 5 .

Consistent and standardized reporting has been a global challenge. Two main problems confound the use of reported COVID-19 mortality in these analyses – younger ages and more under-reporting are both more common in low-income countries with limited testing. Although age is the strongest determinant of COVID-19 mortality, many countries do not report the ages of COVID-19 deaths, and analyses to date are surprisingly inconsistent in attempts to age-standardize. Because poor countries tend to have both lower preparedness scores and younger populations low preparedness tends to correlate with low COVID-19 mortality without age standardization. Second, those countries with the strongest and most transparent public health surveillance systems are likely to detect and report the most COVID-19 deaths. Under-reporting of COVID-19 mortality by factors of 50- or 100-fold is common among low-income countries with weaker testing and public health surveillance 6– 8 .

Recently several groups including the Institute for Health Metrics and Evaluation (IHME) have produced estimates of COVID-19 mortality using total excess mortality and modeling approaches to adjust for low testing and under-reporting 6, 8, 9 . In combination with indirect age-standardization it is now possible to directly compare relative COVID-19 mortality across countries after adjusting for a country’s specific age distribution and its under-reporting of COVID-19 deaths. Although the lack of age specific excess mortality estimates precludes the direct comparison of age standardized excess mortality rates, the comparative mortality ratio, a form of indirect age standardization, can be computed by comparing total excess mortality estimates with an expected COVID-19 mortality based on a global age pattern of COVID-19 mortality rate and age specific population from countries.

The comparative mortality ratio is calculated using the following formula:

C M R = E x c e s s D e a t h s c ∑ i = 0 A μ i g ⋅ p i c

Where CMR is the comparative mortality ratio; c refers to country; g refers to global; i refers to age, with maximum at A; p is population (person years of exposure); and µ is the mortality rate.

A different picture emerges once countries are ranked using comparative mortality ratios, rather than excess mortality estimates 10 . Many high-income countries including the United States, Italy, Germany, Great Britain, Spain, and France had more than 150,000 estimated excess deaths through the first 2 years of the pandemic. Even after adjusting for their large population sizes, their excess mortality rates still rank among the top half of all countries. However, once the older age structure of the population is taken into consideration, the ranking of comparative mortality ratios among these nations falls to the bottom third of all nations. On the other hand, many low- and middle-income countries with lower excess death counts and excess mortality rate exhibit much higher comparative mortality ratios, indicating worse performance during the pandemic, as shown in Figure 1.

Global picture of comparative coronavirus disease 2019 (COVID-19) mortality for countries and subnational regions.

Comparative mortality ratios adjust for different age distributions using indirect age standardization and under-reporting by estimations using total excess mortality.

The correlation between the GHS Index preparedness score and this age- and underreporting-adjusted mortality is shown in Figure 2. The figure and correlation analysis were done using Excel version 2204. Countries with higher preparedness scores have significantly lower relative COVID-19 mortality as estimated by the comparative mortality ratio (r(192) = -.32, p<.001).

Coronavirus disease 2019 (COVID-19) mortality by preparedness score for 194 countries.

Preparedness scoring is by the Global Health Security Index in 2021 2 . COVID-19 mortality is adjusted for age by indirect standardization and for under-reporting by estimations using total excess mortality 6 . The correlation is statistically significant (r(192) = -.32, p<.001).

Demonstrating that better preparedness did correlate with better control of COVID-19 is important as countries and institutions begin to turn from acute efforts to control the pandemic to longer term plans for preventing and controlling the next one. Although it is not surprising that countries with the highest scores in areas such as laboratory testing, disease surveillance, and transparency in reporting would also be the most likely to detect and report COVID-19 deaths, it has been surprisingly difficult to adjust for this bias in measuring COVID-19 mortality. The adjusted measure not only confirms the expected relationship to preparedness but should allow for future comparisons of countries with similar preparedness scores and much different adjusted COVID-19 mortality. This relationship allows for better understanding of the impact of specific public health mandates, health system capacities, vaccines, or therapeutics. Better preparedness, as measured by the GHS Index, the JEEs, and similar indices, before a crisis such as COVID-19 offers countries the best protection against excess pandemic mortality.

 Data availability Source data

The GHS Index scores for 2021 can be obtained from https://www.ghsindex.org/report-model/ and download raw data files.

 Extended data

Open Science Framework: Better Pandemic Preparedness Does Correlate with Lower Covid-19 Mortality. https://doi.org/10.17605/OSF.IO/9F65Q 10 .

This project contains the following extended data:

preparedness and cmr data 2019–21.xlsx (the COVID-19 comparative mortality ratios)

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

 Acknowledgements

We thank Spencer Pease for help in preparing the comparative mortality ratios, Reed Sorensen for the global age patterns of Covid-19 mortality, and Haley Comfort for assistance with Figure 1.

 Haider N Yavlinsky A Chang YM : The Global Health Security index and Joint External Evaluation score for health preparedness are not correlated with countries' COVID-19 detection response time and mortality outcome. Epidemiol Infect. 2020;148:e210. 32892793 10.1017/S0950268820002046 7506172 Bell JA Nuzzo JB : Global Health Security Index: Advancing Collective Action and Accountability Amid Clobal Crisis. 2021. Reference Source Keim ME Lovallo AP : Validity of the National Health Security Preparedness Index as a Predictor of Excess COVID-19 Mortality. Prehosp Disaster Med. 2021;36(2):141–4. 33397547 10.1017/S1049023X20001521 7809220 Duong DB King AJ Grepin KA : Strengthening national capacities for pandemic preparedness: a cross-country analysis of COVID-19 cases and deaths. Health Policy Plan. 2022;37(1):55–64. 34608933 10.1093/heapol/czab122 8522413 COVID-19 National Preparedness Collaborators: Pandemic preparedness and COVID-19: an exploratory analysis of infection and fatality rates, and contextual factors associated with preparedness in 177 countries, from Jan 1, 2020, to Sept 30, 2021. Lancet. 2022;399(10334):1489–512. 35120592 10.1016/S0140-6736(22)00172-6 8806194 COVID-19 Excess Mortality Collaborators: Estimating excess mortality due to the COVID-19 pandemic: a systematic analysis of COVID-19-related mortality, 2020-21. Lancet. 2022;399(10334):1513–36. 35279232 10.1016/S0140-6736(21)02796-3 8912932 Rose SM Paterra M Isaac C : Analysing COVID-19 outcomes in the context of the 2019 Global Health Security (GHS) Index. BMJ Glob Health. 2021;6(12):e007581. 34893478 10.1136/bmjgh-2021-007581 9065770 Economist: The pandemic's true death toll. The Economist;2022. Reference Source WHO: Global excess deaths associated with COVID-19, January 2020 - December 2021. 2022. Reference Source Wang H Pease S Comfort H : Raw Comparative Mortality Ratios by country.2022. http://www.doi.org/10.17605/OSF.IO/9F65Q 10.21956/gatesopenres.14930.r32821 Reviewer response for version 1 Oppenheim Ben 1 Referee https://orcid.org/0000-0003-1487-3534 Madhav Nita 1 Co-referee https://orcid.org/0000-0001-8077-330X New York University, Boston, MA, USA

Competing interests: No competing interests were disclosed.

 16 2 2023 Copyright: © 2023 Oppenheim B and Madhav N 2023 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

Summary:

This research note takes a new approach to analyzing the relationship between pandemic preparedness metrics and COVID-19 mortality. A number of studies have explored the apparent mismatch between preparedness metrics and national performance in controlling the COVID pandemic, using a variety of preparedness metrics, mortality estimates, and statistical modeling techniques. Findings across these studies have been inconsistent, and many analyses have generated questionable results by relying on reported mortality estimates that are confounded by high levels of underreporting.

The authors of this paper compute a comparative mortality ratio (CMR) using excess mortality estimates, an approach that attempts to account both for underreporting and (widely) varying age structures. Using this new metric, they find a negative correlation between the Global Health Security Index and COVID-19 mortality – that is, evidence that countries with higher preparedness scores had lower levels of mortality.

This research note makes a valuable contribution to the ongoing debate over how to assess national performance during health emergencies, as well as the efficacy of preparedness metrics.

Comments:

Data. It would be helpful if the authors added a short section that briefly describes the various data sources used in this analysis. The source of the excess mortality data is not made explicit, and the paper does not indicate the source of the demographic data used to compute the comparative mortality ratio estimates. It would be good to provide a citation and link to the original data, as well as an appendix noting the procedures for data cleaning (e.g. handling missing data, etc.). It could also be interesting and useful for the authors to comment on why they selected the specific excess mortality dataset to estimate the CMR, given that there are several widely-used estimates (e.g. WHO, IHME, Economist) available.

Analysis: The authors perform a correlational analysis between CMR and the aggregate GHSI scores. 

The GHSI is a complex index, which incorporates metrics measuring a number of dimensions of prevention, preparedness, response, and resilience. It could be valuable for this study to dig deeper, and explore whether particular components of the GHSI (for example, rapid response and mitigation) are more associated with country outcomes than others.

The bivariate correlation does not allow the authors to explore the effects of other factors, such as country-specific immunization rates and speed of rollout, that may be associated both with preparedness and COVID mortality. It’s potentially difficult in a cross-national, cross-sectional study, but perhaps worth exploring with multivariate regression models.

The scatterplot in figure 2 shows a handful of extreme outliers; it would be useful to run an analysis omitting these points.

Relatedly, it would also be good for the authors to identify and drop, in a supplemental analysis, any datapoints with levels of reporting failure that might not be effectively adjusted for by the excess mortality estimations. It’s hard to know without knowing which dataset was used, but the extremely low CMR for North Korea (visible in the scatter as well as fig. 1) seems questionable. 

The authors note that indirect age-standardization requires making the assumption that global age-specific mortality rates can be applied to each country in the study. Effectively this seems to mean assuming that 80 year olds in Norway and Namibia (for example) have the same mortality rate. This is a necessary assumption given the lack of age-specific mortality rates by country, but it would be helpful for the authors to comment on the implications of this assumption, and (to the extent possible) the potential direction and magnitude of effect where the assumption is violated.

From a methodological standpoint, it would be interesting to comment on the difference between the CMR and other approaches that have previously been taken in the literature, such as regressing excess mortality on preparedness metrics while controlling for demographic structure (e.g. via variables capturing the proportion of the population above age 65, etc.)

Minor note:

Figure 1 is excellent. But since the analysis uses national-level data only, we would suggest removing the subnational variation from the map; it’s visually informative, but not incorporated into the empirical analysis. The CMR data provided by the authors could certainly be used for an analysis of subnational variation in COVID mortality – for example, using the NHSPI in the case of the United States – but this is beyond the scope of this specific paper.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Partly

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Epidemiology, Biosecurity, Global Health, Social Science

We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.

 10.21956/gatesopenres.14930.r32820 Reviewer response for version 1 Nelson Cassidy 1 Referee https://orcid.org/0000-0003-0934-8313 Department of Biology, University of Oxford, Oxford, UK

Competing interests: No competing interests were disclosed.

 31 1 2023 Copyright: © 2023 Nelson C 2023 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

Summary of the article:

This article provides a succinct insight into an important question: why have countries ranked higher in pandemic preparedness in the Global Health Security Index (GHSI) experienced greater COVID-19 mortality. This issue has been pointed out previously in multiple peer-reviewed publications. The authors address the problem from a new angle, using the comparative mortality ratio (CMR) to allow for indirect age standardisation to compare COVID-19 deaths. The authors demonstrate that the 2021 GHSI score negatively correlates with COVID-19 mortality when estimated using the CMR. 

Specific feedback:

Work clearly presented: A minor stylistic point, but there are not enough commas after clauses and introductory statements which makes the writing less clear. 

Source data: While the GHSI score data and the computed CMRs are provided by the authors, it is unclear where the country-specific parameter data came from to calculate the CMRs. The authors cite multiple sources for adjusted estimates of COVID-19 mortality (a publication by the COVID-19 Excess Mortality Collaborators, the Economist, the WHO). It is not clear as written, but it is presumed that the first cited source is used for their CMR estimates. This source has data openly available on the Global Health Data Exchange with excess COVID-19 death estimates for each country, but other factors such as each country’s age population structure are not in these cited sources. It would be good for the authors to be explicit about the data sources for these estimates as well that went into the calculation of each country’s CMR. 

Replication: The analysis with the provided CMRs can be mostly replicated. 

Calculations of the CMRs cannot be replicated because only the final values are provided (see above). 

Figure 2 was reproducible in R with the data provided, although it is unclear why the authors omitted one country (194 included instead of 195) when CMRs and GHSI scores were available for all. 

For Figure 1, this was mostly reproducible with the data provided at the country level but intra-country territories such as state level did not have data available. It appears CMR values have been calculated for states in the US, Mexico, provinces in Canada, etc. but this data is not provided by the authors.

Analysis: For the correlation analysis, using R, the same values are reproducible r = -0.32, p<0.001 for 195 countries (degrees of freedom 193). It is presumably similar for 194 countries (df 192) as provided by the authors, although as stated above, which country that was omitted is not stated, or clear in the paper.

Conclusions supported by results: The conclusions are mostly supported by the results, but the final statement “Better preparedness, as measured by the GHS Index, the JEEs, and similar indices, before a crisis such as COVID-19 offers countries the best protection against excess pandemic mortality.” is not fully supported as 1) this paper has not shown these results for other indices like the JEE (although GHSI score and metrics like the e-SPAR score are highly correlated) and 2) stating that preparedness as measured offers the “best” protection would require more comparators. All the same, the authors conclude correctly that preparedness does correlate with improved outcomes and that this is important for future long term planning.

 Overall this is a useful piece of research and a needed addition to the discussion on this topic. I think with the points above addressed it is suitable for indexing.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

Partly

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Medicine, Public Health, Infectious Disease Epidemiology, Global Health, Mathematical Modelling, Biosecurity

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

 10.21956/gatesopenres.14930.r32451 Reviewer response for version 1 Graeden Ellie 1 Referee Talus Analytics, Washington, DC, USA

Competing interests: I have worked on several projects funded by the Gates Foundation in the last few years.

 20 9 2022 Copyright: © 2022 Graeden E 2022 This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. recommendation approve-with-reservations

This short report analyzes whether COVID-19 mortality rates correlate with pandemic preparedness, as measured by the Global Health Security Index. The paper, unlike prior analyses, normalizes the mortality data by adjusting for population size, age structure, and rates of case reporting. 

The paper is overall very strong and provides a valuable contribution to the field in re-evaluating a prior finding from the literature and assessing confounding factors, including age structure of the population and reporting discrepancies.

Recommendations prior to publication:

The map in Figure 1 is great. Could the authors add a figure for the counterfactual? What does the analysis/visual look like without the standardizations and adjustment? A four panel figure could show (a) mortality rates by country without adjustments, (b) with adjustment by age, (c) with adjustment from under-reporting, and (d) adjustment with both. That comparison would really drive home the message and help show the contribution of each variable.

It would be helpful to see the regression in figure 2 performed for the JEE scores in addition to GHSI scores, given that the JEE scores are the WHO standard.

Please clarify which estimate of Excess Deaths per country is being used in this analysis. Additional detail around the specific methods of correction and standardization, especially for underreporting would be helpful.

Could the authors please provide additional detail in the text clarifying the age stratification and method of standardization used for analysis? The citation included links to a dataset doi that does not appear to contain any data.

Given that age is only one determinant of COVID-19 mortality, it would be helpful to both cite the assumptions related to age-based mortality rates and include a discussion of additional characteristics that may be relevant.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Partly

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Data analysis for global health security

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.