Novel therapeutic and vaccine development for coronavirus disease 2019 (COVID-19) has advanced at extraordinary speed and scale. Against all historical precedent, multiple products have progressed from discovery through emergency regulatory approval in less than one year. In addition to these novel discoveries, scientific efforts have also focused on assessing the arsenal of existing products to determine whether off-label use of products could impact COVID-19 acquisition, disease progression and mortality. Within weeks of the identification of COVID-19, myriad claims were made about the potential efficacy of approved products, which consequently provoked supply shortages of multiple essential drugs and the initiation of many uncoordinated clinical trials.

We utilized a commercially available electronic health record (EHR) derived de-identified privacy-protected HIPAA-compliant protected health information (PHI)-free secondary-use assented real-world data (RWD) repository (Cerner Healthe DataLabs RWD™, Cerner Corporation, Kansas City, MO, USA). All data was fully anonymized and there was no direct human subject research, and we thus did not seek approval from an Institutional Review Board. The repository comprises records from 2010 to present, with over 500 million care-related records and 160 million distinct patients. Material is encrypted end-to-end and pre-cleaned upon arrival into a data warehouse. Each data element is timestamped, and multiple episodes for a single person are longitudinally tracked with an encrypted key. The instance of the Cerner RWD used for the present study contains longitudinally-linked health records for 100% of patients in the care of 90 U.S. institutions and their affiliated ambulatory care clinics and physician offices, accessed via a private cloud environment with Python and Pandas software and ANSI-standard SQL. The data cleaning, transformation and further analyses were performed with R 4.0.4 (The R Foundation) and RStudio 1.4.1106 (RStudio, PBC).

At the time of our analysis, there were 373,032 PCR-proven, distinct COVID-19 patients in the Cerner Healthe DataLabs RWD™ repository between 01-MAR-2020 and 31-DEC-2020 in 90 tenants (extended data, Table S1; McNair et al., 2021). We retained 347,570 records in the 38 tenants whose contributed cases comprise more than 0.5% of the aggregate sample size (extended data, Figure S1; McNair et al., 2021). Demographic characteristics of all patients are available in Table S1 ( McNair et al., 2021).

After an initial review of 25 vaccines in common use in the US, we narrowed the analysis to two vaccines with signals warranting further attention: an MMR vaccine, and a recombinant adjuvanted varicella-zoster vaccine. For MMR-related analyses, we confined the records to persons aged 25 to 64 years old inclusive at the time of their 01-MAR-2020 and 31-DEC-2020 index episode ( Table 1). In general, such subjects received MMR in connection with their employment as health care workers, first responders, teachers, and other professions in which employers require periodic titer measurement and revaccination with MMR, and the vast majority of vaccinated MMR subjects are female. Not surprisingly, there are relatively few prevalent persons aged 65 and older at the time of their 01-MAR-2020 and 31-DEC-2020 index COVID-19 episodes who received MMR vaccination within the preceding 5 years. In similar fashion, we confined our analyses related to the recombinant adjuvanted varicella-zoster vaccine just to persons 50 years of age or older and excluded subjects receiving that same vaccination at younger ages ( Table 1).

For each PCR-proven COVID-19 subject in the real-world data repository selected by SQL query inclusion-exclusion criteria, we utilized vaccination for influenza within the lookback time horizon as a proxy for vaccine non-hesitancy. This was done to account for potential confounding due to health-seeking behavioral attributes that might be associated with vaccination ( Fireman et al., 2009; Pollard et al., 2017). We required ICD-10-CM U07.1 code denoting COVID-19 disease and a positive PCR result for SARS-CoV-2. We recognize that PCR sensitivity is less than 100% and this composite selection criterion may therefore be overly conservative. We designed SQL to retrieve records from the real-world data repository with selection criteria to provide comparability of cases and controls in terms of (a) a diverse range of comorbid ambulatory care-sensitive conditions (ACSCs) and (b) time span of longitudinal follow-up subsequent to the index episode, as well as likelihood and timing of contact with the health system enabling endpoint ascertainment. The subjects included in our analysis are patients whose EHR records showed that they received at least one medication prescription in connection with their care episodes. We recognize that this selection criterion excludes some people without known ACSCs who presented to an ambulatory clinic/office or who may have been dismissed with no prescribed medications or only with recommended OTC medications, and no follow-up plan.

Persons who presented to the institutions’ emergency departments or who were admitted to hospital as inpatients had, in general, clinical findings and physician orders indicative of more severe COVID-19 disease than persons whose care was delivered in ambulatory settings. Emergency Department patients and inpatients tended to have multiple symptoms, larger numbers of physician orders, larger numbers of episodes of care preceding the index episode, larger numbers of comorbid conditions, and laboratory result values in the records of the index episode consistent with severity of illness such as would require care in an acute-care venue. By contrast, records for many of the ambulatory outpatient episodes lacked any SNOMED-CT observation or finding denoting a COVID-19 symptom, despite having PCR-positive test results associated with the U07.1 ICD-10-CM code.

On preliminary analysis of the prevalent PCR-proven COVID-19 cases in the real-world data ( McNair et al., 2021), prior vaccinations with a recombinant adjuvanted varicella-zoster vaccine or MMR do not appear to have significant protective effects against SARS-CoV-2 infections (extended data, Table S3; McNair et al., 2021), but they do appear to show substantial effects reducing COVID-19 case fatality rate, with the MMR result rendered inconclusive when using additional statistical controls described below (extended data, Table S2; McNair et al., 2021). Because of the null result for a protective effect, for the subsequent analyses we focus on the effects of prior vaccinations in reducing COVID-19 case fatality rate. However, the case fatality ratio (CFR) when expressed as an effect size exhibits substantial heterogeneity among the institutions contributing the cases to be analyzed. I ² values indicate that 99% of the total variance in MMR- and recombinant adjuvanted varicella-zoster vaccine-associated mortality-prevention effect-size across studies is due to systematic (site-wise by source institution) heterogeneity rather than due to random sampling error. Bias of I ² estimates is expected to be negligible for a collection of 90 source institutions as in the Cerner repository of COVID-19 cases and contemporaneous populations of patients receiving care during the pandemic ( von Hippel, 2015).

We determined the absence of disparities in vaccine provisioning by race and gender by the Cochran-Mantel-Haenszel test ( Landis et al., 1978; Mantel, 1963; Mantel & Haenszel, 1959) noting that the exact version of the generalized Cochran-Mantel-Haenszel test does not necessarily result in the same p-value as Fisher's exact test ( Davis, 1986). Evidence-combining was performed by standard methods of meta-analysis ( Lipsey & Wilson, 2001; Singh et al., 2005; Van Houwelingen et al., 2002).

Our several attempts to use covariables available in the EHR-derived RWD repository to devise a suitable propensity score predicting MMR exposure were unsuccessful in achieving acceptable balance. Hosmer-Lemeshow calibration was poor across propensity score quartiles for MMR exposure. Furthermore, the best of our provisional propensity scores for MMR exposure demonstrated residual heterogeneity I ² = 43%, a value that we believe to be too large to permit reliable interpretation of analyses of efficacy endpoints in real-world data (ideally below 10%; see ( Austin & Stuart, 2015; Berger et al., 2017)). The average treatment effect size calculated for the available data is not significant, and the sample size is not presently sufficient to power detection of so small an effect size ( Munoz & Rosner, 1984; Nam, 1992; Wallenstein & Wittes, 1993; Woolson et al., 1986). For this reason, we plan to defer further analysis of MMR non-specific protective efficacy until a larger sample size has accrued or pooling of data and meta-analysis with other repositories (such as the Oxford University – OHDSI Scylla repository) becomes possible. In general, findings were consistent that the presence of MMR vaccination did not yield lower case fatality rates than controls, but given the limitations described above, we could not derive clinically significant results from this dataset.

By contrast to MMR, the available data on the recombinant adjuvanted varicella-zoster vaccine yielded a propensity score having acceptable performance in terms of covariable balance and reduction in heterogeneity after matching ( Figure 1– Figure 4). Here, the overall odds ratio for CFR was 0.43 (95%CI: 0.38 - 0.48). Thus, the case fatality-related non-specific vaccine efficacy (VE) for a recombinant adjuvanted varicella-zoster vaccine vaccination prior to SARS-CoV-2 infection is approximately 100*(1.00 – 0.43) = 57%.

In the present real-world data analysis we did not find evidence that recent prior vaccination of adults with either a recombinant adjuvanted varicella-zoster vaccine vaccine or MMR vaccine conferred substantial protection of individuals against acquiring SARS-CoV-2 infection (extended data, Table S4; McNair et al., 2021). For case fatality rate, inadequate MMR sample size and covariable distributions prevented construction of a propensity score having adequate performance for weighting or matching on covariables to reduce bias and heterogeneity sufficiently to enable reliable analysis. Sample size and event counts were, however, sufficient for the cohort who received a recombinant adjuvanted varicella-zoster vaccine to enable construction of a suitable propensity score, and analysis of propensity score matched cases and controls revealed a clinically meaningful benefit in reduced case fatality rate.

Regarding limitations of the present analysis, we recognize that there may likely be a temporal relation of non-specific protection by vaccines against COVID-19 or other infections. We conducted exploratory analyses [using methods reported by ( Zhang et al., 2020)] of the recency of recombinant adjuvanted varicella-zoster and MMR vaccinations to the date of presentation for the index COVID-19 episode. However, in the real-world data presently available for analysis there were insufficient patients exposed to a recombinant adjuvanted varicella-zoster vaccine between 2015 and 2017 to enable detailed analysis of the effect of the time interval between vaccination and SARS-CoV-2 infection. Therefore, our analysis relied on the recombinant adjuvanted varicella-zoster vaccine exposures between 2017 and 2019, inclusive. Further analysis of temporal relationships and durability of non-specific heterologous protection awaits the accrual of a larger cohort or pooling of data with other investigators’ real-world data repositories.

Beyond the concerns of the present work, we believe that this study illustrates the importance of certain best practices for reliable analysis of real-world data [see ( Berger et al., 2017; Fang et al., 2020; Stürmer et al., 2020). In particular, we recommend the following six rules and practices:

The first of these practices reflects considerations of the difficulty of collecting real-world data involving a large number of mapped, curated variables for propensity score adjustment and inferential modeling. Typically, real-world data tend to consist of dozens or hundreds of variables but a smaller number of data-contributing sites, which means p > n (p is the number of predictor variables and n is the number of observed sites). In addition, exposure counts and outcome events often have over-disperse distributions and have low prevalence. Thus, the data are sparse and unbalanced. This makes it challenging to apply classical statistical analysis methods ( Hastie et al., 2009). Under such conditions, a threshold is applied to focus on patterns and statistical associations within sites having substantial prevalent cases and controls, such that data sparsity will be less influential on analyses and analyses will not be dominated by one or two contributing sites having large numbers of prevalent cases. Minimum prevalence thresholding in RWD analysis is similar to the practice of applying sample selection filters for microbiome analyses of diversity based on minimum operational taxonomic unit (OTU) abundance and minimum total number of 16S rRNA reads ( Bokulich et al., 2013).

In summary, speculation about possible non-specific heterologous vaccine efficacy likely can only be conclusively resolved by appropriate randomized control trials. However, valuable evidence may be obtained from real-world data, provided that adequate samples size and endpoint event counts are available and appropriate adjustments are undertaken to mitigate biases and confounding. Such evidence includes estimates of effect-size and average treatment effectiveness, which may help to inform statistical analysis plans for prospective trials ( Levenson, 2020). While there are several theoretical hypotheses regarding the concept of heterologous trained immunity possibly conferred by live attenuated vaccines and vaccines with novel adjuvants, well-designed controlled studies are needed, before considering heterologous vaccination as a modality for preventing COVID-19 infection or mitigating its severity or outcomes.