Using maternal and neonatal data collection systems for coronavirus disease 2019 (COVID-19) vaccines active safety surveillance in low- and middle-income countries: an international modified Delphi study

Panel composition

This study is the second phase of a larger study entitled “Landscape analysis: Sentinel site readiness for Maternal Immunization Active Safety Surveillance in LMIC” (safeinpregnancy.org) that seeks to define the landscape for integrated maternal immunization active safety surveillance by reviewing the literature (phase 1), building consensus among experts (phase 2), and identifying potential sentinel sites in LMICs (phase 3). An independent Scientific and Technical Advisory Board (STAB) integrated by 12 experts in the field was already established to provide up-to-date and globally representative advice on the larger study. For the Delphi study, all members of the STAB and four additional experts identified by the Bill & Melinda Gates Foundation were nominated to integrate the panel. All the nominated experts had recognized international leadership in maternal immunization, vaccine safety, and maternal and neonatal health. During the recruitment process, as well as experts were selected based of their knowledge in maternal and perinatal health information systems for active safety surveillance. Geographical representation was another criteria taken into consideration. The rationale behind the 16-member panel was that they were large enough to permit diversity of representation while still being small enough to allow everyone to be involved in the group discussion.

To prevent the bias that could be introduced by knowing other leaders' opinions on the subject, the experts answered the questionnaires in each round anonymously and individually. The panel did not meet with each other face to face before the final meeting. The Delphi technique aims to generate consensus, and for this reason, each participant must know the opinion of the group. However, during all rounds, the shared data was de-identified, and in the final meeting, participants declared whether they had any conflict of interest and a moderator sought to prevent some participants from leading the discussion.

Development of questionnaires

A bibliographic search was carried out in PubMed and grey literature. The following terms were employed to identify relevant peer-reviewed literature, reports published by governments or international health organizations (i.e. WHO), and other relevant articles to inform survey development: “COVID-19”, “coronavirus”, “SARS-CoV-2”, “pregnancy” and “pregnant women” were combined separately with the keywords: “active safety surveillance”, “vaccines”, “vaccine clinical trial”, “vaccine development”, “COVID-19 vaccine development”, “COVID-19 vaccine landscape”, “vaccine safety surveillance”, “sentinel site surveillance”, “information systems”, “safety monitoring”, “maternal immunization”, “low income countries”, “low- and middle-income countries” and “LMIC”.

This search identified publications that described factors affecting experts’ opinion on the use of routine information systems for COVID-19 vaccine active safety surveillance. Additionally, this search identified publications describing factors affecting experts' views on pregnant women's inclusion for novel COVID-19 vaccine R&D activities. Reported variables to be collected for maternal immunization safety monitoring were identified and listed. Subject matter experts were consulted to validate a preliminary list of variables based on the literature search. All this information was used to inform the development of the first questionnaire.

The first questionnaire included the following domains: a) appropriateness of using existing maternal and neonatal data collection systems for active safety surveillance, b) feasibility of adapting these systems prior to the pregnancy licensure of novel COVID-19 vaccines, c) benefits versus risks of including pregnant women in COVID-19 vaccine trials; d) feasibility of including pregnant women in the target population for COVID-19 vaccine R&D (in terms of the capacity of existing research networks and acceptability among vaccine decision-makers); and e) feasibility of incorporating existing research networks in LMICs (in terms of capacity and decision-makers’ acceptability). Additionally, lists of maternal, neonatal, and vaccine-related variables (i.e., name of the vaccine, date of vaccination, administration site, lot number) were included in the questionnaire to their prioritize their importance for active safety surveillance. The list of maternal and neonatal variables was selected from GAIA project case definitions piloted in 24 sentinel sites across four WHO regions¹⁷. Questionnaires for the second and third rounds included a selection of questions to address disagreements that were noted in previous rounds. Questions related to selecting variables for active safety surveillance were re-evaluated in round two with new variables suggested by experts in the first round. All three questionnaires included both closed-ended and open-ended questions. The questionnaires can be found as extended data¹⁸.

Data collection

Data were collected through an iterative consensus exercise, starting with two rounds of online, individual surveys followed by a third round, which involved a virtual group discussion and final vote on remaining dissents. Online surveys were self-administered via Survey Monkey™. For both online rounds, panellists were asked to rate questions on a 9-point Likert RAND Appropriateness Scale (RAS)¹⁹. In addition, the rationale for each rating was collected using open-ended questions.

During the second round, experts received the overall rating distributions and comments from the previous round and were asked to re-rate the questions that generated dissent.

The first survey was sent on June 29^th, 2020, to 16 selected experts who had confirmed their interest and availability to participate. Three reminders were sent to participants with partial or no response over two weeks. The second survey was sent on July 22^nd, 2020, to the same 16 experts, followed by four reminders over another two weeks. Completeness and consistency between the survey items were monitored during survey administration.

During the third round, experts met via virtual meeting on August 26^th, 2020, to confirm the generated agreements and address remaining disagreements. Presentations and guided discussions were held in plenary sessions. The main questions to be discussed were developed by researchers with experience in qualitative research (MB, VP and SM). A meeting facilitator (PB) conducted and moderated the discussion (60 minutes). Detailed field notes were taken by one of the researchers (SM). During the plenary session, the system Poll Everywhere^® was used to record the final individual vote.

Data analysis

Panel median rating and disagreement index (DI) were calculated to summarize experts’ ratings and measure consensus. The DI is a continuous scale used to measure panel ratings' dispersion, taken as an indicator of the level of agreement¹⁹. A DI < 1 (including negative values) represents a complete agreement among panellists, while a DI ≥ 1 indicates disagreement among the panel. The RAND/UCLA appropriateness scale (i.e., RAND, a 9-point Likert scale ranging from 1 to 9) was used to classify domains as "appropriate", “inappropriate" or "uncertain", in accordance with: a) the median panel rating, and b) the DI. Domains with median ratings in the top third (score of 7–9) of the appropriateness scale were classified as “appropriate”, those in the bottom third were classified as “inappropriate” (score of 1–3), and those with intermediate median ratings were classified as “uncertain” (score of 4–6). Besides, domains with disagreement among the panel was also classified as “uncertain”. Table 1 provides the definitions for how each item was evaluated for “Appropriateness” and “Agreement”. The analysis was performed using STATA 14.0.

Open-ended questions were analysed using a content analysis. Responses were coded by one researcher (SM) and reviewed by other researcher (MB), and matrices were developed to help interpret the findings. The analysis process was facilitated by a qualitative data management software Atlas.ti 8.4. All rationales that panellists provided to inform their quantitative responses are included in the findings, regardless of whether one or more person mentioned them.

The review was conducted and reported in line with the standards of the COREQ statement for qualitative studies and STROBE checklist for observational studies.

Ethical approval

Approval was obtained from the Tulane University Internal Review Board (Approval number 2019–1453), United States, and the Comité de Ética en Investigación “Norberto Quirno”, Argentina (Approval number 1258/2019). All participants provided electronic written informed consent to participate in the study.

Use and adaptation of existing maternal and neonatal data collection systems in LMICs for active safety surveillance

The expert panel agreed (RAND DI -0.92) on the appropriateness of using existing maternal and neonatal data collection systems in LMICs for active safety surveillance for COVID-19 vaccines, given that 12 out of 16 experts assigned a rating in the top third of the scale (7–9) and the panel median was 9 (Table 3). This question reached a consensus in round 1 and was not re-evaluated in subsequent rounds.

Some participants who gave lower ratings on this item cited limitations in systems and their overall lack of existence. Some participants mentioned that existing systems would need to be adapted and improved to collect meaningful data and link maternal and neonatal information.

The panel consistently disagreed (DI 2.45; 1.18; 1.79 for rounds 1 through 3, respectively) on the feasibility of existing maternal and neonatal data collection systems in LMICs to incorporate new changes before approval or licensure of novel COVID-19 vaccines so that adequate active safety surveillance can be performed amongst pregnant women (Table 3). This indicates a considerable level of experts’ uncertainty regarding this feasibility. Experts acknowledged that there are existing systems that could be modified. However, the variability between systems and locations was noted, and the feasibility of adapting these systems depended on the financial and human resources needed to make these changes before licensure. Support would be required to increase technical, managerial, and staff capacity. Given the accelerated timeline of COVID-19 vaccine development, it was noted that there might not be enough time or resources to adapt these systems before vaccine approval or deployment (Box 1).

Maternal, neonatal, and vaccine variables for active safety surveillance

Table 4 shows the final set of 26 maternal, neonatal and vaccine-related variables prioritized by the experts. Variables were prioritized if they were selected by at least seven panel members (50% of the panel) during the second round.

The following nine maternal variables were prioritized after receiving the vote of at least half of the experts' panel: maternal death, spontaneous abortion, hypertensive disorders of pregnancy, intrauterine growth retardation, threatened preterm labor, maternal age, maternal pneumonia, postpartum hemorrhage, and chorioamnionitis. All variables were prioritized during both rounds, except for postpartum hemorrhage, maternal age, and pneumonia, which were only prioritized during the second round. The following maternal variables were voted less often (by 1 to 6 experts): antenatal bleeding, postpartum hemorrhage, gestational diabetes, history of asthma or other respiratory disorders, endometritis, anemia, maternal gestational weight gain, hyperemesis gravidarum, dysfunctional labor, abnormal placentation, and pathways to preterm birth.

For the neonatal (late fetal period or neonatal) variables, at least half of the experts prioritized the following 11 variables (10 medical conditions and 1 enabling term): neonatal death, preterm birth, stillbirth, congenital anomalies, low birth weight, small for gestational age, respiratory distress, live birth, neurodevelopmental delay, microcephaly and assessment of gestational age. Other neonatal variables were prioritized less frequently: failure to thrive, neonatal encephalopathy, and neonatal seizures. All neonatal variables had very similar frequency distributions in each round.

At least half of the experts prioritized the following six vaccine-related variables were the following: date of vaccination, vaccine lot number, name of the vaccine, dosage, vaccine expiration date and gestational age at vaccination. All these variables were prioritized on both rounds. Other vaccine-related variables less frequently prioritized were: time of reconstitution, time of vaccination, manufacture date, and diluent expiration date. Similar to the neonatal variables, vaccine-related variables presented very similar frequency distributions in each round.

A smaller set of 14 variables were identified by increasing the expert panel's level of agreement to 80%. In this case, the variables that are prioritized were those that received at least 11 votes and are as follows: maternal - maternal death, spontaneous abortion, intrauterine growth retardation; neonatal - neonatal death, preterm birth, stillbirth, congenital anomalies, low birth weight, small for gestational age, respiratory distress; and vaccine-related - date of vaccination, vaccine lot number, name of the vaccine, gestational age at vaccination.

Inclusion of pregnant women in the target population for COVID-19 vaccine R&D

The suitability of the inclusion of pregnant women in the target population for novel COVID-19 vaccine R&D activities was evaluated according to two factors: “Potential benefits versus the potential risks” and “Time appropriateness,” and the results are presented in Table 5. The panel weakly agreed (RAND DI -3.08) that the potential benefits outweigh the potential risks of including pregnant women for COVID-19 vaccine R&D and found this statement appropriate given that the panel median rating was 7 (top third of scale: 7–9). However, six members still reported uncertainty, as indicated by ratings in the intermediate third of the scale (4–6), and one expert considered the risks to outweigh the benefits, as indicated by their reported rating in the bottom third (1–3). The panel agreed (RAND DI -0.93) that it is an appropriate time in the COVID-19 vaccine R&D landscape to consider pregnant women for inclusion in the target population, given that 12/16 experts assigned a rating in the top third of the scale (7–9) and the panel median was 9. These two questions reached consensus in Round 1 of this Delphi study and were not re-rated in subsequent rounds. From experts’ comments, it was observed that ratings on suitability reflected their perception of what recent maternal immunization advances have had on maternal and neonatal health. Some experts viewed that this field has advanced to the point that pregnant women should be included to facilitate decision-making for immunization policy.

Experts' perceptions on the existing evidence that would consider pregnant women an at-risk population for risk of COVID-19 on pregnancy and subsequent consequences to the fetus varied. For some experts that positively support the inclusion of pregnant women in novel COVID-19 R&D activities, this population's exclusion is a violation of pregnant women’s rights and may result in vaccine hesitancy.

The feasibility of including pregnant women in the target population for novel COVID-19 vaccine R&D activities was evaluated according to two factors: “Vaccine R&D decision-makers’ acceptability” and “Capacity of existing research networks”. Experts were also provided the opportunity to suggest other factors that may hinder the feasibility. After analysing open-ended responses from round 1, another factor related to feasibility was identified and evaluated in Round 2: “Regulatory requirements of maternal immunization vaccine studies”. There was no consensus throughout all three rounds (RAND DI 1.73; 2.04; 10.00; for rounds 1 through 3 respectively) regarding decision-makers' acceptability. This indicates a considerable level of expert uncertainty on vaccine R&D decision-makers’ acceptability of including pregnant women in the target population for novel COVID-19 vaccine R&D activities. From the experts’ point of view, decision-makers' acceptability may be conditional on regional differences, liability issues, interactions among decision-makers, and the influence certain groups have on each other, politics, and/or access to the vaccine. The necessity to develop and license a novel COVID-19 vaccine quickly to address the ongoing pandemic was considered a barrier to include pregnant women.

Another barrier outlined was that vaccine developers might not want to take the risk of including this population in vaccine R&D activities when they are utilizing novel vaccine platforms.

One facilitator contributing to the acceptability of vaccine developers and funding agencies mentioned by some respondents was that regulatory agencies are requesting vaccine developers consider the inclusion of diverse populations, including pregnant women, in their pre-licensure studies.

How the pandemic has progressed in certain regions may also affect the acceptability of vaccine decision-makers. For example, countries that were significantly affected in the early stages of the pandemic may have more of a priority to include pregnant women, increasing acceptability of vaccine decision-makers. Differences in acceptability of including this population can be seen between regions with their previous implementation of maternal vaccines (e.g., flu and Tdap), where it was easier to implement in some regions versus others.

During the virtual meeting, panelists pointed out that the region where they currently work (their local working setting) may influence their responses. Given the large variety of decision-makers, their answers may have been influenced by which decision-makers they were most familiar with. Acceptability among academics and medical organizations may be higher, but there is a dependence on those “higher-up” in authority to make the decision.

The panel agreed (RAND DI -0.71) and found appropriate the capacity of existing research networks for including pregnant women for COVID-19 vaccine research, as shown by a panel median of 8 and that the majority of experts (13 out of 16) provided a rating within the top third of the rating scale (7–9). This question reached consensus in round 1 and was not re-evaluated in subsequent rounds. Experts mentioned in the open-ended question responses and agreed on existing networks' capacity to include pregnant women and that they would need minimal support. The panel agreed (RAND DI -0.61) and found appropriate the recommendation to include pregnant women in the target population for novel COVID-19 vaccine R&D activities, indicated by 12 out of 16 experts assigning a rating between 7–9 and the panel median was 8 (top third of the appropriateness scale). Despite this strong recommendation, in the open-ended question, some experts pointed out that it is essential to have better evidence about the burden of the disease that COVID-19 has on pregnant women and neonates or to conduct developmental and reproductive toxicology (DART) studies before including pregnant women in this target population. For some experts that highly recommend the inclusion of pregnant women in novel COVID-19 R&D activities, it is an ethical issue regarding women’s rights to access to vaccines.

They noted that in LMICs, many health workers and front-line caretakers are women of reproductive age. Therefore, the inclusion of pregnant women in trials would allow them access to the vaccine for those health workers.

Including existing research networks from LMICs in COVID-19 R&D activities amongst pregnant women

During the first round, panelists agreed (RAND DI 1.73) that there is uncertainty (panel median 5.5 – intermediate third of rating scale) on the acceptability among decision-makers of incorporating existing research networks in LMICs. During the subsequent rounds 2 and 3, the panel continued to show disagreement (RAND DI 2.04; 2.35) on their opinion regarding this domain (Table 6). In the open-ended questions and during the virtual meeting plenary discussion, panelists mentioned that decision-makers’ acceptability would be determined by the specific network's demonstrated capacity (e.g., capacity to properly care for study participants, ability to document the outcomes of interest or test the hypotheses under evaluation). Consequently, it would influence how much needs to be invested in these networks to ensure sufficient capacity to conduct this work. They also highlighted the potential ethical concerns that may affect decision-makers’ acceptability due to concerns about unethical clinical research among LMIC populations in the past.

In contrast to the previous domain, the panel agreed (RAND DI -0.68) on the capacity of existing research networks in LMICs to participate in maternal COVID-19 vaccine R&D, given that 12 out of 16 experts assigned a rating between 7–9 and the panel median of 7 (top third of rating scale: 7–9). This question reached a consensus in the first round and was not re-evaluated in subsequent rounds. In the open-ended questions and during the virtual meeting, panelists remarked that capacity is dependent on the network and location as there is variability between networks in LMICs. Some experts commented that it might be necessary to conduct this work in high-income countries with greater capacity concomitantly or before utilizing networks in LMICs.

Main findings

In the present study, a systematic approach was used to explore experts’ opinions on the use of existing maternal and neonatal data collection systems and a list of prioritized variables for COVID-19 active safety surveillance, as well as the inclusion of pregnant women and LMICs research network in COVID-19 R&D. A diverse international multidisciplinary panel recommended using maternal and neonatal data collection systems for active safety surveillance in LMICs. There was some uncertainty as to the feasibility of adapting these systems in a timely manner given the accelerated timeline of COVID-19 vaccine development. For active safety surveillance, at least half of the experts supported the prioritization of nine maternal, eleven neonatal, and six vaccine-related variables. The experts found suitable the inclusion of pregnant women amongst the target population for COVID-19 vaccines R&D, although there is uncertainty on its feasibility. Similarly, there is uncertainty and no consensus on the feasibility of including research networks in LMICs for conducting pre-licensure activities amongst pregnant women.

Interpretation

Experts highlighted the importance of including LMICs in pre- and post-licensure/post-authorization activities for COVID-19 maternal immunization, but various disagreements have highlighted specific barriers. In their interim guidance recommendation, the WHO recommends the COVID-19 vaccine for pregnant women based on a benefit vs risk assessment. Given this, the next steps will be to monitor the safety of these vaccines in pregnant women and their potential effects on the fetus and the neonate⁶. To ensure generalizability to LMICs, it is essential to engage them in these R&D and active safety surveillance activities²⁰. Insights from this panel can help guide the next steps to engage LMICs following licensure of COVID-19 vaccines and echo the vaccine community's opinion on the necessity of these activities.

This panel reached a consensus about the capacity of LMIC research networks to be included in R&D activities and their recommendation on the usage of existing LMIC information systems for maternal and neonatal vaccine active safety surveillance of the novel COVID-19 vaccine. However, it was a commonly cited topic for both pre-licensure and post-licensure/post-authorization activities that the feasibility of including LMICs in these activities would require investment in time, resources and improving decision-makers acceptability^17,21,22. Studies of feasibility for multi-country active safety surveillance systems in LMICs noted that an initial investment in capacity is required for such activities¹⁷, which may reflect the panel’s uncertainty in feasibility given that these activities require time. Investment in capacity involves the engagement of vaccine decision-makers. This panel's dissent regarding decision-makers' acceptability on inclusion and comments regarding the feasibility of adapting information systems for active safety surveillance reflect the need to engage stakeholders. To increase acceptability and prioritization of maternal immunization efforts, decision-makers at the global, regional, national, and local levels must be involved in vaccine R&D activities, and their varying baseline awareness and perceptions of maternal immunization understood²³. The panel’s dissent suggests that decision-makers lack of acceptability may pose the most significant barrier to COVID-19 R&D vaccine activities in LMICs. The influence of their awareness, priority, and investment in these activities has been cited as a barrier^24,25. Insights from previous pandemics have highlighted that there is a need for solutions for liability and ethical concerns and promotion of R&D activities to engage decision-makers^21,25. Experts commented that decision-makers have concerns regarding the inclusion of pregnant women for similar reasons. These barriers should be considered and addressed to ensure the timely inclusion of LMICs.

Given the WHO’s interim guidance on use of the authorized COVID-19 vaccines during pregnancy^4,5, it is key to act quickly and utilize already existing systems so that quality data on vaccine safety in pregnancy can be collected from LMICs. The introduction of maternal vaccinations must occur almost concurrently across the globe in the high-, middle- and low-income countries, which requires building upon existing resources and capacity of systems in LMICs²⁶. This panel has reached a consensus regarding maternal and neonatal data collection systems usage in LMICs for maternal and neonatal active safety surveillance for COVID-19. Efforts to ensure access to COVID-19 vaccines in LMICs are already underway¹⁰. Experts highlighted the existence of systems that could be strengthened and adapted for vaccine surveillance, reflecting what other publications have also stated^22,27. The adaptation of these systems for COVID-19 vaccine active safety surveillance is thus a critical next step to ensuring the collection of quality vaccine safety data.

Despite the recommendation of using these existing systems, the investment and necessary adaptation in these systems may be a challenge to collecting high quality vaccine safety data in LMICs. Experts were uncertain about the feasibility of adapting existing LMIC systems on time for COVID-19 maternal immunization. Along with other authors, this panel has cited that resources and investment are needed to adapt these systems^13,28. The urgency to adapt in time has posed a barrier to the establishment of post vaccination maternal and neonatal surveillance systems during COVID-19 pandemic and other outbreaks, such as Ebola²⁵. To address these barriers, it will be critical to plan and act quickly and utilize and leverage already existing systems so that quality data on vaccine safety in pregnancy can be collected from LMICs.

This panel’s prioritization of maternal and neonatal outcomes for vaccine active safety surveillance for COVID-19 provided interim, expert insight. According to the WHO's safety surveillance manual for COVID-19 vaccines, the outcomes specifically mentioned for monitoring adverse events in pregnancy are maternal mortality, stillbirth, miscarriage, neonatal mortality, and congenital anomalies. These outlined outcomes were also prioritized by the experts’ in this Delphi study. The GAIA project has created 25 standardized case definitions for maternal and neonatal outcomes¹³. With the expert insight on the prioritization of these variables given here, a minimal set of maternal, neonatal, and vaccine-related variables necessary for COVID-19 active safety surveillance can be inferred. Besides, this panel's prioritization of maternal, neonatal, and vaccine-related variables can guide the required adaptation of existing maternal and neonatal data collection systems in LMICs for COVID-19 active safety surveillance.

There was established consensus on the need to include pregnant women in the target population for vaccine R&D, and the expert panelists agreed that the potential benefits outweigh the risks which is supported by recent evidence on the severity of disease and DART studies.

The panel agreed that now is an appropriate time in the COVID-19 vaccine R&D landscape to consider including pregnant women in the target population. These findings align with the Institute of Medicine report that recommended that pregnant women be deemed eligible to participate in clinical trials and the WHO interim guidance for the approved vaccines^4,5,29. The ambiguous existing regulations concerning whether and when pregnant women should be included in clinical trials remains one of the main reasons for the persistent underrepresentation of this population in research³⁰.

The panel's dissent over the feasibility of the inclusion of pregnant women in the target population for novel COVID-19 vaccine R&D activities is also concordant with the literature. The concern of the panel was mainly about including pregnant women in early clinical studies to be cautious during a time of high uncertainty with little data available. Exclusionary criteria imposed by IRBs, federal guidelines, and codes of regulation have further contributed to this challenge^31–33. This emerging consensus that systematic exclusion of pregnant women from study involvement ultimately leaves these women and their fetuses susceptible to potentially substantially higher harm due to the lack of rigorous clinical data on the safety of drugs and other treatments have pushed clinicians, academics, bioethicists, and professional societies to increasingly call for a re-examination of the routine procedure of excluding pregnant women from clinical research in recent years³⁴. This panel’s dissent regarding the effect of regulatory requirements on the feasibility of including pregnant women in the target population for COVID-19 vaccine R&D activities suggests that additional guidance and regulation may be required to assist IRBs and research sponsors move toward an equitable, ethical, and adequately protected inclusion of pregnant women in vaccine R&D in the context of COVID-19.

Strengths and limitations

The study has several strengths. First, a methodologically rigorous, transparent, and reproducible process guided the development of this consensus study. Second, the methodology was defined and prior to the study commencing, and it was based on an extensively accepted consensus method technique: the modified-Delphi method. Third, a diverse panel affiliated with a broad spectrum of organizations and who were geographically distant participated in this study. Finally, the questionnaire included both closed- and open-ended questions, providing rich insight to experts’ ratings with qualitative data.

However, there were limitations to this study. First, inherent to any consensus process, there is some level of bias due to the influence of interpersonal dynamics. We tried to mitigate this by conducting two individualized online rounds and asking participants for disclosure of interest. Second, the small panel size did not allow for a subgroup analysis. It would have been particularly interesting to explore experts’ views by the type of country they were representing (high income countries vs. LMICs). Third, a classic in-person meeting was not possible given travel restrictions during the pandemic, so a virtual meeting was conducted. A guided discussion with prompts and probes was developed to encourage and facilitate the discussion to overcome this limitation. Lastly, given time-constraints to report findings, pre-testing and pilot questionnaires were omitted. We tried to ameliorate this by asking five experts in immunizations or maternal health who did not participate in the study to assess the questionnaires' content, their clarity, and the relevance of the included items.

Underlying data

Open Science Framework: Using maternal and neonatal data collection systems for coronavirus disease 2019 (COVID-19) vaccines active safety surveillance in low- and middle-income countries: an international modified Delphi study - Supplementary materials. https://doi.org/10.17605/OSF.IO/ZNS2D¹⁸.

This project contains the following underlying data:

Extended data

Open Science Framework: Using maternal and neonatal data collection systems for coronavirus disease 2019 (COVID-19) vaccines active safety surveillance in low- and middle-income countries: an international modified Delphi study - Supplementary materials. https://doi.org/10.17605/OSF.IO/ZNS2D¹⁸.

This project contains the following extended data:

Extended data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).