Development and application of a hybrid implementation research framework to understand success in reducing under-5 mortality in Rwanda

Development of the amenable U5M implementation research framework

At the start of the research, in 2017, we reviewed existing implementation research frameworks focusing on those applied in LMICs to study implementation strategies, identification of contextual factors, and implementation outcomes^8,16–19. We focused on identifying frameworks that had been used in LMICs to date which were relevant to health systems interventions, through a rapid review in PubMed with search terms “implementation science” and “LMICs.” We also reviewed publications in Implementation Science from the previous three years and conducted outreach and discussion with experts in the field at the time. We identified relevant frameworks to develop a hybrid framework, which was adapted during the case study process to reflect emerging areas where changes were needed.

EBIs to reduce amenable U5M

Reviewing existing guidelines and literature from the MDG efforts, we identified EBIs known to reduce the most common causes of U5M in LMICs for infants and children, and for neonates during the three periods of risk (antenatal, peripartum, and postpartum). This identification of EBIs guided the exploration of implementation strategies, contextual factors, and implementation outcomes (Extended data Tables 1a and 1b)^20–22. These EBIs also guided evidence review and key informant selection.

Extraction and analysis of implementation pathways, strategies, outcomes, and contextual factors

We used an explanatory mixed methods case study to leverage existing quantitative data on selected implementation outcomes (reach, fidelity, adoption) and equity with the qualitative sources describing strategies, contextual factors and selected outcomes (acceptability, feasibility). Reach, also defined as coverage, was calculated from the Demographic and Health Surveys (DHS)^23–26. The Standards for Reporting Implementation Studies (StaRI) and COnsolidated criteria for REporting Qualitative research (COREQ) Checklists were used where relevant and are included in the Extended data^22,27,28.

Desk review. The team conducted a rapid review of available information and published data on the rates and progress of U5M broadly and amenable causes of U5M. We further explored the uptake and implementation of specific EBIs, including policies and strategies. Initial review research was performed through MEDLINE (PubMed) and Google Scholar using the following search terms as inclusion criteria: “child mortality” or “under-5 mortality” and Rwanda, with further searches including terms for specific EBIs (ex. “pneumococcal vaccination,” “oral rehydration salts,”), causes of death (ex. “malaria,” “measles,” “HIV”), or contextual factors (e.g. “community health workers,” “female empowerment”). Other sources included publicly available datasets and reports (Demographic and Health Survey, Countdown 2015, and existing reviews of U5M reduction work in Rwanda)^{20,23–26,29}. The review was limited to amenable causes of death from the neonatal period to early childhood, including EBIs during pregnancy and childbirth deemed critical for child survival and only English-language references were included. Important interventions that contributed to U5M reduction beyond the specific health system EBIs including health system strengthening, education, female empowerment, reproductive health, poverty reduction, improving water and sanitation, and programs designed to improve nutritional status beyond management of acute severe malnutrition were captured as potentially important contextual factors. However, references which dealt solely with implementation of these interventions were not included as beyond the scope of the research, which focused on EBIs directly led by the health sector. The desk review was an iterative process, with additional searches conducted as needed and informed by the primary research described below.

Key informant interviews. With our in-country collaborator (FS), we purposively identified 15 key informants (KIs) who had been involved throughout the process of identifying the burden of the target cause of death and selecting, implementing, and working to sustain the targeted EBIs and U5M reduction more broadly before and during the period of focus (2000–2015). These KIs included current and former Ministry of Health employees responsible for overall U5M reduction efforts, or focusing on specific diseases or intervention areas (Table 2). We also interviewed KIs from implementing non-governmental organizations, multilateral organizations, and donor organizations who had been involved in partner-supported or partner-led activities. Some KIs represented more than one viewpoint based on their experience over the 15 years and were interviewed for each of their multiple viewpoints.

The interviews were conducted by phone or face-to-face using a semi-structured guide (see Extended data for interview guide²²), based on the IR framework, designed to understand EBI selection; the implementation process, from exploration to preparation, implementation, and sustainment; and contextual factors contributing to successes and challenges in each stage. The interviews also helped identify additional data sources important in understanding implementation and related outcomes. Follow-up interviews were conducted as gaps or additional needs were identified. All interviews except one were led by one of the project principal investigators (LRH) with one to two notetakers. Each interview’s duration was approximately one hour. The total number was not designed for saturation but instead for covering the EBIs and was also limited by time and resources. Informed consent was obtained from all interview participants, and all recordings and transcripts were de-identified and stored in password-protected folders and destroyed once the interview coding was completed. The results were disseminated to the Ministry of Health and partners.

Analysis: Following the close of the interview, notes were combined and audio recordings (if permitted) were used to clarify areas as needed. Thematic coding of interviews using inductive and deductive approaches was done by one of the researchers and reviewed by one of the principal investigators for accuracy, with discussion for differences. A priori codes were organized from the framework for EBIs, implementation outcomes, and context. Implementation strategies were informed by the Expert Recommendations for Implementing Change (ERIC) framework³⁰ and additional codes were added as emerging themes were identified.

Additional analyses from the International Center for Equity in Health and geospatial mapping from the Institute for Health Metrics and Evaluation (IHME) were used to understand changes in equity indices for mortality and EBI coverage.

Ethical considerations

The study was determined to be non-human subjects research by the Rwandan National Ethics Committee and Northwestern University reflecting the scope and focus.

Framework

We found that no single framework covered the scope of the work from identification of EBIs to target leading causes of U5M, to selection and testing of strategies to achieve sustainability at scale, understanding the full range of potential contextual factors and exploring outcomes beyond the traditional coverage and effectiveness measures used in evaluation of health systems. We therefore combined and adapted a number of existing frameworks to synthesize a framework designed to guide our work to understand implementation of and outcomes of EBIs to reduce amenable U5M (Figure 1). The frameworks identified as appropriate for guiding our work included:

Figure 1. Hybrid Framework for Understanding Interventions to Reduce Under-5 Mortality.

Reproduced with the permission of UGHE.

The framework was revised as needed as new themes emerged during the Rwanda case study. For example, we recognized that adaptation was an important step during implementation and was sufficiently crucial that we included it as an explicit step rather than as an embedded implementation strategy. This led to the transformation of the four-step EPIS implementation process into the five-step EPIAS implementation process. In addition, this adapted framework expanded the levels at which we captured contextual factors, looking at global, national and subnational, ministry of health and health system, and community/family/individual levels. As the arrows indicate in Figure 1, our framework also explicitly calls out the interplay and non-siloed nature of contextual factors. This framework emphasizes the need to consider the multifaceted nature of these factors important in analyzing health sector implementation outcomes as well as broader influences at the global, national, and health system levels. These factors included those which could be leveraged to support implementation. They also represented barriers, requiring direct or indirect action to address, adaptation of implementation strategies at the local or national levels, or representing reasons for challenges in achieving targeted outcomes.

Development of a theory of change

To put the work to reduce amenable U5M within the broader work to reduce U5M, we also created a theory of change model (Figure 2). This model included the health systems EBIs which were the focus of our research (outlined) and other interventions which could directly or indirectly improve the health status of the children (resiliency), reduce the risk of falling ill or dying during an illness, or improve the ability to access care. These included interventions to improve nutrition, water and sanitation, female empowerment, and economic status. The theory of change takes into account the contextual factors at the four levels that can contribute to the reduction of amenable U5M and U5M more broadly. These factors can influence the implementation of EBIs, health resilience, prevention of disease, and access and quality care, all of which can contribute to or hamper work to reduce amenable U5M.

Figure 2. Theory of Change for Reduction of Amenable Under-5 Mortality.

Applying the framework to understand U5M reduction in Rwanda

We used the framework to develop the key informant guides and overall analysis plan. Guided by the framework, we extracted evidence of the implementation outcomes for EBIs from the available literature and quantitative data, including appropriateness, acceptability, feasibility, effectiveness, coverage or reach, and equity, where possible. We also used the framework to develop an initial set of codes for EBIs, implementation outcomes, and contextual factors. Coding was done manually using directed content analysis with additional codes added based on identification of new concepts, factors, or strategies. No software was used for this first qualitative coding. These results were analyzed using explanatory mixed methods starting with the EBI coverage data by the team and reviewed through a stakeholder meeting in Rwanda to provide feedback and validation, and to identify transferable lessons with the potential for adoption or adaptation in other countries with challenges in reducing U5M. The goals were to extract understanding of how and why Rwanda was able to successfully implement health systems EBIs known to reduce amenable U5M, and where challenges remained in implementation, sustainability, and overall U5M reduction.

Implementation steps and strategies

Implementation of all the EBIs followed the steps from the hybrid framework (EPIAS), starting with Exploration to determine the relative burden of an underlying cause of U5M and identify potential interventions. The Preparation step was carefully conducted and included a range of strategies including: stakeholder engagement; identifying contextual factors that either needed to be addressed (ex. supply chain strengthening or human resources) or that influenced the choice of implementation strategy (ex. task sharing); identifying and collecting any new evidence needed to inform strategies and adaptation; determining funding needs and sources; strengthening needed systems and developing or adapting national policies and protocols to drive systematic adoption; and integration of data needs for monitoring and evaluation (M&E) into routine information systems when possible and ensuring accountability for EBI delivery. Implementation then followed, with the pace of implementation based on EBI rollout needs and readiness. For example, vaccines were rapidly implemented at national scale while others, such as integrated management of childhood illness (IMCI), were rolled out as ongoing learning occurred.

The implementation of all of the EBIs included Adaptation starting soon after initial Implementation step. This adaptation reflected evidence emerging from ongoing monitoring and evaluation or other data sources reflecting how the strategies were put into practice, new global evidence (ex. rapid testing for malaria), or identification of areas where gains in coverage were either not reaching the most vulnerable or diseases, such as malaria, were reemerging. In Rwanda, the work for Sustainment was often included early in the steps through strategies including integration into national standards, ensuring access to long-term funding, fostering a culture supportive of M&E, and inclusion into training and supervision. Throughout these five steps, a number of implementation strategies were identified across the different EBIs (Table 3).

Table 3. Common Implementation Strategies by Exploration, Preparation, Implementation, Adaptation, and Sustainment Stages in Rwanda.

Strategy effectively implemented      Strategy implemented with variable success      Strategy not implemented
Implementation strategy	Exploration	Preparation	Implementation and Adaptation	Sustainment
Donor and implementing partner coordination
Focus on equity
Engagement of key national stakeholders and partners
Engagement of key international stakeholders and partners
National leadership and accountability
Building and strengthening primary healthcare systems
National prioritization of health
Rapid and early adoption of innovations
Development of national policies, guidelines, and standards (including adaptation of global protocols/ guidelines)
Community engagement and education
Data generation by in-country institutions and use
Data use for understanding gaps, prioritization, implementing, adapting
Multi-sector approach
Rapid scale-up
Small-scale testing
Supportive supervision and mentoring for quality
Building on community health worker program and community-based care delivery
Human resources for health expansion
Leveraging and strengthening existing systems (using to implement a strategy)
Task-shifting
Government financing

Implementation outcomes. We found that Rwanda implemented almost all of the EBIs identified as effective in reducing U5M (Extended data Table 1a²²) in LMICs with varying reach but high levels of equity (Table 1; Extended data Figure 1²²). Neonatal mortality reduction was slower and work to implement a number of the neonatal-specific EBIs (Extended data Table 1b²²) became a greater focus towards the end of the study period (Table 1). For example, PCV vaccine coverage was 94.7% in 2014 with a wealth gap of 5%. Measles vaccine coverage rose from 86.9% in 2000 to 95.2% in 2014, with the wealth equity gap hovering around 5% (4% in 2000, 6% in 2014). HIV testing and counseling during antenatal care was above 90% in 2014 with a wealth equity gap for both measures at 3%. Facility-based delivery had a higher wealth gap, but it decreased substantially from 43% in 2000 to 13% in 2014, with coverage increasing from 26.5% to 90.7% during that period.

However, extraction of many of the other implementation outcomes was limited by the available evidence from the primary and secondary resources. The most common outcomes we found from the desk review and KI interviews included feasibility, penetration/reach, and adoption. We also found sustainability outcomes for a portion of the EBIs, defined as maintaining high coverage rates from initial implementation through the end of the study period, and strategies associated with sustainability including incorporation into national policies and budgets. Fidelity and cost were infrequently found except at subnational levels from targeted studies.

Cross-cutting contextual factors. We identified cross-cutting contextual factors that were critical to successful implementation of many of the EBIs in Rwanda (Extended data Table 2²²). Equally important, a number of these also contributed to the decline in U5M by addressing causes not directly related to health. System interventions were important facilitating contextual factors throughout the five steps of EPIAS and represented the four areas from the framework (global, national, health system, and community). These included national leadership and commitment (in policy and finances) to health; a culture of data use for learning and decision-making tied to accountability at all levels; and, importantly, the existence of standards for coordination, collaboration, and communication within the health sector, across ministries, with local leaders, and between government bodies, donors, and implementing partners. Other facilitators included the availability of donor support and in-country implementing partners. The underlying health system design, with its commitment to primary care and decentralization, along with the development of a robust community health worker (CHW) program were also identified as facilitators, building on an underlying focus on equity. Resiliency of the family and child were also strengthened through national economic development, female empowerment through education and policy, development of a national health insurance free to the poorest sectors and emerging work to address nutrition and stunting.

There were also a number of contextual factors which represented challenges in Rwanda, including gaps in components of the health system – such as ambulances and advanced equipment for the care of sick and low birthweight neonates – and cadres of human resources for health.

Example of applying the framework to understand EBI implementation: the case of pneumococcal vaccine introduction

We use the introduction of pneumococcal vaccine (PCV) to illustrate the results of employing the framework to understand the strategies, implementation outcomes, and contextual factors that explain the steps taken and overall results. We highlight examples of implementation strategies and facilitating contextual factors which emerged during the five steps outlined in our framework.

Exploration

Rwanda decided to rapidly move forward with PCV, based on the announcement by GAVI of the availability of grants for PCV; recommendations from WHO; and identification of pneumonia as the leading cause of death among children under 5 in Rwanda (strategies: use of external evidence and local data use for decision-making; factors: data availability, culture of data use)^32–36. This decision was led by the Rwandan Ministry of Health (MOH) and supported by the Vaccine Steering Committee, made up of MOH officials, one of whom chaired it, and national and international partners (strategies: national leadership, multisectoral and donor coordination; factors: national commitment to heath, culture of coordination and intersectoral collaboration)³⁴.

Preparation

Supported by both the MOH and international partners (ex. GAVI, UNICEF, and USAID), Rwanda first performed a nationwide evaluation of the existing cold chain, which was used to address infrastructure gaps (strategies: integration into existing systems, leveraging global resources; factors: focus on primary care systems, donor funding availability). Rwanda also developed national training modules, implemented a cascading train-the-trainers model, revised vaccination guidelines and reporting forms, developed key messages for communities, and conducted a needs estimate (strategies: training with national protocols, integrating into existing monitoring systems, community engagement and education; factors: community trust in the healthcare system, existing monitoring and evaluation and health information systems)³⁵. The MOH procured the very high temperature incinerators (>1200°C) needed for destruction of prefilled glass-containing syringes³⁷. The MOH secured funding in the national budget, the counterpart fund for GAVI (strategy: engagement of key national stakeholders and partners) and chose to immediately roll out implementation nationwide due to confidence in their pre-implementation evaluation of the project, safety, and equity profiles of the vaccine, and a commitment to equity.

Implementation

Implementation of PCV started in 2009, with Rwanda becoming the first developing country to introduce the pneumococcal vaccine³⁸. It rolled out province-by-province as planned and was completed within five months of the start date. The implementation strategy ensured equity by using motorcycles and cold boxes to deliver vaccines and leveraging CHW data to determine which areas needed additional vaccination days. A number of implementation outcomes were achieved, facilitated in part by the strategies chosen (Table 4).

Adaptation

In 2011, in response to high cold storage and incineration temperature requirement for PCV-7, Rwanda switched from using the PCV-7 to the PCV-13 vaccine which also provided protection against six more bacterial strains. These changes allowed for adoption of new vaccines, which had similar storage and disposal needs, with little increase in infrastructure and informed the strategy for introduction of other new vaccines in Rwanda in the future^34,35.

Sustainment

Full vaccination rates with PCV remained consistently high from introduction through the end of the study period and beyond, with coverage at 97–98% since 2010. Strategies that were identified as supporting this sustainment included integration by the MOH into its standard pediatric vaccine schedule; regular data use through monthly monitoring data, which are analyzed at the central level with feedback to address coverage gaps³⁴, and continuation of national funding for PCV, which has been included in the annual national budget since its introduction in 2009.

Rwanda has been highlighted as a country that leads in work to drop U5M. We were able to surface a number of lessons transferable to other countries that wish to accelerate their work to reduce U5M. These emerged from the successful strategies directly related to implementation, as well as from contextual factors, which can be modified to facilitate successful implementation outcomes (Table 5).

Source data

The Demographic and Health Surveys data used in the current study are available from the DHS website.

Rwanda DHS 2015: https://dhsprogram.com/pubs/pdf/FR316/FR316.pdf

Rwanda DHS 2000: https://dhsprogram.com/pubs/pdf/FR125/FR125.pdf

The Institute for Health Metrics and Evaluation (IHME) Low- and Middle-Income Country Neonatal, Infant, and Under-5 Mortality Geospatial Estimates 2000–2017 Local Burden of Disease data used in the current study are available for download from IHME: http://ghdx.healthdata.org/record/ihme-data/lmic-under5-mortality-rate-geospatial-estimates-2000-2017.

Underlying data

Data access is restricted to users with appropriate ethics approval from the committees listed in the Ethical Considerations section. A reader or reviewer may apply to the authors for access by providing a written description of background, reasons, and intended use. If the methodology does not violate the condition of informed consent under which the interview was conducted, and the proposal approved by UGHE and other relevant ethics boards, the user can obtain the data from the corresponding author, and include one of the authors in the project and analysis.

Extended data

DRYAD: Development and application of a hybrid implementation research framework to understand success in reducing under-5 mortality in Rwanda. https://doi.org/10.5061/dryad.kh189324x²²

This project contains the following extended data:

Reporting guidelines

DRYAD: StaRI checklist for ‘Development and application of a hybrid implementation research framework to understand success in reducing under-5 mortality in Rwanda’ https://doi.org/10.5061/dryad.kh189324x²². Accessible here: https://ughe.org/wp-content/uploads/2021/02/3.-StaRI-checklist-IS-Exemplar-framework-methods.pdf.

This work is licensed under a CC0 1.0 Universal (CC0 1.0) Public Domain Dedication license.

Ethics approval and consent to participate: The study was determined to be non-human subjects research by the Rwandan National Ethics Committee and Northwestern University reflecting the scope and focus.

Interviewees were informed about the goals and structure of the project, and verbal informed consent for participation and for audio recording was obtained separately from the interview.