Understanding evidence ecosystems: What influences the production, translation, and use of modeled evidence in Burkina Faso, Nigeria, India, and Kenya?

Ethics approval

This study was approved by Health Media Lab IRB in the United States (HML IRB Review ID: 1001R4DV21, October 8, 2021) and a local IRB in each study country (Comité d’ethique pour la recherche de santé in Burkina Faso, Health Systems Research India Initiative Institutional Ethics Committee in India, Amref Ethics and Scientific Review Committee in Kenya, and the University of Nigeria Teaching Hospital Health Research Ethics Committee in Nigeria). In Kenya, the study was also registered with the National Commission for Science, Technology and Innovation, per local regulations (Ref No. 801251, December 1, 2021).

Study design

The study was coordinated by Results for Development in the United States and funded by the Bill & Melinda Gates Foundation (BMGF).

Before country selection, Results for Development conducted a desk review of modeling activities and actors in seven countries identified by BMGF as focus countries or for known modeling activities (Figure 1). In combination with informal conversations with key stakeholders, the study team used this desk review to understand the level of activity, experience, and variety of actors engaged in modeling in each country. The four selected countries are BMGF focus countries and have modeling ecosystems in various stages of development. They also represent a variety of geographic regions, and include countries from both Francophone and Anglophone Africa: Burkina Faso, India, Nigeria, and South Africa. Kenya was added to this list, with a smaller scope and budget, at BMGF’s request to inform planned activities. South Africa was later removed from the study due to delayed ethical review timelines during the COVID-19 pandemic.

Figure 1. Country Selection Process.

Data collection and analysis were carried out by local research partners in the four selected countries: Centre de recherche en santé de Nouna in Burkina Faso, the Health Policy Research Group University of Nigeria in Nigeria, ACCESS Health International in India, and independent consultant Peter Muriuki in Kenya. The research teams applied a sequential mixed-methods approach informing the purposive selection of informants and probing questions used in the qualitative phase, that entailed in-depth key informant interviews with 230 decision-makers, modelers, and brokers of modeled evidence across each country’s modeling-to-decision-making ecosystem.

The study was advised by a Working Group that included representatives of stakeholder groups in the four final research countries and South Africa. It met four times during the study. Members were selected by the country research teams and played key roles in the modeling-to-decision-making ecosystem in their country. The mandate of the Working Group was to:

Data collection

Results for Development drafted the research tools, including an interview guide and survey. These tools were reviewed by the Global Center for Gender Equality at Stanford University to ensure gender-related issues were taken into consideration and then lightly adapted by the country teams to reflect local contexts, address local IRB requirements, and (in Burkina Faso) translate to French. The interview guide and survey can be found as Extended data (Taddese et al., 2023).

In Burkina Faso, Nigeria, and India, the country research partners generated initial lists of contacts to receive the survey based on their own and the Bill & Melinda Gates Foundations’ knowledge of who was currently active in the modeling space. Snowball sampling was used to expand this list to further stakeholders. The survey was conducted online and included a question about whether the respondents were willing to be contacted for an interview. In Kenya, the team did not conduct a survey due to the study’s more limited scope and budget.

The research teams identified key informants for interviews from the pool of survey respondents who had provided informative survey responses or through additional outreach efforts to key stakeholders in the modeling space who had not responded to the survey. The research teams aimed to interview modelers, knowledge brokers, and decision-makers working in a representative variety of health fields, organizations, and levels of government, and to ensure inclusion of respondents who identified as women. Further snowball sampling was used to identify additional interview respondents from the actors mentioned by key informants. In Kenya, key informants were identified based on the researcher’s own and the Bill & Melinda Gates Foundations’ knowledge of who was active in the modeling space, along with snowball sampling.

Key informant interviews were conducted in-person or remotely via a virtual platform such as Zoom and were audio recorded. The interviews were conducted by one or two members of the research team in each country, on average taking 30–60 minutes.

The research teams in Burkina Faso, Nigeria, and India continued interviewing until they felt they had reached saturation, at 24 or 25 interviews. The researcher in Kenya conducted nine interviews due to the more limited scope and budget. All the Kenya respondents identified as modelers or decision-makers, with none having a purely knowledge brokering role. The table below summarizes the number of survey respondents and key informants in each country and the survey response rate (Table 1).

Data analysis

The research teams conducted a quantitative analysis of survey results in Excel using a template provided by Results for Development. The analysis was used to identify key actors in the modeling ecosystem of the study countries and build a preliminary understanding of the modeling needs of decision makers, including constraints and opportunities which were further explored in key informant interviews. Results for Development created a uniform codebook in Excel to be used to code key informant interviews which were transcribed across all countries. Research teams used software of their choice to analyze the data (Nvivo for Burkina Faso, Kenya, Nigeria teams and QDA Miner Lite for India team). The codes were thematic and broadly grouped by research question. Each research team then had the chance to develop subcodes specific to their context and the data they collected. Each of the research teams coded their first two interviews with a member of the R4D team and held a meeting to discuss their coding, any questions that came up, and ensure uniformity of analysis across countries. The remaining interviews were then coded by one or two members of each research team. We did not test the inter-rater reliability of codes after the initial two interviews.

The research teams presented their survey and key informant interview findings to the Working Group to receive feedback and further insights. Each research team then developed a country case study in PowerPoint and policy brief in Word. These analyses fed into a final project report produced by Results for Development summarizing findings across all countries.

This study has a few limitations. First, as the research teams relied on their knowledge and networks to recruit participants into the study, they may have missed other key individuals in the modeling to decision-making ecosystem. Second, survey response rates were low, particularly the response rates from women were low.¹ Third, the research teams were unable to interview several of the key informants who participated in the survey and missed the opportunity to build on and further explore their responses. Fourth, because we were not able to include South Africa in our research, our sample size of full study countries was even smaller (from four to three), and we acknowledge that South Africa’s robust modeling ecosystem might have influenced our conclusions and recommendations differently. Finally, we found little documentation of country-level public health modeling activities in the study countries. As such, along with the limited scope and condensed timeline of our research, we are likely to have missed some key actors and activities in the modeling ecosystems of the four study countries.

Modeling to policy ecosystem

Key actors in the modeling-to-policy ecosystem include modelers, knowledge brokers and boundary organizations, mechanisms or spaces for knowledge sharing and exchange, and funders. Figure 2 and Table 2 identify these actors and provide the definitions we used in the study). Although these terms, particularly boundary organization, knowledge broker, and translator are often used interchangeably in the literature, the distinctions offered in our definitions allowed us to better characterize and describe how modelers and decision-makers come together to inform policy and practice or program level decisions. We observed each of these actors in the four study countries with variation in prominence and role played.

Figure 2. Key Actors in the Modeling to Decision-making Process.

Modelers

Public universities and research institutions (national and international) play a lead role in developing models in the four study countries.

In Burkina Faso, modeling is conducted almost exclusively by national research institutions and universities.

In Nigeria, modeling is conducted primarily by public and private universities. (local and international) as well as some parastatal research institutions and local NGOs (Non-Governmental Organizations) like Pro-Health Nigeria.

In India, a range of actors, including local public and private research and academic² institutions, local and international NGOs, government think tanks, and the Health Technology Assessment agency engage in modeling.

In Kenya, local universities are the most prominent modelers. But a range of other actors, including the parastatal Kenya Medical Research Institute, bilateral agencies, foreign universities, and regional initiatives, also engage in modeling activities.

Knowledge brokers and boundary organizations

The extent to which research organizations with modeling capacity or stand-alone organizations with boundary spanning roles engaged in knowledge brokering to influence policy or practice varied by country. In countries where we observed more modeling activities, we noted an overlap in modeling and knowledge brokering functions, with most research organizations playing both roles.

In Burkina Faso, government agencies, civil society organizations, iNGOs, and UN agencies serve as boundary organizations – facilitating communication between modelers and decision-makers without conducting any modeling themselves. Communication between modelers who are not directly affiliated with the government and government agencies is facilitated almost exclusively by these boundary organizations.

In Nigeria, the Academy of Science serves as a prominent boundary organization, as well as some local NGOs and UN/bilateral agencies. However, other local NGOs, parastatal organizations, and UN/bilateral agencies both create models and communicate the results directly to the government.

In India and Kenya, all the organizations that engage in knowledge brokering between modelers and decision-makers are engaged in modeling themselves, including local and international NGOs, government think tanks, parastatal research and academic institutions, and the Health Technology Assessment agency.

Mechanisms of knowledge exchange

Across all research countries, government-led advisory groups, working groups, and task forces were routinely cited as key mechanisms for knowledge exchange between modeled evidence and policymaking. Non-government research coalitions also provide a platform for debate and knowledge dissemination, particularly in the countries we observed to have large and complex data ecosystems.

In Burkina Faso, government advisory groups, such as the COVID-19 thematic working group, provide a platform where modelers and decision-makers can come together to discuss modeled evidence around various diseases, most prominently COVID-19.

In Nigeria, disease-specific and general health or data government advisory groups provide this platform, along with the prominent and independent National Council on Health.

In India, independent consortia of researchers such as the COVID-19 consortium and Health Technology Assessment (HTA) consortium provide this platform, while government-led “working trainings” provide unique opportunities for modelers to come together to develop models needed by decision-makers while developing their own modeling capacities.

In Kenya, disease-specific government advisory groups and task forces (most prominently the COVID-19 Task Force) provide this platform, while formal partnerships between government and modeling agencies provide another avenue for communication.

In the four study countries, decision-makers currently engaging with modeled outputs reported a focus on disease-specific initiatives, particularly HIV, tuberculosis, and malaria. In all the countries, modelers from all fields were called on to support COVID-19 modeling. Table 3 presents a list of initiatives for which there were past or present modeling efforts in the study countries, as mentioned by key informants. It is not a comprehensive list of all modeling efforts in the research countries.

Facilitators & inhibitors influencing production, translation, and use of modeled evidence

Our research examines the factors that facilitate and inhibit the production, translation, and use of modeled evidence at the interpersonal, organizational, and environmental levels (Table 4). Interpersonal factors explore capabilities at an individual level, while organizational factors describe the structures, communication, and coordination infrastructure needed to support production, translation, and use of modeled evidence. Finally, environmental factors refer to internal and external conditions that influence the use of evidence, including funding and global events like the COVID-19 pandemic.

Interpersonal factors

Key informants highlighted the importance of having credible modeling capacity across different disciplines to inform policy and practice decisions. Building this type of capacity is time and resource intensive. In addition to individual level capabilities, successful modeling efforts also demand tools and infrastructure for modeling, including modeling software and computers powerful enough to run it.

Modeling outputs can be complex and hard for decision-makers to make sense of and use in informing policy and planning. As a result, they may not understand the value and relevance of the models in addressing the real-life public health questions they are grappling with. Boundary organization representatives reported that when they engage with a decision-maker who understands modeling as a concept, the decision-maker is more likely to promote and accept the usefulness of the model.

Informants also noted that when decision-makers and modelers collaborate in defining the research and policy questions and have regular and sustained engagement with each other, modeled outputs are more likely to be considered in decision making. In this collaboration, it is equally important that modelers communicate and share findings in formats that are easy for decision-makers to understand and use.

Organizational factors

Misaligned time frames for modeling and decision-making were a frequently cited issue, particularly in the context of a health emergency like COVID-19. In such situations, decision-makers demand rapid responses to their evidence needs, but high-quality, rigorous models take time to develop. These misaligned expectations can result in modeling outputs only being published once the window for decision-making has already passed.

Decision-makers often report not having enough time to engage in knowledge exchange activities with researchers due to competing time demands. When consulted and engaged in developing models, however, they are more likely to buy in to the process and outputs and consider them in a decision process. Co-production is a way to build decision-maker trust and increase awareness and understanding of the value of using modeled outputs to inform policy and planning. In addition to formal partnerships, informal relationship-building activities can help to position researchers as trusted experts or partners for decision-makers.

Transparent communication and data sharing between modelers and decision-makers is key to facilitating the use of modeled evidence. Having research institutes and knowledge brokers in the Ministry of Health with direct access to high level decision-makers helps to facilitate knowledge and data exchange in the Ministry. Software and databases that allow for the secure transmission of data are also critical for information sharing.

Through mechanisms like task forces and committees that may be present at the federal and state department levels, decision-makers and research partners can come together to coordinate timelines, co-produce research questions, and engage in dialogue and debate about evidence that is produced to inform decision-making. These types of mechanisms and other government initiatives championing evidence-based decision-making in recent years have created platforms where new kinds of evidence, like modeled evidence, can thrive.

Environmental

The lack of quality data and modeler access to data limits the production of models. Data are often collected by different agencies and can result in fragmented, program-specific outputs that are partial or biased and of limited value to modelers. Further, lack of communication between the modeling community and data gathering entities can lead to a misunderstanding about what is needed.

Some decision makers also expressed that they felt models that were built exclusively on foreign data (such as many of the models produced in the early COVID-19 pandemic) were of limited value for local decision-making.

Sustained funding from internal government and external donor sources is needed to support researchers in generating modeled evidence in a well-established, routine, and sustainable manner. Funding is also needed to ensure intermediaries or boundary organizations are equipped with skills, spaces, and tools to facilitate knowledge exchange and to strengthen capacity for evidence-based decision-making in government through skill and awareness building.

The rapidly evolving nature of COVID-19 spurred widespread cooperation among countries in epidemiologic modeling to rapidly inform measures to curtail the spread of the virus and protect citizens. Informants noted how quickly decision-makers in ministries of health mobilized to create spaces and structures for bringing together decision-makers, modelers, epidemiologists, and other research partners, with clearly defined roles and responsibilities for each group.

Mechanisms that enable exchange between modelers and decision-makers

We observed a range of mechanisms designed to facilitate exchange between modelers and researchers in the four study countries pointing to a shared recognition of the value in bringing diverse stakeholders together to advance evidence use in decision-making through institutional structures (Table 5). Except for the dedicated task forces and committees that were mobilized quickly to respond to COVID-19, these mechanisms provide a space for debating evidence in the health sector broadly, with modeled outputs featuring as one type of evidence. There is an opportunity for country actors and partners to build on the momentum of the modeling-specific activities and structures introduced during COVID-19 – strengthening what has worked well and making improvements where needed.

In some fields (e.g., environmental science), stand-alone entities that are commonly referred to as boundary organizations and often independent of the research and policy sides, play a key role in facilitating knowledge exchange between research and policymaking communities. Knowledge brokers also play a role in facilitating exchange between research producers and users but typically are embedded within research / modeling organizations. We find that these roles apply to the modeling ecosystem as well. Specific mechanisms that aim to facilitate dialogue, debate, and communication in a boundary organization capacity include entities like the Nigeria National Council on Health. Mechanisms operating in a knowledge broker capacity include formal government research partnerships. Our research partners identified advisory groups as a common engagement mechanism for supporting modeling activities in all countries.

Finally, we observed in the small sample of four countries included in this study, that as capacity to produce, translate, and use modeled evidence develops, stand-alone boundary organizations tend to be replaced by knowledge brokers in modeling organizations.