Improving performance of primary care systems in low- and middle-income countries (LMICs) has been suggested as a necessary condition for the achievement of universal health coverage in the age of the Sustainable Development Goals ¹ . High-performing primary care systems not only are the first point of contact for continuous, coordinated, comprehensive and people-centered health services, but also provide critical preparedness and response to public health threats ^{2, 3} .

The Salud Mesoamerica Initiative (SMI) is a multi-country, large-scale primary care performance improvement program. It is the result of a partnership between the governments of the eight Mesoamerican nation-states, the Bill and Melinda Gates Foundation, the Carlos Slim Foundation, the Government of Canada, the Inter-American Development Bank (IADB) and, during earlier stages, the Government of Spain. The program is aimed at improving reproductive, maternal, neonatal and child health outcomes among the poorest, rural populations in participating countries. Intended outcomes include increased availability, utilization, and effective coverage of primary care services and a reduction in preventable health inequities. The program’s approach to performance improvement combines the use of high-powered financial incentives at the government-level and the external verification of public sector, primary care system performance.

Programs and policies aimed at improving the performance of health systems have been at the forefront of many public-sector reforms. Initial waves of reforms in the public sectors of high-income countries were focused on learning and improvement ^{4– 7} ; subsequent waves of reform targeted public sector accountability and organizational best-practice ^{8, 9} . Governments in LMICs adopted and replicated these reforms, oftentimes with the support of multilateral finance institutions and agencies in the official development assistance space.

Several generic types of reforms follow the logic of public-sector interventions ¹⁰ , including (1) political interventions as expressed in policies and regulations; (2) reforms by laws; (3) intervention by audit and inspection, based on continuous evaluation of results and conformity to predefined norms; (4) intervention by management, based on organizational science and management practice, such as continuous quality improvement or change management methods, among others; and, (5) intervention by rationalizing professional behaviors such as the introduction of evidence-based practices and the standard comparison of outcomes by public sector providers.

In LMIC, accountability-driven reforms have flourished under the rubric of Results-based Financing (RBF); the health sector has regularly been at the center of such reforms. The Multilateral Finance Institutions, the Global Fund to Fight HIV/AIDS, tuberculosis and malaria, and the Global Alliance for Immunizations have financed results-oriented, global health programs, some of which have targeted health system improvements.

There are ambiguities in the definitions as well as in the scope and content of RBF programs and policies. In this study protocol, RBF is understood as “any program that rewards the delivery of one or more outputs or outcomes by one or more incentives, financial or otherwise, upon verification that the agreed-upon result has actually been delivered” ¹¹ ; incentives can target health care providers (supply side), households (demand side), or both. Performance-based Financing (PBF), is a prevalent type of RBF in which the incentives are exclusively financial; rewards are only aimed at providers; and the payments are usually adjusted for quality. PBF assumes many forms but, in essence, serves the purpose of reforming the ways in which governments pay health care providers (individuals and facilities) for the provision of services.

Accountability-driven interventions in public sector reforms are designed to reduce the misalignment in incentives between principals (voters, legislative bodies, executive-level leadership, funders, etc.) and their agents (program implementers, providers of care, etc.) ^{12, 13} . Such reforms usually assume that incentives and rewards serve as powerful motivators for the achievement of desirable behaviors among utility-maximizing, rational individuals ^{14– 16} . These assumptions have conventionally been based on principal-agent theory ^{14, 15} , positive agency theory ¹⁶ , and/or rational choice theory ^{17, 18} . In recent years, there have been calls for using a more expansive view of human agency when discussing motivation and decision-making. Under such views, humans are not exclusively motivated by rewards and incentives, but can also be driven to action by intrinsic motivators ¹⁹ . This perspective has also influenced contemporary research on PBF in LMIC ^{20, 21} .

Most of the primary studies that have assessed RBF and PBF programs in LMIC have characterized the effects of financial incentives on provider-level motivation and behaviors ^{21– 31} . However, the evidence on the effects of RBF on large-system reforms targeting government-level improvements is scarce; and studies in LMIC settings are largely absent ^{32– 34} .

The most-recent systematic review of performance-based financing programs in LMIC concluded that PBF is not a single type of intervention and that its effects are dependent on the interactions among multiple variables ³¹ . Most PBF evaluations to date have used a narrow focus such as characterizing changes in health care outputs, while neglecting most other domains of primary care performance improvement ³⁵ . Furthermore, the empirical evidence about how PBF leads to changes in attitudes and behaviors among public sector actors is scarce ³⁵ . Domains that have been under-studied include, among others, whether and how do extrinsic and/or intrinsic motivators affect the behaviors, autonomy and responsiveness of providers and managers of primary care delivery systems; the influence that performance measurement data can have on the behaviors of primary care system actors and stakeholders; and the negative effects of RBF reforms, such as gaming, shirking and cream-skimming.

Public-sector reforms tend to incorporate multiple interventions that generate effects at different levels within organizational hierarchies, and among different actors and stakeholders. Those actors and the environments in which they are embedded interact with each other through time, generating inter-dependencies and, oftentimes, leading to counter-intuitive, emergent, and unintended effects. Furthermore, the implementation strategies and ancillary components of reform programs themselves, such as the provision of technical assistance or change management support, can also trigger system changes that need to be better studied ³¹ .

Beyond accountability reforms, studies on performance management and performance assessment have empirically studied improvement-driven public-sector reforms. Studies of such reforms in the public sector of the United States have identified factors that can drive organizational learning and improvement. For instance, Moynihan and Landuyt ³⁶ found that the most influential predictors of organizational improvement and learning were the use of work-groups as learning forums; the availability of performance information systems that collect, store and disseminate performance data; the existence of a mission orientation that builds a sense of shared vision for success and common purpose; and the existence of organizational slack, such as time and resources that allow people to think and learn.

Improvement reforms are predicated on the assumption that the continuous collection, availability and analysis of performance data and information would lead to organizational improvement and learning ⁷ . However, despite widespread calls for using performance data and information to improve decision-making, the utilization of such data and information can rarely be guaranteed ³⁷ . Also, little is known about the conditions under which performance measurement work or the mechanisms that lead to system improvement. Studies in evaluation science have addressed these issues ^{38, 39} . In this literature, the availability and dissemination of performance evaluation can influence system improvement through multi-level changes on individual, interpersonal and/or collective motivation. Considerable research in evaluation science has been informed by this evidence ^{40– 44} . We did not find, however, any study assessing the effects of evaluation results on health system performance improvement in LMICs.

In SMI, governments agree with the IADB to the implementation of up to three consecutive, 18–24 month programs, aimed at achieving a series of progressively complex health targets (including inputs, processes, outputs and outcomes) that are externally verified by the University of Washington’s Institute for Health Metrics and Evaluation (IHME). Participating governments contribute domestic funds a-priori to attain the agreed-upon targets; once domestic funds are made available and targets are agreed, SMI matches the domestic contribution with grant financing on a 1:1 ratio. Afterwards, the IADB enters into formal performance contracts with each government. In the contract, SMI commits to reimbursing half of the initially invested domestic funds contingent on the achievement of the agreed-upon performance targets ⁴⁵ .

Country-specific performance frameworks with geographical targeting of the poorest, rural populations, were negotiated with each government at the start of the program and have remained stable through time. A pass-or-fail policy was agreed, according to which a government has to achieve 80% or more of the approximately ten (10) targets that make-up any given performance framework to be eligible for the reimbursement of half of the initial domestic contribution. Table 1 lists some of the targets agreed by El Salvador and Honduras.

As the agency in charge of external verification of government performance, IHME conducts a full-scale quantitative measurement that follows SMI’s sequential process of implementation. Before each country program starts, a baseline is collected and its results disseminated. After that, at the end of each 18 to 24-month implementation projects, IHME collects household and facility-based data to evaluate the achievement of agreed-upon results. Phase 1 programs started in a staggered fashion in 2011; phase-2 programs will finish during 2017; a third and final phase will start in 2018 and go into 2020. Program targets during phase 1 were focused on adherence to protocols, availability of resources and, in general, structure and process performance. During phase 2, targets were focused on outputs, and phase 3 will be centered on health outcomes, including but not limited to, coverage of exclusive breastfeeding, increased modern contraceptive prevalence rates, effective coverage of antenatal care and institutional deliveries, post-partum and post-natal care coverage, and in some cases, reductions in the prevalence of anemia and effective coverage of measles vaccination, measured in blood ⁴⁶ . After each round of performance evaluation, results are aggregated and disseminated in each country through policy dialogue workshops convened by the government and involving the IADB and IHME.

SMI’s original theory of change ( Figure 1) hypothesized that the use of supply-side financial incentives directed to central-level ministries in each participating government (Ministries of Finance and Health) would focus their attention on accounting for the achievement of their own agreed-upon health targets. The success of this hypothesis rested on four causal pathways. The first established that the three consecutive, biannual rounds of external verification of performance by IHME would generate sustained pressure on governments for the production of health results. The second pathway proposed that ongoing dialogic, participatory dissemination of data, information and evidence would lead to progressive improvements in the quality of care services and improved, aggregate performance in each participating country’s primary care system. Anticipated population-level health effects were also contingent on increasing domestic pro-poor health spending and expanding the demand for high impact health interventions among beneficiary populations.

While the program’s original theory of change identified several causal pathways, it did not explain how its main interventions would trigger outcomes, nor did it provide, either, a-priori explanations about the role that each country’s policy context could play in moderating the effects of program interventions. The SMI partnership appears to have embraced a high degree of flexibility in implementation to facilitate governmental buy-in. In 2011, the partners agreed on a set of common principles including a focus on external and independent measurement of results, accountability and transparency, and country ownership. These principles established the institutional boundaries that, in turn, allowed the IADB to negotiate country-specific performance contracts, results frameworks and evaluation plans with each participating government. They also granted implementing partners a high degree of flexibility in the design of each country’s multi-phased implementation plans and, also, led to performance contracts based on a few high-order principles (country ownership, a focus on results, pro-equity, cost-effective interventions, measurability and transparency) that were originally agreed among the funders and the IADB, and reflected in the program’s operating model.

In the two countries under study, El Salvador and Honduras, the program’s focus on country ownership led to each government deciding how to best deploy SMI’s non-reimbursable resources and their own domestic financing to increase the likelihood of achieving programmatic success. For instance, El Salvador had undergone a health system reform in the early 2010s, which coincided with the beginning of SMI implementation. The government decided to focus its targets on results that leveraged one of the reform’s central tenets, the provision of universal primary care services through community-centered, Family Health Teams ⁴⁷ . Honduras, in turn, had started in the late 2000s a large-scale pilot of contracting-out and pay-for-performance in the delivery of primary care services ⁴⁸ . The government thus decided to leverage its own performance-driven policies and programs and has thus implemented SMI in municipalities that had already acquired experience with RBF.

The refined program theory and CMO configurations resulting from this study have several anticipated uses and applications. Program implementers like the IADB and Salvadorian and Honduran government actors, for instance, can use the findings to consider introducing adjustments in SMI’s implementation during its third and final phase (2018–2020). Also, given the study’s focus on exploring the linkages between ongoing, pre-existing policy mandates and priorities, it is plausible to expect study findings to be of relevance to further improve the evaluation and subsequent re-design of domestic health policies in El Salvador and Honduras. Program evaluators like IHME and other research groups, can use program findings to enhance ongoing evaluation activities or to inform the design of new evaluations that deepen one or more of the various casual patterns identified. For example, our research team intends to use the emerging CMO configurations in the area of performance management to inform the design of a new study to explore whether and how SMI quality improvement interventions produce gains in primary care performance. We also expect program findings to set the stage for further realist evaluations in other large-scale primary care reform in contexts other than Mesoamerica.

Another source of complexity in this study arises from the significant evidence gaps that we identified and from the multiple fields that would need to be rigorously studied to properly address the various outcomes generated by accountability and performance management reforms. As discussed before, such outcomes can occur at different levels of analysis (individual, organizational and collective) and in different contexts (high- as well as LMIC). Not only does this type of research demand inter- and multi-disciplinary capabilities within research teams, but it also calls for rigorous, systematic assessment and mapping of the evidence gaps ^{79, 80} . Theory-based program evaluations of primary care performance improvement would also benefit from the publication of realist syntheses that rigorously appraise the literature in search of context-mechanism-outcomes and program theory ^{81, 82} . Such studies would not only facilitate the work of research teams currently addressing primary care performance improvement research, but would also strategically shape future health system research agendas, particularly in LMIC.

The research team has faced several challenges in shaping this study’s hypotheses, or preliminary program theory. Many of these challenges are common to other realist evaluations and have been discussed elsewhere ^{53, 83} . One such challenge pertains to settling on an unambiguous and precise definition for what constitutes a mechanism. Several definitions in the literature are of a descriptive nature and focus on well-known features of mechanisms such as them being unobservable, context-specific and being able to generate effects. We settled on a definition of mechanisms as the ideas and opportunities triggered among program actors and stakeholders in response to program interventions ⁵⁷ . Such an approach is consistent with a view of social change according to which the beliefs, choices and opportunities of individual actors and the interactions among them (micro-level) are the main drivers of social change. This approach also recognizes that the “macro,” social and cultural environment in which these individual actors are embedded can shape social change by means of the internalization of collective values, norms and institutions among individual actors ^{84, 85} .

Based on these considerations, this study aims to, first, explore plausible causal explanations based on individual or group-based ideas and opportunities among program actors and stakeholders and, second, to ground those observations on an understanding of the policy and program context in which those actors and stakeholders are embedded. Therefore, we expect that any explanation of primary care system performance improvement needs to address both individual, micro-level, as well as collective, macro-level properties that “are not meaningfully attributed to individuals” ⁸⁴ . Three specific types of mechanisms that explain social change are thus of interest to this study ⁸⁶ .

Situational mechanisms refer to the macro, organizational-level environment in which SMI actors and their social interactions or linkages occur, including domestic policy-makers, ministry of health managers and primary care providers, among others. It also includes SMI stakeholders such as the implementation agency (the IADB) and the external evaluators of performance (IHME). This type of mechanism operates in the direction from macro environment to individual actors (macro-to-micro change). Action-formation mechanisms are those that explain how actors’ ideas and opportunities influence individual behaviors across the primary care system. In this type of mechanism, the interaction between program interventions and context, trigger changes in individuals’ ideas and opportunities that further influence others in the same social system. This type of mechanism can generate effects that spread from an individual actor to additional actors (micro to micro change). Finally, transformational mechanisms provide explanations of how the sum of new behaviors of multiple individual actors in the primary care system bring about change across the entities that conform the primary care system’s macro environment such as norms and institutions (micro-to-macro social change).

Due to limitations in scope, this study can only plausibly characterize some of the “macro” and “micro” mechanisms triggered during SMI’s initial stages up until its current, mid-term stage (2011–2017). It is also plausible to characterize intended and unintended effects generated during this same period. Findings can also be used identify propositions about downstream effects that could occur or not during the final implementation phase (2018–2020) and about the mechanisms that could help sustain desirable effects after implementation ends. Longer-term, transformational mechanisms, their anticipated effects and the underlying context-mechanism-outcome configurations will, however, remain outside our scope of work.

Another challenge in this study refers to the contested nature of the current definitions used to characterize the interventions that conform RBF, PBF or any of the various reforms that use supply-side incentives to drive accountability in public sector actors and, as is the case in SMI, across the entire primary care system. Like others before us, we settled on the definitions provided by Musgrove ¹¹ , but we remain cognizant of the fact that PBF is not a single intervention and that its “ancillary” components can themselves generate system effects ³¹ . In this respect, this study frames SMI interventions as generic types of public-sector reforms aimed at inducing accountability and organizational improvement and learning. Given that the challenges in defining what these large-scale reforms contain in specific contexts –both in high and well as less-developed nations- the use of a realist evaluation approach will likely contribute to the theorizing of how and why specific contexts generate health and non-health effects in primary care performance management reforms.

Finally, given the method-neutrality that is central to realist evaluations, this study also faced the challenge of settling on a final sequence and content of research methods and activities. We decided to, first, follow the steps described by Vareilles and Marchal in relatively similar realist evaluations and studies ^{53, 57} , but also relied on the Realist and Meta-narrative Evidence Syntheses study that provides guidance on how to improve quality and consistency in the reporting of realist evaluations ⁷⁷ . By aligning protocol design to these guidelines, we expect that the furture publication of this study’s findings will adhere to current best practice.

The study’s protocol was reviewed and declared exempt by the George Washington University’s Institutional Review Board (study number 041733). The Ministries of Health of El Salvador and Honduras were informed of the proposed research by the IADB and provided written approval for the research activities.

Ethical approval documentation will be made available on request. The study will employ scrupulous adherence to the highest ethical standards, and current international and local legislation pertaining to research governance. The data collection will operate under explicit informed consent, which will be preserved in study records. Respondents will be given the choice to provide consent verbally on tape before the interviews, or in writing. To maintain anonymity, respondents will reserve the right to review the study outputs and withdraw consent if necessary. All identifying information will be removed from transcripts and stored separately with access restricted to the research team. All transcripts will be stored electronically in password protected cloud services, and physical documents will be securely stored at George Washington University, Milken Institute School of Public Health.