SMI is a results-based, health system-strengthening initiative that provides funding to participating governments for the achievement of population-level outcomes in reproductive, maternal, neonatal, and child health ³⁵ . Participating governments agree with the Inter-American Development Bank (IADB) to implementing up to three consecutive 24-month phases, aimed at achieving a series of progressively complex health targets. Country performance frameworks target the poorest, hardest-to-reach municipalities and have an initial focus on inputs and processes, while progressively prioritizing health outputs and outcomes.

Governments contribute domestic funds a-priori and SMI matches these contributions with grant financing. In the contract between SMI and the government, the former commits to reimbursing half of the initially invested domestic funds contingent on the achievement of the agreed-upon performance targets ³⁶ . A supply-side financial incentive was established, according to which a government has to achieve 80% or more of the targets in its performance framework to be eligible for the reimbursement of half of the initial domestic contribution.

SMI’s Salvadorian phase 2 program include interventions that are common to all participating countries and, also, unique PMM interventions. Of relevance to this evaluation was the inclusion of an in-kind, team-based incentive program implemented between 2015 and 2017, and involving the 75 Community Health Teams (CHTs) participating in SMI.

Participating CHTs agreed to a set of eleven (11) maternal, neonatal and child health targets, linked to the performance framework agreed between SMI and the Ministry of Health (MOH). CHTs that succeeded in achieving 80% or more of their performance targets were eligible for an in-kind incentive from a list of pre-established goods valued at up to $1,000. The payment rule was based on a sliding scale, with the awards varying depending on the number of targets achieved.

A cluster-randomized impact evaluation was conducted to experimentally evaluate the effects of in-kind incentives on CHTs performance ³⁷ . SMI collected household and facility survey data every six months. Results from the impact evaluation will be published separately by the IADB. In response to SMI’s measurement of CHT performance, the MOH increased its own internal processes for CHT performance verification and supervision. The collection and analysis of performance proxies became a continuous process, and the regular monthly meetings between regional MOH offices and CHTs became a routine source for data analysis, internal accountability, action planning, and course-correction.

Performance results from the four waves of impact evaluation data collection were disseminated every six months, among all CHTs participating independent of experimental assignment. Specially convened meetings were hosted by the MOH and attended by central- and regional-level officers from the MOH, CHT coordinators, and SMI experts.

Program performance is externally verified by the University of Washington’s Institute for Health Metrics and Evaluation (IHME) at baseline and before the finalization of each phase. IHME uses quantitative measures of performance obtained through household and facility surveys ³⁸ . After each round of performance measurement, results are aggregated and disseminated in each country through dialogue workshops involving the government, IADB and IHME. Further information about SMI’s original design and theory of change can be found in the evaluation protocol ³⁹ .

Having consistently achieved its performance targets, El Salvador’s SMI program has been a major success to date. As a result, the MOH has received financial incentives for $2,6 million in phases 1 and 2. Table 1 lists targets and the results obtained to date.

El Salvador is a low-income country in Central America. According to the World Bank, annual real growth in gross domestic product (GDP) was 1.9% in the period between 2010 and 2016, the lowest in Central America. Since the end of the civil war in 1992, El Salvador has made continuous progress in human development, increasing access to public services among the poor, reducing inequality by about 5 percentage points between 2007 and 2016, and increasing income in the lowest income quintile ⁴⁰ . Health indicators have also shown major improvements in the last decade with institutional deliveries and immunization coverage reaching 98 and 93%, respectively. Sustained pro-poor social policies in the country helped the country achieve Millennium Development Goal #4 (MDG 4) for under-5 mortality reduction ⁴¹ . Also, El Salvador is one of twenty countries undertaking accelerated implementation of the 2030 global agenda for the Sustainable Development Goals.

In 2009, the government of El Salvador reformed the public health system by introducing changes to the Ministry of Health’s (MOH) model of primary care delivery. Existing facilities were reorganized into integrated networks of healthcare delivery; free services were introduced; and the provision of comprehensive primary care services through Community Family Health Teams (CHT) became the central mode of service delivery. Rural CHTs include a medical doctor, two nurses, and three community health promoters with a nominal catchment of 3000 individuals for the entire primary care team. CHTs identify, enumerate and assess risks in their assigned catchment areas and are responsible for providing a package of approximately 300 primary care-related interventions such as health promotion, prevention, and treatment of common conditions ⁴² . By the end of 2015, El Salvador had established 747 CHTs. In El Salvador, SMI started implementation in 2011 in 14 of the poorest rural municipalities where 75 CHTs operate. Table 2 provides details about the linkage between the PMM framework and the SMI program in El Salvador.

This study used a realist evaluation approach, which provides a valuable standpoint for evaluating policies and programs that attempt to introduce changes in complex adaptive social systems ^{43, 44} such as a PMM system ⁴⁵ . Realist evaluation is a type of theory-driven inquiry based on the premise that an evaluation needs to answer, “ what worked, how, in what circumstances, and for whom” complementing the more conventional evaluation question “ Did the program work?” ⁴³ . The rationale for using a realist approach in this study was described in a previously published study protocol ³⁹ . The appeal we saw in realist evaluation, compared to other possible approaches is its explicit foundations in critical realism, an epistemology that contends that the interactions between context and program interventions generate change through the triggering of underlying, usually hidden, causal mechanisms.

To characterize high-performing primary care systems at scale, we purposefully identified and studied positive outliers, a methodological approach that “can reveal a great deal about intense manifestations of the phenomenon of interest” ⁴⁶ . Such study design also aligns well with the realist evaluation proposition that contexts can trigger to-be-identified mechanisms that, in turn, interact with program interventions and contribute to generating outcomes at various levels.

This paper reports the results of an ongoing evaluation that is concurrent with the implementation of SMI’s second and third phases. The research questions addressed here are: (1) What are the effects of using supply-side incentives at the primary care delivery level, on the performance of the primary care delivery system? How are those effects produced? Under what contextual conditions? And, (2) What are the effects of high-stakes external measurement on performance itself? How are those effects produced? What contextual factors contribute or hinder those effects? In this paper, we report program effects only from the perspective of CHTs; policy-level perspectives will be presented elsewhere.

The steps followed in this evaluation were detailed in the study protocol ³⁹ . They included, first, the development by the research team of a preliminary program theory (PT), followed by the identification of the research methods best suited to test and revise the preliminary PT. Data is then collected and analyzed; the process ends with a synthesis of findings and the generation of a refined PT. Each of these steps will be briefly described below.

The preliminary PT ( Figure 1) that underpins this evaluation shows hypothesized causal linkages between context, SMI’s interventions, causal mechanisms, and program outcomes. The initial PT resulted from a scoping review of the literature on PMM and public sector motivation ^{9, 18– 20, 23, 24, 28, 47, 48} , performance information utilization ^{29, 49– 53} , diffusion of innovations theory applied to health services delivery ^{54– 58} , socio-cognitive theory ^{59– 61} , self-determination theory ^{62– 71} , neo-institutional theory ^{72, 73} , and recent evidence on the performance and motivation of frontline workers in LMIC ^{10, 74– 76} . Additional information was obtained through SMI documents review and a program stakeholder workshop held in early 2017. The preliminary PT is described in detail in the study protocol in Box 1 ³⁹ . Table 3 provides a summary of the data sources used to address specific elements of the preliminary PT and a general schematic of the sequencing and timing of the evaluation components.

In a work environment, motivation is defined as an “individual's degree of willingness to exert and maintain an effort towards organizational goals ⁷⁶ .” Highly motivated workers report higher levels of job satisfaction, which can lead to lower levels of health worker attrition, ultimately contributing to the retention and continued presence of skilled and experienced workforce in a health system. Motivation can be instrumental or extrinsic, driven from contingent rewards (both financial remuneration and non-monetary recognition), or intrinsic, driven from an individual’s inherent desire to perform a task for its own sake ^{77, 78} . Usually a combination of these two facets determine an individual worker’s motivation to perform.

Research in economics, psychology, organizational behavior, and public administration, among other fields, assume that incentives and rewards serve as powerful motivators for the achievement of desirable behaviors among utility-maximizing, rational individuals ^{79, 80} . This approach has fueled the design of various accountability-driven interventions and many interventions. Many public-sector reforms in LMICs and various global health partnerships have been influenced by this approach. However, theoretical and empirical developments in public administration research suggest that workforce motivation can also be explained by intrinsic motives such as public service motivation, a socially learned set of preferences that can be present among individuals working in the public sector ^{9, 81} . To characterize workers behaviors in primary care settings in LMICs, we focused on the theory of self-determination, a macro theory of human motivation that has been used to study motivation in the workforce including in a previous realist evaluation in Uganda ^{67, 71} . The theory has good cross-cultural validation and has demonstrated that individual workers who satisfy internal needs for competence, autonomy and relatedness feel intrinsically motivated and can develop self-motivated behaviors ^{66, 82– 84} .

As detailed in the protocol ³⁹ , realist evaluation is method-neutral. The methods that optimally addressed the evaluation questions that this paper focuses on, under real-world operational constraints included: document review, key-informant interviews, and non-participant observation. We used a purposeful, outlier sampling ⁴⁶ in which a case is defined a top-performing CHT in the SMI phase 2 program in El Salvador. For the purposes of this study, we selected positive outliers, namely the top four (4) performing CHTs as identified by independent performance measurements conducted by SMI in September 2015; March 2016; September 2016; and February 2017.

Interviews were conducted with MOH policy and program experts, and with providers of care in the four participating CHTs (doctors, nurses and health promoters). To acquire information about SMI’s design, implementation and evaluation not likely to be apparent from program documents, we also interviewed IADB and IHME experts. A total of 38 in-depth interviews were conducted between July and November 2017. Interviews were conducted by one member of the research team while a second investigator acted as observer. Table 4 summarizes the profiles of the respondents.

Specific issues addressed in CHT interviews included, (1) Respondent perceptions of the SMI interventions under study; (2) Emotional and cognitive reactions triggered by the use of MOH-level financial incentives, in-kind team incentives, and external measurement of CHT performance; (3) Knowledge of, and perceptions about, context-specific factors that hinder or contribute to workers individual and team-based behaviors; (4) Descriptions of program ancillary components that could explain program effects; and, (5) Unintentional and undesirable effects. Interview guides were published with the study protocol ³⁹ .

Information about the dissemination of results from the external verification of performance was obtained by observing IHME’s presentation of results of overall program performance measurement at the end of SMI’s second phase. One of the authors (CI) attended the workshop that took place in El Salvador during May 2018.

Interviews were first transcribed verbatim in Spanish and were then translated into English. A predefined coding system based on the preliminary PT was developed a priori and used for coding by three of the authors (WM, SW and SM) using NVivo 12. Inductive codes also emerged from the data, further enhancing the coding process. Context-Mechanism-Intervention-Outcome patterns across interviews were synthesized into themes using analysis memos that identified salient domains and recorded exemplary quotes.

We used several criteria to identify emergent themes. First, the frequency of respondents’ perceptions was the starting criterion to identify salient areas; these were consolidated as themes based on their relevance and importance to the study questions and preliminary PT, through discussions and consensus-building among the researchers. We also merged the perspectives shared by central-level policy-makers with the accounts shared by frontline workers to compare, contrast, and ultimately converge perspectives from these two organizational levels of the health system.

Once thematic analysis was completed, the themes emerging from the transcripts were arranged in tables according to their reference to context, interventions, causal mechanisms, and outcomes identified by each individual respondent. We used the resulting tables to identify hypothetical Context-Mechanism-Intervention-Outcome configurations that could plausibly explain the observed program results. Findings were also triangulated by considering data from different sources (IHME external verification data, the results from CHT performance measurement, observation notes, and secondary document analysis). The refined PT was synthesized as narratives and figures. Additional details about the data analysis strategy were included in the study protocol ³⁹ .

To increase the validity of the study in terms of reflexivity, credibility and confirmability, and to enhance the trustworthiness, transparency, and accountability of the research, investigators maintained ‘personal biases memos’ that made explicit all self-identified biases and pre-conceptions that may have effects on the research process ⁸⁵ . Also, all analytic decision notes and memos, biases memos, document analysis syntheses, interview guides, research team meeting agendas and minutes, and analysis outputs including coded transcripts, conceptual frameworks, and tables have been preserved to provide a verifiable audit trail.

The study’s protocol was reviewed and declared exempt by the George Washington University’s Institutional Review Board (study number 041733). The Ministries of Health of El Salvador and Honduras were informed of the proposed research by the IADB and provided written approval for the research activities.

Respondents were given the choice to provide consent verbally on tape before the interviews, or in writing. To maintain anonymity, respondents reserved the right to review the study outputs and withdraw consent if necessary.

Based on the evidence from the achievement in the four high-performance CHTs sampled, several contextual factors appear to have been conducive to the successful implementation of SMI in El Salvador.

The convergence of domestic political and policy priorities, a conducive global environment, and SMI’s focus on performance measurement created a unique window of opportunity for country ownership to take hold. The election in 2009 of the first-ever, left-wing President in El Salvador generated a favorable political environment for SMI implementation. The incoming administration’s priorities opened a window of opportunity for institutional reforms in the health sector, leading to the formulation of new rules for increasing affordability, equity, access, and coverage of publicly provided health services and to the scaling-up of comprehensive primary health care services in the poorest municipalities in the country. Central to this process were the enhancement of the supply of public services through CHTs and the introduction of free primary care services.

Upon SMI launch, the MOH internal organizational environment was characterized by a positive tension for change and a sense of urgency. The Salvadorian reform’s focus on primary health care delivery and SMI’s pro-performance stance supported the perception at high-levels that the government’s and SMI’s priorities were strategically aligned. This was particularly evident in areas such as targeting the poorest rural municipalities, improving equity gains, and expanding coverage for maternal, neonatal and child health services. These factors may have also provided a positive tension for change and a sense of urgency on the part of the government, leading to their full engagement with SMI and agreement to attempt the achievement of high-stakes performance targets. The global focus on achieving the Millennium Development Goals further reinforced SMI’s and the Salvadorian government’s high priority for maternal and child health.

SMI implementation leveraged existing MOH structures, practices and routines, increasing country ownership and generating complementarities between SMI implementation and the MOH’s internal environment. Since SMI has been implemented directly by the MOH, the program has leveraged existing MOH administrative and managerial systems. For example, the MOH has a long-standing tradition of hosting regular, monthly meetings among primary care stakeholders (e.g., regional offices and CHTs) for supervision and coordination purposes; these meetings were also regularly used to transmit top-down instructions and directives. These routines were adopted and adapted by the MOH and SMI and transformed into continuous, ongoing spaces for performance dialogues, a central aspect to SMI’s approach.

SMI implementation exposed weaknesses in MOH management systems early on, which have been reduced during implementation and no longer hinder CHT performance. Early failures in supply management and logistics led to stock-outs that negatively affected CHTs’ ability to perform. Such experiences led to the identification and explicit discussion of the inter-dependencies between CHT performance and health system deficiencies. Improvements in administrative processes ensued and stock-outs and poor availability of supplies were addressed.

Causal mechanisms are defined as the actions and reactions, or behaviors, triggered among program actors by the interactions between the context and SMI’s interventions. Extrinsic and intrinsic motives serve as mechanisms for transforming individual and interpersonal behaviors among CHTs and other stakeholders. The mechanisms hypothesized in the preliminary PT, can be positive sources of motivation as well as negative, balancing causes of de-motivation. Table 5 presents exemplar quotes about the mechanisms identified.

Positive sources of individual motivation (extrinsic and intrinsic). Our results suggest that the Salvadorian reform and SMI’s pro-performance stance may behave as complementary sources of positive, intrinsic motives as well as external drivers of performance, respectively. Furthermore, findings suggest that the intrinsic motivation for joining CHTs are complex and may therefore be triggered by different motives and preferences. When asked about their reasons for working at the community level, CHT respondents identified the acquisition of a sense of competence and the satisfaction of a desire for providing public services, while also acknowledging the benefits from wages and job security.

Extrinsic motives play an important role for individual CHT members who recognize the inherent value of public-sector job security. The joint effect of extrinsic motives like salary, job security and the SMI-specific in-kind incentives for CHT was important for all respondents. A desire for additional bonuses and other forms of external recognition was reported by the majority of CHT respondents.

All frontline workers interviewed expressed a sense of commitment and sacrifice to the communities they serve; to the health of families in their catchment area; and to the achievement of their collective, CHT-specific performance targets. Such degree of commitment plausibly indicates the existence of high levels of intrinsic self-motivation among CHT members that may operate at individual as well as interpersonal levels. Work as a front-line, primary care provider satisfies for most respondents one or more of the following: a sense of competence in their abilities and skills; a sense of autonomy in executing their tasks; and a sense of relatedness to their colleagues and the communities they serve.

Credible data support the qualitative findings regarding commitment and positive motivations expressed by CHT respondents. These include CHT-specific performance information collected by the MOH on monthly cycles of measurement, and the analysis and action-planning reinforced by SMI interventions such as quarterly supervision visits; regular focusing events where teams’ performance is reviewed between CHTs, the MOH, and SMI; and, high-level dialogue meetings where MOH, SMI and IHME discuss the results of the external measurement of system-wide performance.

An intrinsic motive frequently expressed by respondents is the sense of competence obtained from accomplishing their team tasks and thus being able to verify for themselves positive effects on local beneficiaries. Physical and emotional closeness between front-line workers and the community in which they live, come from, and serve, likely increases their social identity and the significance of their jobs, fulfilling a sense of competence and relatedness, which can contribute to the expressions of job satisfaction that emerged frequently in the interviews.

Team-based, interpersonal-level motivation. Our findings suggest that team-level mechanisms play a major role in modifying interpersonal motivation and supporting team cohesion. CHT respondents recognized that SMI introduced major, and positive sources of team motivation. One of the most valued sources of motivation was the sense of contribution, collaboration, and relatedness to other CHTs that was found through the convening of regular performance dialogue meetings. Furthermore, respondents reported a sense of team-based, collective efficacy mediated by the Salvadorian reform requirement that CHTs engage local communities in developing their annual operating plans as well as in accounting their performance results. These are sources of inter-personal or team-based motivation, as the effects they trigger can only be accounted for by collective actions executed by the entire CHT. Such mechanisms are reinforced by the sense of trust that communities develop in high-performing CHTs, thus increasing their members’ individual and team-based social status.

Furthermore, PMM interventions and ancillary components introduced by SMI are implemented in within- and between-teams’ social environments such as ongoing, internal team meetings in each facility, and external learning events convened by SMI and hosted by the MOH. CHTs from different geographical areas get together to learn from each other, share knowledge and experiences, and design improvement plans. The data suggests that such group dynamics can help CHTs make sense of the performance data, leading to the actual utilization of performance information to plan improvements and process innovations. It is thus plausible that these dialogue spaces among peers may trigger social learning mechanisms and set the basis for the organic diffusion of ideas, innovations and behaviors.

When describing the reasons why they enjoy working for their communities, CHT team members consistently reported a sense of empowerment, commitment, team efficacy, and community service. The closeness between team members and beneficiary communities is also a source of inter-personal motivation. Highly-effective CHTs can attain a degree of social recognition that may only be possible in community-embedded teams. Furthermore, for individuals with high levels of public service motivation, energized by fulfilling their sense of competence, autonomy and relatedness, social recognition may further reinforce their intrinsic, pro-social attitudes.

Potential sources of de-motivation. Despite their strong motivation, CHT members oftentimes work against their own personal interests even when this generates undesirable effects for themselves or their families. A frequent theme when describing the challenges of providing outreach services is one of sacrifice which may be both a source of stress as well as an internal drive to help others.

Performance measurement also triggered feelings of stress and anxiety particularly during the early stages of SMI implementation when, according to CHT respondents there were shortages of supplies and other inputs required for meeting CHT performance targets. Also, while respondents in all four CHT reported participating in, and being energized by their involvement in meetings and learning activities, their responses indicated certain preferences and dislikes. For instance, the least appealing activities for many respondents were those that required inter-team comparisons, poorly-delivered feedback by supervisors, and meetings where differentials in goal achievement among CHTs were publicly highlighted.

We identified plausible causal linkages between the relatively small, but well-targeted supply-side incentives brought by SMI at the central level of the MOH; the high-stakes generated by SMI’s measurement of CHT performance every six months; and in-kind, team-based incentives. Findings suggest that SMI interventions and ancillary components; the dynamic institutional environment generated by the health reform; and, the socio-economic contexts in which Salvadorian front-line workers operate, contributed to triggering intrinsic and extrinsic motivation in the four high-performing CHTs studied. Such gains in motivation appear to have been built upon high levels of pre-existing public service motivation among front-line workers and on extrinsic motives such as wages and the expectation of job security and promotions. The introduction by SMI of performance measurement and in-kind, team-based incentives, appear to have reinforced self-motivation among individual team members, also contributing to the generation of team-based motivation.

The learning events scheduled during SMI implementation created a space in which front-line workers, mid-level managers, MOH policy-makers, and SMI staff engaged in enriching performance dialogues and learning and knowledge-sharing. This may have generated several effects. First, it may have focused the attention of CHT and other system actors on making sense of performance data and using performance information to plan future improvements. These behaviors may have also been reinforced by 1) the pro-social, other-oriented and intrinsic satisfaction derived from the sense of competence, autonomy and relatedness described by community health team members; 2) the external motivation triggered by in-kind team-based incentives; and, 3) the extrinsic and powerful incentives generated by increased community recognition and social stature among front-line workers. Second, performance dialogue meetings may have also created safe inter-organizational spaces where social connection, learning, and knowledge-sharing could happen. Finally, these meetings acted as focusing events where high-performing teams were recognized and rewarded, and where dialogue across MOH organizational levels created opportunities to change internal conversations between managerial and operational actors, from conventional top-down control, towards emergent horizontal and, possibly, bottom-up performance management and organizational learning activities. Were this last to be sustained through time, it could set the basis for the emergence of a culture of autonomous, self-directed improvement among CHTs, further reinforcing the positive motivational mechanisms described above.

Our findings identified elements that were not originally included in the preliminary PT that enrich the emerging, refined PT. First, the PT identified theories of human behavior and motivation that were mostly focused on individual-level motives, such as social-cognitive and self-determination theories ^{59, 61, 65, 82, 83} ; it also highlighted interpersonal mechanisms mostly transmitted through diffusion and social learning channels. Second, despite the mention of unintended or negative outcomes in our preliminary PT, the evaluation protocol did not expand on what those effects could be, except as it pertains to gaming, shirking and other negative effects.

The refined PT ( Box 3 and Figure 1) now includes team-based motivation as a specific type of interpersonal motivation, informed by collective efficacy theories ⁸⁶ as well as by theories of identity ⁸⁷ and social identity ^{88– 91} . The refined PT also contains unintended, demotivating effects that PMM interventions could induce in frontline workers such as stress, anxiety, and poor work-life balance.

Finally, the use of powerful supply-side financial incentives by SMI at the higher echelons of the MOH may trigger additional mechanisms and outcomes (positive and negative; intended, and otherwise) that need to be further researched and that will be the subject of a separate paper. We expect the PT to continue to be adjusted as new findings emerge from the realist evaluation of SMI in El Salvador and Honduras.