Evidence gap map of performance measurement and management in primary care delivery systems in low- and middle-income countries – Study protocol

Why is this study relevant to research, policy and practice in LMICs?

There are multiple approaches, frameworks, and conceptualizations for characterizing health systems, measuring and managing their performance, and typifying health system interventions. The study uses a multidisciplinary approach to identify and characterize the relevant literature from different fields and disciplines such as organizational science, health systems research, and public-sector management.

Organizational performance refers to the results generated by an organization and measured against its intended aims. In private sector organizations, performance can be a function of profits, organizational efficiency and effectiveness, quality of goods and services, market share, and customer satisfaction. In public administration, the definition of organizational performance has evolved with the changing framings for the role of the State in the production and delivery of public goods and services⁴. Historically, governments initially emphasized aspects of performance such as the control of inputs and the compliance with standards. Subsequent framings shifted, first, towards a focus on the quantity and quality of outputs, productivity, and efficiency and, in recent years, to outcomes and policy impacts and, in the case of the health sector, to social values like patient- or people-centered health services^4–7.

On the research side, the theories of organizational performance have followed, in general, the evolution of the practice of performance management in high-income countries. According to Talbot⁸, an initial set of theories and frameworks were focused on characterizing associations between individual elements of performance and organizational effectiveness. Afterwards, researchers focused on excellence, quality and organizational culture which led to the development of a first wave of models of performance measurement and management. These models did not account for differences between public and private sector dimensions of performance but were nonetheless adopted by governments around the world. In the 90s, the focus shifted from theoretical perspectives about organizational performance to interest in how to measure goal achievement in public and private sector organizations using performance models such as the Balance Scorecard and others^9–11. Interest in performance measurement and management spread around the world, and international comparisons and benchmarking of performance flourished in various sectors such as governance^12,13, health¹⁴, and education¹⁵.

During the last 40 years, innovations in performance measurement and management in the health sector have been prevalent in the United Kingdom, Canada, Australia, New Zealand, Sweden and the US, among other countries^16–19. Amplified by multilateral finance organizations and some bilateral agencies, performance measurement and management systems have spread among LMIC, sometimes as central aspects of large-scale public-sector reforms and, also, as stand-alone health sector reforms. Some elements of performance measurement and management have spread more than others particularly performance-based financing, pay-for-performance, performance budgeting and contracting and the use of financial incentives (defined below).

The spread of the practice and research of performance measurement and management has also affected the global health architecture and its governance. The interest among donor governments, multilateral finance institutions, bilateral agencies, and global philanthropies started shifting since the late 90s from a focus on funding inputs towards an interest on the production of measurable results aid effectiveness, and accountability. Such shifts in preferences contributed to the emergence of new global organizational forms and partnerships such as the Global Fund to Fight HIV/AIDS, Tuberculosis and Malaria, GAVI the Vaccine Alliance, the Global Finance Facility, and the Mesoamerican Health Facility, to name a few.

In terms of effectiveness, 40 years of research on performance measurement and management have shown that, despite many challenges, such systems can be effective^20–25. There is evidence, also, of the generation of unintended effects in the public and private sectors^22,26–29.

In the area of health systems research, research conducted to date by the Cochrane Collaboration has generated approximately 200 systematic reviews addressing the effective organization of health services. While the majority of these have been focused on issues of relevance to high-income countries research and policy, there is a growing portfolio of reviews focused on delivery and financial arrangements, as well as implementation strategies in LMIC^30–38.

The components of a generic performance management system are delineated in Figure 1 and represent an adaptation of two frameworks. The first, is a generic framework of public management in public sector organizations developed by Pollitt²⁷ and the second is the result of ongoing research on primary care performance in Mesoamerica led by one of this study’s authors (WM)³⁹. Framework components include: 1) A context in which various policies, organizations, programs and health interventions coexist with system actors and stakeholders; 2) one or more performance management interventions; 3) activities for measuring the results from the implementation of primary care policies and programs, and its ensuing data; 4) a process through which raw performance data is made sense of and transformed into performance information; 5) dissemination of performance information among system actors and stakeholders with the intent of making it actionable; 6) performance information use, misuse or non-use; 7) implementation of planned action based on the use of performance information; and, 8) the effects from the implementation of planned action and clinical and managerial improvements (proximal processes, outputs and outcomes, and distal, societal and population-level outcomes).

Figure 1. Generic performance measurement and management system.

The production of actual, measurable performance is a complex, dynamic phenomenon. Real performance can be very hard if at all possible to observe. Its measurement is characterized by lags between the introduction of interventions, the production of effects, and their measurement. These delays can also create a disconnect between action, measurement and results. Once measurement occurs and performance information is available, system actors and stakeholders can respond to the perceived performance gap by using, not using, or misusing such information^40,41. To be effective, performance information needs to apprise subsequent organizational action. Changes in strategic direction or operational tactics would also have to be effectively implemented for outcomes to be generated.

Based on the above theoretical and practice-oriented considerations, the study defines performance measurement and management in a primary care delivery system as the introduction of management systems focused on measuring organizational processes, outputs and outcomes with the proximal aim of informing the introduction of clinical, managerial, programmatic and policy changes and the ultimate goal of contributing to socially valued, population level health outcomes.

Implementation strategies

In this category, the study identified 8 interventions of relevance including: 1) In-service training, a form of positive behavior support aimed at increasing the capabilities of individual primary care system actors⁴⁹; 2) Reminders, manual or computerized interventions that prompt individual providers to perform an action during a clinical exchange and can include, among others, job aids, paper reminders, checklists, and computer decision support systems^42,50–54; 3) Clinical practice guidelines, or systematically developed statements to assist healthcare providers and patients to decide on appropriate health care for specific circumstances^42,55–59; 4) Continuous education, referring to courses, workshops, or other educational meetings aimed at increasing the technical competencies of primary care providers; 5) Clinical incident reporting, or systems for reporting critical incidents and adverse or undesirable effects as a means to improving the safety of healthcare delivery³³; 6) Local opinion leaders, referring to the identification and use of identifiable local opinion leaders to promote good clinical practices^31,60; 7) Continuous quality improvement defined as the iterative process to review and improve care that includes involvement of healthcare teams, analysis of a process or system, a structured process improvement method or problem-solving approach, and use of data analysis to assess changes⁴². It will include lean management as one of the approaches used to improve efficiency and quality in service provider organizations^61–65; and, 8) Supervision, defined as routine control visits by senior primary care staff to providers and facilities^66–72.

Accountability arrangements

The interventions included in this EGM are: 1) Audit and feedback, defined as a summary of primary care provider or facility performance over a specified period of time, given in a written, electronic, or verbal format; such intervention s can occur at individual provider as well as at organizational, facility level^73–78; 2) Public release of performance data, defined as arrangements to inform the public about the performance of primary care providers or facilities in written or electronic formats; and, 3) Social accountability interventions, defined as an accountability arrangement in which community members and/or civil society organizations are involved in the monitoring of performance of primary care providers or facilities⁴⁸.

Financial arrangements

There are many variations in this type intervention and contested definition among them. The interventions of interest to this study are under the general heading of Performance-Based Financing (PBF) but can also include Results-Based Financing (RBF), Pay-for-Performance (PFP), and the use of provider rewards and incentives. For precision purposes, we include the definitions developed by Musgrove⁷⁹ for these terms:

Outcomes included in the evidence gap map

Outcomes were categorized following the guidelines developed for EPOC systematic reviews and adapted for the identification of gaps in research on performance management in primary care systems in LMIC. Relevant outcomes are, therefore, those that can be actionable for the intended users: research groups, funding agencies, and performance measurement and management practitioners in primary care systems in LMIC. Based on these considerations outcomes of interest will be wide in scope; can occur across short- and long-term timeframes; can be observable at various levels within a system (individual, organizational, social); and, can include desirable as well as undesirable, adverse effects. Equity effects constitute a unique, cross-cutting category, the inclusion of which is recommended in evidence gap maps. The priority-setting exercise that will follow the completion of this EGM may result in the identification of primary and secondary outcomes; at this stage, however, the study aims to scope the largest number of relevant outcomes within available operational constraints.

The main categories included in this EGM are listed in Table 2. They include: 1) provider and managerial outputs and outcomes, defined as individual, provider and managerial staff effects, and exemplified by changes in workload, work morale, stress, burnout, sick leave, and staff turnover; 2) patient outcomes, defined as changes in health status or on patient health behaviors; 3) organizational outcomes, defined as organizational-level effects within and across facilities and networks of primary care such as quality of care process improvements, patient satisfaction, perceived quality of care, workforce retention, organizational culture, and unintended outcomes (gaming, shirking, shaming, data falsification, etc.); 4) population-level outputs and outcomes, defined as aggregate, health and equity effects accruing defined populations, including utilization of specific primary care services (for instance, number of antenatal care visits, institutional deliveries, etc.), coverage of services (such as the proportion of pregnant women receiving antenatal care, proportion of pregnant women delivering in facilities; coverage rate of specific vaccines), access to primary care services (for instance, waiting times), adverse health effects or harm, health equity effects, and unintended health effects; and, 5) social outcomes defined as non-health, social, economic, or cultural effects affecting defined populations, such as changes in community participation, non-health equity effects, non-health adverse effects or harm, and other unintended social outcomes. Table 2 lists each outcome category and provides examples of specific types of results within each category.

Overall approach

The team will follow the methodology to produce evidence gap maps developed by 3ie^109,110. The methodology was developed as a tool to systematically map evidence and research gaps on intervention effects for a broad topic area. In doing so, EGMs can help inform strategic use of resources for new research by identifying ‘absolute gaps’ where there are few or no available impact evaluations, and ‘synthesis gaps’ where there are clusters of impact evaluations but no available high-quality systematic reviews. By making existing studies easily available to researchers and describing the broad characteristics of the evidence base, the EGM can also inform the methods and design of future studies. Evidence gap maps can also facilitate the use of evidence to inform decisions by providing collections of systematic reviews that are critically appraised and ready for use by various decision makers. The methods used to develop EGMs are informed by systematic approaches to evidence synthesis and review and include key characteristics such as explicit inclusion/ exclusion criteria and a systematic and transparent approach to study identification, data extraction and analysis. We describe our methods in more detail below.

Criteria for including and excluding studies

The process starts with developing the scope for the EGM by creating an evidence-informed framework that serves for the identification of the interventions and outcomes that are relevant for the domains under study.

Table 1 and Table 2 define the final intervention and outcome inclusion criteria. To be included, studies have to assess the effect of at least one of these interventions on one of the outcomes.

On the supply-side of performance management, the study will exclude interventions focusing on the delivery of vector-control and epidemiological surveillance activities; and services delivered in hospitals. Demand-side interventions will be excluded from the study (financial interventions such as conditional cash transfers; communication for behavior-change interventions; and social marketing interventions, among others).

Types of included study designs

We will include studies designed to assess the effects of interventions, and systematic reviews of such studies, as defined below:

Efficacy trials and systematic reviews of efficacy trials will be excluded. Broadly, efficacy trials determine whether an intervention produces the expected result under ideal/controlled circumstances, whereas effectiveness trials measure the degree of beneficial effect under “real world” settings. However, the distinction between these two types of studies is generally considered as a continuum rather than a clear dichotomy and in practice it can be difficult to clearly categorize a trial as either effectiveness or efficacy¹¹¹. We will therefore draw on the criteria developed by Snilstveit¹¹² et al. to aid the identification of efficacy trials for exclusion from the EGM. The adapted criteria are as follows:

Other inclusion and exclusion criteria

In addition, studies have to be conducted in a low- or middle-income country based on a search template from Cochrane EPOC Group with LMICs defined by World Bank. Moreover, studies have to be published in English in the year 2000 and after. We will include studies regardless of status (ongoing or completed) and type of publication, published (e.g. journal article, book chapter) and unpublished (e.g. report or working paper). Studies published before 2000 will be excluded. Finally, we will also exclude studies exclusively focused on high-income countries, or systematic reviews focusing on a single country.

Search strategy

We have developed a systematic search strategy in collaboration with two information specialists. We developed a detailed search string for searching bibliographic databases and relevant portals. A sample strategy was developed for Medline, (see Supplementary File 1) and covers a detailed explanation of the search terms used based on an initial set of English search terms relevant to the main concepts of our inclusion criteria, including intervention, study design and population (low- and middle-income countries). These were combined using appropriate Boolean operators. All search strategies used in the study will be published along with study results.

We will identify potential studies using three strategies as listed below:

Procedures for screening and data extraction

Following the search, we will import all records into EPPI reviewer 4. Following the removal of duplicates, we will combine manual screening and text mining to assess studies for inclusion at the title and abstract stage. To ensure consistent application of screening criteria for all screeners, we plan to assess the same random sample of 100 abstracts. Any discrepancies will be discussed within the team and inclusion criteria will be clarified if necessary. Following this initial set of 100, we will move to single screening with “safety approach”, whereby there is an option to mark unclear studies for review by a second reviewer¹¹³.

Once all screeners have been trained, we will screen a random sample of 500 abstracts to train EPPI reviewer’s priority screening function. The priority screening function can be used at the title/abstract screening stage to prioritize the items most likely to be ‘includes’ based on previously included documents. Using priority screening in this way allows for the identification of includable records at an earlier stage in the review process so that work can begin earlier on full-text screening and data extraction.

Depending on the number of search hits, we may also make use of EPPI reviewer’s auto-exclude function to auto-exclude studies from the search that have less than a ten per cent probability score of inclusion. This function classifies un-screened studies into ten percent intervals of probability of inclusion, based on keywords included in previously included and excluded studies.

Because of time and resource constraints we will not conduct independent double screening of all studies that will be considered at full text. To minimize bias and human error we will however double screen a sample of studies at the beginning to ensure inter-rater reliability between screeners. In addition, we will take a “safety first” approach as at the title and abstract stage, whereby any studies where the first screener is uncertain about inclusion/exclusion will be screened by a second person¹¹³. All studies identified for inclusion will be effectively screened by a second/third person during data extraction.

We will use a standardized data extraction form in Microsoft Excel to systematically extract data from all included studies, including bibliographic details, intervention type and description, outcome type and definition, study design, and geographical location. We will also assess the extent to which studies incorporate equity, and extract information about if and how studies consider vulnerable and marginalized groups. To do so we will draw on the PROGRESS-Plus framework¹¹⁴ which outlines dimensions that may give rise to inequity in either access to services, or final health outcomes. In particular, we will consider the following dimensions: Place of residence (location of household such as distance from health facility, or rural/urban), ethnicity, culture and language, gender, socioeconomic status and other vulnerable groups (open category to be used iteratively to record details of any vulnerable groups identified during coding).

For each study we will assess if they consider equity for any of these dimensions, and if so how, giving the following options: 1) Contains equity-sensitive analytical frameworks/theory of change; 2) Uses equity-sensitive research questions; 3) Follows equity-sensitive methodologies (sub-group analysis); 4) Contains equity-sensitive methodologies: additional study components to assess how and why (including mixed and qualitative methods); 5) Uses any other methodology that is equity sensitive that is not covered by the other options; 6) Uses equity-informed research processes (who are the respondents, who collects data, when, where etc.); 7) Addresses interventions targeting specific vulnerable groups - Looks at the impact of an intervention that targets specific population groups; and/or, 8) Measures effects on an inequality outcome .

For multi-arm trials testing different interventions, each comparison arm will be treated as an individual study for the coding of interventions. We will report both number of studies and number of papers identified. In addition, we will report on the number of linked studies. Studies will be considered linked if there are multiple papers by the same study team on the same impact evaluation reporting different outcomes or different follow-up periods. If they report the same information, the study will be excluded as a duplicate.

A full list of descriptive data to be extracted is included in the coding tools in Supplementary File 2. We will begin the coding process with a training with the whole research team. This training will involve coding one included systematic review and one included impact evaluation as a group to familiarise all coders with the coding tools. The entire research team will then independently pilot the coding tool on the same small subset of studies to ensure consistency in coding and to resolve any issues or ambiguities. We will start this process with two systematic reviews and two impact evaluations, and test an additional small subset if issues or discrepanies remain in the application of the tool. Data extraction will then be completed by a single coder. To minimize bias and human error we will however review the data extraction of a sample of studies.

We will follow the adapted SURE checklist, available in the 3ie systematic review database protocol for appraisal of systematic review. This checklist is based on the SURE Collaboration checklist for deciding how much confidence to place in the findings of a systematic review, giving systematic reviews a rating of high, medium or low confidence (Supplementary File 3). We will produce a user friendly summary for all reviews of high confidence. All systematic reviews will be appraised by at least two people, and shared with study authors before publication.

Statistical analysis plan and EGM visualisation

Upon completion of the data collection, findings will be initially presented in a visual interactive format using 3ie’s custom built platform and accompanied by a detailed report.

The visual, online EGM will be build by, first, transforming the intervention-outcome framework into a matrix followed by the uploading of cvs files with data for all the studies included in the map (intervention, outcome, study type, impact evaluation study design, systematic review confidence level, geographical location and equity focus). This data will automatically populate the framework matrix to indicate the relative availability of evidence. This will be used to identify and describe absolute evidence gaps (no studies) and synthesis gaps (sizeable impact evaluation literature, but no high confidence SR). In addition the map will contain descriptions of the characteristics of the evidence base by using graphs, figures and descriptive statistics.