Introduction

Gates Open Res

Gates Open Research

2572-4754

F1000 Research Limited

London, UK

10.12688/gatesopenres.12851.1

Research Article

Articles

Leveraging existing program data for routine efficiency measurement in Zambia

[version 1; peer review: 1 not approved]

Homan

Rick

Conceptualization Data Curation Formal Analysis Funding Acquisition Methodology Project Administration Supervision Visualization Writing – Review & Editing https://orcid.org/0000-0001-9827-9238 a 1 Bratt

John

Conceptualization Formal Analysis Funding Acquisition Methodology Project Administration Supervision Writing – Original Draft Preparation Writing – Review & Editing https://orcid.org/0000-0003-3619-7442 1 Marchand

Gregory

Data Curation Methodology Project Administration Software Supervision Visualization Writing – Review & Editing https://orcid.org/0000-0003-2326-1898 2 Kansembe

Henry

Supervision Validation Visualization Writing – Review & Editing https://orcid.org/0000-0002-3730-2905 3 1Health Services Research, FHI 360, Durham, North Carolina, 27701, USA 2Avencion Ltd., Lusaka, Zambia 3Planning Department, Ministry of Health, Government of the Republic of Zambia, Lusaka, Zambia

a rhoman@fhi360.org

No competing interests were disclosed.

31 8 2018

2018

20 8 2018

2018

This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Background: As donor contributions for HIV/AIDS stagnate globally, national governments must seek ways to improve use of existing resources through interventions to drive efficiency at the facility level. But program managers lack routinely available information on unit expenditures at points of care, and higher-level planners are unable to assess how resources are used throughout the health system. Thus, managers cannot measure current levels of technical efficiency, and are unable to evaluate effectiveness of interventions to increase technical efficiency.

Methods: FHI 360 developed the Routine Efficiency Monitoring System (REMS), a relational database that leverages existing budget, expenditure and output data to produce quarterly site-level estimates of unit expenditure per service. Along with the Government of the Republic of Zambia (GRZ) and implementation partner Avencion, we configured REMS to measure technical efficiency of Ministry of Health resources used to deliver HIV/AIDS services in 326 facilities in 17 high-priority districts in Copperbelt and Central Provinces. REMS allocation algorithms were developed through facility assessments and key informant interviews with MoH staff. Existing IFMIS and DHIS-2 data streams provide recurring flows of expenditure and output data needed to estimate service-specific unit expenditures. Trained users access REMS output through user-friendly dashboards delivered through a web-based application.

Results: District health management teams are using REMS to identify “outlier” facilities to test performance improvement interventions. Provincial and national planners are using REMS to seek savings and ensure that resources are directed to geographic and programmatic areas with highest need. REMS can support reimbursement for social health insurance and provide time-series data on facility-level costs for modeling purposes.

Conclusions: REMS gives managers and planners substantially-improved data on how programs transform resources into services. The GRZ is seeking funding to expand REMS nationally, covering all major disease areas. Improved technical efficiency supports the goal of a sustainable HIV/AIDS response.

efficiency routine site-level Zambia dashboards unit expenditure

Gates Foundation

OPP1117063

This work was supported by the Gates Foundation [OPP1117063].

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Introduction

Global demand for HIV/AIDS services is increasing faster than available resources, resulting in a substantial funding gap. In 2016, a total of US$19.1 billion was spent by all contributors (international and domestic) to the global HIV/AIDS response in low- and middle-income countries (LMICs) ¹. The UNAIDS “Fast Track” strategy has accelerated the timeline for ending the public health threat from HIV/AIDS, but intensification of effort comes at a high cost: by the year 2020, the global HIV response is projected to require US$26.2 billion in annual resources, a 37% increase over current spending levels ². This gap could grow even larger, due to increases in treatment costs (more people on treatment and more expensive treatment regimens), demand for emergent interventions such as medical male circumcision and pre-exposure prophylaxis, and flattening trends in international and domestic HIV/AIDS financing ³.

Closing the HIV/AIDS funding gap will require a multifaceted approach, including substantial new resources from international and domestic sources, as well as stronger efforts by national health authorities to increase efficiency of resource use in their respective countries ⁴. But international funding commitments leveled out after the 2009 global financial crisis, and in 2016 worldwide financial pledges declined by 7% over the previous year ¹. More optimistically, recent analyses of HIV/AIDS service delivery programs suggest substantial scope for cost savings through improvements in technical efficiency. Zeng et al. ⁵ noted that existing literature on unit costs of HIV/AIDS interventions shows sizeable variation across programs and countries, much of which is unexplained and likely related to poor governance and weak human resource capacity. The authors conducted an econometric analysis of HIV/AIDS program efficiency in 68 countries, and concluded that a typical country program could double its output if it used inputs more efficiently. Remme et al. ⁶ analyzed several different policy levers for increasing domestic support for HIV/AIDS programs in 14 sub-Saharan countries, and found potential efficiency gains equivalent to 29 percent of current spending. Di Giorgio et al. ⁷ examined ART provision in Zambia, Kenya and Uganda and determined that if health facilities boosted their efficiency levels to an 80 percent level, utilization could increase by 33% - 62%.

To realize these potential efficiency gains, managers at all levels of LMIC health systems need information on current technical efficiency in facilities under their supervision. Much of the existing evidence on technical efficiency at the facility level has come from large multi-country cost studies ^{8–
10} or “one-off” studies conducted in single countries ^{11–
13}. These studies, typically cross-sectional, have generated useful information on differences in costs across sampled services and facilities within LMIC health systems, across countries and regions, and trends in service costs over time. In 2017 the Global Health Cost Consortium (GHCC) created a “Unit Cost Repository for TB and HIV Prevention, Treatment, and Care Interventions,” (see GHCC Study Repository) which seeks to consolidate unit cost information from HIV/AIDS and TB-related interventions in a searchable database. But few cost studies are nationally-representative, which diminishes their utility in guiding facilities not sampled toward higher technical efficiency. Also, release of study results often lags data collection by one or two years, making it likely that reported findings correspond to a program that has changed substantially in the interim. Finally, these studies themselves can be costly and intrusive to service delivery.

In upper-income countries (UICs), health system managers increasingly are using data-driven cost accounting systems to track and improve performance across networks of similar service outlets. Cost accounting can be differentiated from financial accounting primarily by the intended audience. Cost accounting information is used by decision makers inside the organization to improve cost control and efficiency, while financial accounting information is geared toward external groups such as shareholders, lenders and regulators. Cost accounting systems encompass multiple interoperable databases, and enable managers to monitor patterns of resource use, identify cost drivers for cost control, measure total costs of care at various levels of the health system, integrate information on financial inputs and clinical outputs and outcomes, and improve data utilization through use of executive dashboards (see Becker’s Hospital CFO report on advanced cost accounting). But such systems are expensive to install and maintain, and have not yet been adopted within the LMIC context.

Managers of health systems in LMICs clearly need different data and tools to drive efficiency and effectiveness within facilities under their supervision. The Government of the Republic of Zambia (GRZ) with technical support from FHI 360 and Avencion Ltd. is leading an effort to develop an automated Routine Efficiency Measurement System (REMS) that uses existing budget, expenditure, and health output data to produce near-real-time estimates of expenditure per unit of service at the individual health facility. REMS incorporates two main innovations: (1) a computational mechanism called a “resource allocation matrix” (RAM) that enables a standard expenditure report to be translated into a set of allocation weights to the program’s main “service lines”; and (2) a relational database accessed through a web-based portal that can accept RAMs from multiple institutional entities, aggregating allocated expenditures into totals per service line that can then be divided by output to generate estimates of expenditure per unit of service. Output from REMS can be used by health system managers to monitor expenditure patterns over time, guide resource allocation decisions, and identify best practices for lowering unit costs. The REMS build in Zambia covers 326 facilities spread across 17 districts in Central and Copperbelt provinces.

Methods Creating a culture of collaboration and government ownership

From the outset, the REMS technical assistance team within FHI 360 and Avencion recognized the need to form a project implementation group led by GRZ officials. Ensuring ownership by the Ministry of Health (MoH), Ministry of Finance (MoF) and Ministry of Community Development and Social Welfare (MCDSW) was critical to the project’s immediate success and long-run sustainability. Creating a culture where government officials were empowered to provide active and engaged leadership in project planning was foundational. This was accomplished through a concerted effort to engage ministerial senior leadership and, in parallel, by convening regular consultative meetings with integrated teams of government technical experts. Our approach increased investment of time by government senior leadership and subject matter experts. This integrated group of government officials evolved into a REMS core team that championed and guided the project by providing continuous feedback on direction. The core team members were identified through an iterative process of regularly engaging with key leadership within the relevant government ministries to discuss REMS functional requirements, overall implementation, validation and sustainability of the REMS system. REMS core team members were chosen based on their level of commitment, overall interest, and level of influence within the government hierarchy.

At the constitution of the MoH REMs core team, a REMS implementation workshop was held where the core group established team principles, values, and capacity developmental needs which were the basis for the implementation strategy and work plan activities. To establish these principals, values, and capacity developmental needs, the workshop agenda incorporated a team building activity. The results from the core team exercise are shown in Table 1.

Table 1. Core group team building result: values, principles, attributes, strengths and capacity developmental needs.

Core Values and Principles

Family (spouses and children)

Make sure systems and resources are efficiently utilized

Helping poor and providing access to health services and other human needs

Building Africa’s success story; including our culture and community

Preaching and religious beliefs

Equity and truth

Inclusion and leveraging the African Diaspora

Arts and engineering

Implementation Team’s Desired Attributes

Have a winning attitude and pragmatic (making ideas happen)

High performance with a clear focus on targets, objectives and getting things done

Truthful, knowledgeable and with a common understanding

Interoperability and sustainability

Not afraid of change, open minded, innovative and with ability to adapt

Team work/collaboration focused on optimizing talent and expertise

Every team members carries their own weight; accountability and mutual contribution

Supportive, jovial and uplifting

Implementation Team Strengths

Problem solving and analysis

Organizing and teaching

Operating on patients (orthopedic surgeon)

Creativity, innovating and finding different ways of doing something

Following up, speak language of equity, analysis

Implementation Team Areas Requiring Additional Capacity

Technical writing

Coding / programming (software)

Conflict resolution management

Summarizing ideas and concepts without losing value

Evidence based decision making

Dissemination of REMS

The REMS project was fully integrated into the sector wide approach (SWAp) mechanism, which is the health sector coordination platform encompassing various layers of technical and policy activities including technical sub-committees, technical working groups (TWGs), policy review meetings and Annual Consultative Meetings (ACMs). TWGs in the Zambian ministerial context report their activities at periodic Policy Meetings composed of senior leadership and Cooperating Partners. By fully integrating the REMS project within the SWAp framework, the core team could regularly disseminate information to sector partners, create linkages with other complementary initiatives in the health ecosystem, build effective working relationships with partners, share user adoption cases and foster continuous support within government. Led by the ministerial core team, the REMS project was continuously represented in the existing SWAp meetings inclusive of Monitoring & Evaluation (M&E) and Healthcare Finance TWG meetings. As a matter of standard practice, REMS presentations at SWAp meetings were always delivered by a Zambian government health official who could speak to the relevance of REMS to supporting health service delivery.

The M&E technical working group in early 2017 met with many health partners at MoH head office where REMS was presented. The feedback from health partners was that a system that can link health outputs and the financial resources mapped to service delivery was long overdue. Since REMS was receiving external data from disparate systems, it also was providing a periodic quality assurance layer for both the financial and health output data. REMS was further presented in the December 2017 Policy Meeting by the MoH’s Department of Policy & Planning and Healthcare Finance teams, culminating in an official endorsement of REMS by the Permanent Secretary of Health as the GRZ’s preferred efficiency measurement tool in the health sector. Thus, by working through existing ministerial structures, REMS became visible at the senior management level and, in parallel, was continuously validated by health sector technical experts, resulting in official acceptance and buy-in.

Technical description of Zambia REMs build

A key feature and advantage of REMS is that it leverages existing financial and output data, reducing the amount of “new” data needed to generate unit expenditure estimates. Governments routinely produce information on expenditures and outputs, but bringing these two data streams together electronically to produce recurring unit expenditure estimates at the facility level is an innovation. The Zambian MoH uses the Integrated Financial Management Information System (IFMIS) ¹⁴ to track financial expenditures, and the District Health Information System (DHIS 2) to report on service outputs at the facility level. The Zambia REMS relational database creates an electronic linkage between IFMIS and DHIS2 data in two steps: first by stepping down quarterly IFMIS expenditures (which are often reported at the national, provincial, or district level) to the facility level and allocating expenditures at the facility level to specific services, and second by dividing these allocated expenditures by the number of units of output for that service for the same calendar quarter.

Figure 2 provides a schematic of the logic flow used in the REMS database. In the upper left, the existing IFMIS data system is represented by a green cylinder. Additional data sources required by REMS are the allocation weights shown as the green cylinder in the upper right of Figure 2 and the Facility Resource Allocation Matrices (RAMS) shown in the lower right. These data exist as look up tables within the REMS database, and were assembled as follows: Allocation weights are based on input from key MoH informants at national, provincial, and district level and existing planning documents. Allocation weights are applied to the expenditures reported by IFMIS account number. IFMIS account numbers comprise a unique combination of head/department/unit/programme (and in some cases)/activity numbers. For each IFMIS account controlled at each level, informants were asked to identify what portion of the expenditures in that account were being used to support HIV/AIDS services, and of those expenditures, what portion was flowing down to the lower level(s) vs. remaining at the current account level. Portions of expense accounts that were flagged as HIV/AIDS-related but not flowing down further were classified as “above-facility” expenses. Whenever possible, existing planning documents were referenced to help informants agree on proper allocations. Once accounts were allocated to district level (either those expense accounts already reported at the district level or those flowing from national or provincial level accounts to that district), district level informants were asked to allocate these accounts to specific facilities within their district or keep portions of them at the district level as an above-facility expense. Again, existing planning documents which are used in the annual bottom-up budgeting process were a useful resource to help guide allocation decisions.

The facility RAMS (shown in lower right of Figure 2) are used to distribute HIV-related expenditures incurred at a service delivery point (facility) to specific HIV services delivered in that location (facility or community) and also to classify into expense categories (i.e., labor, drugs, other supplies, equipment, etc.). Each RAM comprises a matrix of weights corresponding to unique combinations of resources and the services they support within the facility. RAM weights were derived from a detailed facility assessment in which trained data collectors interviewed clinic staff to determine inputs used to provide specific HIV services within the facility. Data collectors obtained inventories of equipment and supplies used to support service delivery as well as records of HIV-related drugs dispensed in the prior month. Standard unit costs were applied to each resource identified and these data were summarized in an annual prototypical operational budget format for each facility. Each line item in the operational budget was then allocated across the service(s) for which it was used, considering times and locations when services are available, as well as the relative volume of services when resources such as staff or equipment are shared. These details were then summarized in the RAM weights.

By dividing the allocated expenditures for a specific service at a facility by the volume of service provided during the same time period as reported in the DHIS database (not shown in Figure 2) we are able to estimate the unit cost of service provision at the facility by expense category (shown as blue parallelogram in lower right of Figure 2). This unit cost estimate serves as a performance metric that can be compared across facilities and monitored over time to identify promising service delivery practices for replication in other facilities.

The Zambia REMS system is programmed in Microsoft SQL version 13.0 and the annotated source code is available through Zenodo ¹⁵. REMS uses C# for the user interface in which users select parameters of interest (time period, organization, geographic level, service(s) of interest, type of resource(s), facility(s) and total or unit cost. The results of these selections are displayed graphically using HighChart to generate visualizations and REMS allows the user to save the most relevant performance metrics to an individual dashboard (see animation of REM dashboard from REM site).

In the screen shot below ( Figure 1), user-generated results show the unit cost of Elimination of Mother to Child Transmission (EMTCT) services for 6 facilities for the period Jan-Mar 2016. In the left panel, a ten-fold variation is apparent in the unit costs of an EMTCT visit (range: <125 Kwacha ( ^~$13) in Kalwelwe Health Centre to 1,337 Kwacha in Kohima Camp Hospital ( ^~$134)). In the right panel, the volume of services (patient visits for EMTCT) is shown for the same time period and facilities. While we would expect that facilities with larger service volumes would have lower unit costs (as they can spread fixed costs across more visits) some noticeable exceptions exist: Kawama Urban Health Centre reports high volume and high unit costs, while both Kalwelwe Health Centre and Kasanda urban Health Centre exhibit low volumes and low unit costs. REMS users at the District Medical Office are trained to analyze comparative unit expenditures across a range of facilities in their district, and also to track trends in unit expenditure over time; thus, both of these exceptions would serve as prompts for further investigation by the District Health Management team.

Figure 1. Variability in unit cost and output for EMTCT Services, selected facilities in Central Province. Figure 2. REMS system process flow.

Users at a Provincial Management Office (PMO) have broader permissions to monitor performance and resource efficiency in select priority districts or facilities. At the provincial and national level, users can examine whether overall expenditures are being directed to services or geographic areas where needs are greatest, and can monitor trends in service-level versus above-facility expenditure. By leveraging existing data systems and flows, REMS capitalizes on prior investments in health management information systems and increases the value of the data for decision making.

Discussion

In the current environment of scarce resources and growing demand for health services, there exists an urgent need to ensure more efficient use of existing resources. An ongoing problem in LMIC health systems is lack of data to measure and monitor economic performance at the service delivery level. Information on technical efficiency from cost studies is not sufficient for day-to-day program management that seeks to drive efficiency and effectiveness in front-line health facilities. As a business intelligence platform, REMS delivers routine information on unit expenditure by type of service, type of expense, time period and funding source, covering all facilities within a health system. We envision several use cases for LMIC managers and decision makers using REMS information, including the following:

Performance improvement – REMS is enabling Zambian officials at district and provincial levels to carry out routine monitoring of technical efficiency. For the first time, differences in high and low-performing facilities and districts are visible, allowing managers to develop and monitor initiatives that translate into improved efficiency outcomes. Focusing on “outliers” – i.e., facilities or districts with high or low expenditure per unit of output relative to a standard – managers can investigate potential reasons for the differences. For example, differences in program output may be driving unit expenditure variation in similar facilities. If so, what are the underlying reasons for differences in output and how might these be addressed? Another driver of unit expenditure could be staffing levels that are not aligned with patterns of demand in a facility, opening the possibility of reassigning excess staff to busier facilities. Prior to REMS these differences were obscured, making it impossible to identify efficiency differences, or to measure the impact of changes designed to improve efficiency.

Resource allocation at provincial and national levels - Planners in higher levels of the health system can use REMS expenditure roll-ups by district and province to analyze resource flows, ensuring that funding and services are directed to geographic areas where disease burden is greatest.

Assessment of HIV/AIDS financing and sustainability – National-level health officials now have information on total expenditure per unit of service, and when the system is fully operational, how the expenditure burden is distributed across different funding entities such as the Global Fund and PEPFAR. They also will be able to document facility and above-facility-level expenditure, and look at trends in the GRZ share of the HIV/AIDS financing burden.

Support implementation of social health insurance – the GRZ is assessing feasibility of a Social Health Insurance Scheme, but lacks a source of routine information on expenditure per unit of service. REMs generates time-series data on unit expenditures that will enable calculation of benchmark unit expenditures that can be used to set reimbursement rates. REMs data also may help planners decide on the design of the system (i.e., claims versus capitation), and could assist in measuring the administrative burden associated with claims.

Recent experience with REMS users in Copperbelt and Central provinces confirm that REMS data are opening doors to new insights into the economics of service delivery. Users are working together to explore some of the “whys” behind indicators in the REMs dashboard. For example, at User Conferences in both provinces in late 2017, mixed groups comprising MoH planners and accountants from District and Provincial levels were tasked with building REMS charts based on specific scenarios. Facilitator-led discussions assisted users to analyze differences in unit expenditure across service delivery points and over time, focusing on potential causes of variation displayed in the charts. User familiarity with health facilities enabled informed discussions about such factors as differential staffing patterns, procurement of vehicles and drugs, and variability in catchment population leading to disparate levels of output. In certain cases, users could cite no obvious reasons for differences in unit expenditures, highlighting a need for further investigation at the facilities themselves ¹⁶.

In addition to these direct applications of REMS data, we foresee additional indirect benefits that contribute to health system strengthening in various ways. First, existence of routine comparative data on efficiency will engage local staff in efficiency improvement, incentivizing them toward taking a more active role in interpreting and acting on their own data. REMS data also enable supervisory staff to prioritize their support visits to facilities where the need is greatest, and to evaluate whether their support is having the desired effect over time. Second, any sudden or unexpected changes in REMS unit expenditures will trigger scrutiny, and the quality of data streams feeding REMS will be considered as a possible cause of any changes. As an example, while DHIS 2 data are collected at facility level, they are subsequently rolled-up to district and provincial levels for ordinary reporting purposes. If DHIS 2 data are keyed incorrectly or are missing at the facility level, the roll-up may obscure these errors. But REMS uses facility-level DHIS2 data as denominators for unit expenditure estimates, and so data entry errors (or omissions) will be immediately obvious in the district-level REMS dashboard screen that compares unit expenditures across facilities. Similarly, any data quality issues in the IFMIS also should be reflected in the REMs dashboard. Thus, increasing reliance on REMS data will focus greater attention on the quality of data that feed the system. Third, access to detailed budget information at the facility level is uncommon in Zambia and most other LMICs. Detailed facility budgets produced by the REMS facility assessment tool will provide local managers with additional information to improve program implementation and increase transparency.

In the current Zambia REMS build, health planners and managers are using the system to monitor efficiency of HIV/AIDS service delivery. REMS was designed purposely to be adaptable for use in other areas such as malaria, child nutrition, and immunization, as well as other global development sectors outside of health, such as education, nutrition and economic livelihoods. REMs requires a set of conditions to be in place before the system can be implemented. Data requirements include budgets at the service delivery level and above; an electronic financial management system like IFMIS; and use of DHIS-2 or a similar electronic application for tabulating system outputs. These conditions already are in place in many LMICS and across multiple development sectors, creating multiple opportunities to bring routine efficiency measurement and monitoring to other sectors and countries.

In summary, the introduction of REMS in two provinces of Zambia addresses the problem of insufficient information on economic performance of healthcare delivery at the site level. REMS produces near-real-time data on expenditure per unit of service, emulating business intelligence typically produced by enterprise resource planning (ERP) software. In Copperbelt and Central provinces, managers now can compare current unit expenditures across facilities and districts, highlighting those sites that are producing services at high or low levels of efficiency. When corrective actions are taken at the level of the service delivery point, the impacts of these actions on unit expenditures costs can be monitored going forward. For the first time in a LMIC, routine data are available to support economic changes to enable the health system to serve more HIV/AIDS clients with the same resources, reducing the fiscal gap currently facing many countries struggling to control HIV/AIDS epidemics. REMS also can be used to monitor the impact on efficiency of innovations such as task-shifting, which a recent literature review of HIV/AIDS and TB programs suggests can lead to efficiency improvements and cost savings ¹⁷. In addition to the facility-level performance improvements mentioned above, REMS also can be used to estimate above-facility expenditures, which, if managed effectively, may create opportunities to expand service availability within the current resource envelope.

Data availability

The source databases used in the development of REMS include DHIS2, IFMIS, and results of the facility assessments conducted for this project (see Table 2). DHIS2 is a system being used to capture statistical data on health activities throughout the country. IFMIS provides budget and expenditure data for government ministries. The facility assessments give details of inputs, service delivery patterns and resource use for each facility. In the scale-up phase of REMS, we expect that facility assessments will become routine annual exercises to document inputs and resource use at MoH facilities, and to form the basis for activity budgets. Below is a summary table of the various databases from which REMS draws its data inputs:

Table 2. List of data repositories used in this study.

Repository	Dataset	URL
DHIS2	Health Outputs	www.zambiahmis.org
IFMIS	Financial Inputs	Private closed network
REMS Database	Resource Allocations	Private closed network

Access to private closed networks

The full financial / general ledger data extract from the Government of the Republic of Zambia Ministry of Finance IFMIS (Integrated Financial Management Information System) is available to authorized users from the Zambian government and approved partners. Access to back-end system data in IFMIS and the data extract is accessible with written approval from the Ministry of Health Permanent Secretary. For full data availability protocol please contact andrew.kashoka@moh.gov.zm.

The REMS database is hosted on a server at the Ministry of Health in Lusaka, Zambia. User access is limited to approved users from the Ministry of Health, other relevant government agencies and approved partners. Users with full-privileges can access back-end system data and can extract excel and CVS files of resource allocations and expenditures per facility / health service. For full data availability protocol please contact andrew.kashoka@moh.gov.zm.

Software availability

The Zambia REMS annotated source code is available from GitHub: https://github.com/rhoman88/REMS/tree/v1.0

Archived source code at time of publication: https://doi.org/10.5281/zenodo.1341782 ¹⁵

License: Apache License 2.0.

Acknowledgments

We would like to acknowledge the following colleagues for their contributions to this project: Stanley Banda, Strategic Information Officer, Ministry of Health; Chrispin Champa, Health Finance Analyst, Avencion Ltd; and DeAndre Umakonye, Project Manager, Avencion Ltd. We also would like to thank the hundreds of staff of the Zambia Ministry of Health at all levels (national, provincial, district and facility) who assisted us with functional and technical design of REMS, data collection and development of allocation rules.

Joint United Nations Programme on HIV/AIDS (UNAIDS) and The Henry J Kaiser Family Foundation: Donor Government Funding for HIV in Low- and Middle-Income Countries in 2016. Menlo Park and Washington DC;2017. Reference Source

Joint United Nations Programme on HIV/AIDS (UNAIDS): Fast-track update on investments needed in the AIDS response. Geneva: UNAIDS;2016. Reference Source

Piot

Abdool Karim

Hecht

: Defeating AIDS--advancing global health. Lancet. 2015;386(9989):171–218. 26117719

10.1016/S0140-6736(15)60658-4

Vassall

Remme

Watts

: Financing essential HIV services: a new economic agenda. PLoS Med. 2013;10(12):e1001567. 24358028

10.1371/journal.pmed.1001567

3866083

Zeng

Shepard

Chilingerian

: How much can we gain from improved efficiency? An examination of performance of national HIV/AIDS programs and its determinants in low- and middle-income countries. BMC Health Serv Res. 2012;12:74. 22443135

10.1186/1472-6963-12-74

3353196

Remme

Siapka

Sterck

: Financing the HIV response in sub-Saharan Africa from domestic sources: Moving beyond a normative approach. Soc Sci Med. 2016;169:66–76. 27693973

10.1016/j.socscimed.2016.09.027

Di Giorgio

Moses

Fullman

: The potential to expand antiretroviral therapy by improving health facility efficiency: evidence from Kenya, Uganda, and Zambia. BMC Med. 2016;14(1):108. 27439621

10.1186/s12916-016-0653-z

4952151

Marseille

Dandona

Marshall

: HIV prevention costs and program scale: data from the PANCEA project in five low and middle-income countries. BMC Health Serv Res. 2007;7:108. 17626616

10.1186/1472-6963-7-108

1936993

Galárraga

Wamai

Sosa-Rubí

: HIV prevention costs and their predictors: evidence from the ORPHEA Project in Kenya. Health Policy Plan. 2017;32(10):1407–1416. 29029086

10.1093/heapol/czx121

5886164

Tagar

Sundaram

Condliffe

: Multi-country analysis of treatment costs for HIV/AIDS (MATCH): facility-level ART unit cost analysis in Ethiopia, Malawi, Rwanda, South Africa and Zambia. PLoS One. 2014;9(11):e108304. 25389777

10.1371/journal.pone.0108304

4229087

Obure

Vassall

Michaels

: Optimising the cost and delivery of HIV counselling and testing services in Kenya and Swaziland. Sex Transm Infect. 2012;88(7):498–503. 22859498

10.1136/sextrans-2012-050544

3595498

Aliyu

Chuku

Kola-Jebutu

: What is the cost of providing outpatient HIV counseling and testing and antiretroviral therapy services in selected public health facilities in Nigeria? J Acquir Immune Defic Syndr. 2012;61(2):221–225. 22820805

10.1097/QAI.0b013e3182683b04

Bratt

Torpey

Kabaso

: Costs of HIV/AIDS outpatient services delivered through Zambian public health facilities. Trop Med Int Health. 2011;16(1):110–118. 20958891

10.1111/j.1365-3156.2010.02640.x

Rodin-Brown

: Integrated financial management information systems: a practical guide. The Louis Berger Group, Inc. and Development Alternatives, Inc. under the Fiscal Reform and Economic Governance Task Order, GEG-I-00-04-00001-00 Task Order No. 06.2008. Reference Source

rhoman88: rhoman88/REMS: REMS Zambia (Version v1.0). Zenodo. 2018. http://www.doi.org/10.5281/zenodo.1340656

Chrispin

: Personal communication.2018.

Seidman

Atun

: Does task shifting yield cost savings and improve efficiency for health systems? A systematic review of evidence from low-income and middle-income countries. Hum Resour Health. 2017;15(1):29. 28407810

10.1186/s12960-017-0200-9

5390445

10.21956/gatesopenres.13932.r26632

Reviewer response for version 1

Bertozzi

Stefano Michele

1 Referee 1School of Public Health, University of California, Berkeley, Berkeley, CA, USA

Competing interests: No competing interests were disclosed.

4 10 2018

2018

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

recommendation

reject

Overall, the paper is well-written and highlights a very important issue (the need for better data collecting and aggregation) that is worth addressing. The paper is a description of the development and implementation by FHI 360 of a version of a health management information system they call REMS, or the Routine Efficiency Measurement System. The system takes the financial data from the Integrated Financial Management System and service delivery data from the DHIS2 system and uses those data to generate unit-specific efficiency estimates. The authors describe the system and its successful implementation in partnership with the MoH of Zambia. The paper addresses the process that was followed with the Government of Zambia to shape the REMS and ensure that it is responsive to the GoZ’s needs. It also describes the implementation of the system, the characteristics of the system, and some preliminary evidence of the system’s utility to its users.

Concerns:

The paper shoehorns a description of the implementation of a health management information system into a background/methods/results/conclusion format as if this were a research study addressing a research question. This is not a research paper, but rather a well written journalistic recounting of a project designed to improve the collection and analysis of efficiency in Zambia. It doesn’t have a discernible research question; it doesn’t apply discernible research methods to answering a research question.

The paper claims that:

As a business intelligence platform, REMS delivers routine information on unit expenditure by type of service, type of expense, time period and funding source, covering all facilities within a health system.

Implied in the claim is that the REMS system is delivering accurate, useful, actionable data about performance of clinics within the system. It estimates unit expenditure by type of service using Facility Resource Allocation Matrices (RAMS) to allocate facility-level costs among the different activities in facilities. The authors mention that the “allocation weights are based on input from key MoH informants” and that the Facility Resource Allocation Matrices (RAM) “weights were derived from a detailed facility assessment” with interviews but that is the extent of the description about how exactly the matrix determines unit cost. However there is no reported validation of this approach. The reader has no reason to have confidence that the estimates generated by the system are sufficiently accurate so as to be actionable. There is only the implicit suggestion that because the Ministry has started to use the system to assess efficiency therefore the data generated must be useful. While I am a big believer in the power of data that reveals heterogeneity in performance to improve performance across the system, it too much to expect blind trust in a new method to estimate efficiency. How much of the estimated heterogeneity (variation in efficiency among clinics) is real and how much is an artifact introduced by the estimation tool?

The paper implicitly makes the claim that efficiency data are very important for improving system performance (with which I agree, as a general point) and that the REMS system, in providing such data, is thus transforming health system efficiency. While this story is plausible, no evidence is presented that the interest expressed by the MoH in the REMS data are actually leading to transformational changes in health system performance. For example, the paper offers graphs of unit costs of certain types of care across different healthcare facilities showing highly heterogeneous costs per unit of service delivered. However, there is no indication of whether the large differences in cost of EMTCT are just a reflection of differences in maternal HIV prevalence – or whether there are clear differences in the efficiency of the delivery of the services provided. Thus, it is not clear from the paper that the estimation approach used will meaningfully result in better allocation of resources and improve efficiency.

The paper makes several statements that argue that the process followed by FHI 360 was critically important to the success of the implementation. One such example:

Ensuring ownership by the Ministry of Health (MoH), Ministry of Finance (MoF) and Ministry of Community Development and Social Welfare (MCDSW) was critical to the project’s immediate success and long-run sustainability.

All of these statements are plausible, but there is no evidence provided to support the assertions. Would the implementation of a similar system in another country require the same degree of complexity in the implementation process? Is there evidence that the implementation would not have been successful if it had just been implemented by the MoH and subsequently revealed to the other Ministries? etc. As such, they appear to be statements of opinion by the authors that the approach they took was the correct/efficient/ethical approach that should be replicated if another country were to consider implementing such a system. It perhaps goes without saying that it is especially important to qualify such opinions when the authors are those who implemented the approach being assessed. A scientific article would normally make such statements only after having tested and validated the assertions using some scientific method and using observers who are as unbiased as possible.

Thus, the paper in its current form is not a scientific publication but rather a well-written, well-organized report of the development and implementation of a health data system. It should not be published in the form of a scientific publication but rather as a report. Even as a report it should acknowledge (if true) the lack of rigorous validation of its estimation method and call for such validation. Such validation would not only give the Government of Zambia greater confidence in the estimates the system produces but provide much more useful information to other countries considering implementation of a similar system.

In addition, the report should recognize that comparisons of unit costs without any assessment of the sources of the unit cost differences can provide seriously misleading information to decision-makers. If they are going to share examples of revealed heterogeneity being usefully addressed, it would be very helpful to show that the system can identify the source of the cost differences and that those have led to changes that have improved system performance. I often refer to justifiable and unjustifiable heterogeneity (which depends upon the perspective of the observer). As an example, within Zambia there is more than an order of magnitude of difference in HIV prevalence from one locality to another. It stands to reason that the unit cost of HIV services (especially prevention services) would therefore also (justifiably) vary greatly from one locality to another. Revealing (true) heterogeneity among localities can therefore represent differences in efficiency of service delivery (that needs to be addressed with improved management of service delivery) or differences in return to investment in prevention (which might be addressed by reallocating prevention services to localities where the returns are greater). At a minimum the report should temper the enthusiasm of the authors regarding the utility of the raw system output – or it should reveal that the system output is already far more nuanced than the current example suggests.

I would like to acknowledge the assistance of Madhav Nekkar in preparing this review. I accept full responsibility for its content.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Are all the source data underlying the results available to ensure full reproducibility?

Partly

Is the study design appropriate and is the work technically sound?

Are the conclusions drawn adequately supported by the results?

Partly

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

I confirm that I have read this submission and believe that I have an appropriate level of expertise to state that I do not consider it to be of an acceptable scientific standard, for reasons outlined above.

Bratt

John

Competing interests: No competing interests were disclosed.

1 11 2018

Concerns:

Response from the Gates Open Research Editorial Office : The article by Homan et al was originally classified as a Research article by the Gates Open Research Editorial Office due to the presence of methodology and software source code. However, on a re-assessment of the article, and taking the comments of reviewer 1 into consideration, we would suggest that this article should be reclassified as an

Open Letter

. Open Letters discuss policies relevant to a broad research community, present guidelines or announce new initiatives. The article by Homan et al describes the development and implementation of a new initiative, thereby fitting the criteria for an Open Letter .

The paper claims that: As a business intelligence platform, REMS delivers routine information on unit expenditure by type of service, type of expense, time period and funding source, covering all facilities within a health system. Implied in the claim is that the REMS system is delivering accurate, useful, actionable data about performance of clinics within the system. It estimates unit expenditure by type of service using Facility Resource Allocation Matrices (RAMS) to allocate facility-level costs among the different activities in facilities. The authors mention that the “allocation weights are based on input from key MoH informants” and that the Facility Resource Allocation Matrices (RAM) “weights were derived from a detailed facility assessment” with interviews but that is the extent of the description about how exactly the matrix determines unit cost. However there is no reported validation of this approach. The reader has no reason to have confidence that the estimates generated by the system are sufficiently accurate so as to be actionable. There is only the implicit suggestion that because the Ministry has started to use the system to assess efficiency therefore the data generated must be useful. While I am a big believer in the power of data that reveals heterogeneity in performance to improve performance across the system, it too much to expect blind trust in a new method to estimate efficiency. How much of the estimated heterogeneity (variation in efficiency among clinics) is real and how much is an artifact introduced by the estimation tool?

Response from the Authors : There is a lack of clarity here that will be addressed in the revised manuscript. The allocation weights based upon key MoH informants are the weights used to: 1) isolate HIV-related expenditures within the IFMIS accounts and 2) allocate HIV-related expenditures from National level IFMIS accounts to the Provincial or District Level. Key MoH informants also provided information to isolate HIV -related expenditures in IFMIS accounts pertaining to Provincial or District Level, but then existing GRZ MoH planning and budgeting data were used to further allocate these accounts down to the point of care. Once expenditures are tracked to the point of care, the facility-specific RAM is used to allocate those expenses to specific services and resource categories. The facility-specific RAM is derived from facility assessments where facility staff identified the resources used to support each HIV service within the facility. This is a similar process used in micro-costing, where an ingredients approach is used to build up the cost estimates. The facility-specific RAM is constructed in two steps: first, by creating an annual operating budget for HIV services at the facility, based upon actual resources identified during the assessment and the expected volume of services; and second, by using this budget to compute the percentage of total annual resources used for specific combinations of service and resource type. For example, the RAM may tell us that 5% of total annual resources are used for personnel providing HTC services. REMS uses this information to assign 5% of each Kwacha reaching the facility (in each calendar quarter) to “personnel” for HTC services. We recognize that this approach creates some rigidity in the cost estimates. The next version of REMS will allow for making RAM weights dynamic by linking to the output data, so that if there is an upsurge in testing, the 5% allocated to personnel for HTC would be adjusted upward (along with other resources within HTC) and shares allocated to other services would be decreased.

Response from the Authors: While not shown in the graphs, there is an option to created stacked columns or sliced bars that would break down unit expenditure by the type of resource contributing to the results (i.e., staffing, supplies, equipment, etc.). This allows for a comparison of how productively the fixed resources are being utilized, as these are what largely drive efficiency gains. We think of REMS as a thermometer. It can be used to document a fever, but it does not tell the user how to treat the fever. The user is presented with this information to help target investigations of facility performance, identify outliers, perhaps identify data quality issues within the DHIS2 data, etc. If corrective action is taken, REMS, like a thermometer, can be used to assess whether the patient is responding to the treatment, and if not, to catalyze additional investigations. The goal of REMS is to put actionable information in the hands of persons who are asked to manage the health system, information that currently doesn’t exist. Asking health systems to become more efficient users of scarce resources without providing tools to monitor resource use is unlikely to result in more efficient health systems. All too often we see a standard resource package assigned to a facility based solely on its designation in the health system hierarchy, and not tied to the service volumes for which it is expected to deliver.

The paper makes several statements that argue that the process followed by FHI 360 was critically important to the success of the implementation. One such example: Ensuring ownership by the Ministry of Health (MoH), Ministry of Finance (MoF) and Ministry of Community Development and Social Welfare (MCDSW) was critical to the project’s immediate success and long-run sustainability.All of these statements are plausible, but there is no evidence provided to support the assertions. Would the implementation of a similar system in another country require the same degree of complexity in the implementation process? Is there evidence that the implementation would not have been successful if it had just been implemented by the MoH and subsequently revealed to the other Ministries? etc. As such, they appear to be statements of opinion by the authors that the approach they took was the correct/efficient/ethical approach that should be replicated if another country were to consider implementing such a system. It perhaps goes without saying that it is especially important to qualify such opinions when the authors are those who implemented the approach being assessed. A scientific article would normally make such statements only after having tested and validated the assertions using some scientific method and using observers who are as unbiased as possible.

Response from the Authors: We acknowledge the concern about evidence to support these statements and have edited the text to make clear that these are statements of opinion. In this specific example, access to the financial data is dependent upon the MoF, National and Provincial Level systems are run by the MoH and District Level systems are managed by MCDSW; with such inter-Ministerial complexity, it was critical that all three parties were on-board, engaged and in agreement about how REMS was to be built and accessed. The deployment of a user-centered design process was partly in response to previous experiences where a solution was imposed from outside with little (if any) uptake or interest in supporting the system. We have expanded our discussion of this issue in the revised manuscript .

Response from the Authors: We agree. Our award was funded to 1) develop a prototype, 2) show that the GRZ was willing to participate in the design and development of REMS in Zambia, and 3) implement the system in two Provinces. As such what we have now is essentially a functioning proof of concept for which we are actively seeking further investments to take the system to the next level (expand geographically and into additional health program areas) as well as introduce internal feedback loops within the program logic and create a user-interface to generate performance reports and query the facility assessment data. Our intent in publishing this Open Letter is to raise awareness of the work to date in hopes of securing funding for further validation and development of the system.

Response from the Authors: As mentioned above, the current system is configured to allow users to examine sources of unit cost heterogeneity across facilities and over-time. We are not claiming to have produced a perfect performance monitoring system, but rather to have laid the foundation for a more complete and robust system. To our knowledge, no country in Sub-Saharan Africa (other than Zambia) has a public-sector enterprise cost accounting tool for health services currently in place. We are hoping that REMS will enable more effective management of the health system in Zambia in a time when countries are being asked to take on a larger share of the financing of the health sector. Our co-author from the MoH has stated that he believes that a fully built REMS system could achieve a 40% efficiency gain systemwide by reallocating resources to where they can be more productively utilized. We look forward to using REMS system to validate this assertion, as REMS output is used to reallocate resources across facilities and reductions in the unit cost of care are documented over time.

I would like to acknowledge the assistance of Madhav Nekkar in preparing this review. I accept full responsibility for its content.