The study was approved by national ethics review committees in all countries: Africa Medical Research Foundation (AMREF) Africa Ethics and Review Committee (P838-2020, Aug 11, 2020) and National Commission for Science Technology and Innovation (249062, Nov 9, 2020) in Kenya; Makerere University, School of Social Sciences Research & Ethics Committee (MAKSS REC 03.20.402, July 23, 2020) and the Uganda National Council for Science and Technology (SS613ES, Nov 23, 2020) in Uganda; the National Committee on Research in the Social Sciences and Humanities (NCST/RTT/2/6, Oct 19, 2020) in Malawi; the Commission for Science and Technology (2020-100-NA-2020-024, Feb 20, 2020) and the National Institute for Medical Research (NIMR/HQ/R.8a/Vol. IX/3522, Sept 28, 2020) in Tanzania; and Oxford Policy Management Ethical Review Committee (ERC-A4009, July 7, 2020). Approvals were renewed before commencing year two and three fieldwork.

All participants provided informed consent before commencing interviews or group discussions. For Malawi, Tanzania and Uganda, the ethics committee approved use of verbal consent because the research was low-risk, with questions focused on professional rather than personal issues and relatively senior participants. In Kenya, written consent was obtained in line with ethics committee preferences. Where consent was verbal, this was recorded by the evaluator on an interview notes template.

The DHSSi evaluation was co-designed by UNICEF and OPM. OPM implemented the evaluation, under the management of the Evaluation Section of UNICEF’s regional office, with technical input provided by an Evaluation Reference Group comprising UNICEF staff, government representatives and senior subject experts. The OPM evaluation team included national researchers in each DHSSi country and a central technical team of five (including KG and GA). All team members had a background in health research and evaluation, and all were trained to masters or doctoral level. The team included male and female researchers. Data were collected and analyzed annually, within interim reports and recommendations shared over the course of three years. Annual findings and recommendations were reviewed and discussed by DHSSi teams and contributed to updated annual implementation plans.

The evaluation used a mixed-methods theory-based approach. Evaluation questions were structured using the Organization for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) criteria of relevance, coherence, efficiency, effectiveness, sustainability and impact, and corresponded to outputs, outcomes and impacts the DHSSi ToC. Within the theory-based approach, DHSSI’s contribution was assessed by examining evidence for the presence of outputs and outcomes along the ToC chain, the processes through which DHSSi contributed to outcomes, and the conditions that affected progress, as well as stakeholder perceptions regarding the significance of and role played by DHSSi.

The evaluation focused on core DHSSi components of EBP in all countries, efforts to develop an agenda and curricula for management capacity building, and district management training where this was implemented; other activities under DHSSi – such as support for district staff recruitment in Uganda or research on community governance in Tanzania – were either minimally examined or not covered by the evaluation due to restrictions on scope. The evaluation covered DHSSi implementation from January 2019 up to around July 2022.

For each evaluation round, the OPM evaluation team collected data at national and district levels, covering all DHSSi districts in Kenya, Malawi and Tanzania, and three of ten DHSSi districts in Uganda. Methods comprised document review, interviews, group discussions, observation, and analysis of secondary quantitative data.

Document review provided information on DHSSi activities and district planning practice, and included DHSSi programme documents; district government documents, including annual district health workplans; and national documents such as policy guidelines and working group minutes. Document review was conducted by the central evaluation team and country researchers.

Interview participants were selected purposively in discussion with UNICEF, based on involvement with DHSSi activities. National interviews included UNICEF staff, DHSSi implementing partners, and representatives from the MoH or other relevant ministries engaging with DHSSi. At district level, DHMT members were interviewed in three districts per country, usually a senior DHMT member (such as the District Health Officer), and a member responsible for maternal and child health. Additional interviews were conducted for some evaluation rounds in accordance with priority information needs agreed with UNICEF. For example, in year one, interviews also included national and district development partners or health NGOs, to understand coherence and stakeholder engagement in district planning, and members of the wider local government (the central district council or executive that oversees all sectors), to understand their views on relevance. In year three, additional interviews were conducted with the regional health management teams and zonal health resource centres in Tanzania and with zonal officers in Malawi, as DHSSi developed a focus on these regional/zonal structures for sustainability. At international level, interviews were conducted with UNICEF’s regional office, FPD, the BMGF (in year two) and the UNICEF headquarters (in year three) on areas such as learning and DHSSi alignment with organizational strategies. See Table 2 for the annual number and distribution of interviews. A small number of repeat interviews (2–3 a year) were conducted to provide additional information, primarily with UNICEF country offices or MoH staff who were heavily engaged in DHSSi. Interviews generally lasted 1–1.5 hours, with group discussions taking around 2.5–3 hours.

Group discussions were held with DHMT members (or planning team members in Malawi) in each district, to discuss district planning and management practice, experience of DHSSi activities, and in year three, any changes in prioritized bottlenecks.

Participants for interviews and group discussions were initially approached by email or phone, depending on their usual communication channels. A small number (around 1–2 people per year) declined to participate because they felt they had insufficient involvement with DHSSi to comment, and some participants were unavailable due to travel or other commitments; in these cases, alternative participants with similar positions were identified (for example, other DHMT members).

Interviews and group discussions followed a semi-structured topic guide, developed based on document review and evaluation questions. This guide was iteratively adapted in each country during data collection to build on emerging insights and experience with the questioning. Time was taken before each interview or group discussion to introduce the evaluation purpose and establish trust. Almost all group discussions took place in person, and interviews were remote or in-person depending on local COVID-19 guidelines and respondent preferences. In-person discussions were held in offices or quiet nearby locations, depending on participant preference. Only the participants and researchers were present. All sub-national interviews and group discussions and most national interviews were conducted by the evaluation team country researchers, with a small number of national interviews (particularly with UNICEF country offices) conducted by the central evaluation team or jointly between the central team and country researchers. Cross-country interviews were conducted by the central OPM evaluation team.

The year two and year three evaluation rounds included observation of one day of district planning or review activities, to provide additional insights into district practice and DHSSi implementation. Observation checklists were tailored to the specific activity, and detailed notes were taken during observation.

Detailed training and cross-team discussion were conducted in advance of each data collection round to ensure research tools were contextually relevant and to support consistent, high-quality data collection. This discussion included reflection on positionality and personal assumptions about likely effectiveness, to support openness in data collection.

During data collection, the OPM evaluation team conducted regular debrief discussions to support data quality and build on emerging issues. Frequent themes and new insights were discussed during these debriefs to assess whether the data were adequate and to identify any key gaps for remaining data collection.

Interviews and group discussions were audio recorded and transcribed, and notes were taken during interviews and discussions to support ongoing reflection. Transcripts were then analysed using a framework analysis approach ³⁹ , using Excel matrices based on the evaluation questions to summarize and combine data from document review, interviews, group discussions and observations. The framework matrix column structure equated to a typical qualitative analysis coding tree: columns corresponded to the overall evaluation questions and specific information needs under each question, for example, with columns on timing, reach, content and strength/sufficiency of DHSSi support under an evaluation question on efficiency. Columns were adapted based on emerging insights, and additional 'other' columns were used to record any data that did not fit a pre-identified theme. Detailed issues under each evaluation question were then identified based on the data for later writing. The framework analysis was conducted by two people in the central team and by country researchers for Kenya and Tanzania, with distribution of work varying between years.

To examine changes in district planning and management practice, the evaluation employed the Planning and Management Assessment Tool (PAMAT), a rubric developed by the evaluation team to score district planning and management practice in domains related to key practices targeted by DHSSi, including problem analysis and prioritisation, stakeholder engagement, review and implementation of plans. Scores were agreed by the evaluation team based on information from document review (particularly district annual plans and reports from district quarterly reviews), interviews and group discussions.

Quantitative methods involved analysis of routine indicator data to assess changes in bottlenecks prioritized by DHMTs each year, and assessment of change in coverage of priority health interventions based on nine core coverage indicators (such as ANC visits). This assessment of impact was designed primarily to inform results framework reporting and to serve as a baseline for future evaluation, because changes in coverage were not expected within the initial three-year DHSSi timeframe.

Transcripts were not returned to participants due to tight evaluation timeframes, as well as recognition of the complexities and potential disadvantages of this approach. Preliminary findings from each evaluation round were discussed with UNICEF, to ensure accuracy, obtain any missing information and refine recommendations, and evaluation reports were reviewed by the Evaluation Reference Group, which included the MoH in each country. In some countries, UNICEF also shared draft findings with implementing partners or DHMTs for additional feedback. A summary of annual findings was also made available to all participants each year.

The evaluation faced constraints related to data availability and feasible scope of data collection. For example, quantitative analysis faced gaps and inconsistencies in DHIS2 data and in the data on prioritized bottlenecks, and document review was hindered by difficulties with complete and timely document provision. Observation provided important insights, helping to triangulate information in documents and stakeholder interviews, but was only done for one day per country, and in some cases, activity timing meant it was not possible to observe the most relevant activities. These limitations were considered in the analysis and reporting.

Across countries, national and district governments saw work to develop subnational management skills as highly relevant. Good leadership and management were considered critical for effective health service delivery, and a gap, with staff in district management positions often lacking training in the management skills and knowledge needed to perform their roles.

In some contexts, subnational management was considered an area with limited support from other donors, so UNICEF support was particularly welcome (for example, in Malawi). Where there were several existing management capacity building initiatives, UNICEF support for harmonization was considered valuable to enhance coordination; for example, in Kenya, DHSSi conducted a partner landscaping, established a Technical Working Group to bring together MoH departments and partners, and developed a harmonized healthcare management framework and curricula.

While activities to strengthen and professionalize district management capacity were valued in all contexts, each country experienced delays, and progress varied. In Kenya, Malawi and Tanzania, following initial landscaping and stakeholder engagement, management training needs assessments were undertaken and a training curriculum was developed. District management training was then implemented in Malawi during the second year of DHSSi, and in Tanzania during the final months of year three. There was also initial work on management coaching for DHMTs in Malawi, including training of coaches and visits to some districts. In Uganda, DHSSi initially focused on district recruitment in response to staffing gaps among the district health team and frontline health workers. A need to strengthen management training was only identified during year two, leaving limited time for implementation. In Kenya, delays in developing the harmonized curriculum meant training for county teams was not possible within the DHSSi project period.

Speed of progress of management professionalization and capacity development was affected by a range of issues. Some were internal, such as mixed performance among implementing partners and management capacity being a newer area of work for UNICEF that required significant initial investment in understanding country needs and engagement with new stakeholders. Other issues included changes in government focal points and limited government availability during the onset of COVID-19 (a particular challenge in Kenya), constraints on availability of other stakeholders whose input was needed to ensure a shared agenda, and variation in the initial status of government plans for management training. For example, in Malawi, the MoH had already identified a need to strengthen training for DHMTs and developed plans for this; DHSSi built on this existing agenda, which led to a high degree of MoH engagement and faster progress.

Although time for implementation of management capacity building was limited, early experience in Malawi indicated potential to strengthen district practice. DHMT members in each focus district reported improvements in DHMT practice following leadership and management training and coaching visits, for example in areas such as DHMT teamwork and coordination, supervision, and relationships with clinical teams. Experience in Malawi also indicated the importance of continued support following training for effective and sustained improvement: all DHMTs indicated areas where they hoped to make changes but where little progress was made, or areas where initial improvements were not sustained. DHMTs saw the coaching and supervision visits as important to motivate continued action: “ when you know that supervisors are coming, you try to do something” (Malawi DHMT 01). Coaching visits were, however, paused following initial implementation, as UNICEF, government and partners considered that further work was needed to define an appropriate coaching approach. There were several areas of uncertainty, for example regarding the most appropriate personnel to undertake coaching, considering skills, credibility with DHMTs and time available; the appropriate format for visits and balance between technical mentoring and broader coaching; and systems that could be sustained by government, considering the resources needed for further training and coaching visits.

Experience with management capacity building in Malawi also highlighted the influence of the enabling environment, wider systems and other initiatives. For example, some changes that DHMTs hoped to make depended on action by the district council or national government. Other initiatives also supported or incentivized progress, including assessment of relevant practice through the Local Authority Performance Assessments (which affect council funding and assess areas such as DHMT supervision of health facilities), and other partner initiatives on management capacity, which contributed to skills development.

Linked with these wider systems and the enabling environment, a political economy analysis (PEA) study was designed to generate insight on enablers and constraints to effective district practice and inform strategies to address barrier ⁴⁰ . Delays with the PEA, in part due to COVID-19, hindered use of findings to design or adapt DHSSi activities within the implementation period. The PEA did, however, contribute to increased awareness and thinking within Ministries and UNICEF about areas for future action, for example, on ways to strengthen district health financing and public financial management.

DHSSi work on EBP responded to the importance of effective annual planning and review for district management, including use of evidence. As with the work on broader management capacity, national and district governments saw DHSSi support for EBP as relevant to address gaps in district planning, in particular strengthening skills and knowledge for data use and implementation of national planning guidelines; increasing awareness of the importance of planning, and providing funding for district planning processes (including development of plans, quarterly reviews and stakeholder engagement meetings).

UNICEF’s bottleneck and causality analysis methodology for EBP was applied to varying extents in all districts and seen as enabling more effective identification and prioritisation of activities that responded to real needs. For example, DHMTs in Uganda described this method as increasing analysis and data use, contributing to a move away from planning based on routine activities and previous plans: it helped DHMTs to “think wider and broader” (Uganda DHMT01) and to “plan based on the priority of the district, not just what we think should be done” (Uganda DHMT02). The key contributions of DHSSi EBP support varied between districts, countries and years, depending on aspects such as the focus of DHSSi activities (for example, the level of support for stakeholder engagement and reviews varied between countries), and district priorities and gaps (for example, whether funding for planning workshops was available from other sources, and the extent to which skills and methods were new or already established).

While governments described significant improvement, the extent to which DHSSi brought a change in district plans varied between countries, years and districts, and in many cases, the degree of improvement seen in planning documents was more limited than suggested by DHMT respondents (as assessed through document review and use of the PAMAT). For example, there were continued gaps in quality and completion of plans, and in some cases, plans deteriorated between years, for example due to a change in district planning staff. There were also gaps in application of certain EBP steps in some contexts, including limited use of BNA (for example, for just one intervention), and in some cases little integration of BNA within plans, with activities developed through EBP not clearly included in final plans and budgets.

Implementation of plans also remained a challenge. Lack of documentation meant implementation levels could not be rigorously assessed, and DHMT estimates of implementation levels sometimes varied widely within districts. However, DHMTs often estimated that less than 70% of planned activities were implemented, with estimates of less than 50% in several cases. Furthermore, where documentation was available, this often suggested lower implementation rates than those estimated by DHMTs.

Several factors affected progress on EBP, including DHSSi scope and processes, and district and national conditions. In relation to DHSSi activities, a range of issues were apparent. For example, late timing of DHSSi EBP workshops or insufficient follow-on support sometimes hindered use of the analysis for planning and budgeting, and limited reach of some EBP workshops meant only some of the people responsible for planning were familiar with BNA.

Beyond DHSSI, multiple other conditions at district and national levels affected progress in application of EBP. A persistent concern for national and district stakeholders was scope to apply EBP given challenges with current health information systems. Constraints included a lack of DHIS2 data for some programme areas, varied data quality, and difficulty in drawing data from different information systems (such as those for human resources and medicines and supplies), as well as gaps in DHMT access to DHIS2 and in DHIS2 functionality. BNA sometimes helped to highlight data gaps and prompt improvements; for example, in Tanzania, there were indications that BNA contributed to increased focus on data quality among district teams. There were also examples of working around data gaps, notably institutionalized use of a qualitative approach to bottleneck analysis when DHIS2 data were unavailable in Malawi. However, data gaps continued to hinder use of routine data for EBP and also delayed progress on configuring the BNA DHIS2 App. Several stakeholders considered that more work was needed to strengthen underlying health information systems before BNA using this routine data could be conducted effectively.

Fit of EBP, and planning more generally, with DHMT time and skills also proved challenging. Hectic workloads and staff turnover, combined with BNA being an unfamiliar approach that required significant training and support, reduced effective use of BNA. DHMT confidence with BNA and the wider EBP process grew through refresher training and reinforcement over time, but staff turnover hindered cumulative and sustained skills development. Time-consuming and in some cases complicated government planning processes and tools were also a challenge given DHMT time constraints and varied DHMT familiarity with planning, contributing to incomplete and error-prone plans. For example, some planning templates required large amounts of background data, or suitable IT hardware, software and computer skills. Government planning templates also affected integration of the EBP analysis within the plan, as templates for some countries did not include sections for all EBP steps.

Inclusion of activities identified through EBP (and of other planned priorities) in the final district activity plan and budget was also affected by alignment of planning and budgeting systems. In Tanzania, district government budgets are based on the Consolidated Council Health Plan (CCHP) and submitted through the same system, so when solutions from EBP are included in the plan, they are also in the budget. Elsewhere, there were gaps in alignment due to different systems for submitting the plan and budget, and regular delays that meant plans were finalized too late to inform budget submissions. In addition, particularly in Malawi, plans included a long list of activities without secured funding (included to demonstrate need with the aim of attracting partner funds during the year), with widespread comments that district government funds did not cover all high priority activities.

Alignment of partner funding also affected inclusion of activities from EBP and other district priorities in the final district plan and budget. Partner flexibility to follow district plans and priorities varied between districts and partners, but was often considered inadequate by DHMTs. This contributed to partner-funded activities that DHMTs saw as lower priority being included in district plans, and lack of funding for some district priorities. DHMTs considered that more partner involvement to jointly determine problems and prioritize solutions would strengthen EBP and other analysis for planning and encourage alignment of partner funds. DHSSi sought to increase partner involvement, particularly in Malawi and Uganda via partner mapping and stakeholder meetings. However, the effects were limited by lack of DHSSi funding for these activities in some years, by partners not attending workshops or returning mapping templates, and by partner funding often being decided by national rather than district staff. Recognizing the latter, there was preliminary work at national level to support partner alignment in some countries. Several DHMTs also saw EBP as helping to mobilize partner funding by providing evidence of the rationale for proposed activities.

National guidelines and associated district decision space also affected inclusion of activities identified through BNA in the final activity plan and budget. This was a particular issue in Tanzania, where scoring criteria for CCHPs and set ceilings for different budget components affected prioritization. Some DHMTs saw these guidelines and criteria as reducing scope to follow district priorities, and as potentially discouraging use of EBP. Decision space was also affected by political interests , such as the influence of county political structures in Kenya. Perceived flexibility to work within national guidelines or to negotiate with political interests varied between districts.

Many of these factors also affected implementation of plans, including insufficient alignment of partner funds and limited government budgets. Implementation was often hindered by insufficient or delayed government and partner funding, as well as rising costs, a particular issue in 2022 due to high inflation. Insufficient funding in turn reduced motivation to invest time in EBP:

Implementation was also affected by the degree of DHMT focus on the plan during the year. This varied between countries. For example, in Tanzania and Kenya, some DHMTs indicated accountability for implementation, with the staff responsible required to report on planned activities, and assessment of implementation in annual contracts or performance appraisal. In contrast, DHMTs in Malawi reported insufficient awareness of the final plan among programme staff responsible for implementation, and lack of attention to the plan in regular DHMT budgeting and decision-making. This was due in part to a hectic and unstable district context, which led to ‘firefighting’ rather than following plans, low district government budgets that primarily covered administrative costs with little room for planned service delivery activities, and financial reporting systems that were not linked to the plan. DHSSi sought to strengthen attention to the plan and monitoring of activities though support for quarterly reviews, and in Malawi, via development of a monitoring tool to track implementation. This support was welcomed by district teams as increasing focus on the plan and enabling identification of action to support implementation. However, the level of DHSSi support and funding for reviews varied between countries and districts, and was in some cases limited. As well as gaps in frequency, government quarterly reviews in some contexts (particularly Uganda) did not directly review the plan, instead focusing on performance of key indicators related to results-based funding. To strengthen implementation and so the relevance of developing plans, governments suggested greater focus on monitoring and review in future.

A further influence on both development and implementation of plans was the level of DHMT motivation to invest time in EBP. Motivation was reduced by limited funds to implement plans and also by the disconnect between planning and funding systems; for example in Uganda, where district budgets did not require development of plans. In contrast, motivation was sometimes increased by positive experience of EBP helping to justify activities and so secure partner funding, and by the perceived value of EBP for improving service delivery indicators that affected national performance assessments, notably the District League Table in Uganda.

DHMT leadership was also important for development of plans, partner engagement, and implementation. For example, there were cases where senior DHMT members encouraged a focus on planning, and where limited interest or understanding among senior DHMT managers reduced effort or opportunities to plan. Strong leadership was also important to encourage partner attendance at planning meetings and to promote partner alignment with district priorities. There were also examples of effective leadership supporting implementation, for example by encouraging district staff to monitor planned activities, or by identifying efficiencies or raising additional funds to address resource constraints. The significance of leadership supports the expectation within the DHSSi ToC that effective leadership and management can facilitate implementation of plans, but with further work needed to strengthen leadership practice.

A range of further issues affected quality of planning and implementation, including capacity building and support from other partners, and COVID-19, which sometimes reduced engagement in planning meetings and often hindered implementation of district plans.

DHSSi aimed to institutionalize EBP and the professionalization of management within government systems, and to support scale up. In relation to institutionalization, the EBP method is included in national planning guidelines in Malawi, Tanzania and Uganda (and in draft guidelines in Kenya). However, further work is needed to ensure a coherent approach across relevant planning and budgeting guidance. For example, in Tanzania, BNA is part of CCHP guidelines, but assessment of CCHPs by regional and national government does not clearly consider use of BNA, and the focus on compliance with set national priorities and ceilings potentially discourages EBP, as noted above. Budgeting systems also need alignment, for example confirming links between plans and systems for government and result-based funding in Uganda.

The degree of government ownership varied between countries and DHSSi activities. For example, ownership for both EBP and management activities was high in Malawi, with clear MoH direction to secure further resources. However, in some contexts, there were concerns within government about insufficient MoH leadership and little evidence of MoH effort to support scale up. Government ownership of management activities was in general more evident than support for further work on EBP. Scale up of EBP and to a large extent management training were seen by government as requiring partner funds.

Some strategies for sustained support to EBP and management capacity were identified in all countries, and to varying extents implemented. Examples include training of trainer (ToT) approaches, plans for use of regional networks to support districts, and scale up through other UNICEF funding, including to other countries. Pre-service training and linking training with Continued Professional Development were also considered, both areas for future work. In part due to an initial expectation that DHSSi activities would take place over a longer timeframe, strategies for sustainability were not always discussed at the outset of DHSSi, particularly for EBP, and some steps were only initiated at a late stage.

At district level, some DHMTs were confident in their skills to continue EBP, but there are some skills gaps and staff turnover may reduce skills without systems for ongoing support. In addition, DHMT time for EBP has relied on DHSSi-funded workshops. Most DHMTs considered that they could find other sources of workshop funding, including from other partners, government or results-based financing, and some suggested conducting EBP through small meetings at the district health office to reduce the need for funds. However, some DHMTs emphasized that EBP would only be possible with further partner funding. Several DHMTs also noted that continued use of EBP requires DHMT motivation, with some pointing to the importance of leadership and demand from higher levels to motivate sustained EBP practice.