Evaluating the impact of integrated development: are we asking the right questions? A systematic review

Rationale

Twenty-first century global trends such as rapid urbanization and dramatic climate change are forcing the international community to rethink solutions to challenges that are increasingly multi-faceted and interrelated. Indeed, the Sustainable Development Goals (SDGs) – an ambitious framework of 17 goals to end extreme poverty, fight inequality and injustice, and reverse climate change over the next 15 years – emphasize the integration of previously distinct development aims. The agenda states that “[t]he goals and targets we have decided on are integrated and indivisible and balance the three crucial dimensions of sustainable development: the economic, social and environmental” (United Nations, 2015). This evolution in thinking indicates a firm shift away from narrowly isolated sectors of development toward what the authors refer to as “win-win cooperation.” An analysis of how each of the 169 SDG targets is related to others reveals a web of closely interrelated objectives, yet also points out that any policy and program integration founded on these underlying linkages would need to rest on evidence with regard to their means of implementation (Le Blanc, 2015).

Thus, decisions about when and how to most effectively implement the integrated, multi-sector SDG agenda need to be driven by evidence, rather than by assumptions about the amplified results of ‘doing more together’. Importantly, a large volume of research carried out to assess various types of integrated programs – in this case meaning programs that intentionally link their design and delivery (i.e., implementation) across more than one core development sector – suggests that in many cases these cross-sector approaches are successful in achieving positive impacts. But are program evaluators addressing a critical question: is 1 + 1 > 2? For example, does adding a nutritional component to an agricultural program reach more people than doing the same work separately? Does integrating family planning into an environmental conversation program improve the targeted impacts of one or more components? Are these interventions generating amplified impacts that go beyond the sum of single sector interventions? To identify and apply effective integrated program approaches in low- and middle-income countries (LMIC), it behooves us to first understand which programs are being evaluated and how. In this paper we present the results of a systematic review designed to identify to what degree impact evaluations have measured synergies and interaction effects, and if evaluations have, then, which methodologies were used to do so.

Objectives

The objectives of this systematic review were threefold: to identify impact evaluations of integrated development programs; to assess whether these evaluations seek to statistically measure synergistic effects or efficiencies associated with integrated interventions; and, for those that do, to document if synergies are detected.

Stage 1: Establish a sampling frame

The universe of development literature and evidence is extensive. Published articles from this broad field are scattered throughout numerous databases, span a multitude of sectors, and use a diverse range of keywords and terminology (or no indicative terminology at all). We explored many of these specialized databases and possible combinations of databases from which we could specifically identify the most evaluations of programs that integrated two or more traditional development sectors (described below). Fortunately, we found a high-quality database that was not specific to any particular sector that would serve well as a single sampling frame for our review. Moreover, while we were initially interested in a broad range of research designs we felt comfortable with using only impact evaluations as it helped to ensure a baseline level of rigor in study design.

The International Initiative for Impact Evaluation (3ie) Impact Evaluation Repository is an index of all published impact evaluations of development interventions. To be included in the repository, an impact evaluation must be published (as a journal article, book chapter, report, or working paper), take place in a developing country, examine the effectiveness of a specific development intervention, and use a specifically defined experimental or quasi-experimental estimation strategy.

During the creation of the repository, 3ie systematically searched more than 45 databases, search engines, journal collections, and websites with an aim to identify all published development impact evaluations (Figure 2) (Mishra & Cameron, 2014; Jorge Miranda, 2017, personal communication). At the time of our analysis (September 8, 2016), 3ie had reviewed more than 140,000 potential studies, rendering an index of 4,339 eligible studies (Mishra & Cameron, 2014; Jorge Miranda, 2017, personal communication). The repository, including a full description of its inclusion criteria and review methodology, is available here. The search was limited to English language articles that evaluate a development program, project, or policy; to evaluations that occur in LMICs; and articles that use rigorous techniques to identify a counterfactual. Grey literature is included in the database, and the 45 sources include everything from EconLit to Google Scholar (full list can be found here). Key terms to identify impact evaluations included terms like ‘impact’, ‘effect’, or ‘random’, but differ slightly by database. The last 3ie update was completed in July 2016.

Figure 2. PRISMA flow diagram for systematic review with 3ie impact evaluation repository review process.

As part of our due diligence, we sought to confirm that 3ie’s repository was a thorough and sufficient sole-sampling source. A library science specialist audited the methodology 3ie staff used to create the repository. Her objective was to assess whether the searches used were both sensitive (i.e., broad) and specific (i.e., focused) enough to ensure that the vast majority of relevant and eligible impact evaluations were included in the final repository. She reviewed the databases that were used and how they were searched with regard to subject scope, time frame limits, and geographic coverage. She concluded that although some lesser-known and regional databases were excluded from the 3ie repository the likelihood that any significant number of new references would be found was negligible. The databases that the 3ie repository was sampling from were searching many of these smaller databases, so she could find no evidence that the larger databases missed anything the smaller databases held. She concluded that the overall methodology design was strong and its implementation consistent. We therefore feel confident that using the 3ie repository as our sampling frame provided us with a sufficient index of development evaluations.

Stage 2: Screen articles in the repository for integrated development approaches

The purpose of this stage of the review was to identify all of the publications in the 3ie repository that evaluated integrated programs. No terms to denote integrated development have universal agreement. The concept of integrated or multi-sector development in published papers is described by many different terms (e.g., cross-sector, linked, combined, blended). Moreover, authors rarely self-identify their interventions in this way within an article, let alone an abstract. We could not, therefore, rely on key search terms to identify evaluations of programs that were integrated in nature. Instead, we manually reviewed the abstracts of every study in the 3ie repository (as of September 8, 2016) against FHI 360’s working definition of integrated development:

“Integrated development approaches intentionally link the design and delivery (i.e., implementation) of programs across more than one core sector.”

Note that our definition of integrated development encompasses studies that would be classified as “multi-sector” or “multi-disciplinary” by others. More precisely, our definition focuses on the integrated nature of the intervention itself and excludes programs that only integrate different subsectors within a core sector (e.g., health programs that link family planning and HIV/AIDS). Our definition also excludes programs that measure outcomes in multiple sectors stemming from a single-sector program (e.g., measuring both education and nutrition outcomes from an education-only project). Finally, explicit intentionality did not factor in the actual application of the definition, since very few evaluations were detailed enough to illustrate motivation or thought process.

Other definitions of integrated programs imply collaboration between the public and private sectors, integration of services between different established levels in a system (e.g., government ministries and local service providers), include different core development sectors, or require clear implementation processes to support integration (e.g., co-location of services, cross-training staff). While these approaches are often grouped together and utilize some of the same approaches and techniques, we broke out multi-sector integration here from public and private sector (or from multi-level systems integration) to create a group of evaluations that are similar enough to be evaluated together meaningfully.

There are no universal or definitive lists of development ‘sectors’. Global bodies and implementing organizations characterize thematic areas in fluid ways, at times bundling some fields (e.g., health and nutrition), and at other times ensuring they are distinctly separate. For this review, we used core sector categories and illustrative interventions and outcomes found in Table 1. This was shaped by what FHI 360 classifies as core sectors, as well as what 3ie and other development databases classify as core sectors. We used these sector categories to classify interventions as well as outcome measures.

All of the interventions in the studies reviewed fell within these sector categories. We added an “other” category to describe outcomes measured, to capture more amorphous, non-sector specific measures, such as “child labor.” For our review, cross-cutting topics such as gender, youth, civil society, and technology were considered aspects of, and relevant to, the interventions and outcomes in each sector, but not sectors in and of themselves. During the review we had also initially included ‘humanitarian’ as a sector. With further discussion and analysis, it was clear that this sampling frame was not inclusive of the humanitarian sector, nor is humanitarian work represented at the same level of the conventional development sectors included above. Therefore, the final analysis was completed without the humanitarian sector category in either intervention sector or outcome sector measured.

To enhance reliability, two individuals independently reviewed all of the abstracts and identified the sectors represented in the interventions being evaluated. If more than one sector was identified, the study was categorized as “yes” for integrated development; all other studies were marked as “no”. These two reviewers met at predetermined intervals to compare their results, and had an average of 89% agreement.

All discrepancies in coding were resolved at each comparison point after the team discussed the interpretation of the integrated development definition (resulting in 100% agreement). In the few cases in which the two reviewers could not agree on a study’s categorization, a third-party reviewed the abstract and made the final decision. Any study that members of the review team both categorized as integrated development was included in the second round of review. For cases in which the abstract alone did not contain enough information to make a determination, studies were advanced to the next round of review so that a final determination could be made during review of the full text.

Importantly, although the repository includes impact evaluations published in any form, our inclusion criteria for this review required a study to be published in a scientific journal. We excluded grey literature to manage the high volume of eligible papers but acknowledge that there is value in their examination for future reviews. Therefore, only those publications that were peer reviewed moved on to Stage 3.

Stage 3: Characterize evaluations of interventions identified as integrated

Full text articles of the subset of studies on integrated programs that were published in scientific journals were reviewed by two individuals. Each article was compared against a checklist, to ascertain the study’s scope and methodology (our checklist is presented with the corresponding results in Table 2).

In particular, we noted the number of control, single-sector treatment, and integrated sector treatment arms in each evaluation. We further identified those evaluations which employed either a partial factorial or full factorial experimental design. For the purposes of our review, partial factorial designs included at least one single-sector arm (but not all single-sector arms), at least one integrated arm, and at least one control (no intervention) arm. Full factorial designs included all possible single-sector arms, at least one integrated arm, and at least one control arm. Factorial designs are exceptionally rigorous and permit evaluators to determine the effects of multiple interventions on an outcome. Since they include all possible combinations of intervention arms, full factorial designs are able to reveal differential effects of single-sector and multi-sector interventions and measure potential synergistic effects associated with integrated approaches.

Therefore, we specifically reviewed each full factorial evaluation to determine if the authors measured or detected synergy associated with the integrated study arm. For our review, we defined synergy as a statistically significant (p < 0.05) interaction effect in the integrated arm (between two or more intervention sectors), or as instances in which the effect size of the integrated arm of a program was greater than the sum of the effect sizes among the single-sector arms.

Given the extreme heterogeneity of the types of programs evaluated and outcomes assessed, we did not seek to collectively synthesize their substantive findings. Instead, the primary objective was to determine if and how impact evaluations of integrated programs are designed to measure or systematically document the synergy and efficiency assumed in multi-sector development.

Limitations

Creating and applying a definition of integrated development was a subjective process. To address this, we utilized two independent coders and employed inter-coder agreement procedures to enhance reliability in our screening process. Assigning core sectors to studies was also a subjective process, and in some cases assigning sectors to an intervention was difficult (e.g., depending on its particular aim, aid to small-scale farmers could conceivably be an economic development, agriculture, or nutrition intervention). We attempted to mitigate this by providing definitions and examples of core sectors to both reviewers, and once a type of intervention was categorized in one way it remained consistent across studies.

Another potential limitation is that the 3ie repository may not be exhaustive. The 3ie database limits by language (English) and the last update at the time of our searches was completed in July 2016. Limiting to English could systematically exclude publications from certain regions, or exclude certain groups of evaluations. Eligible publications in regional or small databases not included in the search strategy could have been overlooked. Given the size of the repository, however — more than 4,300 publications — including a small number of studies that may be absent in the 3ie repository would not have changed the substance of our findings.

We also recognize that in the past 4–5 years an increasing proportion of impact evaluations are being written up as working papers (Cameron et al., 2016) and may not be published in peer-reviewed journals. Although our judgment is that it would not be significant, we do acknowledge that reviewing to include grey literature could conceivably shift some of the trends we highlight here. Our entire database of impact evaluations for integrated development found within the 3ie repository – peer-reviewed and grey literature – can be found here.

Finally, a large portion of the studies in the impact evaluation repository (and therefore this review) focus on the health sector. This is more a representation of the volume of research conducted in the health sector in general rather than an indication that health-focused approaches have a higher importance or more effectiveness than other sectors. Due to different evaluation cultures within different sectoral communities, and the ease to which some interventions lend themselves to certain types of evaluations, health is almost certainly overrepresented here than if this review had a broader methodological sampling frame.

Further research

There are many avenues which we could not pursue during this research that would further support our understanding of how integrated projects are being studied, and the results of those studies. A review of observational studies, or a systematic review of qualitative or cost analyses studies could also illustrate meaningful trends. There have been targeted reviews on various combinations of sectors that have included deeper examination of the results (since the subject is narrower) (e.g., Lukas, 2008; Ruel et al., 2018). This review can be used to highlight areas that could be further explored with a finer comb. Further reviews of different types of integration – including private and public sector, and multi-sector outcomes for example, mentioned above – could also produce interesting results.