Keywords
integrated, development, multi-disciplinary, multi-sector, evaluation, synergy, interaction effects, SDGs
-
Views
-
Downloads
How to cite this article
Ahner-McHaffie TW, Guest G, Petruney T et al. Evaluating integrated development: are we asking the right questions? A systematic review [version 1; peer review: 3 approved with reservations]. Gates Open Res 2017, 1:6 (https://doi.org/10.12688/gatesopenres.12755.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
Export Citation
Sciwheel
EndNote
Ref. Manager
Bibtex
ProCite
Sente
▬
✚
Systematic Review
Evaluating integrated development: are we asking the right questions? A systematic review
[version 1; peer review: 3 approved with reservations]
PUBLISHED 06 Nov 2017
Abstract
Corresponding author:
Tessa W Ahner-McHaffie
Competing interests:
No competing interests were disclosed.
Grant information:
Bill and Melinda Gates Foundation [OPP1130126] and the FHI Foundation.
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Copyright:
© 2017 Ahner-McHaffie TW et al.
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
How to cite: Ahner-McHaffie TW, Guest G, Petruney T et al. Evaluating integrated development: are we asking the right questions? A systematic review [version 1; peer review: 3 approved with reservations]. Gates Open Res 2017, 1:6 (https://doi.org/10.12688/gatesopenres.12755.1)
First published: 06 Nov 2017, 1:6 (https://doi.org/10.12688/gatesopenres.12755.1)
Latest published: 29 May 2018, 1:6 (https://doi.org/10.12688/gatesopenres.12755.2)
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
There is a newer version
of this article available.
of this article available.
Introduction
Rationale
Twenty-first century global trends such as rapid urbanization and dramatic climate change are forcing the international community to rethink solutions to challenges that are increasingly multi-faceted and interrelated. Indeed, the Sustainable Development Goals (SDGs) – an ambitious framework of 17 goals to end extreme poverty, fight inequality and injustice, and reverse climate change over the next 15 years – emphasize the integration of previously distinct development aims. The agenda states that “[t]he goals and targets we have decided on are integrated and indivisible and balance the three crucial dimensions of sustainable development: the economic, social and environmental” (United Nations, 2015). This evolution in thinking indicates a firm shift away from narrowly isolated sectors of development toward what the authors refer to as “win-win cooperation.” A recent analysis of how each of the 169 SDG targets is related to others reveals a web of closely interrelated objectives, yet also points out that any policy and program integration founded on these underlying linkages would need to rest on evidence with regard to their means of implementation (Le Blanc, 2015).
Thus, decisions about when and how to most effectively implement the integrated, multi-disciplinary SDG agenda need to be driven by evidence, rather than by assumptions about the amplified results of ‘doing more together’. Importantly, a large volume of research carried out to assess various types of integrated programs suggests that in many cases these approaches are successful in achieving positive impacts. But are program evaluators addressing a critical question: is 1 + 1 > 2? In other words, are these interventions generating amplified impacts that go beyond the sum of single sector interventions? To effectively advocate for integrated, multi-disciplinary approaches to development, it behooves us to understand under which circumstances integrating two or more development sectors enhances impacts in amplified or synergistic ways. In this paper we present the results of a systematic review designed to identify if and how synergies and interaction effects between sectors in integrated interventions are being measured.
Objectives
The objectives of this systematic review were threefold: to summarize how integrated development programs are being evaluated; to assess whether these evaluations seek to statistically measure synergistic effects or efficiencies associated with integrated interventions; and, for those that do, to document if synergies are detected. Secondary objectives include documenting other characteristics associated with study design, such as: inclusion of cost analyses or qualitative analyses, types of sectors involved, and regions where the intervention took place.
Methods
Our review consisted of a three-stage process (Figure 1), based on the PRISMA guidelines (Moher et al., 2009), as well as recommendations from Waddington et al. (2012). We first established a sampling frame from which to screen and review published articles. Given our objectives, we required a large, relatively exhaustive development database, or combination of databases, that included a broad range of evaluations not limited to any particular development sector. The first stage in our process, therefore, entailed establishing a sampling frame from which to identify and review the highest number of relevant publications. Once the sampling frame was established, we screened each article’s abstract and identified evaluations of development programs that we defined as “integrated development.” We then reviewed the full text of each article in the integrated development subset and documented characteristics essential to the objectives of our review.
Figure 1. Review process for systematic review on integrated, multi-sector programs.
Stage 1: Establish a sampling frame
The universe of human development literature and evidence is extensive. Published articles from this broad field are scattered throughout numerous databases, span a multitude of sectors, and use a diverse range of keywords and terminology. We explored many of these specialized databases and possible combinations of databases from which we could specifically identify the most evaluations of programs that integrated two or more traditional development sectors (described below). Fortunately, we found a high-quality database that was not specific to any particular sector that would serve well as a single sampling frame for our review.
The International Initiative for Impact Evaluation (3ie) Impact Evaluation Repository is an index of all published impact evaluations of development interventions. To be included in the repository, an impact evaluation must be published (as a journal article, book chapter, report, or working paper), take place in a developing country, examine the effectiveness of a specific development intervention, and use a specifically defined experimental or quasi-experimental estimation strategy. The 3ie review process has no restriction on publication date; however, the systematic review upon which it is based was completed in July, 2016.
During the creation of the repository, 3ie systematically searched more than 45 databases, search engines, journal collections, and websites with an aim to identify all published development impact evaluations (Figure 2) (Mishra & Cameron, 2014; Jorge Miranda, 2017, personal communication). At the time of our analysis (September 8, 2016), 3ie had reviewed more than 140,000 potential studies, rendering an index of 4,339 eligible studies (Mishra & Cameron, 2014; Jorge Miranda, 2017, personal communication). The repository, including a full description of its inclusion criteria and review methodology, is available here.
Figure 2. PRISMA flow diagram for systematic review with 3ie impact evaluation repository review process.
As part of our due diligence, we sought to confirm that 3ie’s repository was a thorough and sufficient sole-sampling source. A library information science specialist audited the methodology 3ie staff used to create the repository. Her objective was to assess whether the searches used were both sensitive (i.e., broad) and specific (i.e., focused) enough to ensure that the vast majority of relevant and eligible impact evaluations were included in the final repository. She reviewed the databases that were used and how they were searched with regard to subject scope, time frame limits, and geographic coverage. She found that some lesser-known and regional databases were excluded from the 3ie repository; however, she determined that these smaller databases would not likely have added a notable number of new references. She concluded that the overall methodology design was strong and its implementation consistent. We therefore feel confident that using the 3ie repository as our sampling frame provided us with a sufficient index of development evaluations.
Stage 2: Screen articles in the repository for integrated development approaches
The purpose of this stage of the review was to identify all of the publications in the 3ie repository that evaluated integrated programs. No terms to denote integrated development have universal agreement. The concept of integrated or multi-sector development in published papers is described by many different terms (e.g., cross-sector, linked, combined, blended). Moreover, authors rarely self-identify their interventions in this way within an article, let alone an abstract. We could not, therefore, rely on key search terms to identify evaluations of programs that were integrated in nature. Instead, we manually reviewed the abstracts of every study in the 3ie repository (as of September 8, 2016) against our organization’s working definition of integrated development:
“Integrated development approaches intentionally link the design and delivery of programs across more than one core sector.”
Note that our definition of integrated development encompasses studies that would be classified as “multi-sector” or “multi-disciplinary” by others. More precisely, our definition focuses on the integrated nature of the intervention itself and excludes programs that:
Only integrate different subsectors of a core sector (e.g., health programs that link family planning and HIV/AIDS); or
Measure outcomes in multiple sectors but do not include multi-sector intervention components (e.g., education programs that measure both education and nutrition outcomes but only deliver education services).
There are no universal or definitive lists of development ‘sectors’. Global bodies and implementing organizations characterize thematic areas in fluid ways, at times bundling some fields (e.g., health and nutrition) and at others ensuring they are distinctly separate. For this review we used the following core sector categories and illustrative interventions. We used these sector categories to classify interventions as well as outcome measures. These categories were used as general guiding parameters rather than strictly exclusive definitions:
Agriculture and food security (e.g., farming, food supply chains, famine prevention);
Economic development (e.g., income, livelihood, cash transfers, microfinance);
Education (e.g., early education, primary/secondary/tertiary school);
Environment (e.g., environmental/land management, conservation, climate change);
Governance (e.g., peace building, conflict management, election monitoring, democracy);
Health (e.g., HIV, tuberculosis, maternal and child health, sexual and reproductive health, non-communicable disease, malaria, immunization/vaccine);
Nutrition (e.g., micronutrients, food fortification, malnutrition, feeding programs, diet diversification); and
Water, sanitation, and hygiene (e.g., water quality, management, supply).
All of the interventions in the studies reviewed fell within these sector categories. We added an “other” category to describe outcomes measured, to capture more amorphous, non-sector specific measures, such as “child labor.” For our review, cross-cutting topics such as gender, youth, civil society, and technology were considered aspects of, and relevant to, the interventions and outcomes in each sector, but not sectors in and of themselves. During the review we had also initially included ‘humanitarian’ as a sector. With further discussion and analysis, it was clear that this sampling frame was not inclusive of the humanitarian sector, nor is humanitarian work represented at the same level of the conventional development sectors included above. Therefore, the final analysis was completed without the humanitarian sector category in either intervention sector or outcome sector measured.
To enhance reliability, two individuals independently reviewed all of the abstracts and identified the sectors represented in the interventions being evaluated. If more than one sector was identified, the study was categorized as “yes” for integrated development; all other studies were marked as “no”. These two reviewers met at predetermined intervals to compare their results, and had an average of 89% agreement.
All discrepancies in coding were resolved at each comparison point after the team discussed the interpretation of the integrated development definition (resulting in 100% agreement). In the few cases in which the two reviewers could not agree on a study’s categorization, a third party reviewed the abstract and made the final decision. Any study that members of the review team both categorized as integrated development was included in the second round of review. For cases in which the abstract alone did not contain enough information to make a determination, studies were advanced to the next round of review so that a final determination could be made during review of the full text.
Importantly, although the repository includes impact evaluations published in any form, our inclusion criteria for this review required a study to be published in a scientific journal. Therefore, only those publications moved on to Stage 3.
Stage 3: Characterize evaluations of interventions identified as integrated
Full text articles of the subset of studies on integrated programs that were published in scientific journals were reviewed by two individuals. Each article was compared against a checklist, to ascertain the study’s scope and methodology (our checklist is presented with the corresponding results in the next section).
In particular, we noted the number of control, single-sector treatment, and integrated sector treatment arms in each evaluation. We further identified those evaluations which employed either a partial factorial or full factorial experimental design. For the purposes of our review, partial factorial designs included at least one single-sector arm (but not all single-sector arms), at least one integrated arm, and at least one control (no intervention) arm. Full factorial designs included all possible single-sector arms, at least one integrated arm, and at least one control arm. Factorial designs are exceptionally rigorous and permit evaluators to determine the effects of multiple interventions on an outcome. Since they include all possible combinations of intervention arms, full factorial designs are able to reveal differential effects of single-sector and multi-sector interventions and measure potential synergistic effects associated with integrated approaches.
Therefore, we specifically reviewed each full factorial evaluation to determine if the authors measured or detected synergy associated with the integrated study arm. For our review, we defined synergy as a statistically significant (p < 0.05) interaction effect between two or more intervention sectors, or instances in which the effect size of the integrated arm of a program was greater than the sum of the effect sizes among the single-sector arms.
Given the extreme heterogeneity of the types of programs evaluated and outcomes assessed, we did not seek to collectively synthesize their substantive findings. Instead, the primary objective was to determine if and how impact evaluations of integrated programs are designed to measure or systematically document the synergy and efficiency assumed in multi-sector development.
Results
We reviewed 4,339 abstracts, comprising the entire 3ie repository as of September 8, 2016. After a two-step screening process, 601 articles were included in our final dataset for characterization (Figure 2, Supplementary File 2). From the initial set of 4,339 articles from the 3ie repository, 3,543 were excluded (2,380 did not meet the definition of integrated and 1,163 were not published in a scientific journal). The full text of the remaining 796 were assessed for eligibility. One hundred and ninety-five were excluded with full text review (193 did not meet the definition of integrated and two were not available to reviewers). This left 601 studies included in the analysis. The list of articles is included here as Supplementary File 3, and each article may also be found in a searchable online database.
Most articles (84%) did not identify the interventions being evaluated as “integrated”, or any other related term (Table 1).
Table 1. Summary of integrated development impact evaluation characteristics.
The majority of evaluations (70%) employed a randomized controlled design to assess the effectiveness of their interventions. However, only 26 (4%) of the 601 studies reviewed used a full factorial design and only 12 (2%) employed a partial factorial design. The majority of evaluations (61%) assessed the effectiveness of an integrated intervention by comparing one or more integrated arms to a no-treatment control only. A minority of evaluations included a comparison of integrated arms only (11%), or contained single-sector arms and integrated arms but no control arm (22%). Few evaluations included qualitative (10%) or cost analyses (7%) components.
With regard to what types of interventions and desired outcomes were being assessed, the three sectors most often represented in the intervention design — in order of highest to lowest frequency — were health, education, and nutrition. The same three sectors were also most common in terms of outcomes measured, with nutrition slightly outpacing education for second most common.
For the 38 studies that represented either a partial or a full factorial design, we assessed whether the effectiveness of an integrated intervention — in terms of study outcomes — was evaluated. Of the 26 that were full factorial, seven reported findings that showed that the integrated arm was most effective (De Brauw et al., 2015; Haque et al., 2010; Leventhal et al., 2016; Nga et al., 2009; Nga et al., 2011; Olsen et al., 2003; Widen et al., 2015). Eight demonstrated mixed findings, or did not report the effectiveness of the integrated arm as compared to the other arms (Awasthi et al., 2013; Duflo et al., 2015; Gilgen & Mascie-Taylor, 2001; Halliday et al., 2014; Jinabhai et al., 2001; Kim et al., 2015; Leventhal et al., 2015; Tahlil et al., 2015). In some cases, the added value of integration was reported in another study, or it was stated that the combination was not intended to have effects on the outcomes of the separate sectors, so the data was not fully analyzed or reported. In terms of mixed findings, some studies demonstrated tradeoffs, where integration added value to certain outcomes, but was deleterious for others. Finally, 11 evaluations found no added value of integration (Attanasio et al., 2014; Dangour et al., 2011; Desai & Tarozzi, 2011; Fenn et al., 2012; Friis et al., 2003; Gilgen et al., 2001; Gowani et al., 2014; Mwaniki et al., 2002; Nahar et al., 2012; Rohner et al., 2010; Walker et al., 2006).
Only three of the full factorial studies incorporated cost analysis, and two of the three found that the integrated arm was cost-effective. The third did not perform a cost analysis specifically on the integrated arm.
We also reviewed the findings reported in the 60 studies that included a qualitative component. We found only 1 study that intentionally documented synergy via the qualitative inquiry – the others used the method to investigate other aspects of the intervention.
Discussion
Our screening of 4,339 records in the 3ie Impact Evaluation Repository identified 601 journal articles that describe studies of programs we defined as integrated. Our full text review of these 601 articles revealed several interesting trends. First, researchers do not use standardized terms for describing integrated development programs. In fact, the majority of authors did not use any term at all to indicate the integrated or multi-sector nature of the interventions they were evaluating. This finding validates our manual screening methodology. Had we used a key term search strategy, we likely would have missed many relevant studies. Interestingly, 46% of the full factorial evaluations addressed integration or synergy in their abstracts, as compared to only 16% of all studies identified as integrated.
Next, only 26 evaluations employed a full factorial design. Though randomized controlled designs are sufficient to confidently detect the impact of these types of programs, a full factorial design is the only design that truly enables researchers to measure whether the impact is related to the synergy presumed to result from integrated, multi-sector programming.
We recognize that full factorial designs are often costly and time-consuming, and may not be feasible in many, or even most, contexts. They may not be necessary for the types of integrated approaches that have been confirmed in the past to deliver synergy. However, for new or less-researched multi-sector models, robust factorial studies will help sufficiently determine the role of integration in their implementation and results. Factorial or not, examining cost efficiencies and qualitatively assessing synergies are other valuable methods that can help determine how integration factors in to the program findings.
Limitations
Creating and applying a definition of integrated development was a subjective process. To address this, we utilized two independent coders and employed inter-coder agreement procedures to enhance reliability in our screening process. Assigning core sectors to studies was also a subjective process, and in some cases assigning sectors to an intervention was difficult (e.g., depending on its particular aim, aid to small-scale farmers could conceivably be an economic development/livelihoods, agriculture, or nutrition intervention). We attempted to mitigate this by providing definitions and examples of core sectors to both reviewers, and once a type of intervention was categorized in one way it remained consistent across studies.
Another potential limitation is that the 3ie repository may not be exhaustive; eligible publications in regional or small databases could have been overlooked. Given the size of the repository, however — more than 4,300 publications — including a small number of studies that may be absent in the 3ie repository would not have changed the substance of our findings. Furthermore, there are many studies in the impact evaluation repository (and therefore this review) that focus on the health sector, but that does not necessarily mean that a majority of integrated programs are focused on health. Due to different evaluation cultures within different sectoral communities, and the ease to which some interventions lend themselves to certain types of evaluations, health is almost certainly overrepresented here than if this review had a broader methodological sampling frame. Other groups have significant evidence bases that are not captured here as they do not fall into the inclusion criteria of the impact evaluation repository, and therefore this review.
We also recognize that in the past 4–5 years an increasing proportion of impact evaluations are being written up as working papers (Cameron et al., 2016) and may never be published. Although our review would have missed unpublished reports, the conclusions we draw in the paper – that synergistic effects are rarely measured in evaluations of integrated development programs – would not likely have changed had unpublished papers been included.
Conclusions
Our systematic review is not intended to determine whether or not integrated development approaches work. We know from the high number of randomized evaluations included here that in many contexts integrated, multi-sector interventions have produced positive impacts. What our systematic review does indicate, however, is that very few evaluations to date were designed to specifically examine the synergistic and interaction effects that are potentially associated with integrated programming. In other words, to what extent is the integration itself producing the results versus other factors? Impact evaluations of new or yet-to-be-proven integrated programs need to be better designed to intentionally assess not only their impact but the explicit value added of linking two or more development sectors, in terms of service delivery outcomes, participant perspectives, and cost.
Addressing these gaps is essential as the international community pivots toward a more cross-cutting global development agenda. Implementing this agenda will likely deploy more promising and innovative silo-breaking programs. We must ensure that our research designs and measurement strategies keep pace accordingly.
Download
Export To
metrics
| Views | Downloads | |
|---|---|---|
| Gates Open Research | - | - |
|
PubMed Central
Data from PMC are received and updated monthly.
|
- | - |
Citations
CITE
how to cite this article
Ahner-McHaffie TW, Guest G, Petruney T et al. Evaluating integrated development: are we asking the right questions? A systematic review [version 1; peer review: 3 approved with reservations]. Gates Open Res 2017, 1:6 (https://doi.org/10.12688/gatesopenres.12755.1)
NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article.
track
receive updates on this article
Track an article to receive email alerts on any updates to this article.
Alongside their report, reviewers assign a status to the article:
Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested
Approved with reservations -
A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit.
Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions
Competing Interests Policy
Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list:
Examples of 'Non-Financial Competing Interests'
- Within the past 4 years, you have held joint grants, published or collaborated with any of the authors of the selected paper.
- You have a close personal relationship (e.g. parent, spouse, sibling, or domestic partner) with any of the authors.
- You are a close professional associate of any of the authors (e.g. scientific mentor, recent student).
- You work at the same institute as any of the authors.
- You hope/expect to benefit (e.g. favour or employment) as a result of your submission.
- You are an Editor for the journal in which the article is published.
Examples of 'Financial Competing Interests'
- You expect to receive, or in the past 4 years have received, any of the following from any commercial organisation that may gain financially from your submission: a salary, fees, funding, reimbursements.
- You expect to receive, or in the past 4 years have received, shared grant support or other funding with any of the authors.
- You hold, or are currently applying for, any patents or significant stocks/shares relating to the subject matter of the paper you are commenting on.
Stay Updated
Sign up for content alerts and receive a weekly or monthly email with all newly published articles
Register with Gates Open Research
Already registered? Sign in
Comments on this article Comments (0)