The views expressed in this article are those of the authors. Publication in Gates Open Research does not imply endorsement by the Bill & Melinda Gates Foundation.

“Why do some behavioral interventions succeed and others fail?” is a question at the forefront of every implementer’s mind. Are there systematic processes that, if followed during the design and implementation of a behavioral intervention, increase the likelihood of the intervention’s success? Specifically, can timely learning or responsive feedback during the implementation of an intervention help increase the likelihood of a project’s success?

The answer to this question occupies the attention of “the Curve” community of practice (CoP) supported by the Bill & Melinda Gates Foundation ( Viswanath et al., 2019). This CoP provides a platform to 20 leading implementation and research organizations and 5 donors that, as a group, have focused on this question since 2018. The Curve provides participating organizations the opportunity to think deeply about this question, to learn from each other’s experience and to develop illustrative examples of responsive feedback, with the objective of providing examples which may be of broader interest within the development field.

Here, we illustrate how feedback can help by providing an example of the process of learning during the implementation of a behavioral intervention in Ghana. In 2016, the tobacco control team within the Global Policy and Advocacy (GPA) team at the Bill & Melinda Gates Foundation decided to fund an impact evaluation of a social marketing campaign to prevent tobacco use in Ghana. The tobacco team felt that an objective impact assessment would not only help guide the design and measurement of future social marketing campaigns, but could also contribute to an investment case for prevention-related communication. A preventive approach for tobacco control is extremely important in Africa, where the prevalence of tobacco use is still relatively low and the tobacco industry sees a prime opportunity for market growth. The impact evaluation was designed to respond to several critical questions: which messages are most effective; which media are most effective; how cost-effective is the campaign; and, perhaps most importantly, what is the magnitude of the campaign’s impact.

The implementing organization, Good Business (GB), had previously designed social marketing interventions in Botswana and Uganda. These interventions had been implemented with the aim of changing knowledge, attitudes, and perceptions around tobacco use. GB had hired researchers to measure the effects of its intervention among adolescents who were exposed to project activities through indicators such as the number of girls who spontaneously listed smoking as something they were opposed to and the number of girls who thought smoking made them more attractive. This ability to track behavior among teens exposed to the intervention had enabled GB to feel confident about continuing with the implementation of planned activities. While results from the campaigns were generally positive, the foundation’s tobacco control team wanted independent verification that the approach worked, and that positive changes in attitudes and norms related to smoking could be attributed to the campaign. In other words, they wanted a comparison of the effects of the campaign with what would have happened in its absence. The tobacco control team felt that a population-based impact evaluation would add to the evidence base and help them build an investment case for using social marketing to prevent teen tobacco use in sub-Saharan Africa.

The Tulane University School of Public Health and Tropical Medicine (Tulane) was selected to conduct the evaluation using a quasi-experimental, panel design. The evaluation relied on population-based household surveys conducted with randomly selected 13–17 year-old girls in project and non-project areas in Ghana. The full methodology and results of the evaluation are detailed in Hutchinson et al., 2019a. These surveys were conducted before activities were initiated and at the end of the project. A unique, additional feature of the evaluation was a feedback loop comprised of multiple rounds of mobile phone surveys designed to measure exposure and message comprehension among the project’s target audience during the project’s 20-month implementation period. The level of exposure to a social marketing intervention is considered a key determinant of its anticipated success ( Hornik, 2002). Indeed, some experts will not consider assessing the population level impact of an intervention that reaches less than 20% of its target audience.

The SKY campaign built upon GB’s experience of implementing a social marketing teen tobacco prevention intervention in Botswana. The Botswana campaign had shown how important a determinant peer influences were on teen tobacco use ( Persons, 2017). Figure 1 shows the Theory of Change (ToC) of the SKY Girls tobacco control intervention. Based on the Botswana experience, the Ghanaian ToC emphasized the importance of a multi-channel brand and community platform to motivate teenage girls to join a movement that fostered a sense of belonging. By connecting to other girls and by expressing themselves through positive choices, the ToC expected adolescent girls to gain self-confidence and a sense of identity that would enable them to reject tobacco use.

In Ghana, formative research conducted by the implementer to design the campaign showed that teen girls were usually aware of the health harms of tobacco and were reluctant to smoke, but the paramount need for social inclusion among fellow teenagers often dictated their decision-making. Teen girls in Ghana lacked the confidence to resist the social pressure of smoking. The choice to smoke had become part of a teenage girl’s struggle with identity formation.

The SKY social marketing campaign was designed to unite girls behind the idea of ‘being true to themselves’, to create an environment that enabled girls to express their true feelings, including their desire not to smoke, and to feel supported by a community of peers. In partnership with the Ghanaian advertising agency Now Available Africa, GB launched a platform focused on girls’ empowerment and resilience. Using content around passion points such as music and fashion to draw girls in, the campaign weaved in messaging about choosing not to smoke as the smart, ambitious, aspirational choice.

To reinforce the message, the campaign was executed across multiple media channels, including free magazines, radio programming, free original movies, social media, and live events in Accra. The local agency worked closely with teens to develop content owned by the audience and in direct response to their interests. As part of this campaign, Ghanaian teens were featured throughout the media channels – in the magazine, in live discussions on radio, and on social media.

We focus our discussion on the interaction between the implementers and the evaluators, on the type of feedback provided and how the feedback provided enabled implementers to adapt the design of the intervention to the media and cultural context of Ghana.

A primary program officer (JP) from the tobacco control team of the Bill & Melinda Gates Foundation managed both the implementation (primed by GB) and the evaluation (primed by Tulane) grants with support on the evaluation component from a colleague in the foundation’s Global Delivery team. The primary program officer believed that evaluators should maintain their independence but should provide insights and learnings to the implementer on a regular basis to optimize the use of data in management decision-making. The evaluation grant was designed to work in tandem with the implementation of the project: research timelines were aligned with implementation timelines, indicators were linked to the project’s theory of change (ToC).

With this management model in mind, in December 2016, the primary program officer organized an in-person, project kickoff meeting attended by the UK-based implementer, the US-based evaluator and the Seattle-based donor. This provided an opportunity to discuss project objectives and evaluation approaches. Following the kickoff, a cadence of monthly Skype calls was put in place which included the primary program officer, her colleague from Global Delivery (SA), the implementer and evaluator. These meetings occurred consistently over the 20-month duration of project implementation.

From the outset, evaluators focused on learning about the project design, its underlying assumptions and the activities planned. Understanding the project’s ToC and learning about the outcomes expected from the planned activities had immediate implications for the design of survey instruments used for assessing exposure to project activities and measuring the impact of the campaign on outcomes.

As part of their contribution to this interaction, GB reviewed and commented on survey instruments and provided input on measurement indicators they considered most meaningful from a programmatic perspective. For example, given their previous experience with the challenges of retaining very young adolescents’ attention during an interview, implementers insisted on shortening the length of the household survey questionnaire. Once the baseline survey had been completed, Tulane shared the findings with GB for their interpretation and programmatic decision-making.

The findings showed that exposure to the SKY campaign was high in Accra. Of girls who lived and attended school in Accra, 84% were exposed to at least one of the SKY channels and 38% were exposed to four or more channels. Of the girls exposed to SKY, 58% said SKY was important to them, with 39% citing the free magazine as the best way to connect to SKY.

The data suggests that the campaign was able to reduce the pressure girls felt to smoke cigarettes and increase teen girls’ confidence in making decisions for themselves. About half of girls exposed to the campaign felt that SKY helped them to reject shisha and cigarettes, think about their future and personal goals and feel confident to make decisions for themselves ( Hutchinson et al., 2019b). Four out-of-ten girls exposed to SKY felt that it helped them connect with other girls in their peer group – a part of GB’s goal of building a community of teens that supports girls in rejecting the things they didn’t want.

In the multivariate analysis, the intervention was associated with an 11 percentage point decrease in the perceived pressure to smoke cigarettes, a 10 percentage point increase in the likelihood of conversations with friends about smoking, and similar magnitude of increases in girls’ perceived ability to make choices and in their disagreement with the justification for shisha use.

In Botswana, GB had used a multi-channel strategy which used a radio show hosted by teen girls, television, the SKY Girls magazine and outdoor events. Social media was a critical component of the Botswana social marketing teen tobacco control campaign. The SKY Girls Facebook page was the hub of the community, allowing girls the freedom to express what they liked (“ sure ka yone”/SKY) and what they did not (“shapo ka yone”), to share their views and to be supported by other girls. The SKY Girls Facebook page was the third most popular page for teen girls in Botswana and was central to the social marketing campaign’s strategy of amplifying the message.

In contrast with Botswana, the Ghana baseline household survey, conducted by the evaluators prior to the start of the campaign, showed that radio listenership was extremely low among Ghanaian teenage girls. A large proportion of Ghanaian girls were not available online. The use of Facebook, which had been central to the campaign in Botswana, was relatively low in Ghana. Adding to the complexity, television could not be used in Ghana because of the risk of campaign messages contaminating the comparison group. Taken together, this meant that a substantially different channel strategy would have to be adopted for the social marketing campaign in Ghana with no reliance on mass media and reduced reliance on social media.

Several other factors made the Ghanaian context different. Adolescent girls’ mobility was much more restricted in Ghana than in Botswana. To the implementers, this highlighted the importance of schools as a place to reach adolescent girls. School-based activities, which were limited in Botswana because they were considered “uncool”, and were not part of the original plan in Ghana, became vital for the social marketing campaign in Ghana. School-based activities helped build awareness of the brand quickly without the need to advertise on television. The local advertising agency used, Now Available Africa, recommended schools as a great means of reaching girls and ensuring a depth of engagement and being more cost-effective. They also emphasized that school activations were not considered “uncool” in Ghana.

It took several months after the baseline household survey for the mobile phone surveys to be implemented. These phone surveys began providing data on campaign exposure. GB were surprised by the initial low level of exposure. This also meant that the number of teens captured in the phone survey who were exposed to SKY was small – limiting an analysis of the profile of Ghanaian girls who were exposed to the campaign - an analysis that is extremely important for marketers to ascertain whether a campaign is reaching those it plans to reach.

To overcome this limitation and provide insights on girls who were exposed to the campaign and to study message comprehension and message effectiveness, the evaluators decided to analyze the SKY Girls database. This was a database on teens exposed to project activities, maintained by GB and Now Available Africa. Analysis of this database was not originally part of Tulane’s scope of work. The analysis showed that Facebook was less effective in delivering messages than the SKY Girls magazine. Earlier analysis of the baseline household survey had shown that a small proportion of adolescent Ghanaian girls used Facebook. Based on both insights, the implementers chose to reduce use of Facebook as a platform for messaging about the harms of tobacco, and instead used it as a medium to engage adolescents on how they made decisions on important issues. Thus, both the channel and messaging strategy changed based on insights from multiple sources of data. GB felt that they were able to adapt their content strategy to match the way in which adolescent girls were using different communication channels.

As the approach taken in Ghana was considerably different from the one in Botswana, it was important for implementers to feel confident that the programmatic strategy they were choosing – a strategy that was very different from what they had expected to use – led to increased exposure to SKY among teenagers.

Subsequent rounds of mobile phone surveys showed steadily increasing levels of exposure over time. Surveys showed that the channels being used in Ghana were working and that there was a steady increase in exposure to the campaign. Analysis showing that adolescents really liked the project brand helped increase the implementer’s confidence in the new approach.

The intervention evolved considerably, as results from multiple rounds of mobile phone surveys came in. Rather than relying on Facebook and interactive radio shows, schools became the catchment area for the intervention. The implementer had expected to reach people through live, public, events, but this was not feasible in Ghana since Ghanaian girls’ mobility was low.

Evaluators used data from the baseline household survey to conduct segmentation analysis - which groups individuals into a small number of homogenous segments – to aid campaign message development. This helped the implementer develop an understanding of sub-groups within the population who could be targeted with messages that may be particularly appealing to them. The implementer also conducted focus group discussions (FGDs) to understand how exposure to certain project activities, such as the movie or brand activations, were received. Continuous listening to adolescents’ response to project activities through quantitative (and some qualitative) tools helped ensure that implementers understood the social and cultural context of adolescent girls’ behavior. In Botswana, adolescent girls spent more time out of the house with their friends than in Ghana where girls were more influenced by parents who exercised greater control over their movement. This made reaching adolescent girls through social media challenging in Ghana and shifted the project’s focus towards school-based activities. There was also a more direct attempt to keep parents in Ghana informed and to secure their support for SKY.

The Ghanaian experience helped intervention designers to modify their initial ToC. Figure 2 shows the modified ToC of the SKY Girls tobacco control intervention. The modified ToC included a greater emphasis on the risk associated with certain types of tobacco, notably shisha. The outcome measures were tightened and rationalized in the new version of the ToC. Finally, the modified ToC took the desired behavior, non-use of tobacco by teenage girls, explicitly into account.

The feedback loop established in Ghana – a combination of the use of phone survey data for programmatic purposes and feedback gathered through the implementers’ regular interaction with the audience through in-person and online interactions - enabled GB to implement a different intervention than what they would have implemented in the absence of insights about the consumer.

A key feature of the feedback loop was the close involvement of the donor at all stages of project design, implementation and evaluation. Decisions could be taken quickly since the donor understood the types of issues that came up on the implementation and the evaluation side. This allowed implementers to shift to new channels and strategies that were more suitable to the Ghanaian context. The creation of free full-length movies is one such adaptation; the movies spoke directly to the Ghanaian teen audience, access and screening were designed around community needs, and the channel worked well with the evaluation design – distribution could be closely managed to minimize contamination of the control, non-project areas. Evaluators appreciated that the donor team did not micromanage the project, enabling them to have flexibility to make changes to the surveys when needed and possible. They felt that the inclusion of an evaluation expert on the donor team made it easier for them to provide the rationale for their analyses and survey design choices to implementers and for the intent-to-treat approach used for analyses of impact using the baseline and endline household surveys.

No data are associated with this article.