Facebook Powered Measurement and Evaluation for Iron Folic Acid Health Intervention in India

Target audience

Our target audience, for both the campaign and the survey, was women who were active Facebook users, aged 15 to 49 and living in the states of Madhya Pradesh and Uttar Pradesh. According to data pulled from Facebook’s Audience Insights tool, there are 6.5 million women fitting that profile in Uttar Pradesh and 2.9 million in Madhya Pradesh.

Our goal for the survey was to gather enough responses on each survey to achieve a margin of error of approximately 3%.

Table 1 shows the population of our target audience that was on Facebook at the time of the survey (provided by Facebook’s audience targeting tool), the number of survey completions by targeting method (Reach targeting and Conversion targeting as tracked in Facebook Analytics), and the resulting margin of error. Data is shown for both Madhya Pradesh and Uttar Pradesh.

The survey

The survey questions were developed in collaboration with UCSF and the University of Utah. The survey was initially developed in English and translated into Hindi with the help of Gates India and our partners at Rabbithole, an India-based marketing firm.

Once the questions were finalized, we created the surveys in Typeform, a cloud-based data collection app. We chose Typeform primarily because their forms are very well designed for mobile devices, which we knew would account for the vast majority of responses. Typeform also has Facebook Pixel and Google Analytics integration, which allowed us to gather some data about how users interacted with the form.

To separate responses by state and by targeting method (described below), we used four separate Typeform surveys. The surveys were identical, with the exception of one question about which district the respondent lived in which was state-specific Ad Targeting.

We distributed the links to our survey using ads created and delivered through Facebook’s Ad Manager. Ad Manager enabled us to target ads based on location, gender, age (among a huge variety of other variables).

On Facebook, a “campaign” can be any number of ads delivered to any number of target audiences with a specifically defined goal. Facebook then targets the distribution of your ads in a manner intended to maximize the value of your campaign. For example, if your campaign goal is “conversions,” Facebook will deliver your ads to users it believes are likely to “convert” -- in this case, click through the ad and submit a survey response. A “reach” goal, on the other hand, is optimized to display the ad to as many people as possible within your target audience.

Since Facebook’s ads algorithm targets conversion ads to people it believes are likely to convert (in our case clicking through and completing a survey), we were concerned that the audience sample would not be representative of the broader population. For this reason, we chose to test both “conversions” and “reach” in each state, for a total of four ad campaigns.

We also divided our overall audience into age group segments. Based on previous experience with other audiences around the world, we suspected that older users would be more likely to click through and complete the survey, and we wanted to test that theory. We also wanted to see if age played a role in helping to predict (or at least associated with) any of the variables. Additionally, if some age groups were underrepresented in the results, we could adjust our ad targeting to ensure we collected a meaningful sample.

Note that Facebook is not passing any information about individual users to our survey page. In fact, users did not share any personal information with us directly or through Facebook, and we cannot connect an individual user with their submission. Additionally, because we ran the survey using a third-party tool, Facebook does not see the data that users submit through the survey, and individual submissions are anonymous.

However, using Facebook’s analytics tool, which requires the implementation of a “Facebook Pixel” into the survey itself, we are able to see aggregated data about the users who clicked on our ads and submitted each survey. That allows us to make observations about our overall sample -- including age, gender, education, and location. This data should be helpful in testing the validity of our audience sample.

Ad creative

The ads themselves were created by India-based creative agency Rabbithole with UpSwell guidance. Per our usual content strategy, we set out from the start to test a variety of messages that might motivate users to complete the survey while not biasing our sample by being specific about the topic or goals. The initial creative was informed by strategies that we have developed over the years. This strategy puts a heavy emphasis on the things that motivate people to take actions on social media. In this case, the intended action was to complete the survey which is a relatively high bar. We know from past experience that people are motivated to take actions that help them define themselves on social media. We also know that people are more likely to take action when they believe doing so will help other people. A review of the top performing content, below, provides a good example of how these concepts manifested in the creative.

Retargeting and lookalike audiences

In addition to providing a way to test the validity of our audience sample, Facebook’s Pixel also provides a way to create “custom audiences” of people who have answered some or all of the questions in the survey. Facebook does not allow us to see which individuals have been added to a custom audience, but once a custom audience has enough users, Facebook allows us to target ads to the people in the audience. Additionally, custom audiences can be used to seed the development of, what Facebook calls, “Lookalike Audiences.” Lookalike Audiences use a large number of variables to identify people who are similar to the individuals who make up the initial custom audience. Although we have used Lookalike Audiences to powerful effect for other initiatives, we will need to test their effectiveness for SBCC campaigns at a future date.