Performance monitoring and accountability: The Agile Project’s protocol, record and experience

Urban settings for PMA Agile

Cities have become home to growing, underserved poor communities. More than half of the world’s population currently lives in cities and this urbanization is accelerating to 70 percent by 2050, especially in Africa and Asia. Cities benefit from economic growth but their governments struggle to accommodate rising demands for services. Reaching urban women and girls with reproductive health services has become a social welfare imperative.

Cities also offer the advantages of spatial proximity to health services with transportation systems, population density, and structural networks among individuals and institutions to support a private health sector that ranges in type from retail kiosks and pharmacies dispensing over the counter medications to specialist doctors and hospitals that cater to more elite client needs. To monitor the implementation of any large-scale health initiative aimed at reducing urban disparities and improving health equity requires gathering information on the private, as well as public, health sectors. Social marketing projects largely target urban populations to reach a market that enables subsidizing commodities for the poor. They function by providing a range of high-quality, affordable, and novel brands of contraceptive and sexual health products to the market through well established distribution and supply chain mechanisms.

Selection of PMA Agile sites

PMA Agile activities officially began November 2016 with the first set of five country and 12 city geographies decided in July 2017: Burkina Faso (Ouagadougou and Koudougou); Democratic Republic of Congo (DRC) (Kinshasa); India (Ferozabad, Indore, Puri and Shikohadbad/Tundla cities); Kenya (Kericho, Migori and Uasin Gishu counties); and Nigeria (Lagos, Ogun and Kano). PMA Agile site location was based on collaborating with existing PMA implementing partners and considerations of local intervention activities, government health administration interest and willingness to act on results and input from Gates Foundation staff. Capitals of two Francophonic countries (Abidjan in Cote d’Ivoire and Niamey in Niger) were held for future consideration, with Niamey subsequently added in early 2019 as the 14^th PMA Agile site.

Implementing partners (IPs)

IPs’ capacity and connections with local stakeholders were key for the successful dissemination and actionability of PMA Agile results. The implementing partners were the Center for Research, Evaluation Resources and Development and the University of Ibadan for Nigeria, the Indian Institute for Health Management Research in India, the Institut National de Statistiques of the Government of Niger, the Institut Superieur de Sciences de la Population in Burkina Faso, the International Center for Reproductive Health in Kenya, and the University of Kinshasa School of Public Health for DRC.

Data elements

PMA Agile’s data system has four main elements: a baseline and quarterly follow-up health facility survey, a semi-annual client exit interview survey of male and female clients, a follow-up phone interview of consenting female clients, and a youth survey based on respondent-driven sampling. Table 1 provides an overview of each element’s purpose, mode of administration, sample, eligibility criteria, target sample size and periodicity³.

Health facility or Service Delivery Point (SDP)⁴

Respondents for the SDP questionnaire were primarily the in-charge/manager of the health facility; however, once the respondent has given consent for the SDP to participate, other personnel at the facility occasionally contributed answers based on expertise/knowledge of the subject matter. These other respondents may include medical staff, pharmacists or accountants. All SDPs that participate in the baseline SDP survey become eligible for subsequent quarterly follow-up surveys.

The size of the SDP sample was determined using the proportion that provides three or more contraceptive methods. In Kenya, the first PMA Agile site, this proportion was 77% of SDPs based on data from five earlier rounds of PMA surveys. With 80% power, alpha of 0.05, and allowing for a 5.5% margin of error, the required simple random sample size was 204 health facilities. After allowing for 15% non-response, the sample size for SDPs was fixed at 220 across all Agile sites and evenly divided into 110 public and 110 private facilities. Lists of registered public and private health providers were obtained from relevant official authorities. The lists included facility names, type of facility and addresses. The facilities were stratified by public and private and then the proportionate distribution of facility type was calculated. The 110 facilities in each sector were then randomly selected. If a site had fewer than 110 facilities, all facilities in that sector were selected to be surveyed⁵. This panel of SDPs was then visited quarterly for follow-up surveys. Preliminary field checks were made to assess the accuracy of the lists but more often, if a sampled facility at baseline was found to be non-existent, closed or transformed into another type of facility, it was replaced with another facility of the original type drawn from the list.

Mobile-phone based survey forms, akin to those used by PMA (see Zimmerman et al., 2017), were developed to consent and interview the in-charge or owner of the health facility on a quarterly basis. The baseline questionnaire or form is about 30 minutes in duration with the quarterly follow up about 15–20 minutes. Consent rates for baseline and retention rates for continued quarterly survey participation have been relatively high across sites, as will be seen below. Incentives were not given to SDP survey participants. However, in Nigeria, retention of the participation of private health facilities over time required providing an additional incentive (a PMA Agile-branded wall clock).

Client exit interviews

The CEI survey was aimed at capturing the service experiences of adults seeking ambulatory health care. It targeted interviews with 10 clients per sampled facility. This number is based on a sample power calculation using a modern contraceptive prevalence of 50%, assumed to be fixed across all Agile sites, a margin of error of 3% and design effect of 2. This resulted in a sample size of 2106 clients which divided by 220 health facilities resulted in 10 clients per facility.

Eligibility criteria for the CEIs were: female clients 18 to 49 years old or male clients age 18 to 59 years. Clients were recruited systematically or sequentially by the field interviewer (known as the resident enumerator or RE) as they exit the sampled SDPs over the course of one or two interview days. The RE was provided the average daily client volume for the SDP, obtained during the baseline survey and a sampling interval. For example, if the SDP saw an average 150 clients per day, the RE was given a sampling interval of 15 to select 10 clients. The RE used a random start number between 1 and 15 and began recruitment with the Nth client who exited. REs worked in pairs at large health facilities, such as hospitals, and also position themselves at the outpatient and primary care clinics for survey recruitment. At small facilities, such as pharmacies, the same systematic selection procedures were followed, and REs could work in pairs depending on client volume. CEIs were generally completed outside the pharmacies.

Clients were approached to participate after they received or attempted to receive care from the SDP. Trained team enumerators introduced themselves, explained the Agile survey to clients and consented the client to participate. Clients consenting and completing the survey were provided with $1 equivalent in cell phone airtime or offered a material good of equivalent value as compensation for their time.

The CEI was approximately 20 minutes in length and collected information on client experience and satisfaction with the health site’s services, with family planning content prioritized. The CEI was fielded in the second and fourth quarterly surveys (Q2 and Q4) each year, or two times over a 12-month period. Participation rates were above 50% in all settings and ranged from 4% non-consent in Kenya to 35% non-consent in Nigeria among female clients (data not shown).

CEI Follow-Up

To assess contraceptive practice, only female clients were recruited for the CEI phone follow-up survey. Upon completion of the CEI, the female client was asked if she was willing to participate in a follow-up interview to occur in approximately 4 months. If she consented, she was asked for a primary and secondary phone number (cell or landline) at which she could be reached. Often female clients provided their male partners’ phone numbers and the script used at the beginning of the call was general to avoid disclosing any confidential health behavior.

The phone follow-up interview asked about the female’s adoption (among those who were not using a method at the time of the CEI) and continued use or switching of contraceptive methods and continued satisfaction with the health facility visited. The four-month interval was selected to enable re-supply of short-term methods such as the three-month injectable and to optimize on retention of the client sample. In the absence of much published literature on participation rates for follow-up surveys administered by telephone in developing country settings, it was expected that approximately half of the client sample would be female and that half would consent to the phone follow-up, leading to approximately 500 clients re-interviewed. In actuality, the average proportion of CEIs with females across the 14 sites and all rounds was 64.7% and was highest in Niamey, Niger at 92.5% and lowest at 32.1% in Puri, India. Follow-up participation rates ranged from 37.3% in Shikohadbad-Tundla, India to 96.6% in Migori, Kenya, with an average of 70.2%.

The RE team set aside specific days to conduct the CEI follow-up in a project office. They were provided their individual list of consenting females, typically ones they had interviewed themselves, and phone numbers to call. Direct touch-dialing enabled the RE to avoid having to enter (or mis-enter) the client’s phone number. The relatively high retention rates across sites is undergoing analysis of the underlying factors. One related factor appears to be recognition by the female client of the RE who originally interviewed her and thus willingness to be re-interviewed.

Youth RDS Survey

The Youth RDS Survey (YRDSS) was borne out of a need to measure contraceptive awareness, procurement and use among urban adolescents and youth as they enter a period of probable sexual activity. The target sample was unmarried female and male youth ages 15–24 years. Capturing this information from youth clients at health facilities, especially unmarried females, was likely to be biased due to social and familial sanctions on sexual activity and contraceptive use. In this age group, it is suspected that youth may be procuring contraceptives via other means, making their use effectively “hidden” to clinic staff and compromising the accuracy of clinic-based survey measures. Their sexual partners, relatives or other adults may be assisting with procurement. As a set of special studies, PMA Agile collaborated with youth-serving organizations in Abidjan and Nairobi, and a third has recently been launched in Lagos, to survey unmarried youth using respondent-driven sampling (RDS) methodology. This sampling method, which can be adjusted post-enumeration to weight to a known population distribution, takes advantage of youth networks for rapid recruitment and reach into diverse communities.

The sample sizes were powered on the estimated modern contraceptive prevalence level for unmarried females 15–24 years obtained from the most recent PMA2020 survey. “Seed” respondents recruit three additional respondents, who each recruit another three until the desired sample size and gender balance, which was monitored daily, was reached. The survey was self-administered on a tablet, with an attendant RE available to guide the respondent’s use. The findings were disseminated in country once the technical report and briefs on selected topics were produced⁶. All Agile questionnaires were translated (and back-translated) into the local languages when required.

Recruitment of resident enumerators

Desired attributes of resident enumerators include: completion of secondary school, English or French literacy, local language fluency, residence in the selected city, a minimum of 21 years of age, not a paid health worker, not a health activist, no physical restrictions in conducting fieldwork, familiarity/experience with cell/smart phones, and personal awareness and support of family planning as a health intervention. In addition, preferred personal traits include maturity, self-confidence, dependability, trustworthiness, ability to protect confidentiality and respondent privacy, and social interaction skills. Recruited REs receive one week of hands-on intensive training, a smartphone and airtime.

RE/field supervisor teams

Agile field teams were composed of six to eight interviewers and one or two field supervisors. Each city had one field team. The field supervisor assisted in the baseline selection and recruitment of SDPs to participate in the surveys. S/he also supported and oversaw the systematic sampling of clients at SDPs and their follow-up phone interviews. Each interviewer was assigned approximately 25–35 SDPs to interview each quarter depending on the geography and conducted 250–350 CEIs and another 150–200 phone interviews every six months. Field staff were salaried and retained for the entire Agile data collection period.

Mobile phone data collection and transmittal

The collection of SDP and client data was completed with a smart mobile phone. All countries except Nigeria used JHU collect forked from the ODK collect community version 1.4.8 for the first two quarters of data collection. For all subsequent quarters of data collection, countries downloaded the latest version of the community ODK collect application as available on the Android PlayStore, prior to data collection for each quarter. Community ODK Collect versions used for data collection ranged from v1.17.1 to v1.23.3. Nigeria used the available latest versions of the application Survey CTO ranging from v2.40 to v2.60 through their 6 quarters of data collection. Nigeria also used community ODK collect v.1.17.1 and v1.25.1 to leverage its dialer app feature for the phone follow-up surveys conducted in the their 3rd and 5th quarter of data collection, respectively. Each RE was provided a basic smartphone with good functionality in Android OS (level 4.1 or higher) with adequate memory and GPS receiver having 6-meter accuracy. The smart phone had the enumeration templates to be used to record the information for each SDP. The RE uploaded each case record from the interviews to a secure cloud-based server after the interview was completed. If there was no network reception at the end of the interview, the RE stored the interview on the phone until she reached network availability and then transmitted the record to the server.

Data are initially stored on a Google Cloud Server with access retained only by designated members of the data management and PI team. Data are downloaded off the cloud server daily to a secure server maintained either by the partner institution or the Agile project at Johns Hopkins University (JHU). Once data collection within a round was finished, all data were deleted from the cloud server and maintained only within the in-country partner’s and JHU’s systems.

Figure 1 illustrates the data collection and transmission cycle. A quarterly cycle can take between 11 to 17 weeks.

Figure 1. Schematic of PMA agile data collection, transmission, archiving flows.

Implementation schedule

Figure 2 provides an overview of the surveys implemented in each Agile site by year, and then in Table 3 by round and coverage of SDPs, CEIs and CEI follow-ups. The estimated population of each Agile city is also provided for context.

Figure 2. PMA Agile Data Collection Schedule.

Data cleaning and quality monitoring

Use of ODK allows constraints and limiters to be built into the questionnaire minimizing entry errors. The date and time stamps from ODK and GPS coordinates allowed determination of the locations and times of data collection. Since these were monitored on real time basis, where unusual patterns are seen, messages for correction were sent to the team and corrective actions taken as needed.

Data cleaning checks occurred throughout data collection and after completion. Analytic routines (e.g., with Stata *.do files) were prepared and executed and generated data quality indicators that were reviewed further for outlier or illogical values by both the in-country survey IPs and the PMA Agile team at JHU. Table 4 illustrates for [Agile site] one routine for weekly monitoring of completion status for three types of Agile data. Data managers at IPs tracked progress toward reaching the sample targets on a daily basis and worked with field supervisors to trouble shoot as needed.

Dashboards

Once data files were cleaned for duplicate records, outlier or illogical values, or missing records, another set of analysis files generated the pre-selected core performance indicators, such as the proportion of SDPs reporting method-specific stockouts at the time of survey. The indicator metrics were integrated into a digital database, aka “dashboard”, which displayed the quarterly indicator data and trends therein. Public users could then view the performance statistics for the SDPs and clients on separate dashboards. Special dashboards with password access were prepared for the two large projects, TCI and DKT International. The dashboard was built such that participating SDPs could also access their own data using a unique ID provided to them. This would allow them to view their indicator data over time in relation to others in the sample (all with identities masked).

Figure 3A illustrates quarterly trends in one of the dashboard indicators—average number of client visits in the past month for specific contraceptive methods for urban Kericho county in Kenya among public and private SDPs (top and bottom panels respectively). Fluctuations are evident over the six quarters. Public SDPs do not dispense emergency contraception. An increasing trend in monthly client visits for injectables and higher numbers of client visits in quarters 3 and 6 for implants are visible. Tracking client visits in public facilities is indicative of demand and can be juxtaposed with stockout levels over the same quarters to assess net performance of the commodity supply chain and meeting client needs. Private consumption of contraceptives (seen in the lower panel of Figure 3A) shows more fluctuation. The average number of return client visits appears relatively stable, except for injectables, while those for new client visits is greater, especially for ECs and implants in early quarters.

Figure 3.

Sample screens from PMA Agile dashboard of quarterly average number of client visits by method for public and private health facilities (A) and contraceptive prevalence among two rounds of client exit surveys in Kericho county, Kenya. Dashboard URL for (A): https://www.pmadata.org/pma-agile-dashboard-kenya-sdp (Note that public facilities did not dispense emergency contraception at time of surveys); Dashboard URL for (B): https://www.pmadata.org/pma-agile-dashboard-kenya-cei.

Figure 3B illustrates the client indicators over two rounds of data collection in Kericho. Modern contraceptive prevalence (mCPR) is assessed among all clients as seen from the client dashboard example in Figure 3B. Although mCPR is usually measured for childbearing aged females only, the figures here are for both female and male clients, where the latter have a female partner age 15 to 49. The mCPR is 61.7% in the first round conducted in Quarter 2 and 67.8% in Quarter 4. Differences over the two quarters by client age and method mix are also shown.

The dashboards for SDPs and client indicators are publicly accessible for the 14 settings in the 6 countries at www.pmadata.org/technical-areas/pma-agile. After accessing the dashboard of interest, users can filter the indicators for each setting, public/private sector, type of facility or for clients by background characteristic (e.g., gender, age) and facility type at which the interview was conducted.

Stakeholder meetings

Agile carried out a range of dissemination activities to promote data utilization. Foremost among these were stakeholder meetings organized by the IPs usually co-sponsored with the local public health department. In addition to national and local government health officials, health staff from non-governmental and research organizations and from international projects and donor agencies were invited. An example where a close connection was forged for data utilization was the use of PMA Agile data for the family planning cost implementation plans for Kenyan counties. These stakeholder meetings also enabled local confirmation of measured and observed trends in the indicators, as was voiced in Ogun state, Nigeria, and Ouagadougou and Koudougou in Burkina Faso.

In addition to the above two main dissemination efforts, PMA Agile has produced summary briefs on SDP and client indicator trends, capturing the dashboard information, to disseminate at stakeholder meetings⁷. These have been necessary where internet connectivity is weak and helpful in expanding knowledge about the measures and their interpretation. Technical reports are prepared for the YRDS and other special studies; and as quarterly data have accumulated, has begun a series of analyses for journal publication and conference presentations. Three annual partners meetings have been held to exchange findings, best practices and lessons learned.