Performance monitoring and accountability: The Agile Project’s protocol, record and experience

The Performance Monitoring and Accountability 2020 (PMA2020) project implemented a multi-country sub-project called PMA Agile, a system of continuous data collection for a probability sample of urban public and private health facilities and their clients that began November 2017 and concluded December 2019. The objective was to monitor the supply, quality and consumption of family planning services. In total, across 14 urban settings, nearly 2300 health facilities were surveyed three to six times in two years and a total sample of 48,610 female and male clients of childbearing age were interviewed in Burkina Faso, Democratic Republic of Congo, India, Kenya, Niger and Nigeria. Consenting female clients with access to a cellphone were re-interviewed by telephone after four months; two rounds of the client exit, and follow-up interviews were conducted in nearly all settings. This paper reports on the PMA Agile data system protocols, coverage and early experiences. An online dashboard is publicly accessible, analyses of measured trends are underway, and the data are publicly available.


Introduction
When monitoring and evaluating (M&E) the performance, progress and impact of large-scale population-level interventions, the standard practice in developing country settings has been to rely on cross sectional surveys conducted by third parties usually at the beginning and the end of the project to provide information on changes in outcomes of interest.These specialized surveys are usually extensive in scope and rich in detail and instrumental for global monitoring (Boerma & Sommerfelt, 1993), but their deployment is also resource-and time-intensive.When findings from these surveys become available, their dissemination is often much later than needed to modify the design or continued implementation of the project.Moreover, it is often necessary to relate the findings with contextual information from other data sources in order to gain insights on why and how the project succeeded or not.In short, the traditional M&E approach is not designed for tracking and acting on performance results on an ongoing basis (Nordberg, 1988;Rowe, 2009).
At the same time, health information systems, while increasingly digitized, are constrained in the types of measures available, their selective coverage of facilities and clients, accuracy of gathered data, and timeliness of reporting.Efforts to eliminate these deficiencies are growing, especially to address prevention and control of large-scale epidemics of infectious diseases (Braa et al., 2007;Walsham & Sahay, 2006).The lack of systematic and programmatically relevant, continuous and timely information available at subnational levels, however, has posed a formidable challenge to nimble and effective project decision-making and response (see Guenther et al., 2014;Maina et al., 2017).PMA Agile was designed to move away from the traditional M&E approach by establishing a near-continuous monitoring system that collects, links and aggregates data at different levels on a focused set of indicators in a cost-effective manner.The Agile system was developed to reduce the lag time between steps in a project learning process: recognizing a project's information needs, identifying sensitive performance indicators, collecting relevant primary and secondary data, analyzing the collected data, producing actionable insights, and enabling the use of the insights to adjust and fine-tune programs.
PMA Agile evolved out of a combined interest of the research and evaluation staff at the Bill and Melinda Gates Foundation and the PMA 1 project at the Johns Hopkins Bloomberg School of Public Health to develop an innovative data system that could track performance of two large projects 2 at the subnational level, in this case urban areas and their poor.In Agile's first year, the selection of urban geographies for monitoring was dependent on these two projects' own plans for locating their program resources.It later became clear that Agile would need to proceed more independently with geographic selection, than originally planned in order to realize its objectives in the awarded project period.The eventual selection of Agile geographies did not affect its design or four objectives which are to: 1. Provide a flexible, continuous and cost-effective data collection system that can triangulate with routine and other survey information; 2. Serve as an adaptable, replicable M&E platform for program implementation addressing needs of the urban poor; 3. Measure core indicators that reflect program performance at the health facility and client levels; 4. Promote actionable findings to enable evidence-based decisions by government officials, non-government stakeholders and researchers.
This paper describes the protocols, record and experiences, to date, of PMA Agile to accompany the findings that are already available on public dashboards.Because the data will become publicly available, this description provides important background to inform current and future users.

Protocol
Urban settings for PMA Agile Cities have become home to growing, underserved poor communities.More than half of the world's population currently lives in cities and this urbanization is accelerating to 70 percent by 2050, especially in Africa and Asia (United Nations, 2019).Cities benefit from economic growth but their governments struggle to accommodate rising demands for services.Reaching urban women and girls with reproductive health services has become a social welfare imperative.
1 Agile retains the core innovation of PMA (formerly PMA2020), where women are recruited from or near the selected enumeration area and trained to collect data using smartphones on a repeated and quick-turnaround basis (Zimmerman et al., 2017).
2 At the time, the first of the two projects was these were The Challenge Initiative, and urban-focused family planning initiative, located in the Bill & Melinda Gates Institute for Population and Reproductive Health at Johns Hopkins Bloomberg School of Public Health (tciurbanhealth.org), the second and was an expansion social marketing project implemented by DKT International (dktinternational.org).Both foundation investments were initiated in 2016.

Amendments from Version 1
We have updated our manuscript to address comments from the reviewers.The most notable changes include the addition of a section on research from this project, numbers for each IRB, and other clarifying edits.
Any further responses from the reviewers can be found at the end of the article Cities also offer the advantages of spatial proximity to health services with transportation systems, population density, and structural networks among individuals and institutions to support a private health sector that ranges in type from retail kiosks and pharmacies dispensing over the counter medications to specialist doctors and hospitals that cater to more elite client needs.To monitor the implementation of any large-scale health initiative aimed at reducing urban disparities and improving health equity requires gathering information on the private, as well as public, health sectors.Social marketing projects largely target urban populations to reach a market that enables subsidizing commodities for the poor.

Sample selection and size
Health facility or Service Delivery Point (SDP) 4 Respondents for the SDP questionnaire were primarily the in-charge/manager of the health facility; however, once the respondent has given consent for the SDP to participate, other Nairobi -1300 Lagos -1300

One time
3 Questionnaires for each can be accessed at https://pmadata.org/technical-areas/pma-agile. 4 We use health facility and SDP interchangeably.
personnel at the facility occasionally contributed answers based on expertise/knowledge of the subject matter.These other respondents may include medical staff, pharmacists or accountants.All SDPs that participate in the baseline SDP survey become eligible for subsequent quarterly follow-up surveys.
The size of the SDP sample was determined using the proportion that provides three or more contraceptive methods.
In Kenya, the first PMA Agile site, this proportion was 77% of SDPs based on data from five earlier rounds of PMA surveys.With 80% power, alpha of 0.05, and allowing for a 5.5% margin of error, the required simple random sample size was 204 health facilities.After allowing for 15% non-response, the sample size for SDPs was fixed at 220 across all Agile sites and evenly divided into 110 public and 110 private facilities.Lists of registered public and private health providers were obtained from relevant official authorities.The lists included facility names, type of facility and addresses.The facilities were stratified by public and private and then the proportionate distribution of facility type was calculated.The 110 facilities in each sector were then randomly selected.If a site had fewer than 110 facilities, all facilities in that sector were selected to be surveyed5 .This panel of SDPs was then visited quarterly for follow-up surveys.Preliminary field checks were made to assess the accuracy of the lists but more often, if a sampled facility at baseline was found to be non-existent, closed or transformed into another type of facility, it was replaced with another facility of the original type drawn from the list.
Mobile-phone based survey forms, akin to those used by PMA (see Zimmerman et al., 2017), were developed to consent and interview the in-charge or owner of the health facility on a quarterly basis.The baseline questionnaire or form is about 30 minutes in duration with the quarterly follow up about 15-20 minutes.Consent rates for baseline and retention rates for continued quarterly survey participation have been relatively high across sites, as will be seen below.
Incentives were not given to SDP survey participants.However, in Nigeria, retention of the participation of private health facilities over time required providing an additional incentive (a PMA Agile-branded wall clock).

Client exit interviews
The CEI survey was aimed at capturing the service experiences of adults seeking ambulatory health care.It targeted interviews with 10 clients per sampled facility.This number is based on a sample power calculation using a modern contraceptive prevalence of 50%, assumed to be fixed across all Agile sites, a margin of error of 3% and design effect of 2. This resulted in a sample size of 2106 clients which divided by 220 health facilities resulted in 10 clients per facility.
Eligibility criteria for the CEIs were: female clients 18 to 49 years old or male clients age 18 to 59 years.Clients were recruited systematically or sequentially by the field interviewer (known as the resident enumerator or RE) as they exit the sampled SDPs over the course of one or two interview days.The RE was provided the average daily client volume for the SDP, obtained during the baseline survey and a sampling interval.For example, if the SDP saw an average 150 clients per day, the RE was given a sampling interval of 15 to select 10 clients.The RE used a random start number between 1 and 15 and began recruitment with the Nth client who exited.REs worked in pairs at large health facilities, such as hospitals, and also position themselves at the outpatient and primary care clinics for survey recruitment.At small facilities, such as pharmacies, the same systematic selection procedures were followed, and REs could work in pairs depending on client volume.
CEIs were generally completed outside the pharmacies.
Clients were approached to participate after they received or attempted to receive care from the SDP.Trained team enumerators introduced themselves, explained the Agile survey to clients and consented the client to participate.Clients consenting and completing the survey were provided with $1 equivalent in cell phone airtime or offered a material good of equivalent value as compensation for their time.
The CEI was approximately 20 minutes in length and collected information on client experience and satisfaction with the health site's services, with family planning content prioritized.The CEI was fielded in the second and fourth quarterly surveys (Q2 and Q4) each year, or two times over a 12-month period.Participation rates (the percentage of clients consenting to be interviewed) were above 50% in all settings and ranged from 4% non-consent in Kenya to 35% non-consent in Nigeria among female clients (data not shown).

CEI Follow-Up
To assess contraceptive practice, only female clients were recruited for the CEI phone follow-up survey.Upon completion of the CEI, the female client was asked if she was willing to participate in a follow-up interview to occur in approximately 4 months.If she consented, she was asked for a primary and secondary phone number (cell or landline) at which she could be reached.Often female clients provided their male partners' phone numbers and the script used at the beginning of the call was general to avoid disclosing any confidential health behavior.
The phone follow-up interview asked about the female's adoption (among those who were not using a method at the time of the CEI) and continued use or switching of contraceptive methods and continued satisfaction with the health facility visited.The four-month interval was selected to enable re-supply of short-term methods such as the three-month injectable and to optimize on retention of the client sample.
In the absence of much published literature on participation rates for follow-up surveys administered by telephone in developing country settings, it was expected that approximately half of the client sample would be female and that half would consent to the phone follow-up, leading to approximately 500 clients re-interviewed.In actuality, the average proportion of CEIs with females across the 14 sites and all rounds was 64.7% and was highest in Niamey, Niger at 92.5% and lowest at 32.1% in Puri, India.Follow-up participation rates ranged from 37.3% in Shikohadbad-Tundla, India to 96.6% in Migori, Kenya, with an average of 70.2%.
The RE team set aside specific days to conduct the CEI follow-up in a project office.They were provided their individual list of consenting females, typically ones they had interviewed themselves, and phone numbers to call.Direct touch-dialing enabled the RE to avoid having to enter (or mis-enter) the client's phone number.The relatively high retention rates across sites is undergoing analysis of the underlying factors.One related factor appears to be recognition by the female client of the RE who originally interviewed her and thus willingness to be re-interviewed.

Youth RDS Survey
The Youth RDS Survey (YRDSS) was borne out of a need to measure contraceptive awareness, procurement and use among urban adolescents and youth as they enter a period of probable sexual activity.The target sample was unmarried female and male youth ages 15-24 years.Capturing this information from youth clients at health facilities, especially unmarried females, was likely to be biased due to social and familial sanctions on sexual activity and contraceptive use.In this age group, it is suspected that youth may be procuring contraceptives via other means, making their use effectively "hidden" to clinic staff and compromising the accuracy of clinic-based survey measures.Their sexual partners, relatives or other adults may be assisting with procurement.As a set of special studies, PMA Agile collaborated with youth-serving organizations in Abidjan and Nairobi, and a third has recently been launched in Lagos, to survey unmarried youth using respondent-driven sampling (RDS) methodology.This sampling method, which can be adjusted post-enumeration to weight to a known population distribution, takes advantage of youth networks for rapid recruitment and reach into diverse communities.
The sample sizes were powered on the estimated modern contraceptive prevalence level for unmarried females 15-24 years obtained from the most recent PMA2020 survey."Seed" respondents recruit three additional respondents, who each recruit another three until the desired sample size and gender balance, which was monitored daily, was reached.The survey was self-administered on a tablet, with an attendant RE available to guide the respondent's use.The findings were disseminated in country once the technical report and briefs on selected topics were produced6 .All Agile questionnaires were translated (and back-translated) into the local languages when required.

Outcomes measured
PMA Agile is indicator-driven, i.e., it measures the core indicators in the service supply, quality and consumer demand

Training and data collection
Recruitment of resident enumerators Desired attributes of resident enumerators include: completion of secondary school, English or French literacy, local language fluency, residence in the selected city, a minimum of 21 years of age, not a paid health worker, not a health activist, no physical restrictions in conducting fieldwork, familiarity/ experience with cell/smart phones, and personal awareness and support of family planning as a health intervention.In addition, preferred personal traits include maturity, self-confidence, dependability, trustworthiness, ability to protect confidentiality and respondent privacy, and social interaction skills.Recruited REs receive one week of hands-on intensive training, a smartphone and airtime.

RE/field supervisor teams
Agile field teams were composed of six to eight interviewers and one or two field supervisors.Each city had one field team.The field supervisor assisted in the baseline selection and recruitment of SDPs to participate in the surveys.S/he also supported and oversaw the systematic sampling of clients at SDPs and their follow-up phone interviews.Each interviewer was assigned approximately 25-35 SDPs to interview each quarter depending on the geography and conducted 250-350 CEIs and another 150-200 phone interviews every six months.Field staff were salaried and retained for the entire Agile data collection period.

Mobile phone data collection and transmittal
The collection of SDP and client data was completed with a smart mobile phone.All countries except Nigeria used JHU collect forked from the ODK collect community version 1.4.8 for the first two quarters of data collection.For all subsequent quarters of data collection, countries downloaded the latest version of the community ODK collect application as available on the Android PlayStore, prior to data collection for each quarter.Community ODK Collect versions used for data collection ranged from v1.17.1 to v1.23.3.Nigeria used the available latest versions of the application Survey CTO ranging from v2.40 to v2.60 through their 6 quarters of data collection.Nigeria also used community ODK collect v.1.17.1 and v1.25.1 to leverage its dialer app feature for the phone follow-up surveys conducted in the their 3rd and 5th quarter of data collection, respectively.Each RE was provided a basic smartphone with good functionality in Android OS (level 4.1 or higher) with adequate memory and GPS receiver having 6-meter accuracy.The smart phone had the enumeration templates to be used to record the information for each SDP.The RE uploaded each case record from the interviews to a secure cloud-based server after the interview was completed.
If there was no network reception at the end of the interview, the RE stored the interview on the phone until she reached network availability and then transmitted the record to the server.
Data are initially stored on a Google Cloud Server with access retained only by designated members of the data management and PI team.Data are downloaded off the cloud server daily to a secure server maintained either by the partner institution or the Agile project at Johns Hopkins University (JHU).Once data collection within a round was finished, all data were deleted from the cloud server and maintained only within the in-country partner's and JHU's systems.
Figure 1 illustrates the data collection and transmission cycle.
A quarterly cycle can take between 11 to 17 weeks.

Implementation schedule
Figure 2 provides an overview of the surveys implemented in each Agile site by year, and then in Table 3 by round and coverage of SDPs, CEIs and CEI follow-ups.The estimated population of each Agile city is also provided for context.

Data quality monitoring, cleaning, preparation for analysis
Data cleaning and quality monitoring Use of ODK allows constraints and limiters to be built into the questionnaire minimizing entry errors.The date and time stamps from ODK and GPS coordinates allowed determination of the locations and times of data collection.Since these were monitored on real time basis, where unusual patterns are seen,   messages for correction were sent to the team and corrective actions taken as needed.
Data cleaning checks occurred throughout data collection and after completion.Analytic routines (e.g., with Stata *.do files) were prepared and executed and generated data quality indicators that were reviewed further for outlier or illogical values by both the in-country survey IPs and the PMA Agile team at JHU. Table 4 illustrates for [Agile site] one routine for weekly monitoring of completion status for three types of Agile data.Data managers at IPs tracked progress toward reaching the sample targets on a daily basis and worked with field supervisors to trouble shoot as needed.

Data analysis Dashboards
Once data files were cleaned for duplicate records, outlier or illogical values, or missing records, another set of analysis files generated the pre-selected core performance indicators, such as the proportion of SDPs reporting method-specific stockouts at the time of survey.The indicator metrics were integrated into a digital database, aka "dashboard", which displayed the quarterly indicator data and trends therein.Public users could then view the performance statistics for the SDPs and clients on separate dashboards.Special dashboards with password access were prepared for the two large projects, TCI and DKT International.The dashboard was built such that participating SDPs could also access their own data using a unique ID provided to them.This would allow them to view their indicator data over time in relation to others in the sample (all with identities masked).
Figure 3A illustrates quarterly trends in one of the dashboard indicators-average number of client visits in the past month for specific contraceptive methods for urban Kericho county in Kenya among public and private SDPs (top and bottom panels respectively).Fluctuations are evident over the six quarters.Public SDPs do not dispense emergency contraception.An increasing trend in monthly client visits for injectables and higher numbers of client visits in quarters 3 and 6 for implants are visible.Tracking client visits in public facilities is indicative of demand and can be juxtaposed with stockout levels over the same quarters to assess net performance of the commodity supply chain and meeting client needs.Private consumption of contraceptives (seen in the lower panel of Figure 3A) shows more fluctuation.The average number of return

Data dissemination
Table 5 provides an overview of PMA Agile's dissemination activities, which are described herein.

Stakeholder meetings
Agile carried out a range of dissemination activities to promote data utilization.Foremost among these were stakeholder meetings organized by the IPs usually co-sponsored with the local public health department.In addition to national and local government health officials, health staff from non-governmental and research organizations and from international projects and donor agencies were invited.An example where a close connection was forged for data utilization was the use of PMA Agile data for the family planning cost implementation plans for Kenyan counties.These stakeholder meetings also enabled local confirmation of measured and observed trends in the indicators, as was voiced in Ogun state, Nigeria, and Ouagadougou and Koudougou in Burkina Faso.
In addition to the above two main dissemination efforts, PMA Agile has produced summary briefs on SDP and client indicator trends, capturing the dashboard information, to disseminate at stakeholder meetings 7 .These have been necessary where internet connectivity is weak and helpful in expanding knowledge about the measures and their interpretation.Technical reports are prepared for the YRDS and other special studies; and as quarterly data have accumulated, has begun a series of analyses for journal publication and conference presentations.Three annual partners meetings have been held to exchange findings, best practices and lessons learned.

Research
PMA Agile is well-positioned to contribute to research in several areas.In particular, the project is examining topics such as:trends in stockouts among SDPs; individual and SDP factors associated with contraceptive adoption and discontinuation; the consistency of self-reported contraceptive use over time; characteristics associated with changes in contraceptive use involving traditional methods; constructing a client-based quality of care index; response patterns in CEI and CEI/Follow-up; and hidden contraceptive behaviors of youth.

Cost
The costs of externally sponsored survey data collection efforts are often difficult to obtain and may not cover the same set of 7 An example of the Burkina Faso SDP brief can be accessed at https://www.pmadata.org/sites/default/files/data_product_results/PMA%20Agile-BF-Ouagadougou-SDP-French2.pdf.A client indicator brief is also available in English and French as well.The total PMA Agile award from the sole funding agency, The Bill & Melinda Gates Foundation, was for $4,993,285 including indirect costs of 10% for the grant period November 15, 2016 to May 31, 2020.Subawards were budgeted at $2,743,626 (63% of total direct costs).The YRDS studies ranged from $142,676 to $220,987 over the three sites and are not included in the unit cost calculations.

Discussion
Establishing and maintaining a well-functioning M&E system that routinely collects data on the supply chain systems and management (Mukasa et al., 2017) to report on the performance of public and private family planning programs plays a critical role in addressing gaps in access to contraceptive information and services in low-and middle-income countries (LMICs).In this paper, we described the design, organization and implementation of a reliable and standardized M&E system that regularly collected, linked and aggregated data at different levels on a focused set of indicators in a cost-effective manner.Facility and client data can be linked, enabling an appreciation of the consumer environment (see Ahmed et al., 2014;Larson et al., 2019).The system has made a needed contribution in rapidly producing survey estimates of the indicators at a sub-national level using mobile phone technology and a dedicated small team of enumerators and supervisors.
The PMA Agile platform demonstrated it is possible to regularly collect data on over 2300 Service Delivery Points (both public and private) to track client volume and commodity It provided valuable learning tools for health workers and less expensive means for program managers to obtain local and actionable information to improve city services.At stakeholder dissemination events, local public health providers and officials often confirmed the results' profiles as aligning with their own perceptions.
Despite its innovations and strengths, PMA Agile also encountered implementation challenges.In several geographies, the local teams had to resolve issues related to incomplete master SDP lists.Facilities were found to be not operational, closed, or their addresses had changed by the time of the survey.In a few settings the distinction between a public or private SDP was blurred in practice.The systematic sampling of clients in advanced facilities often required a second enumerator, where client volume could slow completion of interviews and where otherwise casual interviewing could incur other types of information bias, selection bias and sampling error (Eisele et al., 2013).One critical perspective missing from PMA Agile's provision of a total assessment of the health system's performance in family planning is that of providers.Their interactions with clients, counseling skills and technical competence are important to evaluate (Hutchinson et al., 2011;Solo & Festin, 2019) and could be added to the platform on a regular basis.Resources permitting, this addition could be a useful means to assess human resource needs.
The PMA Agile platform was designed to be replicable, expandable and adaptable obtaining data to scale with the potential to be linked with population, spatial, administrative and other types of information for district-level planning.It has been implemented following standardized protocols with strict quality control across all aspects of sample selection, data collection, analysis and dissemination.Ideally external data systems should complement publicly established ones and not duplicate effort or require new resources.However, in the case of family planning, the quality of LMIC health information systems data have typically been weak and confined to government facilities.Since considerable contraceptive care is obtained from private providers, neglecting measurement of this sector's contribution can significantly bias the understanding where access gaps exist.The PMA Agile platform can also support implementation research and as such, its potential will hopefully be exploited in the coming years.

Data availability
De-identified data from PMA Agile are publicly available from each individual country.To request PMA Agile data, please email the relevant country-specific address: Burkina Faso Agile Data Request burkinafaso.agile.data@pma2020.org;DRC Agile Data Request drc.agile.data@pma2020.org;India Agile Data Request india.agile.data@pma2020.org;Kenya Agile Data Request kenya.agile.data@pma2020.org;Niger Agile Data Request niger.agile.data@pma2020.org;Nigeria Agile Data Request nigeria.agile.data@pma2020.org.
There are no restrictions on who can apply to access the data.Those interested in using the data will be asked to complete a form that includes the purpose of the analysis, and confirmation of various data use considerations.*Baseline is defined as the first quarter each respective survey was implemented.For SDPs (Q1), for CEI baseline (Q2), for CEI follow-up (Q3).†SDP units lost to follow-up are included in the quarterly and total average costs.‡Q1 costs include start-up/prep costs sales and stock outs.It interviewed nearly 34,000 female and 16,000 male clients of childbearing age irrespective of their current contracepting status and reached over 18,700 of the female clients for follow-up phone interviews, across 14 urban settings in six countries, Burkina Faso, DRC, India, Kenya, Niger and Nigeria and within a 26-month timeframe.
The platform also rapidly posted indicator data on 14 publicly accessible dashboards at the SDP and client levels for stakeholders to view and monitor program progress.To understand procurement of contraceptive supplies by unmarried young persons, three youth respondent-driven sampling surveys in three major cities were also conducted.By design, the PMA Agile system leveraged stakeholder engagement early in planning, implementing and monitoring family planning services.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Version 1
Reviewer Report 03 June 2020 https://doi.org/10.21956/gatesopenres.14298.r28828 © 2020 Naravage W. This is an open access peer review report distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Wanapa Naravage
Faculty of Public Health, Thammasat University, Khlong Luang, Pathumthani, Thailand I am pleased to provide my comments for the study entitled "Performance monitoring and accountability: The Agile Project's protocol, record and experience" as follows.In general, the study by Angelwicz et al. give an interesting multi-site study on the performance monitoring and accountability (PMA).The main contribution of the paper offers an excellent opportunity for stakeholders for tracking and acting on performance results on an ongoing of the two large projects.Also, the protocol provides enough details for reader to repeat the PMA.However, I have some additional comments about the paper listed below: Introduction: On page 3 of 19, the last paragraph on the left, it is stated that "...innovative data system that could track performance of two large projects…" I would recommend that the paper should provide more information on the key inputs of family planning innovations as it will allow replication by others if they have the similar interventions.In addition, it would be helpful if the paper provides information on when the two large projects start. 1.
Protocol: On page 3 of 19, the first paragraph "…to 70 percent by 2050."Please provide the reference.

2.
CEI Follow-up: On page 5 of 19, the first paragraph, "…, only female clients were recruit."In the client exit interview both women and men were recruited, but at the CEI follow up, only women were recruited.Please provide rationale why men were not recruited.
In addition, please provide additional information of the gender of interviewers who do the client exit interview and follow-up telephone interview.As contraceptive methods are a bit sensitive when talking unlike other products, women may feel uncomfortable to disclose their practice about contraceptive practice with a male interviewer.

3.
Ethical approval: On page 5 of 19, please provide the ethical approval number of each study site.

4.
Thank you for your kind consideration.
Is the rationale for, and objectives of, the study clearly described?Yes

Is the study design appropriate for the research question? Yes
Are sufficient details of the methods provided to allow replication by others?Yes

Are the datasets clearly presented in a useable and accessible format? Yes
Competing Interests: No competing interests were disclosed.
Reviewer Expertise: Sexual and reproductive health, family health, contraception and health systems.
I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.
Just to confirm that our characterization of the Agile data system as being "innovative" pertains to data collection as opposed to "family planning innovations", such as delivery interventions or contraceptive development advances.The two large projects were themselves innovative and we have included links to their websites for interested readers to explore, along with the start year (2016).
Protocol: On page 3 of 19, the first paragraph "…to 70 percent by 2050."Please provide the reference.
We have added a reference for this statement.
CEI Follow-up: On page 5 of 19, the first paragraph, "…, only female clients were recruit."In the client exit interview both women and men were recruited, but at the CEI follow up, only women were recruited.Please provide rationale why men were not recruited.
Male clients could have been re-interviewed but we did not pursue this for the following reasons: 1) Male clients accounted for about 1 in 3 clients interviewed.The project's resources were constrained and we were not sure how much effort would be required to successfully reach male clients.In total we interviewed nearly 33.9 thousand female clients and of them about 18.7 thousand were successfully interviewed for the follow-up (70%).If we had the resources, we would have endeavored to recontact the male clients.
2) The focus of the follow-up was to measure contraceptive continuation, switching or adoption.Generally speaking, this information is most accurately obtained from the female partner.
In addition, please provide additional information of the gender of interviewers who do the client exit interview and follow-up telephone interview.As contraceptive methods are a bit sensitive when talking unlike other products, women may feel uncomfortable to disclose their practice about contraceptive practice with a male interviewer.
PMA Agile only used female interviewers for the SDP, CEI, and CEI follow up interviews.
Ethical approval: On page 5 of 19, please provide the ethical approval number of each study site.
The ethical approval numbers are as follows, we have added this to the "ethical approval" section of the paper on pg 10:

○
For the dashboards, it may be helpful to uses dates instead of Q's, or at least include years.
Is the rationale for, and objectives of, the study clearly described?Yes

Is the study design appropriate for the research question? Yes
Are sufficient details of the methods provided to allow replication by others?Partly Are the datasets clearly presented in a useable and accessible format?

Not applicable
Competing Interests: No competing interests were disclosed.

Reviewer Expertise: Family Planning, Monitoring and Evaluation
We confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.
General Questions and Comments: • Overall, the paper does an excellent job of describing the overall process used to collect the various types of data.However, it would be helpful to add more detail about the calculations used to produce indicators.Many of the questions in the next section are aimed at identifying specific additions to the methodology section.
The exact calculation for each indicator is available on the project dashboard site (https://www.pmadata.org/pma-agile-dashboard-kenya-cei).On each page of the dashboard there is a circled question mark; when the cursor is moved to this question mark, the calculation of the indicator appears.-Hidden contraceptive behaviors of youth.
Insofar as use of the data for the two projects, we established separate dashboards for the project staff at TCI and DKT International to self-monitor.
• For the cost information, it would be helpful to know the relationship to overall costs to the sub-contracted costs that are used to determine the costs of specific data collection methods.Without this information it is not possible to determine if the approach is cost effective or not.
We provided the total amount of the project grant ($4,993,285)  Because this is a protocol paper, the type of information gathering and review suggested -which we agree would indeed be very valuable and a major undertaking -is beyond the scope of the present effort.
o Are there sustainability plans in place with countries to transfer some of this data collection to routine monitoring already done at the facility level by countries?
There was a dedicated effort to integrate the Agile Nigeria data collection into the Federal Ministry of Health's family planning dashboard at the facility level.This was precluded by an IRB condition not to disclose facility identities.Other challenges would have been alignment on type and measurement of indicators and dates of data collection.In every location, however, Agile results have been disseminated to the relevant local government and non-government stakeholders, usually through in-person meetings.
• The relatively high retention rate for CEI follow ups is an interesting finding -hopefully this can be used as motivation of other surveys to use cell phones for follow up and will allow for more interesting and affordable panel data!

Agree!
Questions about Sampling and Methodology: Overall, it would be helpful to have more detail about your sampling methodology, including weighting.Specifically: • Your CEI calculations are based off an assumption of 50% mCPR -is this for married or all women?Can you give the range of mCPR in your cities?
The CEI client sample was powered with consideration to the male/female composition and overall daily client volume (wherein we consulted the PMA2020 SDP data), rather than the mCPR.
No constraint on marital status was used.We do not have a range of mCPRs for the sites but we publish on the dashboard the % of clients who reporting currently using modern contraception.
• In your CEI, you interview both males and females -are they given the same questionnaire?Are they from the same sample (interviewing 1 out of x number of people who leave the facility)?Can you talk more about the motivation to include men in your sample and what you hope to learn from these interviews?What were the sample sizes of men in your cities?
Yes, the same questionnaire is used for male and female clients.Yes, clients are selected without regard to gender.The client samples are usually 10 times the number of facilities.Men comprise approximately 30% of these across all sites (see Table 2 bottom).Because males often purchase condoms for use with female partners, we wanted to capture their consumption and use behaviors.
• Do you have weights in place given that you have the same number of public and private facilities but there are probably not an even number of private and public facilities in the city?
Yes the public access data files have SDP (facility) weights.The sampling protocol and weight construction can be obtained at the following site: https://www.pmadata.org/sites/default/files/2020-06/PMA%20Agile%20Sampling%20and%20Weighting%20Protocol.pdf • On page 5, can you explain the difference between participation rates and non-consent?
Participation rates and non-consent mean the same thing: the percentage of clients that consented to be interviewed.We have clarified this point in the text.
• What was the thinking behind the even split among public and private facilities?
It was to ensure an adequate number of each type.It was not always 100+100 because a number of Agile sites were small and had less than the target number.If less than 100, the full census of available facilities was conducted.We also oversampled the SDP sample by 10%, so 220 facilities, to allow for attrition over time.The facility weight takes into account the publicprivate distribution and the oversampling.
• Across your sites, did you have information on the share of contraceptive users that interviewed but for now, we cannot weight the Agile client data (see next bullet).
Do you apply weights to the CEI interviews?If so, what information is used to create the weights?
No. Because the REs did not uniformly record the total number of clients approached, we do not have a daily selection probability which would be needed to weight the client data.A forthcoming analysis describes response rates for the client data.
• Since the geographical area covered by data collection does not appear to necessarily be a standard administrative delineation, it would be helpful to have additional information on how denominators are constructed.Are they combined administrative units?Are they based on facility catchment areas?
The geographic coverage is the administrative boundaries for the urban cities in Burkina Faso, Niger, India, and DR Congo.For Kenya, it is the urban area of the 3 counties.For Nigeria it is the boundaries of Lagos and Kano city states and the urban are of Ogun State.For health facilities, the denominators are those within the administrative boundaries.No catchment areas were used for sampling.
Questions Related to mCPR Calculations: Questions Related to Specific Figures: • Figure 3A: outlining the two panels in black boxes would help distinguish them as two different graphics.
We agree and have added black outlines to each panel.
• Figure 3B: having the top left panel axis not extend to 0 is misleading.
For all of the graphs on the PMA Agile dashboard, we scale the axis to fit the range of numbers included, and generally don't extend the axis to numbers out of range.Although this may appear to exaggerate changes, as described in this comment, it also makes the results easier to read.To reduce misinterpretation and guide the viewer on the extent of change over time, all values appear on the graphs.
• For the dashboards, it may be helpful to uses dates instead of Q's, or at least include years.
We appreciate this point.After testing the dashboard with dates, we found that, in nearly all cases, the graph labels are not readable if dates (even just years) are added.This is partly because the Q's sometimes straddled years (between 2018 and 2019).However, we note that the timing and duration of each Q is presented in slide 2 in all dashboards, as well as Table 3 and

Figure 3 .
Figure 3. Sample screens from PMA Agile dashboard of quarterly average number of client visits by method for public and private health facilities (A) and contraceptive prevalence among two rounds of client exit surveys in Kericho county, Kenya.Dashboard URL for (A): https://www.pmadata.org/pma-agile-dashboard-kenya-sdp(Note that public facilities did not dispense emergency contraception at time of surveys); Dashboard URL for (B): https://www.pmadata.org/pma-agile-dashboard-kenya-cei.
They function by providing a range of high-quality, affordable, and novel brands of contraceptive and sexual health products to the market through well established distribution and supply chain mechanisms.Implementing partners (IPs) IPs' capacity and connections with local stakeholders were key for the successful dissemination and actionability of PMA Agile results.The implementing partners were the Center for Research, Evaluation Resources and Development and the University of Ibadan for Nigeria, the Indian Institute for Health Management Research in India, the Institut National de Statistiques of the Government of Niger, the Institut Superieur de Sciences de la Population in Burkina Faso, the International Center for Reproductive Health in Kenya, and the University of Kinshasa School of Public Health for DRC.
Selection of PMA Agile sites PMA Agile activities officially began November 2016 with the first set of five country and 12 city geographies decided in July 2017: Burkina Faso (Ouagadougou and Koudougou); Democratic Republic of Congo (DRC) (Kinshasa); India (Ferozabad, Indore, Puri and Shikohadbad/Tundla cities); Kenya (Kericho, Migori and Uasin Gishu counties); and Nigeria (Lagos, Ogun and Kano).PMA Agile site location was based on collaborating with existing PMA implementing partners and considerations of local intervention activities, government health administration interest and willingness to act on results and input from Gates Foundation staff.Capitals of two Francophonic countries (Abidjan in Cote d'Ivoire and Niamey in Niger) were held for future consideration, with Niamey subsequently added in early 2019 as the 14 th PMA Agile site.

Table 1 . Features of elements for PMA Agile data system. Element Purpose Mode of administration Sample Eligibility criteria Target sample size Periodicity
environments known to influence and be of value to program officials, contraceptive and other health practices, such as commodity stock flows, client volume, client satisfaction or medication or product use adherence.Key indicators for PMA Agile are listed in Table 2, and are grouped under dimensions of supply, service quality, and demand.Additionally, it can incorporate new measures in any subsequent round of data collection desired by local stakeholders.Ethique pour la Recherche en Santé (MOH 2018-02-027); University of Kinshasa School of Public Health Institutional Review Board (ESP/CE/070/2017); Indian Institute for Health Management Research Ethical Review Board (19/12/2017-15/ 01-2018); MOH-Niger Comité National d'Ethique pour la Recherche en Santé (027/2020/CNERS).All participants provided consent in accordance with country specific approved consent procedures.

Table 2 . PMA Agile data system components, indicators and health access dimension addressed. Data unit/Indicator Dimension Health facility
*Additional project-specific indicators are included on a site-specific basis.

Table 3 . Coverage characteristics of PMA Agile data system.
*These population estimates are obtained from official census sources (Kenya) when possible but can be dated (Nigeria).

Table 1 :
Overall, the paper does an excellent job of describing the overall process used to collect the various types of data.However, it would be helpful to add more detail about the calculations used to produce indicators.Many of the questions in the next section are aimed at identifying specific additions to the methodology section.This project focuses on urban areas -what would stay the same and would change for this methodology to work in rural areas or nationally?It is assumed that there is already a lot of research using this data that is already underway, but can you add a section discussing what type of research this data lends itself to and how it has been used to monitor the two projects discussed in footnote 2?For the cost information, it would be helpful to know the relationship to overall costs to the sub-contracted costs that are used to determine the costs of specific data collection methods.Without this information it is not possible to determine if the approach is cost effective or not.Looking at Table 2, it would be nice, since this is a survey methodology paper, to have a discussion of the other methods of data collection (HMIS, LMIS, SPA, UNFPA facility surveys, etc.) and the added value of PMA Agile.A cost comparison, though it may be difficult to collect cost information from others, would be very interesting.Additionally, you can add lessons learned from PMA Agile and what you would suggest governments add to their ○ routine data collection.Are there sustainability plans in place with countries to transfer some of this data collection to routine monitoring already done at the facility level by countries?In your CEI, you interview both males and females -are they given the same questionnaire?Are they from the same sample (interviewing 1 out of x number of people who leave the facility)?Can you talk more about the motivation to include men in your sample and what you hope to learn from these interviews?What were the sample sizes of men in your cities? Do you have weights in place given that you have the same number of public and private facilities but there are probably not an even number of private and public facilities in the city?What was the thinking behind the even split among public and private facilities?Across your sites, did you have information on the share of contraceptive users that receive methods from public versus private when creating your sampling frame?Do you have estimates on the scale of non-registered facilities and what proportion of contraceptive users receive their methods from them?Also, what is the turnover of registered health facilities -how often would a longer-term project need to refresh the sampling frame?In general, can you elaborate on weights?Are large clinics and small clinics weighted the same?Or are there weighting options based off the size of the population they serve?Having these weights would allow you to look at both the facility and women as the unit of interest.For example, you could then report both the percentage of facilities with stockouts and the percentage of women who go to a facility with a stockout.Since the geographical area covered by data collection does not appear to necessarily be a standard administrative delineation, it would be helpful to have additional information on how denominators are constructed.Are they combined administrative units?Are they based on facility catchment areas?
○ ○ ○ ○ ○ ○ ○ ○ ○ ○ Do you apply weights to the CEI interviews?If so, what information is used to create the ○ weights?○ Questions Related

to mCPR Calculations: Questions Related to Specific Figures:
Figure 3B: having the top left panel axis not extend to 0 is misleading. ○ • This project focuses on urban areas -what would stay the same and would change for this methodology to work in rural areas or nationally?A rural application of the same sampling methodology would only require ensuring access to a complete listing of public and private health facilities, stratifying by the same broad facility types, and selection a number that is adequately powered and proportional to the rural: urban distribution and within facility strata.Appropriate weighting would be needed but together with the urban data would generate national estimates.Ideally the client samples will also have weights to generate representative estimates.Telephone follow-up would depend on phone access or other means of timely re-interview.•It is assumed that there is already a lot of research using this data that is already underway, but can you add a section discussing what type of research this data lends itself to and how it has been used to monitor the two projects discussed in footnote 2?
• Looking at Table 2, it would be nice, since this is a survey methodology paper, to have a discussion of the other methods of data collection (HMIS, LMIS, SPA, UNFPA facility surveys, etc.) and the added value of PMA Agile.A cost comparison, though it may be difficult to collect cost information from others, would be very interesting.Additionally, you can add lessons learned from PMA Agile and what you would suggest governments add to their routine data collection.