Monitor to Innovate With Feedback Loops: Process Evaluation Protocol for an Anemia Prevention Intervention

Background: With the proliferation of the digital age, Information and Communication Technologies paired with feedback loops have the potential to facilitate process evaluations which include continuous iteration and innovation of intervention delivery. Objective: To describe how a multilevel social norms eld trial is using feedback loops to innovate and enhance intervention delivery. Methods: We use a mixed-methods process evaluation design. Surveys and structured observation monitor delity to implementation and receptivity to implementation activities among study participants. Quantitative data evaluates implementation dose, coverage, exposure, and reach of intervention activities, and qualitative data will delve more deeply into reasons for high or low functioning. Iron folic acid supplement supply and demand are also monitored for stock-outs. Data collected from 130 intervention villages is processed, visualized, and triangulated in near to real-time via Real-time Monitoring for Knowledge Generation (RPM4K), a locally developed software application. Data visualization products facilitate the examination of monitoring data to mitigate bottlenecks and identify and implement tweaks to the RANI Project’s intervention delivery strategy on an ongoing basis. Discussion: Feedback loops assist with decision-making to facilitate timely course corrections. Feedback loops can also engender a shared understanding of ground realities for a geographically dispersed and culturally diverse team. Leveraging feedback loops, we identify opportunities to provide on-going supportive supervision for our community facilitators promoting joint problem-solving, and communication. Monthly media and hemoglobin level demonstration strategies are informed by participant engagement and receptivity. Stock-outs of iron folic acid tablets activate contingency plans to mobilize local stakeholders and advocate for timely resolutions. Unintended effects are monitored based on ongoing feedback from community facilitators. Conclusions: Documenting our processes can inform the future implementation or scale up of similar projects embracing feedback loops to iterate and innovate their intervention delivery. have not been tested before through the RANI Project. Test results are shared as color coded (green – anemia free; yellow – mild anemia; orange – moderate anemia; red – severe anemia) blood shaped cards visualizing severity of anemia or lack thereof. Post testing, recommendations for behavioral actions and nudges are tailored according to participant’s test results. Measurement and monitoring of anemia levels at the village-level are recorded and visualized using heat maps on RPM4K subsequently to monitor trends at the village-level.

We detail the development and use of a locally developed process evaluation application called Real-Time Performance Monitoring for Knowledge (RPM4K) and how it enhances the ability of frontline community facilitators, managers, evaluators and stakeholders to respond to evolving ground realities.

Background
Anemia Endemic among Women of Reproductive Age Anemia affects roughly a third of the world's population, making it a major global public health problem that impacts maternal and child mortality, cognitive and physical performance1, and work capacity and productivity.2,3 In India, more than 50 percent of women of reproductive age have anemia. 3 The largest contributor to anemia worldwide is iron de ciency. 4 Despite concentrated efforts and the presence of several government programs (e.g., National Iron Plus Initiative and Anemia Mukt Bharat),5 which offer women of reproductive age free iron supplements6-9, anemia prevalence remains high.3 India's Demographic Health Data (2015 -16) shows that only thirty percent of pregnant women consumed IFA for more than 100 days during pregnancy.
Several studies have examined supply side issues to IFA consumption10-13, but studies on demand side challenges including knowledge and social barriers are limited.14 Recently, researchers have begun to ask questions about the role that community-level factors, such as social and gender norms, can play in women's IFA consumption. [15][16][17][18] The Reduction in Anemia Through Normative Innovations (RANI) Project seeks to bridge this gap with a multilevel intervention focused on increasing demand for IFA by in uencing anemia-related social norms. 19 The RANI (Reduction in Anemia through Normative Innovations) Project The RANI project is a cluster randomized trial launched in 2019 in the Angul district of Odisha. Residents live in rural areas and primarily identify as Hindu. 20 Like the rest of the country and the district of Odisha, almost half of all women of reproductive age are anemic. In the study, there are 89 selected clusters of villages, which we randomized into treatment and control on a 1:1 basis. The usual-care control arm receives no intervention, whereas the treatment arm receives the RANI project components. Fifteen clusters (40-41 villages) are selected and 4000 women (2000 in each arm) living in the selected clusters are randomly selected to take part in data collection.
The RANI Project's overall implementation approach, based on its goal to improve IFA consumption through a social norms-based intervention, includes a number of intervention components (see Figure 1 for a visual description of each intervention component), which require monitoring and iterating the intervention delivery strategy at multiple ecological levels throughout the implementation period. Ethical approvals to conduct the RANI project have been acquired from appropriate institutions within the United States and in India (see  for the full intervention protocol19).

Process Evaluation Framework
In public health interventions, process evaluation allows researchers and practitioners to understand what works, what does not work, for whom, and under what conditions. It also serves the important function of providing ongoing feedback so that interventions can be responsive to the changing conditions in the eld. 21-23 Process evaluations also help determine whether delity to the implementation protocol was   maintained during the course of the intervention period, whether intended participants were reached, and  if any unintended consequences resulted from the project.24 Health care delivery and outcomes can be improved by using innovations (i.e., new ideas, technologies, and practices) supported by scienti c evidence. Over the last couple of decades, process evaluation methodologies have been evolving.21 They were originally centered on qualitative research alongside trials and were conducted to provide a deeper understanding of the disease condition, implementation issues and mechanisms of the intervention.22 Now, there is growing consensus that qualitative and quantitative data (mixed methods) can help facilitate trial implementation, identi cation of the effectiveness of "active ingredients", and research translation. As methods have advanced, integration of feedback loops is emerging as the next methodological frontier for process evaluations.

Use of Feedback Loops
Feedback loops are mechanisms that allow intervention information to ow back to the program implementers on an ongoing basis so that appropriate changes in intervention delivery strategy can be adopted throughout the implementation period. Incorporating feedback loops into the process evaluation design ensures interventions are meaningfully and strategically responsive to emerging requirements. In modern times, these loops increasingly rely on mobile and other forms of digital technology, which improves accuracy and reduces the time lag between events on the ground and program adaptations. As such, feedback loops can improve decision-making throughout the intervention.
Historically, institutionalizing feedback loops as an integral part of process evaluations has proven to be challenging.26-29 Many scholars29 have characterized the need for feedback loops "as a bold challenge to current orthodoxy aided by developments in theory, methods, and practice" (p. 3). They call for an approach that promotes interaction between project designers, implementers, researchers, and decisionmakers to encourage adaptation through learning. Such an approach would center on agility, responsiveness, a culture of experimentation, responsiveness, and relevant, timely, actionable data as its key characteristics. Furthermore, scholars advocate moving away from current dominant models of static intervention design and implementation, which tend to be modeled after trials that do not permit projects to be responsive to the complexities and unpredictability of implementation challenges that arise in social and behavioral intervention trials.29 The use of feedback loops is mainstream in the commercial sector for corporations such as Uber, eBay, and Airbnb. In sum, the synergy between feedback loops integrated within process evaluations and enhancement of implementation delivery is still an under-explored area, especially in the context of randomized control trials (RCTs)41. One of the monitoring goals of the RANI Project was to reduce this knowledge and implementation gap. Speci cally, we describe the process evaluation methods we are adopting in a multilevel intervention to increase iron folic acid supplement use for anemia prevention in Odisha, India.

Methods
We use a mixed methods approach which has largely bene ted from key pieces of literature35,41,48 that detail systematic approaches to designing and incorporating feedback loops to facilitate adaptive management for health improvement interventions, and we have adapted them on an ongoing basis to meet the RANI Project's speci c needs. Quantitative data is collected via surveys and structured observation is used to monitor implementation delity (i.e., dose, coverage, exposure, reach of intervention activities, village-level supply of and demand for iron folic acid supplements), and receptivity among program participants. In parallel, monthly qualitative reports provide insights into barriers and facilitators to intervention delivery. Overall, our globally dispersed team leverages near to real-time data to mitigate bottlenecks and pain-points as well as leverage opportunities to enhance implementation delivery. We collectively identify, agree upon, and incorporate tweaks to our intervention delivery strategies on an ongoing basis.

Process Evaluation Aims and Questions
The RANI Project process evaluation has three aims. 1. To assess delity to implementation while guiding intervention planning and delivery for the larger trial via feedback loops.

2.
To monitor the quality of intervention delivery and receptivity to the intervention among its intended audiences.
3. To monitor the supply and demand of iron folic acid supplement in intervention delivery sites to mitigate possible supply chain disruption.
Our process evaluation questions are anchored in the RANI Project's process evaluation framework and aims. They were nalized in collaboration with our stakeholders (Table 1), who serve as key members of the feedback loop decision-making sub-team (i.e., Principal Investigator, Project Director, M&E (Monitoring and Evaluation) Technical Lead, Intervention Implementation and Evaluation Manager) within our project.
Following the nalization of our evaluation aims and questions, we developed a logic model and M&E planning framework (Additional File 1). This framework details key inputs, outputs, outcomes (short, intermediate, and long-term) and links them with data collection instruments. It outlines the frequency of data entry, responsible parties, reporting frequencies, data visualization tools, and a plan for disseminating our ndings. Furthermore, the framework served as a guiding document when developing the wireframe and nal design for our process evaluation software application, called the Real-time Performance Monitoring for Knowledge (RPM4K).
Real-Time Performance Monitoring for Knowledge (RPM4K) RPM4K is a software application customized and developed locally for the RANI Project process evaluation. It facilitates intervention planning, delivery, performance monitoring, and automated production of data visualization and reports using near to real-time data to identify and inform feedback loops ( Figure 2). RPM4K application offers a dual mobile application and a web-based interface.
The process evaluation data lifecycle centers and revolves around our Community Facilitators and their Cluster Supervisors. RANI Community Facilitators, who serve as frontline workers to deliver intervention components, use the system to enter eld level data with the help of a hand-held (mobile) device. Their Cluster Supervisors use the mobile as well as the web interface for entering supervision data and also monitoring intervention delivery and quality by reviewing data entered by RANI Community Facilitators.
Key stakeholders use the web interface to monitor the pace and delivery of intervention activities of both Cluster Supervisors and Facilitators and utilize aggregated data to identify feedback loops and implement tweaks to the intervention. Access to RPM4K is tiered at multiple levels.
Community Facilitators are only able to access their own performance data as well as intervention data for the villages they work in. Similarly, their Cluster Supervisors can only access performance data for the Community Facilitators they manage and the villages they work in. Key members of the process evaluation team (e.g., Process Evaluation Lead, State Implementation Manager, Project Managers etc.) are able to access and view intervention data for all villages and performance data for all personnel. However, the system administrator is the only person with access to all structural functionalities of RPM4K. They are responsible for mapping intervention villages to Cluster Supervisors or Community Facilitators and de ning data access for all users. At the beginning of each month, the system administrator is responsible for uploading intervention plans onto RPM4K. They direct RANI Project's process evaluation data lifecycle, quality control, and technical assistance requests from RPM4K end users. At month's end, data on key performance indicators is auto aggregated into dashboards and reports in the form of easy to interpret heat maps and visual reports. Raw data for speci c time periods or the entirety of the intervention delivery period can also be downloaded from RPM4K by users with de ned data privileges.

Data Collection
We collect, process, visualize, and triangulate data from 130 intervention villages in near to real-time using RPM4K. Data collected ranges from the individual level (e.g., community facilitator performance and hemoglobin levels) to the village-level (e.g., IFA supplement supply and demand). Key performance indicators for each ecological level are summarized in Table 2 (to be placed at the end of this paragraph). A data quality assurance protocol is in place to review various levels and layers of data in a collaborative manner.
Monthly Qualitative Summary Report RANI Project Community Facilitators submit a monthly qualitative report summarizing their re ections, perceptions, and experiences regarding the quality of their intervention sessions, self-assessment of their performance, participant and community reactions to intervention activities, barriers and facilitators and unintended consequences, if any (See Additional File 2 for the full interview guide). All open-ended responses get captured in RPM4K in Odiya, the local language, and then get translated and summarized in a monthly report written in English.

Quantitative Measures and Instruments
The majority of quantitative data is collected and entered into RPM4K through the Community Facilitators Input Form (submitted by Community Facilitators), the Community Facilitators Evaluation Form (submitted by their Cluster Supervisors). These forms have dedicated sections dealing with key performance indicators for each intervention component (i.e., T4, Hemoglobin testing, RANI Comms, IFA supply and demand status, etc.). The forms also have built-in GPS and activity image capture features to associate location and visual data with intervention activities. All forms are submitted within four days of completing an intervention activity. See Additional Files 3 and 4 to review our data collection instruments and screenshots of RPM4K interface, dashboards, and data visualization products.

Dose
Dose is operationalized as the total number of intervention activities (e.g., T4 sessions, community engagement meetings, RANI Comm video demonstration sessions, hemoglobin testing sessions and results demonstrations) reported on RPM4K and summarized at the village-level.

Reach
Reach is measured as the total number of women of reproductive age, their families (i.e., husbands and mothers-in-law), frontline workers and policy makers in the village exposed to intervention activities.

Receptivity
Receptivity to intervention components is operationalized as the total number of likes versus dislikes shared by program participants following the delivery of an intervention activity by RANI Community Facilitators.

Community Facilitator Performance
Community Facilitator performance is assessed and monitored using three distinct tools implemented at different timepoints. The rst tool measures the baseline professional abilities of RANI Community Facilitators when they are rst onboarded as part of the RANI Project implementation team. Each RANI Community Facilitator is classi ed on a scale of 1 (Exceptional) to 3 (Needs Improvement) points on six measures -1) their prior work experience in participatory learning activities or community mobilization, 2) communication skills, 3) professional network, 4) familiarity with the intervention area, 5) references from prior colleagues or supervisors, 6) logistical preparedness related to job requirements. The second tool for assessing and monitoring Community Facilitator performance is integrated within RPM4K. It consists of a self-assessment scale reported by Community Facilitators following every intervention session followed by monthly Cluster Supervisor assessment of Community Facilitators. The Community Facilitator self-assessment scale consists of three measures which ask 1) how easy or di cult it was to deliver the intervention session they're reporting on, 2) how much the Community Facilitator thinks participants understood each session, and 3) how they would rate their overall performance for the session they just delivered. The response options consist of ve-point Likert scales ranging from very di cult to very easy, did not understand at all to understood all of the content, and very poor to very good, respectively. Similarly, Cluster Supervisors conduct random visits covering at least 10% of all intervention sessions delivered by a Community Facilitator during any given month. They perform intervention session evaluations and report Community Facilitator performance by observing and assessing their pace, accuracy, con dence, and ability to hold the audience's attention when delivering intervention components. The response options consist of ve-point Likert scales ranging from too slow to too fast and inaccurate to accurate; three-point Likert scales ranging from not at all con dent to very con dent and not at all focused to very focused. The third and nal tool is utilized and reported with varying periodicity ranging from monthly to quarterly assessments while Community Facilitators work in the eld. Each RANI Community Facilitator is classi ed on a scale of 1 (Exceptional) to 3 (Needs Improvement) points on several measures, which assess programmatic and human resource related performance such as a Community Facilitators' ability to deliver intervention content and respond to intervention feedback loops in a timely manner, need for supportive supervision, and community mobilization skills.

IFA Supply and Demand
In 2019, the RANI Project's implementation leadership team based in Bhubaneshwar, Odisha conducted an initial assessment to map the IFA supply chain, supply status and monitoring techniques in intervention villages before intervention delivery began. Using this information, they operationalized how to monitor IFA supply status, developed tentative ideas for streamlining IFA supply in the intervention villages, and designed an IFA supply and demand monitoring module for RPM4K. IFA supply status is measured as the total number of supply points reporting adequate supply, shortage of supply or stockouts. Similarly, IFA demand is measured as the total number of supply points reporting an increase in demand. Hemoglobin levels Hemoglobin testing gets conducted monthly among 15 women, using a Hemocue meter, which provides instant results in terms of hemoglobin concentration in gm/dL (grams per deciliter). This group consists of ve women who volunteered as repeat testers every month and ten women who have not been tested before through the RANI Project. Test results are shared as color coded (green -anemia free; yellowmild anemia; orange -moderate anemia; red -severe anemia) blood shaped cards visualizing severity of anemia or lack thereof. Post testing, recommendations for behavioral actions and nudges are tailored according to participant's test results. Measurement and monitoring of anemia levels at the village-level are recorded and visualized using heat maps on RPM4K subsequently to monitor trends at the villagelevel.
Identifying, Documenting and Implementing Feedback Loops Generally, process evaluation data are collected and analyzed at the end of the intervention delivery period, which limits the understanding and insights stakeholders have on barriers and facilitators on the ground8. The merger of feedback loops with applications such as RPM4K allow complex, multicomponent interventions like RANI to assess ground-realities to innovate intervention delivery strategies in near-to-real-time. To do this systematically and e ciently, for every scenario that emerges, the M&E team rst identi es the tailoring variables.48 They are de ned as variables that serve as indicators of when an intervention delivery strategy requires optimization or calibration by the intervention delivery team. For the RANI Project, the optimization or calibration takes the form of feedback loops. 4] Policy advocacy to accommodate IFA supply or demand disruption: In the Angul district of Odisha, the government provides daily IFA supplementation for pregnant women and weekly IFA for adolescents for free. While adolescents and pregnant women can acquire IFA free of cost, non-pregnant women of reproductive age remain uncovered. Keeping this coverage gap in mind, the RANI Project intervention team assesses and shares requirements for providing IFA supplements to non-pregnant women with local o cials and frontline workers on a monthly basis. Every month after village-level hemoglobin testing is completed, aggregated test results are shared with local health care workers and higher level administration at the intervention sites. This information is supplemented with a demand for IFA as per the aggregated anemia levels of pregnant and non-pregnant women of reproductive age in the intervention villages. IFA stock is usually replenished and delivered within a fortnight. In case of any delay or shortfall, the issue is escalated rst at the block level and then to the district level, if required.

5] Engaging in participatory consensus building with all stakeholders.
The decision-making process we've adopted is participatory and based on 360-degree feedback from all key stakeholders. The intervention plan for a month is developed during monthly meetings of the intervention team with feedback incorporated from the process evaluation sub-team prior to seeking feedback from our intended program participants and local stakeholders. Next, the village-speci c intervention plans are shared with women of reproductive age in our intervention villages, members of local in uential groups, policy makers (Sarpanch, Ward members etc.) and frontline workers before nalization. Once the feedback loops are implemented, RANI Community Facilitators solicit comments from the participants on the session content and facilitation quality. Comments are also sought by Supervisors and the intervention management team during monitoring visits via informal opinion polls. The quantitative ratings are recorded in RPM4K's mobile-based forms, whereas qualitative feedback is compiled in monthly qualitative reports shared with all stakeholders. Field-level ndings, intervention achievements and qualitative ndings are shared with the process evaluation team on a weekly basis. These discussions, along with interaction with the eld teams, is used to tweak RANI's intervention delivery strategy on a monthly basis. Monthly coordination and advocacy meetings are also conducted with block and district level government counterparts from the Departments of Health, Women and Child and Odisha Livelihood Mission. Village-wise Hemoglobin status along with demand for IFA tablets is discussed during these interactions. The District Collector (local government leadership) holds quarterly review meetings with all o cials from the local departments involved with RANI. Through these interactions, local government o cials also provide feedback and suggestions for improving the coordination and convergence of the RANI project with other ongoing anemia and health improvement programs in the district.

Survey Data Triangulation and Analysis
Beyond using feedback loops to inform RANI intervention tweaks, we plan to triangulate RANI's endline survey data with our monitoring data to assess whether intervention delivery factors (such as village-level reach, Community Facilitator performance over time, testing coverage, implementation coverage, IFA supply and demand) in uence intervention exposure levels at the village-level. We will also explore whether and how 1) these intervention delivery factors in uence psychosocial (e.g., self-e cacy, knowledge, awareness, risk perception), normative or behavioral (e.g., IFA use and adherence) outcomes at the village-level controlling for village-level demographics (i.e., age, income, education, caste) and structural factors (e.g., proximity to supply points). Finally, we will assess whether intervention exposure at the village-level mediates the relationship between village-level intervention delivery factors and psychosocial, normative, or behavioral outcomes.

Discussion
The purpose of this paper is to shed light on the RPM4K, a process evaluation and monitoring system being implemented by the RANI Project. It is built on the idea that even the best designed interventions are not able to foresee all changes that occur as project activities are rolled out. The overall environment is likely to change (as was the case in this study with the advent of COVID-19 that required signi cant delays and adaptations) as are the social and political realities on the ground. Furthermore, changes brought about by the intervention can themselves precipitate other disturbances in the system -as was the case when our campaign to generate demand for IFA resulted in shortages in supply.
Given these realities, and many others that cannot be anticipated at the beginning of the intervention, it is critical to build feedback loops into the overall implementation plan so that changes can be incorporated on an ongoing basis. This requires at least three criteria to be met. First, the different sequential steps in data processing (collecting, analyzing, and reporting) have to occur with minimal delays. In the RANI Project, this meant data collection in real (or close to real) time, automated algorithms built into the analysis step, and a designated person for analysis and reporting on a predetermined timeline.
Second, adaptations made to the intervention rollout plans because of monitoring feedback have to be visible to the team. When data collectors and others involved in data processing come to view their efforts as being inconsequential (because adaptations are not being made to the intervention based on feedback received), few incentives remain for them to continue their work with full commitment. Conversely, the ability to see changes in the intervention as a result of one's data collection, processing, or reporting activity can serve as powerful motivators to continue and even improve one's performance. The RANI Project held monthly meetings with facilitators, supervisors, and RANI's state implementation manager during which time changes made to the implementation schedule and activities were shared with the team.
Finally, it is also important that the overall team buy into a culture of innovation. This is likely the most di cult task, particularly for projects perceived as being conceptualized and implemented externally, with minimal local stakeholder input, and ones thought to be driven solely by outcomes, without regard for underlying processes. The idea that a project is open to innovations means that failures are anticipated, allowed to happen without retribution, and framed in terms of learning opportunities for approaching the task in a way different from how it was originally done. The process evaluation protocol for the RANI project, detailed in this paper, provides a blueprint for other programs or trials aiming to harmonize responsive feedback loops, (near to) real-time data, and adaptive tailoring variables to enhance their intervention delivery and overall impact. Clinical Trial Registry-India (CTRI) (CTRI/2018/10/016186) on 29 October 2018. All participants went through a verbal and written informed consent process before data collection.

Consent for publication
All participants provided written informed consent before participating in the study, which included consent to publish anonymous quotes from individual participants.

Availability of data and materials
Not available

Competing interests
The authors declare that they have no competing interests.

Funding
This study was funded by the Bill and Melinda Gates Foundation. The Bill and Melinda Gates Foundation did not take part in data collection, analysis, or writing of this manuscript. All authors had full access to the data in this study and take responsibility for the integrity of the data and the accuracy of the data analysis. 10.1146/annurev-clinpsy-032511-143152 Tables   Table 1. Process evaluation questions 1. To what extent was delity to implementation maintained during the implementation of the T4 sessions, community engagement meetings, media demonstrations, and hemoglobin testing sessions and demonstrations? a. What were the total number of implementation sessions conducted per village? b. Were community facilitators adequately supported during implementation delivery? c. Were T4 sessions informative and interactive for our program participants? d. What was the level of exposure to our media products at the village-level? e. What were our program participants' reactions and receptivity to our implementation activities?
f. What were the barriers and facilitators faced by our community facilitators and our target population?
g. Were there any potential unintended consequences (neutral, positive or negative) as a result of our program?
h. Was there adequate supply of Iron Folic Acid supplement at the village level?
i. Did demand for Iron Folic Acid supplement increase or decrease at the village-level during the implementation period?
2. Did we reach RANI's target audience with our implementation activities? a. What was the total reach of our implementation activities?
b. Who did we reach with our implementation activities?