<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.2 20190208//EN" "http://jats.nlm.nih.gov/publishing/1.2/JATS-journalpublishing1.dtd"><article xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" article-type="letter" dtd-version="1.2" xml:lang="en">
    <front>
        <journal-meta>
            <journal-id journal-id-type="pmc">Gates Open Res</journal-id>
            <journal-title-group>
                <journal-title>Gates Open Research</journal-title>
            </journal-title-group>
            <issn pub-type="epub">2572-4754</issn>
            <publisher>
                <publisher-name>F1000 Research Limited</publisher-name>
                <publisher-loc>London, UK</publisher-loc>
            </publisher>
        </journal-meta>
        <article-meta>
            <article-id pub-id-type="doi">10.12688/gatesopenres.13028.1</article-id>
            <article-categories>
                <subj-group subj-group-type="heading">
                    <subject>Open Letter</subject>
                </subj-group>
                <subj-group>
                    <subject>Articles</subject>
                </subj-group>
            </article-categories>
            <title-group>
                <article-title>Using &#x2018;Theories of Change&#x2019; and responsive feedback to design a digital service business for patent and proprietary medicine vendors in Nigeria</article-title>
                <fn-group content-type="pub-status">
                    <fn>
                        <p>[version 1; peer review: 2 approved]</p>
                    </fn>
                </fn-group>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author" corresp="yes">
                    <name>
                        <surname>Wright</surname>
                        <given-names>Richard L.</given-names>
                    </name>
                    <role content-type="http://credit.niso.org/">Conceptualization</role>
                    <role content-type="http://credit.niso.org/">Funding Acquisition</role>
                    <role content-type="http://credit.niso.org/">Investigation</role>
                    <role content-type="http://credit.niso.org/">Methodology</role>
                    <role content-type="http://credit.niso.org/">Supervision</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Original Draft Preparation</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Review &amp; Editing</role>
                    <uri content-type="orcid">https://orcid.org/0000-0003-2223-912X</uri>
                    <xref ref-type="corresp" rid="c1">a</xref>
                    <xref ref-type="aff" rid="a1">1</xref>
                </contrib>
                <contrib contrib-type="author" corresp="no">
                    <name>
                        <surname>Gleek</surname>
                        <given-names>Abi</given-names>
                    </name>
                    <role content-type="http://credit.niso.org/">Conceptualization</role>
                    <role content-type="http://credit.niso.org/">Investigation</role>
                    <role content-type="http://credit.niso.org/">Methodology</role>
                    <role content-type="http://credit.niso.org/">Project Administration</role>
                    <role content-type="http://credit.niso.org/">Supervision</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Original Draft Preparation</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Review &amp; Editing</role>
                    <uri content-type="orcid">https://orcid.org/0000-0003-2228-4852</uri>
                    <xref ref-type="aff" rid="a2">2</xref>
                </contrib>
                <contrib contrib-type="author" corresp="no">
                    <name>
                        <surname>Bergin</surname>
                        <given-names>Nora</given-names>
                    </name>
                    <role content-type="http://credit.niso.org/">Conceptualization</role>
                    <role content-type="http://credit.niso.org/">Investigation</role>
                    <role content-type="http://credit.niso.org/">Methodology</role>
                    <role content-type="http://credit.niso.org/">Project Administration</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Review &amp; Editing</role>
                    <xref ref-type="aff" rid="a2">2</xref>
                </contrib>
                <contrib contrib-type="author" corresp="no">
                    <name>
                        <surname>Williams</surname>
                        <given-names>R. Algy</given-names>
                    </name>
                    <role content-type="http://credit.niso.org/">Funding Acquisition</role>
                    <role content-type="http://credit.niso.org/">Project Administration</role>
                    <role content-type="http://credit.niso.org/">Resources</role>
                    <role content-type="http://credit.niso.org/">Supervision</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Review &amp; Editing</role>
                    <xref ref-type="aff" rid="a2">2</xref>
                </contrib>
                <contrib contrib-type="author" corresp="no">
                    <name>
                        <surname>Agha</surname>
                        <given-names>Sohail</given-names>
                    </name>
                    <role content-type="http://credit.niso.org/">Conceptualization</role>
                    <role content-type="http://credit.niso.org/">Funding Acquisition</role>
                    <role content-type="http://credit.niso.org/">Methodology</role>
                    <role content-type="http://credit.niso.org/">Supervision</role>
                    <role content-type="http://credit.niso.org/">Writing &#x2013; Review &amp; Editing</role>
                    <uri content-type="orcid">https://orcid.org/0000-0001-7049-9933</uri>
                    <xref ref-type="aff" rid="a3">3</xref>
                </contrib>
                <aff id="a1">
                    <label>1</label>Chief Sustainability Office, Unilever PLC, Bebington, WIrral, CH63 3JW, UK</aff>
                <aff id="a2">
                    <label>2</label>Every1Mobile, Brighton, Sussex, BN1 4GW, UK</aff>
                <aff id="a3">
                    <label>3</label>Integrated Delivery, Bill &amp; Melinda Gates Foundation, Seattle, WA, 98109, USA</aff>
            </contrib-group>
            <author-notes>
                <corresp id="c1">
                    <label>a</label>
                    <email xlink:href="mailto:Richard.L.Wright@unilever.com">Richard.L.Wright@unilever.com</email>
                </corresp>
                <fn fn-type="conflict">
                    <p>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>13</day>
                <month>6</month>
                <year>2019</year>
            </pub-date>
            <pub-date pub-type="collection">
                <year>2019</year>
            </pub-date>
            <volume>3</volume>
            <elocation-id>1493</elocation-id>
            <history>
                <date date-type="accepted">
                    <day>13</day>
                    <month>5</month>
                    <year>2019</year>
                </date>
            </history>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2019 Wright RL et al.</copyright-statement>
                <copyright-year>2019</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <self-uri content-type="pdf" xlink:href="https://gatesopenresearch.org/articles/3-1493/pdf"/>
            <abstract>
                <p>In a paper titled &#x201c;Responsive feedback: Towards a new paradigm to enhance intervention effectiveness&#x201d;, Viswanath 
                    <italic toggle="yes">et al</italic>. argue that dominant models of intervention design do not account for the complexity and unpredictability of implementation challenges. Particularly in the behavioural sciences, intervention designs need to consider many factors that will be uncertain, or unknown, at the beginning of a new project. </p>
                <p> This letter describes how we were able to respond to feedback during the design phase of a proof-of-concept project to create a digital service business for Nigerian patent and proprietary medicine vendors (PPMVs). Our approach was to create an initial &#x2018;Theory of Change&#x2019; (ToC) based on a similar project with Kenyan shopkeepers. This ToC was revised following user feedback and a landscape analysis with key stakeholders. The new ToC required us to access additional funding to create a &#x2018;digital ordering&#x2019; facility for the PPMVs. Digital ordering provides a mechanism whereby we can reduce the prevalence of counterfeit medicines, offer the PPMVs credit and group-buying facilities, and reduce supply chain costs through co-distribution with fast-moving consumer goods.</p>
                <p> An important learning point was that while our focus was on designing a platform to meet users&#x2019; needs, changes in regulation meant that we spent considerably more time than anticipated meeting the needs of multiple stakeholders. However, the importance of ensuring stakeholders&#x2019; continued buy-in cannot be underestimated and has likely increased the sustainability of the project in the longer term.</p>
                <p> As Viswanath 
                    <italic toggle="yes">et al</italic>. suggest, for responsive approaches to be widely adopted needs more flexibility than exists in current funding models and project plans. Both funding bodies and grantees will need to be more responsive to feedback coming from the field.</p>
            </abstract>
            <kwd-group kwd-group-type="author">
                <kwd>Theory of Change</kwd>
                <kwd>Responsive feedback</kwd>
                <kwd>Patent and Proprietary Medicine Vendors</kwd>
                <kwd>Primary healthcare</kwd>
                <kwd>Nigeria</kwd>
                <kwd>digital</kwd>
            </kwd-group>
            <funding-group>
                <award-group id="fund-1" xlink:href="http://dx.doi.org/10.13039/501100002992">
                    <funding-source>Department for International Development, UK Government</funding-source>
                </award-group>
                <award-group id="fund-2" xlink:href="http://dx.doi.org/10.13039/100007190">
                    <funding-source>Unilever</funding-source>
                </award-group>
                <award-group id="fund-3" xlink:href="http://www.gatesfoundation.org">
                    <funding-source>Gates Foundation</funding-source>
                    <award-id>OPP1172363</award-id>
                </award-group>
                <funding-statement>This research project was funded by the Gates Foundation [OPP1172363] and by TRANSFORM, a joint initiative between Unilever &amp; the Department for International Development (DFID).   The views expressed in this article are those of the author(s) and not of their respective organisations or funding bodies. Publication in Gates Open Research does not imply endorsement by the Gates Foundation.  </funding-statement>
                <funding-statement>
                    <italic>The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.</italic>
                </funding-statement>
            </funding-group>
        </article-meta>
    </front>
    <body>
        <sec sec-type="intro">
            <title>Introduction</title>
            <p>
                <xref ref-type="bibr" rid="ref-5">Viswanath 
                    <italic toggle="yes">et al</italic>. (2019)</xref> note that current dominant models of intervention design do not account for the complexity and unpredictability of implementation challenges. In the behavioural sciences, intervention designs are affected by the enabling environment (political, regulatory, and structural factors) as well as social, cultural, and economic characteristics of the target beneficiaries.  Given that many of these factors will be uncertain or unknown at the beginning of a new project, it would seem reasonable to allow time to define interventions before finalising budgets and timelines. This letter describes how we responded to feedback during the design phase of a proof of concept project to create a digital service business for Nigerian patent and proprietary medicine vendors (PPMVs). </p>
            <p>A Nigerian PPMV is defined as &#x2018;a person without formal training in pharmacy who sells orthodox pharmaceutical products on a retail basis for profit&#x2019; (
                <xref ref-type="bibr" rid="ref-2">Brieger 
                    <italic toggle="yes">et al</italic>., 2004</xref>). PPMVs constitute an important component of the private sector health system, particularly for poorer and more rural communities but are not well supported to provide quality health services (
                <xref ref-type="bibr" rid="ref-1">Beyeler 
                    <italic toggle="yes">et al</italic>., 2015</xref>). However, with so many medically trained cadres operating as PPMVs, an opportunity exists to use them to improve and expand services for primary healthcare and family planning. The need to strengthen this vital health care delivery channel has been recognised by national and state-level government. We believe that a key component of this must be to establish a mechanism to support the PPMVs that both improves their service delivery while, at the same time, meeting their business and financial needs. This is the aim for NaijaCare.</p>
            <p>The project described here was set up to deliver a proof of concept concerning the viability of NaijaCare, a for-profit digital service for PPMVs, which would allow them to improve their service delivery while growing their businesses.  As such, part of the project involved identifying potential revenue streams for a digital service business which was also capable of delivering positive economic, social and health impacts for the PPMVs and their customers.</p>
            <sec>
                <title>UJoin</title>
                <p>The inspiration for NaijaCare came from UJoin
                    <sup>
                        <xref ref-type="other" rid="FN1">1</xref>
                    </sup>; a digital service for shopkeepers (known as &#x2018;duka owners&#x2019;) in Nairobi&#x2019;s slums.  UJoin provides duka owners with a digital community, which they access using their mobile phone. The community provides education and mentoring, credit to buy stock, and a &#x2018;loyalty scheme&#x2019; through which discount vouchers are sent to their customers&#x2019; mobile phones. </p>
                <p>The &#x2018;loyalty scheme&#x2019; is key to UJoin&#x2019;s ToC as it allows us to create demand for some of the dukas&#x2019; stock in two ways (see 
                    <xref ref-type="fig" rid="f1">Figure 1</xref>). First, we improve product affordability by sending discounts to customers&#x2019; phones. Second, the loyalty scheme provides a &#x2018;direct-to-consumer&#x2019; communications channel.  This is realised through UAfya, a digital community that provides young women with health education and peer-to-peer sharing on a broad range of issues.  UAfya not only drives demand for some of the products stocked by dukas but also enhances the broader wellbeing of the young women. </p>
                <fig fig-type="figure" id="f1" orientation="portrait" position="float">
                    <label>Figure 1. </label>
                    <caption>
                        <title>Theory of change for UJoin.</title>
                    </caption>
                    <graphic orientation="portrait" position="float" xlink:href="https://gatesopenresearch-files.f1000.com/manuscripts/14138/12d112be-ed61-42bb-954b-fec54e4943b0_figure1.gif"/>
                </fig>
                <p>UJoin was set up on the premise that we could meet the needs, and drive behavioural change, for low-income households by leveraging the relationships between the duka owners and their customers. We felt that using this market-based approach could have benefits over more traditional direct contact approaches. First, the relationship between the duka owner and customer is long-term and based on trust and this could provide for a series of interventions that address multiple behavioural changes. Second, through the market we could create demand for consumables such as soap, menstrual health products, and nutritious foods; improve the livelihoods of duka owners; and create clear returns on investment for private sector partners investing in UJoin.</p>
            </sec>
            <sec>
                <title>NaijaCare</title>
                <p>We based the initial NaijaCare ToC on UJoin (see 
                    <xref ref-type="fig" rid="f2">Figure 2</xref>).  NaijaCare has the same business building blocks as UJoin but, in addition, we believed that there could be a route to impact by using the digital community to improve the PPMVs&#x2019; service delivery. There were three elements to this; (i) Education and mentoring to increase the quality of customer interactions; (ii) Support of the introduction of consumer diagnostics such as rapid diagnostic kits for malaria, which could identify consumer health problems and create further demand for PPMV products, and (iii) A referral mechanism, which in the first instance made it easy for the PPMVs to refer customers to providers of family planning services. The evidence is that current levels of PPMV referrals to clinics are low (see 
                    <xref ref-type="bibr" rid="ref-1">Beyeler 
                        <italic toggle="yes">et al</italic>., 2015</xref> for a review).</p>
                <fig fig-type="figure" id="f2" orientation="portrait" position="float">
                    <label>Figure 2. </label>
                    <caption>
                        <title>Theory of change for NaijaCare.</title>
                        <p>PPMV, patent and proprietary medicine vendors.</p>
                    </caption>
                    <graphic orientation="portrait" position="float" xlink:href="https://gatesopenresearch-files.f1000.com/manuscripts/14138/12d112be-ed61-42bb-954b-fec54e4943b0_figure2.gif"/>
                </fig>
            </sec>
        </sec>
        <sec>
            <title>Designing NaijaCare</title>
            <p>Initial design workshops with PPMVs revealed some similarities and some differences with duka owners. A key insight was that PPMVs saw themselves as business owners rather than health providers. So, like duka owners, their primary interest was in improving their businesses. There was a strong interest in credit for stock, need for stock control and business planning.  While these insights did not affect our basic ToC, it became clear that the first release of NaijaCare needed to focus on business benefits rather than health provision.</p>
            <p>In parallel, we conducted a landscape analysis, consulting with organisations such as the Pharmacist Council of Nigeria (PCN), the National Association of Patent and Proprietary Medicine Dealers (NAPPMED), and the Society for Family Health (SFH).  What became clear from these consultations was that there were a range of ongoing developments, all of which would need to consider in our design. </p>
            <p>Key design changes resulted from reports that PPMVs are a major route for the supply of sub-standard and counterfeit medicines (
                <xref ref-type="bibr" rid="ref-4">Health Communication Capacity Collaborative, 2016</xref>) and sell medicines that are not covered under their licence (e.g., 
                <xref ref-type="bibr" rid="ref-3">Fajola 
                    <italic toggle="yes">et al</italic>., 2011&#x2013;2012</xref>). Further, we discovered that imminent regulatory changes would create three tiers of PPMVs and require them to register with PCN. This context meant that the model built with duka owners was not enough to meet the requirements for PPMVs; a digital service that supported these businesses would need to mitigate the legal contraventions; facilitate any necessary registration; and target each PPMV with content that was appropriate to their tier. What had started as process of designing a digital platform that met users&#x2019; needs now needed to consider the needs of multiple stakeholders and gatekeepers. </p>
            <p>Based on the landscape analysis and the business needs of the PPMVs, we felt we needed to alter our ToC (see 
                <xref ref-type="fig" rid="f3">Figure 3</xref>) to include a digital ordering facility that would enable PPMVs to order medicines from an assured provider.  This required additional in-market relationships and technology development.  Naturally, these additional activities led to a requirement to increase the budget envelope beyond that originally allocated. </p>
            <fig fig-type="figure" id="f3" orientation="portrait" position="float">
                <label>Figure 3. </label>
                <caption>
                    <title>Revised theory of change for NaijaCare.</title>
                    <p>PPMV, patent and proprietary medicine vendors.</p>
                </caption>
                <graphic orientation="portrait" position="float" xlink:href="https://gatesopenresearch-files.f1000.com/manuscripts/14138/12d112be-ed61-42bb-954b-fec54e4943b0_figure3.gif"/>
            </fig>
            <p>Digital medicine ordering allows us to create a secure supply chain of assured medicines, with PPMVs only able to access those medicines covered under their licence. Further, digital ordering can provide a digital trading history that may de-risk the provision of highly-valued credit, share distribution costs through adding further products to the basket (e.g. soaps, toothpastes, skin creams) and reduce the unit cost of stock for the PPMVs by enabling group-buying.  </p>
            <p>It is our hope (to be tested) that that digital ordering will become a major reason for the PPMVs to engage with the platform and to conform with the relevant laws and regulations. It will reduce the potential friction in their lives and can be used to reassure their customers of the quality of their products. Finally, we hope that it could provide an affordable way for PPMVs to register with PCN through taking small payments as part of each transaction.</p>
        </sec>
        <sec sec-type="discussion | conclusions">
            <title>Discussion and conclusions</title>
            <p>The transfer of an existing ToC provided the starting hypotheses for NaijaCare. This was tested through qualitative and ethnographic research in our user-centred design process. Further, the appropriateness in the new context was explored with a new set of stakeholders.  This led us to change our ToC and add the development of digital ordering in parallel with the creation of our &#x2018;Minimal Viable Product&#x2019; (MVP) community. The community has now launched and is being refined based on user interactions and feedback.  In addition to qualitative feedback we are now introducing quantitative measures; for instance, if the completion rates on a course are low then we revise the course. </p>
            <p>Our expectation is that, once our intervention has stabilised, we will test it more formally, providing more quantitative evidence for its effectiveness and our theory of change. Eventually, we may choose to perform a randomised controlled trial. However, we believe that it is important to spend considerable time in a more responsive mode &#x2013; designing data gathering to enable decision making rather than as proofs of efficacy.</p>
            <p>We would support the call by Viswanath 
                <italic toggle="yes">et al.</italic> for a more flexible approach to behavioural interventions, whereby a theory of change is specified together with an associated level of uncertainty. If this can be accepted by all stakeholders (donors, governments, grantees), then projects can be designed in such a way to maintain flexibility in the early stages, including the use of responsive feedback mechanisms.</p>
        </sec>
        <sec>
            <title>Data availability</title>
            <p>No data are associated with this article.</p>
        </sec>
    </body>
    <back>
        <fn-group>
            <fn>
                <p id="FN1">
                    <sup>1</sup>By March 2019, we had 3,000 UJoin duka owners and over 20,000 customers redeeming over 100,000 vouchers through the loyalty scheme. UAfya is a vibrant community providing advice on subjects such as handwashing, nutrition, and breastfeeding. Both UJoin and UAfya have been developed through user-centred design and testing and iterating minimal viable products.</p>
            </fn>
        </fn-group>
        <ref-list>
            <ref id="ref-1">
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Beyeler</surname>
                            <given-names>N</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Liu</surname>
                            <given-names>J</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Sieverding</surname>
                            <given-names>M</given-names>
                        </name>
</person-group>:
                    <article-title>A systematic review of the role of proprietary and patent medicine vendors in healthcare provision in Nigeria.</article-title>
                    <source>

                        <italic toggle="yes">PLoS One.</italic>
</source>
                    <year>2015</year>;<volume>10</volume>(<issue>1</issue>):<fpage>e0117165</fpage>.
                    <pub-id pub-id-type="pmid">25629900</pub-id>
                    <pub-id pub-id-type="doi">10.1371/journal.pone.0117165</pub-id>
                    <pub-id pub-id-type="pmcid">4309565</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-2">
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Brieger</surname>
                            <given-names>WR</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Osamor</surname>
                            <given-names>PE</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Salami</surname>
                            <given-names>KK</given-names>
                        </name>

                        <etal/>
</person-group>:
                    <article-title>Interactions between patent medicine vendors and customers in urban and rural Nigeria.</article-title>
                    <source>

                        <italic toggle="yes">Health Policy Plan.</italic>
</source>
                    <year>2004</year>;<volume>19</volume>(<issue>3</issue>):<fpage>177</fpage>&#x2013;<lpage>182</lpage>.
                    <pub-id pub-id-type="pmid">15070866</pub-id>
                    <pub-id pub-id-type="doi">10.1093/heapol/czh021</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-3">
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Fajola</surname>
                            <given-names>A</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Asuzu</surname>
                            <given-names>MC</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Owoaje</surname>
                            <given-names>ET</given-names>
                        </name>

                        <etal/>
</person-group>:
                    <article-title>A rural-urban comparison of client-provider interactions in patent medicine shops in South west Nigeria.</article-title>
                    <source>

                        <italic toggle="yes">Int Q Community Health Educ.</italic>
</source>
                    <year>2011&#x2013;2012</year>;<volume>32</volume>:<fpage>195</fpage>&#x2013;<lpage>203</lpage>.
                    <pub-id pub-id-type="pmid">23353561</pub-id>
                    <pub-id pub-id-type="doi">10.2190/IQ.32.3.c</pub-id>
                </mixed-citation>
            </ref>
            <ref id="ref-4">
                <mixed-citation publication-type="journal">
                    <collab>Health Communication Capacity Collaborative (HC3)</collab>:
                    <article-title>Landscape of Antimalarial Medicines in Nigeria. </article-title>
                    <year>2016</year>.
                    <ext-link ext-link-type="uri" xlink:href="https://sbccimplementationkits.org/quality-malaria-medicines/wp-content/uploads/sites/15/2016/10/Landscape-of-Antimalarial-Medicines-in-Nigeria.pdf">Reference Source</ext-link>
                </mixed-citation>
            </ref>
            <ref id="ref-5">
                <mixed-citation publication-type="journal">
                    <person-group person-group-type="author">

                        <name name-style="western">
                            <surname>Viswanath</surname>
                            <given-names>K</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Synowiec</surname>
                            <given-names>C</given-names>
                        </name>

                        <name name-style="western">
                            <surname>Agha</surname>
                            <given-names>S</given-names>
                        </name>
</person-group>:
                    <article-title>Responsive feedback: Towards a new paradigm to enhance intervention effectiveness [version 1; peer review: 4 approved].</article-title>
                    <source>

                        <italic toggle="yes">Gates Open Res.</italic>
</source>
                    <year>2019</year>;<volume>3</volume>:<fpage>781</fpage>.
                    <pub-id pub-id-type="doi">10.12688/gatesopenres.12937.1</pub-id>
                </mixed-citation>
            </ref>
        </ref-list>
    </back>
    <sub-article article-type="reviewer-report" id="report27385">
        <front-stub>
            <article-id pub-id-type="doi">10.21956/gatesopenres.14138.r27385</article-id>
            <title-group>
                <article-title>Reviewer response for version 1</article-title>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author">
                    <name>
                        <surname>Fogarty</surname>
                        <given-names>Linda</given-names>
                    </name>
                    <xref ref-type="aff" rid="r27385a1">1</xref>
                    <role>Referee</role>
                    <uri content-type="orcid">https://orcid.org/0000-0002-9378-083X</uri>
                </contrib>
                <aff id="r27385a1">
                    <label>1</label>Jhpiego, Baltimore, MD, USA</aff>
            </contrib-group>
            <author-notes>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>24</day>
                <month>1</month>
                <year>2020</year>
            </pub-date>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2020 Fogarty L</copyright-statement>
                <copyright-year>2020</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <related-article ext-link-type="doi" id="relatedArticleReport27385" related-article-type="peer-reviewed-article" xlink:href="10.12688/gatesopenres.13028.1"/>
            <custom-meta-group>
                <custom-meta>
                    <meta-name>recommendation</meta-name>
                    <meta-value>approve</meta-value>
                </custom-meta>
            </custom-meta-group>
        </front-stub>
        <body>
            <p>This open letter - &#x201c;Using &#x2018;Theories of Change&#x2019; and responsive feedback to design a digital service business for patent and proprietary medicine vendors in Nigeria&#x201d; - provides good rationale for the letter, is accessible and well-referenced, and recommendations are clear. The letter is well-written and timely: the need for a more flexible approach to international health intervention design given the complexity and unpredictability of many global health challenges and contexts is even acknowledged by USAID through investments in Developmental Evaluation (through the Global Development Lab/Monitoring, Evaluation, Research and Learning Innovation Program). However, the letter does not spend time describing differing views and opinions. Though those differences may be obvious, it would strengthen the letter to add a sentence or two describing the current donor and development trends and expectations that make longer, more flexible intervention design periods difficult or even unacceptable. However, the letter makes a very clear and compelling case for such an approach.&#x00a0;</p>
            <p> </p>
            <p> I have a couple of questions about the note. First, affordability of well-being products was an outcome of the loyalty scheme in Kenya and in the first TOC for Nigeria. Why was it dropped for the final TOC? Though I realize the TOC is quite focused on the PPMVs, the goal of improved client health outcome seems directly supported by affordability. Second, and a very minor question, why the change from &#x201c;consumer&#x201d; to &#x201c;client&#x201d; to describe beneficiary of improved health? In this case, i.e., for this letter, it might be useful to harmonize these (just because this probably isn&#x2019;t the right place to discuss such a detail) and so it doesn&#x2019;t distract.</p>
            <p> </p>
            <p> Finally, I noticed two places where very minor edits might be helpful: 
                <list list-type="bullet">
                    <list-item>
                        <p>In the Designing NaijaCare section, para 2, last sentence: There is a &#x201c;we&#x201d; missing. I think it should be: &#x201c;&#x2026;all of which 
                            <underline>we</underline> would&#x2026;&#x201d;.</p>
                    </list-item>
                    <list-item>
                        <p>Last paragraph on page 4, first sentence: &#x201c;It is our hope&#x2026;&#x201d; there are two &#x201c;that&#x201d; words together. I think only one is needed.</p>
                    </list-item>
                </list>
            </p>
            <p>Does the article adequately reference differing views and opinions?</p>
            <p>No</p>
            <p>Are all factual statements correct, and are statements and arguments made adequately supported by citations?</p>
            <p>Yes</p>
            <p>Is the Open Letter written in accessible language?</p>
            <p>Yes</p>
            <p>Where applicable, are recommendations and next steps explained clearly for others to follow?</p>
            <p>Yes</p>
            <p>Is the rationale for the Open Letter provided in sufficient detail?</p>
            <p>Yes</p>
            <p>Reviewer Expertise:</p>
            <p>Public health monitoring, evaluation and research, reproductive and maternal health, HIV, systems strengthening, human resources for health, social sciences.</p>
            <p>I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.</p>
        </body>
    </sub-article>
    <sub-article article-type="reviewer-report" id="report27386">
        <front-stub>
            <article-id pub-id-type="doi">10.21956/gatesopenres.14138.r27386</article-id>
            <title-group>
                <article-title>Reviewer response for version 1</article-title>
            </title-group>
            <contrib-group>
                <contrib contrib-type="author">
                    <name>
                        <surname>Evans</surname>
                        <given-names>W. Douglas</given-names>
                    </name>
                    <xref ref-type="aff" rid="r27386a1">1</xref>
                    <role>Referee</role>
                    <uri content-type="orcid">https://orcid.org/0000-0002-7559-1592</uri>
                </contrib>
                <aff id="r27386a1">
                    <label>1</label>Milken Institute School of Public Health, George Washington University, Washington, DC, USA</aff>
            </contrib-group>
            <author-notes>
                <fn fn-type="conflict">
                    <p>
                        <bold>Competing interests: </bold>No competing interests were disclosed.</p>
                </fn>
            </author-notes>
            <pub-date pub-type="epub">
                <day>17</day>
                <month>6</month>
                <year>2019</year>
            </pub-date>
            <permissions>
                <copyright-statement>Copyright: &#x00a9; 2019 Evans WD</copyright-statement>
                <copyright-year>2019</copyright-year>
                <license xlink:href="https://creativecommons.org/licenses/by/4.0/">
                    <license-p>This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>
                </license>
            </permissions>
            <related-article ext-link-type="doi" id="relatedArticleReport27386" related-article-type="peer-reviewed-article" xlink:href="10.12688/gatesopenres.13028.1"/>
            <custom-meta-group>
                <custom-meta>
                    <meta-name>recommendation</meta-name>
                    <meta-value>approve</meta-value>
                </custom-meta>
            </custom-meta-group>
        </front-stub>
        <body>
            <p>This is an interesting and valuable paper on the use of theories of change and responsive feedback to improve a digital business model for patent and proprietary medicine vendors in Nigeria. The impact of this project is potentially very large and the study is well conceived and the paper well written and clear. I have just a few minor suggestions for revisions: 
                <list list-type="order">
                    <list-item>
                        <p>It is worth noting that the Viswanath 
                            <italic>et al.</italic>, 2019 paper and the responsive feedback model in general is part of a much larger literature on evidence&lt;--&gt;practice feedback loops. The Precede-Proceed model, RE-AIM, and related theories which are part of this tradition, have been studied extensively, and should be noted as background. Responsive Feedback is an extension and outgrowth of these well established models.</p>
                    </list-item>
                    <list-item>
                        <p>It would be valuable to readers if the article said more about the data that were&#x00a0;gathered as the project in Nigeria developed and how those data were analyzed and interpreted. What specific findings led to changes in the ToC reported here?&#x00a0;More specifics would be helpful.</p>
                    </list-item>
                    <list-item>
                        <p>How will we know if the program in Nigeria is a long-term success? The authors could provide more discussion on potential future research, questions to be answered, and challenges likely to be faced.</p>
                    </list-item>
                    <list-item>
                        <p>The article overall is a bit short on citations to other research related to&#x00a0;evidence&lt;--&gt;practice feedback loops, which relates back to point 1.</p>
                    </list-item>
                    <list-item>
                        <p>This project clearly applies a marketing framework and approach, yet marketing is never mentioned nor are there any citations to relevant literature on marketing theory and methods.</p>
                    </list-item>
                </list>
            </p>
            <p>Does the article adequately reference differing views and opinions?</p>
            <p>Yes</p>
            <p>Are all factual statements correct, and are statements and arguments made adequately supported by citations?</p>
            <p>Yes</p>
            <p>Is the Open Letter written in accessible language?</p>
            <p>Yes</p>
            <p>Where applicable, are recommendations and next steps explained clearly for others to follow?</p>
            <p>Partly</p>
            <p>Is the rationale for the Open Letter provided in sufficient detail?</p>
            <p>Yes</p>
            <p>Reviewer Expertise:</p>
            <p>Health communication, social marketing, digital interventions, applied behavioral research methods</p>
            <p>I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.</p>
        </body>
    </sub-article>
</article>
