Stakeholder Interviews to Inform Best Practice for Public Facing COVID-19 Wastewater Dashboards

Ethics

Written informed consent was obtained from all participants as part of study sign up, and all analyses were performed on anonymized data. This study was approved by the ethics committee of the London School of Hygiene and Tropical Medicine (approval number: 28778, granted 15^th June 2023).

Research on a sample of online SARS-CoV-2 WW dashboards was conducted to develop research themes and interview questions. Both the EU Sewage Sentinel System for European Dashboards²⁴ and COVIDPoops19²⁵ were used to identify specific WW dashboards. These websites were put together by research stakeholders to act as a hub for stakeholders view independent dashboards. Our initial review of dashboards focused on the granularity and presentation of WW data and the inclusion of clinical data. This gave rise to questioning how and what epidemiological data should be hierarchized in dashboards. We therefore proposed a qualitative interview study of stakeholder perspectives on this question.  

Participants were recruited from organizations with dashboards included in the COVIDPoops website, through informal networks, and via social media. We adopted a purposive sampling approach to include a variety of backgrounds and forms of expertise, including among researchers, health providers, public health workers, government employees, and academics across Europe and North America. There were no prior relationships with the study participants and the interviewer.

Participants were asked to fill out a consent form, including questions on demographics and basic information to help with interviews. Demographic questions included gender, country of work, country where WW data are collected. To help with interviews, we requested a link to the dashboard used in their country of work, their profession, how long they have worked in this role, and if they identified as a Developer/Implementer of dashboards or as a User/Stakeholder. The interviews were conducted over Zoom and were transcribed by a transcribing software (Zoom v 5.16.0). In preparation, a pilot interview was carried out, in which it was identified that a PowerPoint presentation would be useful to guide the interviewees through the questions.

Interviews consisted of a 30–45-minute semi-structured interview, carried out throughout August 2023 by DM, who was carrying out this study as part of her MSc, and this was explained to all study participants. Only the interviewer and interviewee were present at the call. The interview guide was separated into three sections:

Following the transcription of the audio, the transcriptions were uploaded into NVivo, a qualitative coding software. Codes were developed based on the Braun and Clarke Thematic Analysis²⁶. More specifically, a theoretical thematic approach was used to code sections that were relevant to the original objectives, and an open coding process was used to allow for modification. Transcripts were not returned to the interviewee for comment.

The original parent codes included: experience with dashboards, process of development, gaps of knowledge regarding the content within a dashboard, epidemiological hierarchization of the graphic, and epidemiological hierarchization of WW units of measurement. Within these parent codes, sub sections were created based on the frequency and importance of a section.

Demographics of study participants

A total of 32 people were contacted via email, with 14 interviews conducted, resulting in a response rate of 43.8%. Reasons for non-participation were non-response or not being able to participate (interviews took place in August, a vacation period). No repeat interviews were required. Participants were based across Europe and North America, and worked in 10 countries, but collected WW data from 15 countries, including within Africa and Asia. Seven participants identified as both a User/Stakeholder and Developer/Implementer while five identified as just a Developer/Implementer and 2 as just a User/Stakeholder (Table 1). Participants had worked with WW data for an average of 12 years.

There were three identified themes from the coding process (Table 2): the development of WW dashboards, the content of WW dashboards, and the challenges of implementation.

Theme 1: Development of WW dashboards

1.1 SARS-CoV-2 experiences

While some countries did not begin investing in WW tracking until the COVID-19 pandemic many others already had WW tracking in place and were able to transition their equipment from a previous different target to SARS-CoV-2. Only four participants mentioned WW tracking that were already conducted in their countries, with applications including polio, illicit drug use, faecal contamination of bathing waters, and specific bacterial/viral projects conducted through academic institutions. Eight participants did not start working with projects focused on WW tracking until 2020. One participant mentioned that while dashboards had existed pre-COVID-19, “…the technical reason for why we didn’t develop more, was that we didn’t have a good reason for deploying them”. In fact, when the Scottish government decided to conduct routine monitoring of SARS-CoV-2 in June of 2020 this was the first national surveillance system for a pathogen that had been done independently of the UK.

1.2 Strengths of WW dashboards

During the initial days of the pandemic, WW tracking systems were put together very quickly. A developer explained that,

One participant described how their team discussed and promptly executed initial research without securing funding. These unprecedented circumstances were new territory for countries and a stakeholder described it as if “Everybody was trying to invent the wheel”, and many others expressed that there were no standardization or protocols for countries to follow. Nevertheless, experts were communicating between countries, and the European Commission supported member states. Two participants working in government appointed health roles shared that until the intervention of the EU many governments were not interested in WW epidemiology.

1.3 Identifying stakeholders

The first step that developers took when designing dashboards was to identify the audience and stakeholders. For example, an epidemiologist might prefer exportable data to create models themselves, while government/health policy makers require a plain explanation of the data. All participants reported that the dashboards were not originally designed for the general audience, and instead for professionals such as national health security and safety agencies, local health authorities, researchers, and policy makers. Initially WW tracking was driven by academics, but as the pandemic continued public health officials, policy makers and the public became increasingly interested in WW tracking, meaning that the audience widened. Additionally, initiatives such as the EU WISH (https://www.eu-wish.eu/) were developed to enable collaboration on cross order health threats by improving disease surveillance. One participant commented, “I think [EU WISH is] really good because it's the public health institutes that are that are involved in this. And I said before at the start of the pandemic and wastewater surveillance, it was really driven by academics. And there's a disconnect. So I think this will be really good because it is the public health institutes plus their affiliated academic partners.”

1.4 Dashboard development

All participants expressed the common theme of dashboard creation following an iterative process. As one developer stated, “The dashboard had to be correct in all regards so to reduce the corruption, or the possibility for corruption”. Each developer explained that the ideal process was done in stages with user feedback present at each stage, and four developers credited team collaboration for success. Only one developer mentioned the use of specific guidelines that were followed, and this was the Government Design Principles²⁷. In brief, these are eleven principles that have been developed for UK governmental services where the user and their needs are identified, any project should be simple, translatable, easy to use, accessible, and meets the needs of the user. Two developers mentioned using the ONS coronavirus survey carried out in the UK as an exemplar for data communication²⁸. One user expressed that their dashboard could not succeed until statistical support was sought out for analysis of the data. Two developers stated that a major improvement was simply having more data points, one of them stated, “…we first only had qualitative data detected and not detected, and later on we were able to solve for trends and quantitative results”.

Challenges were encountered during dashboard development, such as balancing data privacy and clarity, “What should be the resolution? So the provinces that was in the end allowed, but not the communities.” The presentation of data evolved during the pandemic, alongside there being more data to display. Several participants commented that explanations of the data, and guides for users of the dashboards were included in the later stages,

1.5 Current use

As users were engaging with the data, use varied from checking the dashboards weekly, to not at all. Three users explained that they rely more on weekly reports provided to health officials instead of checking the physical dashboard. A stakeholder explained that the Center for Disease Control (USA) dashboard provides automated updates for users so even if the dashboard is not checked daily, it will still contact stakeholders. Provision of dashboards to a wide audience also presented research opportunities,

1.6 Future use

As of August 2023, some countries have either started to switch off their dashboards, terminate programs, or have taken a hiatus in the use of WBE. In fact, a couple of weeks before the interviews, the Welsh government discontinued the development of the dashboard and stopped clinical surveillance of SARS-CoV-2 (although some moderate data collection has since resumed). The discontinuation of SARS-CoV-2 WW data collection has encouraged researchers to think beyond COVID-19, for example by expanding towards a multi pathogen surveillance system including seasonal respiratory and gastroenteritis diseases^29,30, as well as interest in monitoring for antimicrobial resistance.

Theme 2: Content of WW dashboards

2.1 Presentation of WW data and associated metrics

When asked about how users would prioritise graphical illustration of epidemiological data, eight stakeholders explained that the dashboard of the country that they worked in favoured virus quantification over time. These same stakeholders considered this the most valuable information to include. The most popular reason for this preference was that they valued seeing how the virus load changed over time. One user stated that,

The only challenge described was that sometimes line graph representations can be over too much time. One user expressed that,

Only two participants preferred the use of a geographical map. The reason for the geographic preference was that these stakeholders preferred to look at the bigger picture, a stakeholder expressed that, “… we have an eagle’s eye view. We look from the top, so, we are interested in the occurrence of the presence of the virus”. Although it was common for participants to compliment the design of the map, they admitted that it was not realistic globally, a stakeholder explained that, “It depends on the infrastructure that is there, in the European regions, there's nice coverage of networks, but that is not the case in all locations of the world”. For many countries the information that needs to be included in a geographical display is either not public data or not part of a public system. In fact, many challenges were expressed about the design of a map and the most common being that the data changed too much depending on population level leading to changing boundaries. It is very difficult to show this data in one graphic especially in large countries, a developer explained that,

When stakeholders were asked about measuring the units of SARS-CoV-2 in WW, even with varying levels of expertise, every stakeholder emphasized the importance of normalizing the data ie. at least accounting for population size in the units of measurement reported. Accounting for the population served by the sampling location enables comparison of data between locations, assuming that the methods of data collection are identical. Thus, each of the stakeholders ranked the SARS-CoV-2 gc/L as the lowest priority, and one developer stated that, “I wouldn't be very comfortable showing non normalized data at all, because of misinterpretation”. Five stakeholders identified normalizing by population as their top priority, and in fact it was the most common unit used throughout each of the dashboards. However, it was identified as a challenge when the population is not mapped against a sewer shed and it might not work with diverse or temporally changing population sizes. After normalizing by population, stakeholders preferred normalizing by WW flow. This was identified as beneficial because it corrects for rainfall and other contaminants in sewer systems. Next, it was preferred to normalize by a faecal indicator. However, there were mixed responses on the reliability of this method. While three stakeholders prioritized this unit in their dashboards, others countered the benefits and one stated that,

another stakeholder said,

Nevertheless, the most common faecal indicators mentioned were the pepper mild mottle virus (PMMOV), ammonia, and crAssphage. Lastly, two stakeholders valued the importance of the qualitative metric. One of these stakeholders expressed why they chose this method by explaining,

One developer explained that their dashboard compares by percent change metrics instead to discourage comparisons among sites.

2.3 Clinical surveillance

The question of including clinical surveillance in dashboards was unanimously described as valuable. In fact, the only reason that it was not included in certain dashboards was due to a lack of funding. Most stakeholders claimed that clinical surveillance aided in validating the WW data and would prefer to be able to plot the case data alongside. In fact, one participant emphasized the fact that,

Nevertheless, it was frequently stated that under-reporting from clinical surveillance has increased in recent months, this has resulted in WW data becoming the early warning system it was originally proposed to be, and in some circumstances the sole source of information.

2.4 Variant surveillance

When users were asked about their opinion on variant surveillance, users unanimously mentioned the importance of conducting the research, however, they were divided on including this information on a dashboard. The answers were dependent on identifying the audience of the dashboards. Nine users prioritized the importance of it, while five expressed concerns with including this type of data. Many users shared that they thought variant data was more academic than informative, others claimed that genomic sequencing is not advanced enough to validate the data, and others said that they are not sure about including this data. One developer explained their hesitancy by explaining, “You're not treated any differently based on what variant you have. Your doctor's making the same decisions. We're recommending the same protective actions.” Regardless of these reservations, every stakeholder in support of including variant surveillance emphasized the importance of communicating the information to the users in an accessible language to avoid confusion.

2.5 Data transparency

The topic of open dashboards versus restricted dashboards had varied opinions among participants. Overall, stakeholders emphasized the importance of having open data transparency for everyone. It was frequently mentioned across interviews that if taxpayers are paying for the program, then they should have access to the data. Although no stakeholder explicitly mentioned restricting data, many stakeholders described having to be cautious with sharing data openly. Some stakeholders cited government hesitancy and confidentially concerns and explained that they have restrictions with sample size. For example, a stakeholder explained,

Similarly, in the European dashboard the site is restricted because of data agreements with member states. In this case the data provider remains the owner of the data. One stakeholder explained that the raw data is restricted because “You don't want someone taking your data and doing open manuscript and publishing before you actually get the chance to do it”. Two stakeholders mentioned having to separate the data into an internal dashboard and a public dashboard. The internal dashboard included a data download option and identifiable information for specific WW samples sites. In fact, a developer explained this separation by stating, “One of the big failings initially was we tried to make a website that would work for everyone, and it worked for no one”. Regardless of the current dashboard status, most of these data transparency perspectives can be summarized by the following comparison stated by a stakeholder,

Theme 3: Challenges of the implementation of dashboards

3.1 Communication and trust

The main challenge identified across every interview was minimizing the risk of misinterpretation. Developers needed to report the data in a language that everyone who had access to the dashboard could understand, and a developer stated that,

One stakeholder explained that dashboards have global interest however, for a long time their dashboard was only in the national language and translations were included to meet the demand of international interest. Another communication challenge was updating dashboards. Dashboards need to explain a potential lag in data reporting or other delays, and one developer stated that “We'll occasionally get questions from health departments about why our internal dashboard shows one thing, while the COVID Data tracker is showing something different”.

A concern with communicating data based on geographical locations is an issue of provenance, if that catchment falls over multiple different health boards or different geographies of interest, the information is diluted which limits interpretation. The representativeness of a site for a specific locality and the coverage achieved was difficult to communicate for many audiences. A developer stated that,

Interest from clinicians or in public health specialists was challenging in some settings. A developer explained this hesitancy by stating that “This happens every time you have something new and innovative...you have to prove the value”.

3.2 Acceptability

Initial development of WBE perhaps lacked engagement with people working in public health. One stakeholder stated that, “The link or collaboration with public health was missing and very few of the people at the monthly meetings were from public health backgrounds”. Instead, many of the original stakeholders were people with environmental backgrounds and many participants claimed that these two worlds were not communicating effectively. A common theme identified was that WBE only works if the health sector is the implementor. This theory was exemplified by a stakeholder stating that, “The WW people, they are very helpful, but they are scared that they have to do the job, so in other words that they would have to pay…”, suggesting that there is a reluctance for those involved in WW tracking to directly pay for the tracking itself.

In the case of acceptability many stakeholders explained that there was discourse between clinical and WW teams, and one implementor stated “There is also a lobby of the medical sector, who want to maintain their unique position in society for doing disease surveillance”. Overall, stakeholders mentioned that initially the health sector was not willing to accept this different type of surveillance. A stakeholder explained their frustration by claiming that, “There are too few doctors anyways and they don't have time, and they don't need to have this bureaucratic burden...This additional burden”.

During the pandemic many practitioners were not utilizing dashboards because they did not understand what dashboards added to their practice. A stakeholder explained that the practitioner’s perspective was that,

While that perspective was for COVID it was stated that as WW tracking moves to other pathogens the perspective of WBE and dashboards may change because of the potential to inform clinical care.

3.3 Ethical and security concerns

A common theme across the interviews was that with WW sequencing individual cases should not be identifiable, a developer stated that,

Privacy concerns were stated to be more of an issue with small catchment areas such as with university monitoring.

In the United States, since 9/11 it was reported that sewerage systems have been identified as a potential target of a biothreat. As a result, geolocated data is considered a privacy concern, potentially preventing WW information being public data. Another stakeholder specified that there are legal implications behind releasing WW data, for instance “Who owns that WW and consequently the information encoded?” The conclusions that can be drawn from WW are advanced enough to be able to infer health-related risks, and stakeholders expressed concern that interest from the private sector could affect life insurance or healthcare insurance.

Innovation and experimentation

The WW tracking experience that countries had before the start of the SARS-CoV-2 pandemic impacted not only how quickly dashboards were developed but also the reliability and acceptability by policy makers. We found that until governmental bodies were convinced that WW tracking could be a valuable method many academic institutions were operating with their own resources. Even if funding was provided, researchers were operating on temporary support because the duration of the pandemic was unknown. The methods of data collection and the metrics included within these dashboards were fully dependent on the resources available to these teams. The lack of resources may have contributed to methodology gaps in sampling and analysis of WW data. Looking forward, we note that continued funding for infectious disease monitoring using WW varies across settings, with considerable investments in the USA³⁵ and the European Union²⁴, and an uncertain funding environment from charitable and governmental research funding bodies. Both the collection and presentation of WW data are important to improve and refine, both for endemic diseases and in advance of the next pandemic.

Data presentation

The data displayed on a dashboard is dependent on the question that stakeholders are seeking to answer, which may be stakeholder dependent. However, there was a strong preference to observe trends in time along with comparisons within a local and/or national area. The second preference for display is through a geographical map. This graphic is dependent on the availability of catchment area data, how willing a country is to show sewer-catchment sites and the limits/boundaries of these systems. This information is crucial for policy makers that are interested in comparing the presence and intensity of the virus to other regions. There is a tension in the detail of information provided: on the one hand providing dashboards with up-to-date granular detail of infection trends has been described as a form of democratisation where failures in the social safety net can be seen and addressed³⁶, and in this study the open data approach facilitated innovation, and on the other hand this granular detail may be seen by some as a security risk and could have implications for healthcare access. Community, stakeholder and policy engagement is required to decide if the democratisation of data outweighs the perceived risks of making data available. As we gave interviewees a preference from just two options, we may have been over-simplistic in our questioning – additional variations in display options could give further granularity on how best to visualise information.

Even though stakeholders prioritize normalization of units, the specific factor of normalization is dependent on the data collected, and the technology available. Normalizing by population is recommended to compare across sites with differing catchment populations³⁷. In some areas the size of the population fluctuates and the amount that the population travels (in and out of a catchment area) may affect the interpretation and requires methodological development. Normalizing by WW flow can be important in areas affected by heavy rainfall, or accounting for the effects of industrialisation, but again the precise data requirements and methods available to account for this require development and is an ongoing area of research. These conclusions assume that methods for data collection are identical across sites – a further challenge, beyond the scope of this study, is comparisons between data where methods differ.

If available, clinical surveillance is considered by the interviewees to be crucial to include in dashboards, and supports conclusions made elsewhere³⁷. Clinical surveillance may include the number of cases, deaths, hospitalization, etc., which on their own will underestimate community burden or be less responsive to changes in incidence. Having both clinical and WW data, summarised and explained in an appropriate manner can provide a more informative appraisal of community burden which can be useful for departments of public health and have aided departments in communicating risks and preventive actions. However, if there is intense and frequent clinical sampling occurring in a population, WW data will show the exact same conclusions. As a result, WBE may not be an early warning system unless there is a significant uncertainty in clinical sampling or delays in accessing complete clinical data. Clinical data was less likely to be displayed on dashboards, due resource constraints while developing dashboards. Considerable support for the user would be required to aid interpretation of dashboards containing both information and should be the subject of future research.

The inclusion of SARS-CoV-2 variant surveillance is important only if it will influence health policy changes or clinical practice. For example, detection of new variants could indicate importation and emergence. Examples of where this may be epidemiologically important are if the variant corresponds to vaccine escape, if the community has a very low incidence (such as seen in Australia in 2020), or if variants indicate chronic carriage. These insights could offer information that changes a health officers’ response or clinical practice, but concrete examples are currently limited. One overview from the USA provides some examples³⁴. The interview participants expressed that while variant data is valuable, the information is perhaps more academic, although early indication of variants with additional vaccine resistance would be useful to know. The risk of misinterpreting variant analysis due to lack of understanding can be problematic, especially in public facing dashboards, and suitable guidance for data interpretation is needed.

Implications

From the three themes analysed in the present study certain recommendations can be made to facilitate development of dashboards that are interoperable. Access to displayed data on the dashboard should not be restricted and should ideally be openly accessible. The preferred time-intensive graphic needs to represent a pre-determined time, focussing on recent weeks. Based on all the analysed factors that may influence geographical representation the spatial design may not be practical currently, especially if there are concerns around security and identifiable factors that are dependent on sewer-catchment locations. However, if stakeholders regard geographical comparisons as important, investments should be made to develop a way forward that considers ethical and security concerns. Clinical surveillance should be included in a dashboard, especially because WW is still classified as supplementary to this data. Similarly, there needs to be more research done with variant surveillance because there remains hesitancy around the value of providing health policy makers with this data. Standardization of dashboards may not necessarily be a research or public health priority. Instead, evaluation of approaches helps to establish best practice, which may of course result in more uniformity is dashboards, but allows for innovation and adaptation.

A strength of the qualitative approach adopted in this study is the generation of data on the processes which shaped the development of dashboards in practice. This allows us to see the emergent and iterative nature of the innovation, as well as how the data translations afforded by different dashboards are subject to multiple interpretations on account of their data and use contexts. Qualitative interviews are inevitably oriented to the generation of accounts that are situated in their specific local contexts, and thus may not have generalisability beyond these and the perspectives of the participants involved. In our study, interviews have generated findings of generic value when considering to what extent standardisation is possible and feasible when translating WW analyses into data presented via dashboards. A limitation of this study is that the initial coding of transcribed interview data informing the discussion of analytical themes were generated by one person (DM). As a rapid response study to an intervention development, the study was inevitably limited in its sample size and recruitment potential, and the purposive sampling may limit the representativeness of the findings. The participants of this study are all from high-resource settings in Europe and the USA whereas the WW tracking has utility across all settings. This bias is largely due to a majority (at the time of the study) of WW dashboards being for sites in high-resource settings and the purposive sampling led to a bias towards dashboards from Europe. Further research using a more representative sampling technique would be invaluable to determine whether dashboards differ in other settings, and the potential reasons for this.

This study provides insight on how dashboards were developed during an acute period of a pandemic, and highlights best practice that were developed along the way. The varying experience of research groups and the initial reactions of governing bodies during the early days of the COVID-19 pandemic impacted the development of SARS-CoV-2 WW dashboards. Although in the beginning countries encountered varying degrees of challenges, today countries across Europe and North America remain focused on improving the future of WBE.