The use of insecticides is a core component of global malaria control strategy, limiting contact between mosquito vectors and human populations ( Bhatt et al., 2015; Wilson et al., 2020). The most common strategies target the adult stages of the vector, primarily insecticide treated nets (ITNs) to prevent nighttime biting, sometimes supplemented by indoor residual spraying (IRS) to kill mosquitoes that rest inside dwellings ( WHO, 2023) ( Killeen, 2020). However, the continued efficacy of pyrethroid insecticides in malaria vector control is threatened by the widespread emergence of resistance in mosquito populations ( Hemingway et al., 2016; Ranson & Lissenden, 2016). The long-term use of pyrethroid class insecticides throughout sub-Saharan Africa in both vector control and agriculture has resulted in a selective pressure in Anopheles mosquito populations towards adaptations that provide a protective effect against these tools ( Hancock et al., 2020; Moyes et al., 2020).

To manage insecticide resistance, it is necessary to implement detailed, routine resistance monitoring programmes on vector populations. The World Health Organization (WHO) provides guidance to national control programmes on identifying insecticide resistance in mosquito populations ( WHO, 2022a). The data generated through monitoring of resistance in vector populations allows control programmes to be adjusted appropriately, maximising the efficacy and longevity of interventions ( Kouassi et al., 2023; Moyes et al., 2020; WHO, 2022a). The WHO tube bioassay is a straightforward method for identifying evidence of insecticide resistance in a population, with 25 adult mosquitoes exposed to papers treated with insecticide for one hour. A mosquito population is considered resistant to an insecticide if mortality is less than 90% 24 hours post-exposure (with 90–98% mortality considered an indicator of possible resistance) ( WHO, 2022b). Increasingly importantly, the WHO tube assay can be used to evaluate the ability of a synergist to restore susceptibility to a pyrethroid-resistant mosquito population. If mosquitoes pre-exposed to the synergist piperonyl butoxide (PBO) subsequently exhibit increased mortality following exposure to a discriminating concentration of a pyrethroid insecticide, relative to those exposed to the pyrethroid alone, then this is evidence that resistance results from upregulation of P450 metabolic enzymes ( Edi et al., 2014; Moyes et al., 2021).

The results of synergism testing are used to guide decision making when choosing to deploy ITNs containing PBO as part of a vector control programme ( WHO, 2023). It is therefore important that the threshold at which a mortality increase following PBO pre-exposure represents evidence of synergism and therefore metabolic resistance should be supported by robust statistical evidence, which will be dependent on sample size. A PBO assay should be designed so that an indication of synergism can be reliably distinguished from random fluctuations in mortality due to the variability inherent to bioassays ( Praulins et al., 2022). Currently, WHO guidance indicates that with a sample size of four tubes of each treatment (a so-called ‘4x4’ design), this synergism threshold is a 10% difference in 24hr mortality between pyrethroid-only and pyrethroid-PBO treatments ( WHO, 2022c).

Where a sample size is fixed, identifying the effect size (mortality difference) that can be reliably detected by a statistical analysis is entirely dependent on the variability of the outcome. In the context of a WHO resistance assay with a fixed ‘4x4’ sample size, identifying the mortality difference that can be reliably detected requires quantification of the variability in mortality measured between tubes of the same treatment. Known sources of variation in a mosquito bioassay include genetic differences between mosquito populations, environmental differences (e.g. temperature, humidity, time of day), the number of mosquitoes per test item (tube in this case), and experimental technique (e.g., reagents, user variability/experience, measurement tools). Beyond the variability explained by known parameters is a level of unexplained variation (known as ‘noise’). While all variation is attributable to some cause, there are limits to what can be controlled or measured in a mosquito bioassay. Nonetheless, procedures to standardise these definable sources of variation and thus reduce the variability of the outcome make it possible to detect a smaller effect size than would otherwise be possible with the same sample size. The variability remaining after standardising procedures (the ‘noise’) can be used to calculate the minimum mortality difference between treatments that can be detected with the experimental design.

The purpose of this study was to identify and quantify as far as possible the sources of variation in WHO tube bioassays and use an estimate of the remaining level of ‘noise’ to determine appropriate sample size guidance for detecting a difference in 24hr mortality between pyrethroid-only and pyrethroid-PBO exposure treatments. This will help generate more robust data on which to determine the presence or absence of synergism and thus metabolic resistance mechanisms in a mosquito population and better inform decisions about where to distribute pyrethroid-PBO ITNs.

This study aimed to quantify and identify sources of variability in WHO tube bioassays when assessing the mortality of pyrethroid-susceptible and pyrethroid-resistant An. gambiae exposed to permethrin and PBO. This variability information was used as the basis of a simulation framework for the sample size of tubes needed to reliably detect mortality differences. We demonstrate that the power to detect a 10% difference in mortality is dependent on what absolute mortality values are being compared (i.e. 90% vs 100% is not equivalent to 50% vs 60%). We show that a ‘4x4 design’ (four tubes of each treatment) can detect an increase in mortality from 90% to 100% with >90% power even when variance is assumed to be very high. However, this scenario where one treatment group has mean mortality close to 100% is a special edge case, as proportional data cannot have a value larger than 1 thus there is very low variance around the mean. This would also apply if mortality was close to 0%. Consequently, statistical comparisons where the mean of one group is close to 0% or 100% mortality have high power. Where both treatment groups have mortality that is not close to 0% or 100%, a 10% difference is much harder to detect. We demonstrate that for the ‘worst-case-scenario’ where one treatment group has 50% mortality (as it is furthest from 0% or 100%) the smallest mortality difference that can be detected with 80% power is 22.5%. For any middle ground where mortality for both groups is between 50% and 100%, power to detect a mortality difference can be obtained from the linked R Shiny application.

When evaluating the sources of variability, we observed that for both the pyrethroid-susceptible and resistant strain most variation occurred between testing days rather than within them. Furthermore, we observed the overall variability of the pyrethroid-susceptible Kisumu strain was greater than that of the pyrethroid-resistant Tiassalé 13 strain (when pre-exposed to PBO), with the strains having similar levels of within-day variation, but the susceptible strain approximately twice as variable between-days. However, it should be noted that due to the highly controlled rearing and testing environment, the variance estimates here are indicative of the minimum variability. Consequently, in less rigorously controlled conditions variance would be expected to be higher and thus a larger sample size needed. The R Shiny application provides sample size estimates across a range of variance assumptions.

When assessing the sources of variation in 24hr mortality, the number of mosquitoes in the tube and the total dry weight of mosquitoes were identified as predictors of the outcome, together contributing a total of approximately 10% of variability in the data. Although the WHO guidelines state that 25 mosquitoes (or as close as practically possible to 25) should be exposed in each tube, in practice this varies somewhat from tube to tube due to the challenges of manipulating live mosquitoes (which varied from 20 to 35 in our experiment). Additionally, it is recommended ( WHO, 2022f) that some measure of the size of mosquitoes used for bioassays is taken as a quality control measure and to aid the interpretation of results. Temperature and humidity, within the very narrow range of values permitted within in our climate-controlled insectary (26 ± 2 °C and 80 ± 5% RH), did not impact 24hr mortality. In conditions where temperature and humidity vary outside of these narrow ranges, temperature and humidity are well established to impact the outcome ( Mseti et al., 2024). The mean wing length was not predictive of 24hr mortality in this study, with dry weight a more predictive indicator. However, in an equivalent study on WHO bottles with the same mosquito colonies run in parallel, increasing wing length was found to have a negative effect on 24-hour mortality ( Praulins et al., 2024), with the same paper finding not detecting a statistical effect of dryweights on mortality. Overall, our observations indicate that approximately 90% of the variability (or ‘noise’) observed in WHO tube bioassays under these testing conditions was due to parameters not considered or quantified here. The observation that between-day variation was high, but within-day variation was comparatively low, indicates a systematic and unmeasured difference between testing days. This may be attributable to differences in the mosquitoes used on each testing day, potentially due to differences in larval environment during rearing. The number of mosquitoes in the tube was observed to have a positive effect on 24-hour mortality, which may be attributable to the greater density causing mosquitoes to collide with each other and the tube more frequently and so experience greater exposure to the insecticide ( Praulins et al., 2022). Taken together, these findings indicate that performing all tests on the same day and keeping mosquito numbers consistent at 25 per tube would reduce variance by approximately two thirds, compared to allowing numbers to vary from 20 to 30 and performing the test over three days.

It was observed that 24hr mortality with the pyrethroid-resistant strain exposed to both permethrin and PBO declined with repeated use of the same permethrin-treated papers. This equated to an approximately 11% reduction with each use, up to the fourth exposure, with no subsequent decrease thereafter up to the maximum of seven. To confirm this observation this experiment was repeated in full (with the one change that papers were stored in the fridge, to remove their being stored in the first experiment with Tiassalé 13 as a potential cause), yet the decline with testing day was statistically identical across the same across the two repeats. To determine if this was due to a decline in available insecticide on the papers, chemical analysis of these samples was performed ( Table 2) which indicated no difference in total AI content on the papers across zero and seven uses, though measuring total AI content may not be sensitive enough to detect changes in surface availability which may affect bioactivity. However, it is important context that in both experiments the same cohort of mosquitoes was used for the repeated tests on the same papers, and mosquitoes tested on each day were the same age. Each subsequent test therefore used mosquitoes which had pupated on a later day than the previous test, having been reared in the same larval environment. Consequently, it is not possible to disentangle the effects of paper use with time in the larval environment (time to pupation). Future work could be done that standardised time-to-pupation to test papers multiple times, to test the hypothesis that the reduced mortality observed in this study was in fact a result of the number of times paper were used.

The concern with underpowered sample sizes is not just a risk that the effect will not be detected but that comparisons that are detected (p<0.05) by underpowered sample sizes will have an inherent risk of overestimating the effect size, as has been demonstrated in previous simulation studies ( Ioannidis, 2008). Consequently, the mortality difference observed in underpowered samples is not a reliable estimate of the true effect. Current WHO guidance for identifying evidence of metabolic resistance in a mosquito population states that a 10% difference between pyrethroid-only and pyrethroid-PBO exposure in a 4x4 WHO tube assay is indicative of PBO synergism and so metabolic resistance. Where pyrethroid-only mortality is 90% and pyrethroid-PBO is 100%, our simulation study indicates that this 10% difference can be reliably detected with >90% power even when assuming a very high level of within-day variation. However, under the ‘worst-case’ scenario where the mortality of either group is 50% (assuming the level of variation observed in our assays with the pyrethroid-resistant strain), a 4x4 design is only powered to detect a 22.5% difference with 80% power. Additionally, our simulation study determined that if the exposures are performed over multiple days, rather than simultaneously within the same day, the synergistic effect must be approximately 5% larger than otherwise to be reliably detected. Finally, it should be noted that that statistical significance is not the same as biological significance, thus a difference that can not be reliably detected by the analysis may still be important in practical terms.

As this work on noise in WHO tube assays was done in parallel with an equivalent study on WHO bottle assays using the same mosquito colonies, insecticides and operators ( Praulins et al., 2024 in review), a direct comparison can be made between the variability of each bioassay. In the bottle, the overall coefficient of variation for the susceptible Kisumu strain was 0.682 compared to 0.719 observed here for the tube test, making the assays very similar in overall variability. However, the proportion of this variation that is attributable to within-day and between-day is slightly different between assays. While in both assays most variation is between days rather than within them, this ratio is particularly pronounced in the bottle assay where standard deviation within:between was 0.350:1.236 for the tube compared to 0.145:1.788 for the bottle. This may indicate that while the bottle assay itself is more consistent than the tube assay, the coating of the bottles may introduce greater variation compared to the making of insecticide papers used in tube assays.

The power simulation approach used here demonstrates a framework for taking a dataset of observations for a given bioassay and using it to produce sample size guidance for conducting an assay that is powered to detect a defined mortality difference. While typically sample size calculations are performed by individual research groups for a given planned experiment, potentially resulting in inconsistent quality and robustness, this framework allows an accessible, uniform approach. The end-user need only input their target effect size through a user interface, either accepting default assumptions about variability or manually setting these if they have specific estimates from their own lab. This ensures a uniformity of robustness and allows results to be compared across time or between studies. The underlying statistical methods can also be updated over time as guidance on how to perform the assay changes. Additionally, as all simulations have been performed in advance, the computational demands to perform the power analysis are greatly reduced for the end user, who simply needs to open a web browser to use the application. As the WHO tube bioassay is arguably the most straightforward of all WHO mosquito bioassays in terms of designing power simulations, it is ideal for a functioning proof-of-concept to adapt for other bioassay designs. The broad approach is readily applicable to other laboratory bioassays such as WHO cones or tunnel tests, and semi-field experiments such as Experimental Hut Trials ( Challenger et al., 2023).

Ethical approval and consent were not required.