Maximising power with the minimum number of mosquitoes: Designing robust sample sizes for the WHO Cone Bioassay

Literature review

To investigate the use of the WHO cone bioassay for the bioefficacy evaluation of pyrethroid ITNs we conducted a literature review of studies published between 1998 and 2021. The search was conducted in 2021 using the keywords “bio-efficacy” or “cone bioassay tests” or “Insecticide treated nets” and “long lasting insecticidal nets” and later updated in 2024 to include any subsequently published papers. We used inclusion criteria to identify studies with Anopheles mosquitoes that reported 24 hr mortality. The search identified 2,487 titles (148 from PubMed and 2,338 from PubMed Central), which we screened to identify relevant papers. Duplicate papers already found in PubMed were removed from the PubMed Central search results leaving a total of 632 papers for screening. Further screening for papers that met the inclusion criteria narrowed the results down to 55. The selected papers were thoroughly analysed, and specific details were recorded to ensure a comprehensive evaluation of study methodologies. This structured approach ensured that all relevant methodological details were consistently captured across the reviewed studies. The information which was extracted is detailed in Table 1.

Using WHO cone bioassay data from multiple laboratories to measure variability in mortality

Existing WHO cone bioassay data was obtained based on pyrethroid-susceptible Anopheles gambiae exposed to the same six new, unwashed pyrethroid-treated nets which were circulated around seven test centre laboratories. The purpose of evaluating this data was to identify an appropriate range of variability estimates for the simulation study.

Outcomes assessed were between-net and between-net piece variability (between-replicate variability could not be calculated in most cases due to data aggregation). Variability between parameters of interest and outcome (24 hr mosquito mortality) were analysed using generalised linear mixed models (GLMMs) with binomial link function within the ‘lme4’ package in R (version-4.3.3). To quantify variability in the outcome in each laboratory, random effect terms for ‘net’ (different nets of the same product), ‘net piece’ (different pieces from the same net) were included in the GLMMs. When considering overall net and net piece variability across the laboratories, ‘Testing lab’ (variable for different laboratory test centres) was considered as a fixed effect term as well as the random effects listed.

Statistical significance of model parameters was assessed using log-likelihood ratio tests, comparing the explanatory power of different models with and without the inclusion of a given parameter. Variability values associated with ‘nets’ and ‘net pieces’ were extracted from random effects estimates of the GLMM, with standard deviation used as the unit of variation. To visualise the spread of data between different nets, violin plots with a bee swarm function were created using the ‘ggbeeswarm’ package.

Simulation study

Here we use the simulation principles of Johnson et al. (2015) and the sample size framework of Mechan et al. (2024) to perform simulation-based power analysis. To briefly summarise, this simulation-based power analysis process for calculating sample size involves simulating a very large number of plausible datasets from variability estimates. Subsequently, each simulated dataset is analysed individually using generalised linear models (GLM) to determine how often an effect is detected. The aim is to establish the minimum sample size (i.e. number of nets, net pieces, and replicates) to reliably distinguish a difference in mean mosquito mortality between different treatment groups/ITN products. The threshold at which a sample size was deemed sufficient was 80% power (meaning an 80% chance of detecting the effect, per convention).

The power of different experimental designs was evaluated. A plausible range of combinations of effect sizes (differences in mean mosquito mortality), sample size (number of nets, net pieces, and replicates), and variability assumptions (informed by our analysis of existing data sets) were assessed ( Table 2). For each unique combination of simulation parameters, 5,000 plausible datasets were simulated. Each simulated dataset was analysed using a binomial GLM with net type as fixed effect. Following this, p-values were extracted from each GLM. The power for each parameter combination was reported as the proportion of 5,000 analyses for which p<0.05. By comparing the power for different sample sizes, we identify the minimum number of samples (nets, net pieces, and replicates) needed to detect a given effect size (e.g. a 15% difference in mosquito mortality). Unless otherwise stated we report power for a given experimental design at the point of maximum uncertainty (‘Umax’), meaning a 15% mortality difference represents the difference between 50% and 65%. To maximise accuracy of power estimates close to 80% (78-82%), the threshold at which a design is either sufficiently powered or not, we performed follow-up evaluations of these experimental conditions with 15,000 simulations. For the purposes of reporting here, power estimates values >79.5% were rounded up to 80%.

Evaluating the 2013 WHO guidance for phase I laboratory testing

As we cannot show all the possible designs for a WHO cone bioassay experiment here in this article, we focus the results on the practical example of the sample size recommended in the 2013 WHO guidance for phase I laboratory testing (“Guidelines for laboratory and field-testing of long-lasting insecticidal nets”) ( WHO, 2013). Phase I laboratory testing requires a sample size of four nets, one net piece, and ten replicates on each ITN treatment group (for a total of 200 mosquitoes per treatment group).

We evaluate the power of a WHO cone bioassay experiment to detect mortality differences with this fixed sample size, as well as showing the impact of increasing or decreasing sample size beyond this. This is not to say that the findings are limited to this context, but instead it is used for illustrative purposes. The open access associated WHO cone bioassay sample size Rshiny application ( Cone Bioassay Sample Size app) allows the reader to investigate power for other plausible study designs. This is particularly beneficial for visualising the dynamic relationship between the different number of nets, net pieces, and replicates at the same time, which is challenging to communicate in a static format. This Rshiny application was created in R (version-4.3.3).

Characterising the reporting and application of the cone method from the literature

The methodological details extracted from the selected papers identifies great inconsistency in how methods are referenced and how much methodological detail is reported, as well as the details of how the tests themselves were performed ( Figure 1). Only some authors referenced guidelines in the Methods sections of the published studies and many referenced guidelines that were years out of date at the time of publication. Out of the 55 papers selected for analysis 15% did not reference any guidelines, while 5% referenced published papers instead of guidelines. Of the papers that did reference WHO guidelines or other associated documents, 4% used the 1998 document “Test procedures for insecticide resistance monitoring in malaria vectors, bio-efficacy and persistence of insecticides on treated surfaces: report of the WHO informal consultation, Geneva, 28-30 September 1998” ( WHO, 1998), while 18% used the 2005 guidelines “Guidelines for laboratory and field-testing of long-lasting insecticidal nets” ( WHO, 2005). There were no papers that referenced the 2006 guidelines “Guidelines for testing mosquito adulticides for indoor residual spraying and treatment of mosquito nets” ( WHO, 2006) or the 2024 guidelines “WHO Prequalification of Vector Control Products Implementation guidance (Modules 3 and 5) Bioassay methods for insecticide-treated nets: Cone test” ( WHO, 2024c). Only 7% referenced the 2011 guidelines “Guidelines for monitoring durability of long-lasting insecticidal nets under operational conditions” ( WHO, 2011), while 47% referenced the 2013 guidelines “Guidelines for laboratory and field-testing of long-lasting insecticidal nets” ( WHO, 2013), and only 2% referenced the 2016 guidelines “Test procedures for insecticide resistance monitoring in malaria vector mosquitoes: second edition” ( WHO, 2016). A further 2% of studies referenced both WHO (2011) and WHO (2013) guidelines. In terms of how many years out of date the referenced guidelines were, the selected papers varied widely, with 7% using guidelines that were up to date, and 57% used guidelines that were 3 years out of date. The remaining 36% of studies referenced guidelines that were 5-18 years out of date. There is a need for greater consistency in the referencing of guidelines and adherence to up-to-date guidelines to ensure the reliability and comparability of results across studies.

Figure 1. Standardisation in WHO cone bioassay testing.

(A) Proportion of referenced guidelines used for cone bioassay testing. (B) Number of years since the referenced guidelines became outdated. (C) Types of netting used as a negative control. (D) Net sample piece sizes. (E) Testing age of mosquitoes. (F) Number of mosquitoes used per cone.

Methodological details extracted from the papers included the sources of the nets used in the studies, which were diverse and included direct sourcing from the chemical company, local markets, homes, research institutes, storage facilities, and the Ministry of Health. However often it was not stated at all where the nets were sourced from. Additionally, the storage conditions for the nets varied widely, with some studies not stating the storage conditions and others using variously foil, plastic bags, or envelopes to store the nets, sometimes in the fridge.

The size of the net sample used in the bioassay varied, with most studies using 25 × 25 cm samples, while others used similar sized pieces at 30 × 30 cm or 20 × 20 cm samples, or instead did not break down nets and instead did cone bioassays on whole nets. A total of 35% of papers did not state the size of net piece tested. They reported the total number of nets used, which varied from one to 932. For each net, they indicated the number of net pieces collected, with a range of one to ten net pieces per net. Additionally, they detailed the number of cones in each net piece, which ranged from one to 16 cones.

The papers provided a range of information regarding their mosquito sampling. The age of the mosquitoes used in the bioassay also varied, with the majority (45%) of studies using 2–5 day old mosquitoes, while others used 2-4 day (2%), 2-3-day (9%), 3-4 day (4%) or 3–5 day (18%) old mosquitoes. Some studies (20%) did not state the age at all, and some used wild caught mosquitoes of unknown age (2%).

The number of mosquitoes exposed per cone varied, with most studies using five mosquitoes per cone (73%), while others used seven (2%), ten (19%), eight to ten (2%), nine to 12 (2%) or 20 (2%) mosquitoes per cone, or did not state the number (13%). The negative control netting used in the studies varied, with non-specific “untreated” nets being the most common. Finally, papers typically specified the overall sample size for the study, which varied widely from 20 to 20,000 mosquitoes.

Determining appropriate variability assumptions for the simulation study

Variation in 24 hr mortality between nets (of the same product) and between net pieces (from the same net) in the WHO cone bioassay was quantified using testing data with the same samples across multiple laboratories. With pyrethroid-susceptible An. gambiae Kisumu exposed to a pyrethroid-only ITN the between-net variation differed across laboratories, with standard deviation (SD) values ranging from 0.55 to 2.00 ( Figure 2 and Table 3). Variability between pieces from the same net also differed between laboratories but to a lesser extent (SD range: 0.08 to 0.85) compared to between nets ( Table 3).

Figure 2. Percentage 24 hr mortality of ITNs tested across multiple laboratories.

Plots highlighting 24 hr mortality of pyrethroid-only insecticide treated-nets (ITNs) (LLIN 1 to LLIN 6) samples tested across different laboratories (Lab A-F, I) with pyrethroid-susceptible An. gambiae s.s. mosquitoes. Black dots represent raw percentage mortality values from individual bioassays (one dot represents 25 mosquitoes).

When measuring overall variability across laboratories by combining all data, the between-net SD was 0.641 and between-piece SD was 0.249 (across laboratories in Table 3). Replicate variability could not be directly assessed due to the data being aggregated. As in practice many laboratories may have substantially higher variability, to be conservative we take these summarised variability values as a pragmatic minimum/‘floor’ variability estimate. We use this floor estimate, as well as the range of values observed to establish a conservative set of three categories of variabilities: ‘moderate’, ‘high’, and ‘very high’ ( Table 4). The preset variability assumptions are intended for use where robust estimates are not available prior to testing; it would be reasonable for researchers to assume lower variability for their own power calculations if indicated by robust laboratory-specific data.

Limits of interpretation of the fixed 2013 sample size guidance

Here we investigate mortality differences that can be reliably detected with different sample sizes using variability assumptions in the pre-set categories established above. We use the sample size guidance in the 2013 phase I laboratory testing documentation as a starting point, then demonstrate the impact of increasing or decreasing the number of samples beyond this. In the 2013 WHO guidance, sample size is fixed at four nets of each treatment group, with one piece from each net and ten replicate cones on this piece (for a total of 200 mosquitoes per treatment group).

As would be expected, the probability of detecting a difference increases as the values being compared against moves away from 50% (in either direction) ( Figure 3). Thus 60% vs 70% requires less samples to reliably detect a given difference than 50% vs 60%. Assuming our ‘moderate’ estimate (between-net SD of 0.6, and a between-replicate SD of 0.3) the smallest difference that can be detected by this sample size guidance with, at least 80% power at the point of maximum uncertainty (‘Umax’, where the reference value is 50% mortality), is 17.5% (power: 90%). A 10% difference is only reliably detectable with this sample size guidance and ‘moderate’ variability when reference mortality is 82.5% or above (i.e. 82.5% vs 92.5% can be reliably detected).

Figure 3. Power to detect 10% mortality differences between treatments in cone bioassays.

Power to detect a 10% difference in 24 hr mortality between pyrethroid-treated nets at determined variability levels in the WHO cone bioassay following the 2013 WHO guidelines (four nets, one net piece per net and ten replicates per net piece). The levels of variability (Var) assessed are: ‘moderate’ (‘M’) standard deviation (SD) values = Net 0.6, Replicate (Rep) 0.3, ‘high’ (‘H’) SD values = Net 1.2, Rep 0.6 and ‘very high’ (‘V-h’) SD values = Net 1.6, Rep 0.9. The red dashed horizontal line represents 80% power threshold.

As shown in Figure 3, assuming greater variability leads to less power to detect a 10% difference. Assuming ‘high’ variability (between-net SD of 1.2, and a between-replicate SD of 0.6) still allows a 17.5% difference to be detected at the ‘Umax’ (power: 84%). However, if assuming ‘very high’ variability (between-net SD of 1.6, and a between-replicate SD of 0.9) the smallest mortality difference between treatment groups that can be detected by this sample size at ‘Umax’ is 20% (power: 89%).

How does changing the sample size impact power?

Here we explore how changes to the sample size guidance would impact the power to detect different mortality differences. We investigate power using ‘moderate’ and ‘high’ variability estimates mentioned above, assessing how power decreases as the variability increases.

Net number

‘As outlined above the smallest difference that can be reliably detected with the 2013 WHO guidance at the ‘Umax’, with ‘moderate’ net variability assumption, is 17.5% (power: 90%) ( Figure 4A). If instead just three nets were used, a 17.5% difference could still be detected (power: 80%). Detecting a 15% difference requires five nets (power: 88%), with a 10% difference outside the scope of our simulations (ten nets: 79% power). However, as reference mortality approaches 100%, a 10% difference becomes more feasible to detect with four nets. The threshold at which a 10% difference becomes detectable (under the same net piece, replicate, and variability assumptions) is approximately 82.5%.

Figure 4. Power to detect mortality differences between treatments when alternating nets.

The power to detect different 24 hr mortality effect sizes between pyrethroid-treated nets considering alternate numbers of nets, when net variability and replicate variability is assumed ‘moderate’ (A) and is assumed ‘high’ (B). Black solid line represents power of the 2013 WHO guidance (four nets) across different mortality differences at the maximum point of uncertainty ('Umax'). ‘Moderate’ standard deviation values: between-net = 0.6 and between-replicate = 0.3, ‘high’ standard deviation values: between-net = 1.2 and between-replicate = 0.3 keeping the number of net pieces fixed at one and number of replicates fixed at ten as per the 2013 WHO guidance. Red dashed horizontal line represents 80% power threshold, black dashed vertical lines represent true mortality difference of 5%, 10%,15%, 17.5% and 20%.

Under our ‘high’ net variability assumptions, a 20% difference can be reliably detected at the ‘Umax’ using three nets (power: 84%), and a 15% difference with six nets ( Figure 4B). A 10% difference is outside the scope of simulations but becomes detectable with four nets (as per the 2013 WHO guidance) when reference mortality is 82.5% or higher.

Net piece number

The power to reliably detect a difference between treatment groups increases with the number of net pieces, with the largest increase from one to two net pieces but the additional benefit diminishing noticeably above three net pieces ( Figure 5A). We fix the number of nets at four and replicates at ten as per the 2013 WHO guidance.

Figure 5. Power to detect mortality differences between treatments when alternating pieces.

The power to detect different 24 hr mortality differences between pyrethroid-treated nets across different numbers of net pieces, when net variability and net piece variability is assumed ‘moderate’ (A) and assumed ‘high’ (B). Black solid line represents power of the 2013 WHO guidance (one net piece) across different mortality differences at the maximum point of uncertainty ('Umax'). ‘Moderate’ standard deviation values: between-net = 0.6, between-net piece = 0.3 and between-replicate = 0.3, ‘high’ standard deviation values: between-net = 1.2, between-net piece = 0.6 and between-replicate = 0.3, keeping the number of nets fixed at four and number of replicates fixed at ten as per the 2013 WHO guidance. Red dashed horizontal line represents 80% power threshold, black vertical lines represent true difference % at 5%, 10%,15%, 17.5% and 20%.

With the standard sample of one net piece, a 17.5% difference can be detected with the 2013 WHO guidance sample size at ‘Umax’ when the variability is ‘moderate’ ( Figure 5A). A 15% difference can be detected using two net pieces (power: 95%) under these assumptions. A 10% difference can be detected with three net pieces (power: 84%).

With ‘high’ variability assumptions (keeping replicate variability ‘moderate’), the 2013 WHO guidance can detect a 17.5% difference at ‘Umax’ ( Figure 5B). A 15% difference can be detected with two net pieces (power: 91%) and 10% with four net pieces (power: 86%).

Replicate number

Assuming a ‘moderate’ level of net and replicate variability, the 2013 WHO guidance can detect a 17.5% difference between treatment groups (power: 90%) ( Figure 6A). Keeping the number of nets fixed at four and pieces fixed at one, detecting a 15% and 10% difference is outside of the range of scope of simulations at ‘Umax’ when altering the number of replicates. However, if the reference mortality is just slightly higher than the ‘Umax’ (55%), a 15% difference becomes detectable with ten replicates. Furthermore, a 10% difference becomes detectable when the reference mortality is 82.5%. It is noteworthy that under these assumptions a 20% difference is still detectable with fewer replicates than the 2013 WHO guidance, with just six replicates giving 82% power.

Figure 6. Power to detect mortality differences between treatments when alternating replicates.

The power to detect differences in 24 hr mortality between pyrethroid-treated nets across different numbers of replicates when net variability and replicate variability is assumed ‘moderate’ (A) and is assumed ‘high’ (B). Black solid line represents power of the 2013 WHO guidance (ten replicates) across different mortality differences at the maximum point of uncertainty ('Umax'). ‘Moderate’ standard deviation values: between-net = 0.6 and between-replicate = 0.3, ‘high’ standard deviation values: between-net = 1.2 and between-replicate = 0.6, keeping the number of nets fixed at four and numbers of net pieces fixed at one as per the 2013 WHO guidance). Red dashed horizontal line represents 80% power threshold, black vertical lines represent true difference % at 5%, 10%, 15%, 17.5% and 20%.

Assuming a ‘high’ variability, the 2013 guidance can detect a 17.5% difference between treatment groups (power: 84%) ( Figure 6B). Reliably detecting a 10% and 15% difference under the 2013 WHO guidance is outside of the range of scope of simulations at ‘Umax’. However, if the reference mortality threshold is increased to approximately 62.5% a 15% difference becomes detectable. A 10% difference becomes detectable at a reference mortality at approximately 82.5% or higher. As with ‘moderate’ variability assumptions, at ‘high’ variability with fewer replicates than the 2013 WHO guidance, a 20% difference is detectable with just seven replicates giving 82% power ( Figure 6B).

Optimising sample size of nets, net pieces and replicates to minimise the number of mosquitoes needed for testing

As shown above there is a trade-off between the ability to detect small differences and the size of the sample needed. For researchers performing a WHO cone bioassay the limiting resource is typically testing-age female mosquitoes. Identifying the optimal trade-off between detecting a sufficiently small difference and the mosquitoes needed to achieve this can minimise the resources needed for an experiment. Excessive effort can be avoided if identifying very small differences is not necessary to answer the research question, and wasted effort avoided by not performing experiments underpowered to answer the question.

As shown above the 2013 WHO guidance to use 200 mosquitoes per treatment group is powered to detect a 17.5% difference at ‘Umax’, assuming ‘moderate’ to ‘high’ variability. However, decreasing the number of replicates from ten to six still gives sufficient power to detect a 20% difference under the same conditions, reducing the number of mosquitoes required from 200 to 120. This highlights the trade-offs between power and sample size.

Increasing the number of net pieces per net has the greatest mosquito cost, yet using only a single net piece means a core source of variability (between-net piece) is not accounted for. Up to this point, this paper has considered the impact of varying net, net piece, and replicate sample sizes independently, yet in practice all three may be adjusted at the same time to optimise power. We show here that for different research objectives in terms of the effect size to be detected, there are optimal combinations of net, net piece, and replicates that maximise power for the fewest mosquitoes (refer to Table 5).

Supporting sample size calculations for the WHO cone bioassay with a user-friendly application

To allow end-users to readily design their own experiments with WHO cones, we provide a browser-based Rshiny application. This app allows researchers to take advantage of the simulation-based framework outlined above, without the need for access to computer programmes or hardware. Users can input their minimum effect size and level of variability (for net, net piece, and replicate number), with the output all of the viable experiments they could perform that would provide robust data. Additionally, the app provides an interactive visualisation of how power changes with different numbers of net, net pieces, and replicates, across a fine-scale range of variability values (allowing tailoring to a laboratory’s level of variation). Click here for access to the application for interactive visualisation to explore other combinations of sample size to support experimental design for the WHO cone bioassay: Cone Bioassay Sample Size app https://fmechan1.shinyapps.io/who_cone_app/.