Planning and communicating prototype tests for the Nano Membrane Toilet: A critical review and proposed visual tool

Case study: The NMT, a complex product

Sanitation, the containment, transport, and treatment of human excrements, is a topic of high significance for human development (Jahan, 2016): UNICEF (2017) stress the importance of safe sanitation for children’s health and that improving (access to) sanitation could reduce child mortality. Lack of sanitation has been linked to reduced cognitive development in children (Sclar et al., 2017) as well as stunting, caused by environmental enteric dysfunction (Budge et al., 2019), and to a risk of assault, particularly for women and girls practicing open defecation (Jadhav et al., 2016; Miiro et al., 2018).

Urban sanitation poses particular difficulties, due to the lack of piped water and prohibitively high cost of sewer systems (Cobbinah & Poku-Boansi, 2018; Parnell et al., 2007). The most commonly promoted sanitation systems in cities of the Global South involve toilets that use little to no water, i.e. dry toilets, and store the faecal material onsite. Examples of this are pit latrines, pour-flush toilets, urine-diversion dry toilets (UDDT), and septic tanks (Semiyaga et al., 2015). When full, these toilets are emptied, and the faecal sludge is transported and either treated and reclaimed or discharged, or discharged without treatment. However, there are problems with these sewer-less sanitation services, such as high fees for emptying services, collection and transport trucks not being able to access the houses, high transport costs to treatment facilities or the altogether lack of such facilities (Strande, 2014). Another obstacle for the success of these technologies is public acceptance: Users can consider the pedestals of dry toilets uncomfortable, dirty, or malodorous, and they may worry their children could fall into the pit (Mkhize et al., 2017; Roma et al., 2013).

To find a solution to these problems, the Bill & Melinda Gates Foundation (BMGF) initiated the Reinvent The Toilet Challenge (RTTC) to “create a toilet that:

As result, research institutions and companies worldwide are now developing waterless, non-sewered sanitation technologies (Bill & Melinda Gates Foundation, 2013). The reinvention of the toilet requires unconventional thinking. Niemeier et al. (2014) discuss the challenges of building technologies for the development context: “If we are to resolve global inequities in access to innovations that improve health, we must adopt new approaches to engineering design that reflect the unique needs and constraints of low-resource settings”. They further mention how “efforts like the [RTTC] reflect the kind of integrative thinking that must occur at the beginning of a design initiative […]”. One example of a reinvented toilet is the NMT, conceived by researchers at Cranfield University (Parker, 2014). The NMT is a household-level, onsite sanitation system that looks similar to a porcelain water flush toilet (Figure 1 and Figure 2). It uses combustion and membrane processes to treat the mechanically separated solid and liquid waste streams. With all its components, the NMT would not just replace the currently existing dry toilet technologies, but also the associated faecal sludge management services, thus offering a form of safely managed sanitation (WHO & UNICEF, 2017). It is not simply a human waste receptor, but rather a miniature faecal sludge treatment facility.

Figure 1. Conceptual schematic of the NMT and its components.

The front end comprises the mechanical flush with its rotating bowl and rubber swipe, the collection tank with the grid and weir and the screw conveyor. The back end consists of the dryer, the combustor, and the membrane bundles.

Figure 2. Nano Membrane Toilet front-end prototype.

Naturally, there are numerous considerations to be made during the development of such a technology. The NMT combines entirely novel technologies with already existing ones. However, even for the well-established technologies, their application for this specific purpose is novel, and requires further research in order to miniaturise, integrate and optimise for off-grid functionality. At the same time, from a user’s perspective, not much should change when transitioning from using another dry toilet or a porcelain flush toilet to the NMT. To fulfil the RTTC’s demand for an aspirational design (Bill & Melinda Gates Foundation, 2018), it should be comfortable, appealing, and simple to use. It should take into account the preferences and customs of users from a diverse range of cultural backgrounds. Fejerskov (2017) emphasizes the importance of considering the users of a newly developed technology: “[…] a technology developed in isolation from those who are supposed to benefit from it cannot be expected to wield predictable outcomes.” There is research on the preferences of toilet users in various contexts, from industrial nations like the Republic of Korea (Lee, 2019) and Canada (Morales et al., 2017), to a focus on elderly users (Dekker et al., 2011) to low- and middle income countries (Austin-Breneman & Yang, 2017; Nelson et al., 2014). Hence, developing the NMT entails developing a user-friendly user interface—in software development projects this would be called “front end” (Reza & Grant, 2007)—and a “back end” comprising several sub-technologies, and integrating them into the overall system.

The RTTC has thus led to an unusual case of PD at this scale: It asks for a product that connects existing notions of a toilet’s function and design with never-before-seen technologies. A viable solution to the problems associated with dry sanitation must simultaneously satisfy users’ ideas of aesthetics and comfort, and adhere to high standards of safety and reliability.

The NMT is a complex product, which is, in fact, a combination of sub-products that have to be developed individually and then integrated into the overall system. The original design brief for the NMT was the RTTC, which included important user-centred objectives of aspirational design and affordability, as well as objectives aiming at sustainability and at solving the problems of urban non-sewered sanitation (Bill & Melinda Gates Foundation, 2013). From this, initial design ideas were conceived involving membrane treatment of liquids and water recovery through condensing beads, as well as the drying and coating of solids (Parker, 2014). Later design stages discarded the condensing beads and a combustion process was devised to replace the coating of solids. Considering that the user of the toilet would usually not interact directly with the treatment processes, these were not subjected to user testing yet. The pedestal, the part of the toilet with which the user interacts, mainly differs from a porcelain water flush toilet in its mechanical flush. It was developed as the result of studies among potential users in Ghana and subsequent agile innovation processes (Tierney, 2017). Several iterations of prototypes were produced to develop a mechanical flush, until it could be tested in real-use scenarios.

This mechanical flush – a rotating bowl and rubber swipe activated by moving the toilet lid – separates the user from a tank underneath the toilet pan (Tierney, 2014; Tierney, 2017). Solids are separated through settling and displacement, transported by an screw conveyor (Mercer et al., 2016) and subsequently dried and combusted (Fidalgo et al., 2019; Onabanjo et al., 2016), while the liquid fraction is extracted through a weir and purified through membrane processes (Kamranvand et al., 2018; Wang et al., 2017), driven by the heat of the combustion, which is transferred via a heat exchanger (Hanak et al., 2016). The toilet pedestal, including the mechanical flush, the screw conveyor, and the liquid weir are considered the NMT’s front end. The dryer, combustor, and membrane components are considered its back end. The NMT is envisioned to be independent of water- or sewer connections and energy neutral, or even have a positive net power output (Kolios et al., 2018). However, the back end components have not yet been integrated and combined with the front end to produce a fully functioning prototype of the NMT. Such tasks are envisioned to be conducted in the near future.

At the moment, all sub-products which the NMT comprises are in an iterative phase of building and testing prototypes. The front end has been re-designed as result of field tests involving target users of the NMT (Hennigs et al., 2019). Prototypes of the dryer and combustor (Jurado et al., 2018) have been tested in the lab in several iterations. In addition, the recovery of electrical energy by reverse electro-dialysis is under investigation.

This means that the individual components and sub-products of the NMT are developed enough to plan for integration of all sub-products into a complete prototype. Such a prototype would then be tested in laboratory tests, and once its safe operation was sufficiently proven, it could be deployed for user-centred field tests. The aim of such tests, and concurrent further sub-product improvements, would be to optimise the operational settings of the entire system. The recently published ISO 30500 standard (ISO, 2018) could provide the benchmark performance values the prototype has to achieve. Once the prototype meets these values, its design can be polished for usability and manufacture. Final reliability and user tests of this polished design would ensure the NMT’s usability and reliable functionality throughout its lifetime, and when passed, allow this design to confidently be tested for ISO 30500 standard-compliance, making it a market-ready product.

Consequently, there are still numerous prototype tests which need to be planned and conducted. The development and testing of prototypes of the NMT’s various sub-products to date have not been guided by a visual test planning tool. Instead, prototypes of components were developed and tested by the teams working on these components. In the case of the front end, a prototype was developed by one team and then tested by another (Hennigs et al., 2019). It could therefore be possible that the prototype tests to date could have been conducted more effectively had they been planned in a more coordinated manner. Similarly, the communication within and between the various teams working on the NMT could benefit from a more consolidated terminology and shared understanding of the development process and the associated testing activities. A visual test planning tool could thus facilitate and improve the planning and communication of testing activities in the future.

Literature review

Using our own publications and those of our colleagues on the project, the development history of the NMT was established as that of a complex product and outlined in the section above. Subsequently, the Scopus and Google Scholar databases were used for an exploratory search of peer reviewed literature, with a focus on literature reviews covering a range of publications, to advance the understanding of the broad field of prototype testing in PD. Search terms included review, prototyping, prototype testing, product development, technology development, testing, and others. Using these terms in various combinations, promising publications were identified and studied individually. In the search, several repeatedly occurring aspects of testing prototypes were identified and then further investigated in a more targeted search on the same databases. The aspects were types of prototype tests, phases of the PD process, iteration, usability testing and user centred design, reliability testing, testing for standard compliance, design of experiments, back end and front end testing, and visualisations of PD processes. They were chosen for further investigation because of their repeated occurrence in the literature and their apparent applicability for the context of this paper. Again, the focus of the resulting search was on reviews of the existing literature rather than original work, as the aim was to obtain a wide understanding of multiple fields of study, rather than an in-depth analysis of a single one. Similar search terms were used, with the addition of the identified testing aspects. The identified literature was then analysed to extract the information relevant to prototype testing activities, particularly for the development of complex technologies. This analysis yielded the section ‘relevant aspects of prototype testing’.

Creating a visual test planning tool

A visualisation of the prototype testing processes for the PD of the NMT was then conceived, with the aim to consolidate the collected information. A simple linear PD process model was chosen, divided into three phases, with parallel strands of testing for the front end and back end, considering different types of testing, DOE, usability testing, reliability/durability testing, and international standards at different stages of the process, and possible iteration loops throughout the process.

Types of prototype tests

Boës et al. (2017) define testing in the context of PD as “exposing a physical system to a condition or situation in order to observe the system’s response.” They then clarify the physical system as a representation of the product or one of its components, the condition or situation as a “use case as a whole or its effect on a subsystem”, and the system’s response as “the performance of a desired function [or] an undesired failure mode.” Using this definition, they propose four categories of testing activities according to the type of knowledge that is generated. First, trial and error tests can be used to gain a basic understanding of the development project and to explore the design space. Secondly, experiment tests resemble experimental work in fundamental scientific research in their structured approach in order to identify influencing factors and develop “necessary system knowledge”. Thirdly, verification tests are usually pass/fail tests to determine if the system- or component prototype fulfils the requirements set at the beginning of the PD process. Lastly, validation tests determine whether the product addresses the underlying user needs, rather than the requirements set by the product developer. They are commonly conducted with a fully functional prototype.

Camburn et al. (2017) comprehensively review the state of the art in prototyping techniques, strategies, and guidelines, and devise a visual framework to connect prototyping objectives to techniques. They clearly distinguish prototypes from design concepts by stating that prototypes are always tied to tests. They discuss literature on preparing for prototyping, enhancing prototype performance, reducing cost and time, and fabricating prototypes, before reflecting on prototyping science. While they don’t specifically identify types of prototype tests, they do list frequently cited prototyping objectives, namely active learning, exploration, communication, and refinement. They further develop scales of prototype distinction, i.e. “system (isolated or integrated); media (virtual or physical); requirements (relaxed or final); and scale (reduced or final)”. They also discuss iterative prototyping, parallel prototyping, requirement relaxation, subsystem isolation, scaled prototyping, and virtual prototyping as individual prototyping techniques.

There seems to be the implication that there are different types of tests being conducted throughout the PD process that differ in their level of formality, in their approach, and in the knowledge they are designed to produce. A similar observation can be made about the prototyping tests for the NMT. We have identified three distinct types of prototype tests conducted so far, namely user tests, laboratory tests, and field tests. These types of tests can coincide and overlap: just for one generation of a front-end prototype, Hennigs et al. (2019) conducted user surveys and interviews (user tests), photography and image analysis (laboratory tests), and field tests on the material choice of a rubber swipe and the long-term robustness of the prototype (field tests). A fourth type of tests are international standard tests. In the case of the NMT, these would be described in the standard ISO 30500:2018 Non-sewered sanitation systems (ISO, 2018).

Phases of the product development process

In the case of usability testing, Rubin & Chisnell (2008) distinguish three phases of testing: First, exploratory tests are conducted in the early stages of PD, to test its basic design, i.e. whether users find it intuitively appealing. Then, assessment tests, conducted about halfway through the development process, expand the knowledge on the product’s usability, i.e. whether users can perform the intended tasks on the product. Lastly, the validation/verification tests at the end of the cycle tend not to inform further iteration, but rather confirm that all previously identified problems have been resolved, and that the entire product can be used as intended. In these phases, the basic design, early to well-developed prototypes and the final product are tested on potential users to identify their likes and problems.

In his Stage-Gate model, Cooper (1990) considers five stages between a product idea and a “post implementation review”, namely 1. Preliminary assessment; 2. Detailed investigation (business case) preparation; 3. Development; 4. Testing and Validation; and 5. Full production and market launch. Royce’s (1970) Waterfall model comprises seven steps to develop a large computer program: 1. System requirements; 2. Software requirements; 3. Analysis; 4. Program design;5. Coding; 6. Testing; and 7. Operations. Similarly, other models of PD processes are also sorted into phases or steps (Boehm, 1988; Forsberg et al., 2005). While testing is a single distinct step in most of such models, the various types of prototyping activities and prototype tests throughout the PD process could also be imagined to be separated into phases as the product’s maturity increases. Rubin & Chisnell’s (2008) three phases (exploration, assessment, validation/verification) provide a sufficient level of distinction for the present work on the NMT.

For this purpose, the objective of the exploration phase is to explore potential solutions to the design brief, to discard unviable ones, and to gather an understanding of the required development process, i.e. to develop the questions that need to be answered in the assessment phase.

The objective of the assessment phase is to expand the knowledge about the potential product solutions, to answer the questions developed in the exploration phase. The outcome of this phase should be one single, functional prototype.

The objective of the verification and validation phase is to ensure that all questions have been answered, that the product functions as expected and is, in fact, a solution to the design brief.

Iteration

Iteration occurs throughout the PD process and can have different causes and outcomes (Wynn & Eckert, 2017). With its earliest forms dating back to the 1930s (Larman & Basili, 2003), the sequential testing and refinement of a prototype (Christie et al., 2012) can be welcomed as a driver of positive design change, or seen as a wasteful, costly delay in a PD project (Ballard, 2000; Le et al., 2010; Wynn & Eckert, 2017), but it is undeniable that iteration occurs in nearly every PD process, particularly for complex products (Wynn & Eckert, 2017).

It is common to develop software user interfaces iteratively (Nielsen, 1993). For complex physical products, every iteration-step of building and testing a prototype can be associated with high costs (Tahera et al., 2019). It is therefore important to consider when and how many iteration-steps should be undertaken. Camburn et al. (2017) emphasize that iteration should occur often and early in the PD process, and should be encouraged the higher the potential for performance increase and the lower the cost per iteration. The higher the product’s maturity, the higher the cost of iteration will likely be. There is a multitude of publications discussing the complex nuances of iteration in PD, and Wynn & Eckert (2017) offer a comprehensive overview and terminology of this field of research: They differentiate between micro-level and macro-level iterations, as well as between three iterative functions of progress toward completion, correction of errors, and coordination of actors, decisions, or workflows. They assign multiple iterative stereotypes to each function: Progressive iteration comprises the stereotypes exploration, concretisation, convergence, refinement, and incremental completion. Corrective iteration covers new work, rework, and churn. Lastly, coordinative iteration encompasses the stereotypes governance, negotiation, parallelisation, comparison, and concentration (Wynn & Eckert, 2017). This illustrates the complex role that iteration plays in PD.

For the development of an NMT testing tool it is mainly important to know that, while iterative loops can occur throughout the PD process, they value for cost reduces as product maturity increases (Wynn & Eckert, 2017), and to consider which test results should trigger or prevent an iterative loop. With the main goal of the PD process being a marketable product, any proof that a prototype does not represent such a marketable product should be a trigger of iteration. Failed tests, like the shortfall against benchmark values, can be seen as such a proof. International or internal standards can provide such benchmark values. Cost is a deciding factor, and will have to be considered by a project manager when deciding whether to iterate further or not. However, the complex calculations weighing iteration-cost against the cost of having a less well-tested product would go beyond the scope of the present work.

DOE

Laboratory-based tests commonly involve the observation of a (sub-) product’s condition and/or outputs in relation to its inputs. As mentioned above, a comprehensive understanding of the systems’ outputs respective to their inputs and process variables is required. Often, there are several inputs and/or outputs for one component, and inputs can interact with each other to create second- or higher-order effects on the outputs (Montgomery, 2009). For example, potential factors that can affect the processes in a combustion chamber are the amount of fuel, its flux, its moisture content and calorific value, the process temperature as well as the flux, pressure, temperature, humidity, and oxygen content of the inflowing air (Jurado et al., 2018).

To test all inputs and their interactions, across their entire range, is either very difficult or impossible. It would take hundreds of tests to assess every factor’s influence on the combustion process. Some factors cannot be controlled; some cannot be changed without affecting another.

Based on the work of statistician R.A. Fisher (Fisher Box, 1980; Yates, 1964), DOE uses statistical approaches to address such problems, to minimise the time and effort required for a set of experiments while maximising the validity, reliability, and replicability of information gathered from them. The basic principles of DOE, initially developed for agricultural research, are (Cortes et al., 2018; Fisher, 1935):

It may occur that these principles have to be compromised to some extent for practical reasons, or that complex processes are to be investigated. Within the DOE-toolbox are methods such as split-plot design (Kulahci & Tyssedal, 2017; Lee Ho et al., 2016), fractional factorial design, response surface methodology, and random effects models (Montgomery, 2009) for such cases. It is thusly possible to achieve a high level of understanding from comparably few experimental runs. For example, Ilzarbe et al. (2008) found in their bibliographical review of 77 DOE applications in the field of engineering, that with an average of 5.06 factors to be investigated, 77% of the studies achieved this goal with 30 or fewer experiments, and 50% with 20 or fewer.

DOE finds application in PD efforts of various kinds: Pineau et al. (2019) used a fractional factorial design to assess which design factors of coffee vending machines impacted the sensory experience of the product the most. Gumma & Durgam (2019) improved the structural performance of a car’s body using a multi-model DOE sensitivity study, including simulations and experimental model testing. Sano et al. (2019) studied Bayesian optimisation techniques to reduce the number of experiments necessary to obtain the information with which they could improve production parameters for orally disintegrating tablets.

Thus, DOE encompasses a wide range of statistical tools for planning how to conduct tests, and how to analyse the results later on, to maximise the statistical validity of the lessons learnt. It does, however, not give any advice on what to test, or why. Other problems with DOE can be that statistical models developed through its use do not accurately reflect the observed processes (Deaconu & Coleman, 2002), or that it gives false credibility to results that stem from badly conducted experiments or the incorrect application of DOE principles. For example, modern technical processes and systems can and must be tested differently to fields of crops (Collins et al., 2011).

Reliability and Durability Testing

Reliability estimation, a part of reliability engineering, comprises reliability tests and the analysis of the data gathered in those tests. Kapur & Pecht (2014) define reliability as “the ability of a product to function properly within specified performance limits for a specified period of time, under the life-cycle application conditions”, i.e., how long a product functions without needing repair. This means that reliability tests are carried out to assess the likelihood of the product—or its components—failing over time. They are usually conducted on prototypes of high maturity, on randomly selected products fresh off the assembly line, or even products that have been in use for a certain time. Similar to DOE, statistical approaches are used to calculate a level of confidence with which a failure will occur in a given time (Kapur & Pecht, 2014).

Durability is a measure of a product’s lifetime, i.e. the how long the product can be used until repairs become more expensive than replacing it (Garvin, 1996). In other words, “durability is a particular aspect of reliability” (Tahera et al., 2019). This implies that, while the calculation of durability will involve repair costs and other economic factors, the durability tests to assess likelihood of failure will often be similar or identical to reliability tests. This similarity is reflected in existing literature, where both terms are often used in conjunction (Lee et al., 2015; Munson et al., 2018), and sometimes seemingly synonymously (Bluvband, 2012; Jimenez, 2017).

For both, the challenges lie in accelerating the product’s lifetime: It is not feasible to test a statistically significant number of product units over a number of years in order to assess their reliability and durability for this timespan. Therefore, a reliability/durability engineer attempts the realistic emulation of real use scenarios and environmental conditions in a shortened period of time by applying potential stresses, like shock, vibration, or climatic conditions in rapid succession, periodically, or simultaneously (Cheon et al., 2015; Donovan & Murphy, 2005; Zanoff & Ekwaro-Osire, 2010). For this aim, accelerated life testing (ALT) is used to determine a product’s time until failure by compressing its lifetime in a short period, usually weeks or months. Highly Accelerated Life Testing (HALT), in contrast, is a technique to determine the most likely failure points of a product but compressing its lifetime into a very short period, usually hours or days (Silverman, 2006).

Difficulties with these approaches lie in the complexity of combined stresses and failure modes, particularly on complex physical products. It is difficult or impossible to “model multiple (or competing) failure mechanism[s] to support reliability testing methods” (Bhamare et al., 2007). Furthermore, reliability / durability engineering does not consider the user’s experience, but rather focuses solely on the product’s reliable functionality. Thus, reliability / durability tests may miss important inputs, as (mis-)use is an important factor in the lifetime of a product, and important outputs, as the user experience may be a more important factor in design changes than increased reliability and durability. For example, a sturdier handheld device may be more reliable and durable, but too heavy or impractical to use.

Testing for technical standard compliance

International “technical standards are established norms or requirements applied to technical systems. They are a crucial aspect of almost all industries […]” (Shin et al., 2015). They play an important role in technology development (Østebø et al., 2018), by providing “a benchmark for quality and acceptability in the market place” and guidance on the “safety, reliability, efficiency and interchangeability” of products (Tyas, 2009). The testing procedures and performance requirements outlined in technology standards form the basis for the process of ensuring a product is compliant with the standard before being released to market. This does, however, not mean that the first time the standard should be consulted is at the end of the PD process. Instead, the performance, safety and other requirements provide the benchmark to which even early prototypes can be compared, and the testing procedures and-protocols can be adapted or used directly to test prototypes of sufficient technological maturity.

Examples of standards being used as benchmarks during product testing are for a wireless fire alarm (Feo-Arenis et al., 2016), packaging of products (Nolan, 2004), or sensor interface circuits for the automotive industry (Ohletz & Schulze, 2009). Another example is the ‘syngina test’, the standardised test for tampon absorbency. The standard was developed by the American Society for Testing and Materials, when a link between tampon size and toxic shock syndrome was discovered, but customers could not reliably buy tampons of similar absorbency from different brands (Vostral, 2017).

While international standards provide this much-needed guidance, it is important to remember that they are not infallible and may overlook important aspects especially of innovative technologies. For example, Mjör (2002) noted that, for dentistry equipment, “parameters measured in the standards are often not predictors of clinical performance”, and often lacked clinical backing. For the case of the ‘syngina test’, Vostral (2017) discusses the issue that the test is merely a very coarse approximation of a menstruating human body. Narayanan & Chen (2012) discuss the fact that in the early stages of competing, similar technologies, the setting of a standard can result in a “winner-take-all outcome”, as seen in the competition between Betamax and VHS video systems. Moreover, Hu (2010) lists potential threats that international standards could pose to innovation, such as exclusion of innovative start-ups from the market and lack of incentive for leading companies to innovate beyond a minimum-standard level of quality, but also mentions that the benefits of standards toward innovation outweigh the limitations.

The ISO 30500 Standard

The attempt to assist innovation through standards can be applied to the development of sanitation technologies as well: the ‘International Organization for Standardization’ (ISO) recently published the standard ISO 30500:2018 Non-sewered sanitation systems. It “specifies general safety and performance requirements for design and testing as well as sustainability considerations for non-sewered sanitation systems”, and thus aims “to support the development of stand-alone sanitation systems […] and promote economic, social, and environmental sustainability […] ” (ISO, 2018). In the document, requirements for performance, materials, safety, maintenance, and sustainability are listed and testing procedures are described in great detail. The ‘Annex A – Test methods and additional testing requirements’ comprises 33 pages, and the main document 34. The range of tests covers a comprehensive list of aspects concerning the safety, quality, and usability of non-sewered sanitation systems, but does not necessarily account for the statistical variation in measurements, as would be considered in DOE. For example, only one unit of a new sanitation system is to be tested for the ISO 30500 standard. Similarly, the standard does not fully support a UCD approach. While the ease and safety of use are described as requirements, and consideration is given for variations in cultural requirements like the distinction between users preferring the squatting or seating positions, the standard cannot account for the broad variety of user preferences according to their physical, cultural, and social needs. Such considerations are given in UCD.

Usability testing and User Centred Design

The term ‘User Centred Design’ (UCD), first coined and publicised by Norman & Draper (1986) in the context of software design, encompasses a collection of processes and methodologies that follow the basic principles of a human-centred approach, which are now described in the international standard ISO 9241-210:2010 (ISO, 2010):

Usability testing, is only one, albeit essential, part of the entire UCD process (Bastien, 2010), and comprises in itself a range of possible tests: from complex experimental designs as produced with DOE methods and involving large numbers of participants to rather informal tests with a single potential user as participant. Considering the aim of such testing is often to obtain qualitative information for the design process, rather than obtaining statistically relevant design parameter values, less formalised, qualitative methods are the focus of Rubin & Chisnell’s (2008) ‘Handbook of Usability Testing’. While there are cases of user tests designed using DOE principles (e.g. Jensen et al., 2018), these remain the exception, as the more common approach for user tests seems to be a qualitative one (Bastien, 2010) .

Usability testing and UCD appear to be particularly important for products that have a high degree of complexity but are used by a broad spectrum of users, with varying degrees of expertise, for example a tubeless insulin pump (Pillalamarri et al., 2018). Another application would be for products specialised for a certain user group, like a motorcycle tool for one-handed users (Sudin, 2013). However, a large portion of usability tests is still conducted in software development, for example for a drill rig control system (Koli et al., 2014).

Distinction between front end and back end – lessons from software development

For the case of the NMT, a distinction between the toilet seat, bowl, and flush, i.e. the “user interface” or front end, and the treatment system, or back end, can be made. This is analogous to software products like web sites (Chen & Iyengar, 2003). The front end and back end of a system require different testing in the PD process (Bertolino, 2007; Sánchez Guinea et al., 2016; Sneed, 2004). For example, while the front end is the part of the product with which the users will interact, the back end is usually only of indirect concern for them, as long as everything operates as expected. Hence, the front end should undergo user testing early on (Chuang et al., 2011), while the back end might only require limited user testing at later stages, to ensure maintainability by trained personnel. On the other hand, the back end of the NMT will likely require more extensive laboratory testing than the front end, as the treatment processes are more compicated than the user interface.

It might therefore be sensible to consider testing for the front end and back end independently, and in a separate step actualise integration testing of the individual components.

Visualisations of PD processes

As mentioned above, visualisations can be beneficial for product developers to plan and communicate their work (Keller et al., 2006). Wynn & Clarkson (2018) give a very comprehensive overview of a large body of work on process models in PD, including visualisations thereof. They categorise models focusing “on the large-scale organisation and management of design and development” as macro-level models. Examples of these are well-known models such as the Stage-Gate model (Cooper, 1990), the V-model (Forsberg et al., 2005), the Waterfall model (Royce, 1970), and the Spiral model (Boehm, 1988). While all these models differ in their philosophy, reflected in the shape of their visualisations, they have some common characteristics: They all describe a continuously progressing process towards the final product, and while they all mention testing at some point in this process, they do not seem to consider testing throughout its entirety. Generally, the importance of the role that testing plays in the various models reviewed by Wynn & Clarkson (2018) is not reflected in their visualisations. This further emphasizes the importance of developing a visualisation of testing activities in PD, not just for the NMT.

As the NMT project does not follow a specific PD process model, a visual description of the testing activities could conceivably be shaped in a number of ways. However, for the sake of simplicity, it seems sensible to base it on the progression of the project to date. Recently, two teams worked on designing and testing the NMT’s front end, and two teams were allocated to researching, developing, and testing the back end membrane and combustion processes respectively. As described in the previous section, a separation into parallel testing activities for front end and back end seems sensible. Furthermore, the entire process has progressed steadily, and even though iteration occurred, a linear model describes the overall development most aptly.

The end point of many models is the launch, or deployment, of the product, although some consider also the operation and maintenance of the product (Wynn & Clarkson, 2018). For the case of the NMT, the envisioned final test of a prototype, one which is practically identical to the final product, is envisioned to be the testing for compliance with the ISO 30500 standard, conducted by a licensed laboratory. Therefore, this test would signify the endpoint of a visualisation of the NMT testing strategy.

Summary

A testing tool for the development of a novel sanitation system should consider all aspects to provide a comprehensive overview of the considerations that need to be made when planning prototype tests. We believe that a combination of existing concepts – not only in testing – is a common occurrence in PD. An example is Lean Six Sigma, a now established concept itself, which combines the two management methodologies ‘Lean’ – a methodology to “remove non-value activities from the [PD] process” and ‘Six Sigma’ – a methodology to reduce variability and thus defects and errors in the process of concern (Alexander et al., 2019). Furthermore, since the dawn of computer aided engineering, a combination of virtual and physical testing has been (Van Der Auweraer & Leuridan, 2005) and continues to be promoted by experts (Tahera et al., 2014). Likewise, the testing practice in private enterprises is likely to be more experience-based and will often combine various concepts to varying degrees. However, a visualised combination of different aspects appears to be a novel conclusion.

To incorporate all discussed aspects, a tool for testing of prototypes in PD should thus:

Synthesis: A Visual test planning tool

Derived from the considerations described above, a visual test planning tool for the NMT could look like the flow chart shown in Figure 3.

Figure 3. Visual test planning tool.

A simple linear PD process model was chosen, divided into the three phases exploration, assessment, and validation/verification, with parallel strands of testing for the front end and back end, considering different types of testing, DOE, usability testing, reliability/durability testing, and international standards at different stages of the process, and possible iteration loops throughout the process. As value over cost per iteration decreases with product maturity, iteration loops are marked as encouraged, possible, and costly over the course of the three phases.

Starting from the problem description, and potentially using input from already existing international standards, a design brief is the first step of the PD process. This will prompt first concept ideas, thus beginning the exploration phase. Designers will then develop potential solutions to the problem, which will be realised in the first prototypes, both for the front end and back end. The front-end prototypes will often have low functionality and are mainly used to communicate the designers’ vision. They can then be tested with potential users, to assess whether the designers are ‘on the right track’, and their proposed solutions could be accepted by users. The back-end prototypes will likely be first breadboard prototypes of single components, to prove the viability of the concept ideas. Several competing designs might be tested simultaneously, and iteratively, to refine the initial designs. In this phase, more iterations should be encouraged, as producing and testing prototypes is relatively cheap and yields a lot of useful information.

In the following assessment phase, further developed prototypes can be constructed. These might be prototypes of sub-products or the whole product, and they are likely to already have a certain degree of functionality. It is on these prototypes, and increasingly developed iterations thereof, that a variety of tests will be conducted to learn about the technical, functional, and aesthetic aspects of the NMT. Using DOE and UCD methods, and tests from international standards as benchmarks, components and (sub)systems are tested in laboratories and field tests towards functionality and usability. HALT methods can be used to identify and mitigate likely failure points. While first tests will still be conducted on separate prototypes for the front and back end, user tests and functionality tests can be conducted simultaneously at later stages of sub-product maturity, when an integrated prototype is constructed. Several iterations are possible, until satisfactory component and system maturity is reached, and competing designs can be developed and tested simultaneously. It can be a difficult decision to define a cut-off point for further iterations. The developer has to have confidence that the entire system will safely function as intended. The minimum performance values of an international standard can provide helpful guidance to ensure this confidence.

In the final phase of validation and verification, a finalised design, maybe already produced on the product’s assembly line, is tested for reliability and durability using ALT methods. DOE methods can be applied to improve statistical validity of tests. Final user tests ensure all user-related problems have been mitigated. If no serious problems arise, the product can be sent to be tested towards compliance with an international standard. In the case of the NMT, this would be the standard ISO 30500:2018 Non-sewered sanitation systems.

There is a possibility that tests reveal problems which necessitate a return to much earlier stages of the PD process. However, this should be avoided by completing an appropriate number of test iterations during the exploration and assessment phases, as it would entail significant cost to iterate at such a late stage.