Route Optimization Tool (RoOT) for distribution of vaccines and health products

Ensuring the delivery and availability of health products, including temperature-sensitive vaccines, is vital to saving lives in low- and middle-income countries (LMICs). In many LMICs routes are hand drawn by logisticians and are adjusted based on vehicle availability and product quantities. Easy-to-use real-time supply chain tools are needed to create or adjust routes for available vehicles and road conditions. Having more efficient and optimized distribution is especially critical for COVID-19 vaccine distribution. Route Optimization Tool (RoOT) works best for planning routes for 50 health facilities or less, in two minutes. We develop RoOT using a variant of a Vehicle Routing and Scheduling Algorithm (VeRSA) that is coded in Python but reads and writes Excel files to make data input and using outputs easier. RoOT can be used for routine operations or in emergency situations, such as delivery of new COVID-19 vaccine. The tool has a user-centric design with easy dropdown menus and the ability to optimize on time, risk, or combination of both. RoOT is an open-source tool for optimal routing of health products. It provides optimized routes faster than most commercial software and is tailored to meet the needs of government stakeholders We trained supply chain logisticians in Mozambique on using RoOT, and their feedback validates that RoOT is a practical tool to improve planning and efficient distribution of health products, especially vaccines. We also illustrate how RoOT can be adapted for an emergency situation by using a test scenario of a cyclone. Currently, RoOT does not allow multi-day routes, and is designed for trips that can be completed within twenty-four hours. Areas for future development include multi-day routing and integration with mapping software to facilitate distance calculations and visualization of routes.


Introduction
Vaccines save millions of lives every year and save billions of dollars by reducing health care costs and preventing lost productivity 1 , and by yielding an estimated 10-to 25-fold return on investment 2 .Despite their effectiveness, global vaccination coverage has plateaued at 85% since 2010 and, in 2017, an estimated 20 million children lacked access to routine vaccination services-approximately 60% of whom lived in resource-constrained countries 3 .One reason for stagnating coverage rates is inefficiencies in the immunization supply chain, which is increasingly challenged by population growth, new vaccine introductions, currency and policy fluctuations, and the introduction of new technologies and supply chain practices 4 .This is even more critical today with the COVID-19 pandemic, where efficient and effective distribution of the COVID-19 vaccine is critical to curbing the pandemic 5 .Many governments and stakeholders have been waiting for a COVID-19 vaccine since the pandemic hit.Now that the vaccine doses are arriving in low-and middle-income countries (LMICs), it is important to plan for the efficient and optimized distribution.
Direct delivery of health products to health facilities, by districts or provinces, is one of the most effective interventions to improve product availability and quality.This method is not only cost effective but gives health workers more time to provide services, and improve coverage, that would instead be spent traveling to pick up health products.Based on VillageReach's experience leading supply chain system design efforts using modeling tools in the Democratic Republic of Congo, Mozambique, Pakistan, and Zambia 6-8 , direct delivery reduces stockouts, and contributes to improving equity.Additionally, health products, especially vaccines, are most at risk of temperature excursions during transit [9][10][11] ; having products packed well and delivered efficiently to health facilities reduces risk to vaccines and maintains their potency.This is important for all vaccines, but especially so for COVID-19 vaccines due to limited supply, high demand, and the special ultra-cold chain requirements for some COVID-19 vaccines.
For direct delivery, one of the most important decisions 12 made by governments is effective routing that delivers products safely and quickly.Currently, routes are hand drawn by logisticians, and are not optimized for transit time or road conditions to improve availability and maintain vaccine potency.Based on the literature review summarized in the next section, there is no simple and practical tool available to allow logisticians to adjust plans quickly when a situation changes, such as when a vehicle breaks down or a road is impassable due to flooding.Direct delivery distribution depends on vehicle availability, and how much product each vehicle can hold and this varies month to month..There are currently no supply chain tools and methods that logisticians can use in real-time to easily decide which vehicles to use for specific routes, and to quickly calculate if they can carry the required product volumes.
In 2019, Mozambique experienced Cyclone Idai, which devastated many health facilities across the country.Supply chains were disrupted leading to stockouts and as many health products became unavailable 13 .Hence, an approach was needed to adjust supply chains to account for the road infrastructure that was destroyed, making some areas completely inaccessible, as well as changes to storage locations as some facilities were damaged 14 .Due to the cyclone's damage make-shift health facilities were set up in other areas to serve communities which needed to be incorporated into new distribution routes.Even for health facilities whose physical infrastructure was not impacted, their ability to maintain cold storage for vaccines was affected with disruptions in electricity in addition to the increasing numbers of people they were serving.This called for a shift in design of the supply chain system and highlighted the need for a quick and easy way for logisticians to distribute health products where they were needed most.
Based on our experience in Mozambique, we recognized the need for a new fast and easy-to-use tool that can be used by governments and organizations who do not have the time, resources, ability, willingness, or political capital to conduct extensive optimization or simulation modeling that can sometimes take days to run.Hence, we designed the Route Optimization Tool (RoOT), keeping in mind delivery of health products in routine and emergency settings, as well as the time and skills of government users.RoOT is designed to be used by the warehouses or facilities that are distributing products -whether there is one big provincial warehouse or multiple large health facilities distributing to smaller health facilities.

Landscaping of existing methods and tool
Before developing RoOT, we did a literature review of the existing tools and methods available for optimizing distribution of health products based on minimizing risk of spoilage of health products.While there is extensive literature on the importance of transportation for health supply chains,

Amendments from Version 1
Based on reviewer comments, we have clarified and answered several questions.We also did another round of copy editing to check for grammar and readability to reduce repetition, and made changes based on that.
The abstract is updated to bring forth the key points, i.e. the tool works best to give an optimal solution for 50 health facilities or less, in 2 minutes.The full-text, we: • Added other routing tools (like GraphHopper and Openrouteservice) suggested by the reviewers to make sure the landscaping of other tools is complete.
• Added a table summarizing Nielson's usability heuristics applied in RoOT • Added more details on computational performances, and clarified comparison with commercial software • Updated Figure 7, which previously had an error • Added references where missing and requested by reviewers

• Address all reviewer comments
Any further responses from the reviewers can be found at the end of the article infrastructure risks and financial systems are rarely addressed, and these are usually responsible for most of the network disruptions 15 .Often, the routing decision is based on the shortest distance, as it presumably reduces cost and leads to faster delivery.However, other risk factors, such as road failure from flooding, road sink, or bridge collapse, could make a recommended route infeasible 16,17 .
One option for optimizing routes is to incorporate the risk of road and infrastructure failures into the primary objective, instead of solely minimizing transit time or distance.Penalty parameters can be used as a way to incorporate the probability of road failure into an objective function, thus enabling the optimal routes to avoid unreliable roads, as in Hamedi et al. 17 Studies show that using a minimum risk approach identifies routes that avoid critical roads and thus decrease risk 18 .
While analyzing risks for routing delivery of health products is not common 15 , risk is often used as the main objective when transporting hazardous materials 19,20 .Risk of spilling hazardous materials is addressed with the probability of accidents due to speed, road conditions, and busy intersections 21 .Accident rates are also assessed due to the time of day, weather conditions, and type of road 20,22 .Using risk minimization, routes that avoid these dangers, are preferred even if travel time is increased to ensure safe transport and delivery of materials.
In redesigning the supply chain for distribution of health products including vaccines and temperature sensitive products, we investigated different optimization tools that are available, including tools that perform inventory optimization, network optimization (minimize cost or distance) or route optimization.Since one of the main contributors to waste and spoilage of vaccines and temperature sensitive products is not maintaining effective cold chain during transport, we sought optimization tools that capture risk as well as transit time 23 .However, most decision support systems for logistics focus on inventory control and network optimization, and are not easily adapted to balance risk with transit time for route optimization 24 .
During the landscaping analysis, we identified 18 vehicle routing software packages and 15 supply chain software packages that could be used in our context 25 .Most of the routing software used in commercial logistics focuses on distribution efficiency to minimize cost and time and does not explicitly include risk as an objective.A user cannot easily modify these tools to tailor the objective functions and constraints for health products.Additionally, commercially available tools are costly, require special installation and training, take hours to provide an optimized solution, and are too complex or not "light".
We narrowed down the 15 identified supply chain logistics software to two that had capabilities for optimizing routes: Coupa (previously called LLamasoft) Supply Chain Guru® Cloud-Based Supply Chain Design Software, and Global Logistic Competence (GLC) 26,27 .Our prior experience with using these tools was that they were complex to use, costly, and required advanced skills.Also, based on our experience, they require detailed data that is often not readily available in LMICs, such as geographic coordinates or details of road networks.The user interface of the existing tools is complex for those who may not be familiar with it which requires technical assistance that can be costly to some governments.
There are routing tools, such as GraphHopper and Openrouteservice that can help users in planning efficient routes.However, they do not consider vehicle storage capacity, cold chain, road conditions, vehicle conditions, and vehicle assignment to facilities.All of this information is needed by logisticians to plan health product delivery.The landscaping analysis identifies a clear need for a light-tool that can be used easily by a variety of stakeholders, without the need for advanced user skills or significant financial investment.Additionally, any software tool needs to consider risk of health product spoilage.

Model description
The optimization model in RoOT is a variation of a vehicle routing optimization problem 28 that is tailored to address the needs of a cold chain distribution of temperature sensitive health products (such as vaccines) and ambient temperature health products (such as syringes or medicines).RoOT is designed to be easy to use, and is based in Microsoft Excel, which is used by many governments and local organizations working on supply chains in LMICs.VillageReach's experience in supporting government users in Mozambique was leveraged with multiple discussions with government stakeholders to ensure the optimization model could address common supply chain questions as summarized in Box 1.

Box 1. Modeling support needed by government users
RoOT is designed to meet the modeling needs of government users, based on questions that government stakeholders are typically interested in modeling: • How does changing the resupply frequency impact the quantity delivered at each distribution?
• What adjustments are needed when a new vaccine or health product is added to a distribution route?
• What vehicle is needed and how should routes be adjusted if a new health facility is added to distribution route?
• What is the transport cost for each distribution route option?
• What changes in routes are needed if a road is unavailable or if the road condition changes (e.g., rain, flood, conflict or natural disaster)?
• What adjustments are needed when storage capacities change, either decreasing due to natural disaster or requirements of product (e.g., requiring ultra-cold chain) or increasing with added cold chain when new refrigerators are added?VillageReach and the University of Washington agreed on the following requirements for the Route Optimization Tool (RoOT).The tool should: • Be Microsoft Excel based to be easy to use and easy to modify data by governments and technical partners.
• Be usable for routine operations, but also in emergency situations.
• Consider all health products, including those that need cold chain, such as vaccines.
• Consider the availability and reliability of vehicles.
• Consider the road conditions that may change seasonally (e.g., flooding) or in emergencies.
• Provide routes with departure times and quantities of health products for delivery.
• Minimize transit time and risk to vaccines.
• Calculate cost of routes.
• Execute quickly and provide results within minutes.
The steps we took to design a practical and simple tool are described in the section Usability.
In addition to the tool being easily usable, it is also important that the tool provides a solution quickly, since most government users would not have the time to use a tool that would take hours to run.As described in more detail in the section Computational performance, we tested several available optimization algorithms and observed that it may take hours to produce a feasible solution, and even longer to determine an optimal solution.We decided to develop our own optimization algorithm that we could fine-tune to provide a solution to our optimization model quickly (see Computational performance).

Multiple objectives
Efficient distribution is critical in supply chains to ensure the supplies reach facilities in time and that there are no stockouts.Risk of temperature excursion is a critical consideration for all vaccines, but more so for the COVID-19 vaccine given the specialized temperature requirements, high demand, limited supply, and associated costs related to vaccine procurement.
RoOT allows users to balance transit time with risk by entering a weight for transit time, denoted W t , between 0 and 10, and then the weight for risk, denoted W p , which is calculated as W p = 10 -W t .The tool normalizes the values of each objective, so the weights affect each objective similarly.For example, weights of 5 and 5 give equal importance to transit time and risk, due to the normalization.The objective in RoOT is: (risk for all routes).

p W W
If the user enters W t = 10, then the model will only optimize transit time, and if W t = 0 then the model will only optimize the risk from unreliable roads and vehicles, whereas any value in between will balance minimizing transit time and risk.

Constraints and assumptions
The constraints of the optimization model in RoOT, as in Box 2, reflect the basic assumptions highlighted in Box 3.

Box 3. RoOT model assumptions
The assumptions in the model are: • Distribution routes are completed within one day, a maximum 24-hour time period.
• Vehicles start and return to the same distribution warehouse.
• There is sufficient supply of health products at the distribution warehouse to meet the requested demand.
• Health facilities can properly store the entire quantity of health products requested.
• Distribution routes use roads which vehicles can access (transport over water or on foot is not considered).
RoOT assumes all routes will be completed in a single day, Hence, only considers single-day routes for distributions, and does not consider overnight stays or multi-day routes.This means that the maximum time limit for a route, as set by the user, must be 24 hours or less.One way the user can get around this is by choosing 24 hours, which could represent 3 days of 8 working hours each.We also recognize that drivers or health workers may not spend their entire 8 hour work day on deliveries.Hence, the start time and return time should reflect the start and return for the deliveries, not the work day. Figure 1 illustrates the "parameter" input sheet where the start time and return time is specified.The second assumption is that vehicles start and return to the same distribution warehouse.Although vehicles for distribution may be available at different locations, they would need to pick up health products at a specified distribution warehouse.Hence, it is a reasonable constraint and assumption that each route starts from and returns to the distribution warehouse.
The model is designed to meet the demand of health products, irrespective of supply.It assumes that there is sufficient supply to meet the demand requested by health facilities.If supply is limited, the user should adjust the requested demand to indicate the stock that will be delivered.Therefore, RoOT is constrained to meet the total demand.Additionally each health facility is visited exactly once by one vehicle.
Moreover, RoOT does not use facility storage capacity as a constraint and assumes that facilities can properly store the quantity of health products they will receive.However, if the quantity requested by the facility exceeds its storage capacity, RoOT provides an immediate warning to the user but it is still possible to execute the optimization model.
RoOT will schedule as many routes as needed to fulfill the distribution.If only one vehicle is available, that vehicle may be assigned several routes.If more vehicles are available, RoOT assigns the number of routes to available vehicles as evenly as possible.For example, if there are three routes and two vehicles are available, the model may assign two routes to one vehicle and one route to a second vehicle.
The user enters the cold and ambient transport capacity for each vehicle, and this is a constraint the model considers when optimizing routes.The user can enter any type of land vehicle and its transport capacity.For example, a motorcycle's cold storage capacity may be to transport a small vaccine carrier with some additional space to carry ambient products, like syringes or essential medicines.On the other hand, a refrigerated truck would have a larger transport capacity for cold and ambient products, which the user would input into the model.
The last assumption is that the vehicles can travel on roads.
To include distribution to an island that requires transport by boat, we selected a location that is accessible by land vehicles and assumed that a boat would meet the vehicle at that location.A similar adjustment can be made to meet vehicles on roads if foot access is required.The constraints in the optimization model ensure that routes use roads that are accessible.A road that is in poor condition, but passable, is allowed and a penalty for the road condition is added into the risk objective function.

Usability
User centered design and ease-of-use was important in the tool from the initial design.There are several optimization and modeling supply chain tools available but due to their design and complex interface, they are not easily used or adopted by government logisticians or technical partners supporting governments (see section Landscaping of existing methods and tool for more information).To focus on user centered design, we first mapped out the workflow for using the tool as shown in Figure 2, and analyzed the usability by using a modified version of Nielsen's usability heuristics 30 , which are an industry standard in user interface design.
Based on the workflow and principles of information architecture, three principles for improving the usability of the tool were identified: (1) reduce input data errors, (2) immediate feedback to users, and (3) reduce data input time.Nielson's usability heuristics 31 used in this tool are summarized below in Table 1.
To reduce user errors during data input, we used dropdown menus wherever possible for the user to select options from a list rather than typing them out or copying from another sheet, based on Nielsen's 1 st usability heuristic that users should get immediate feedback to make informed decisions, and 5 th heuristic that a system should be designed to prevent any errors.Hence, the RoOT Excel input file has dropdown menus for: • selecting the start and return location for distribution, • indicating whether a health product requires cold storage, • vehicle availability, • vehicle condition, and A common challenge with any tool using multiple databases is ensuring that facility names, products, and other input information are spelled consistently throughout so that the back-end algorithm can associate them.However, often facilities have multiple variations in spellings with slight changes across different databases.It is also tedious for the user to keep track of correct spellings across multiple sheets or databases.
To address this and make the data input process easier, and to reduce errors, the users enter the names of all facilities only once, in the "center_capacities" sheet, and the names are automatically replicated across all other sheets.Similarly, the names of health products are entered only once and automatically replicated across relevant sheets.
In addition to RoOT's user guide based on the 10 th usability heuristic, we added brief instructions on every input sheet for easy reference by the user and to make data entry easier 29 .We also color-coded the cells to clearly indicate where the user needed to enter data, and where it was automatically calculated for pre-processing.
Further, users may want to not consider a vehicle type for some route or may not be delivering some health products.
To reduce the cognitive load on users when modifying this data, we included a yes/no dropdown for health products and available/not available for vehicles to indicate whether the model should include them in the analysis based on Nielsen's The design should speak the users' language.Use words, phrases, and concepts familiar to the user, rather than internal jargon.
• Familiar words like "yes"/"no" instead of binary codes (0 and 1) are used, when identifying cold storage needed for products • Instructions are provided on each sheet • User guide with screenshots are available for users for additional support • Color-coded cells indicate where the user enters data, and where it is automatically calculated 2 nd and 6 th heuristic 31 .This allows the user to enter data using words and phrases that they are familiar with, making data entry easy and reducing errors.

Computational performance
In operations research, it is well-established that the computation time to solve a vehicle routing problem increases significantly as the number of health facilities increases, or as the number of available vehicles increases 28 .However, getting quick results within a few minutes was an essential requirement by users and, hence, we developed our own optimization algorithm for RoOT that is a variant of a Vehicle Routing and Scheduling Algorithm (VeRSA) 32 .
VeRSA embeds an indexing rule in a branch-and-bound framework to quickly construct a feasible solution in seconds, instead of hours.VeRSA calculates two indices based on the problem constraints to decide which vehicle to use for the route and which facility to visit next.Every time that a new facility is visited and added to the solution, the index is recalculated to choose the next best center to visit.Therefore, the indices are used to create good and feasible routes in a timely manner.The routes that are determined in one or two minutes perform nearly as well as the optimal solutions that may take hours to determine.For RoOT, we adapted the indexing algorithm used in VeRSA for health product and vaccine distribution specifically.We also embedded specific constraints directly into the feasibility check in VeRSA to speed up computation.This version of VeRSA is coded in Python, and reads and writes Excel spreadsheets.Hence, the Python code is invisible to the user, making RoOT easy to use while providing timely results for logisticians.
In earlier versions of VeRSA, the number of products also increased computation time.However, in the final version of VeRSA used in RoOT, the computation was streamlined by aggregating the products into two categories: those requiring cold storage (such as vaccines) and those kept at ambient temperatures.We do not assume a temperature range for the cold chain, or for the ambient temperature products.When the user identifies the vehicles with cold storage, they can determine an appropriate temperature range that satisfies the health products' requirements (e.g., 2-8°C for many routine vaccines).The Python code disaggregates the two categories of products into specific names and quantities of products in the Excel output file.Thus, there is no limit to the number of products that can be input in RoOT as long as it fits in the two categories, and it does not impact computation time in the optimization.Hence, RoOT can be used to plan routing for integrated health supply chains that deliver vaccines and other health products, such as family planning, malaria etc.,.
RoOT can also be used in routing vaccines for campaigns and routine immunization simultaneously, instead of distributing through parallel supply chains.
Computation time in modeling and routing software is an important factor for users.Solving large-scale vehicle routing problems can often take hours or days to obtain an optimal solution.However, government logisticians or technical partners supporting governments, often cannot wait to run modeling problems for long and often require quick solutions they can use.
To understand how RoOT's computation time compared with other available optimization software, we ran several numerical tests.(For details, see the results reported in Petroianu et al. 25 ) Overall, RoOT performed very well.For 10-20 facilities, the performance of RoOT's indexing method was similar to the best of the available software packages, and produced an optimal solution within 2 minutes.Hence, the default computation time in RoOT is set to 2 minutes.Additionally, we also tested RoOT for a greater number of health facilities to reflect distributions in larger countries.For 50 health facilities, RoOT provided good results in 2 minutes, while the other available software packages took much longer 25 .For example, when results were compared testing with 50 health facilities, the other available packages were run for two minutes and could not reach the result that RoOT calculated in less than 10 seconds.For 100 facilities, RoOT determined a feasible solution within 2 minutes; however, this solution did not perform as well as a solution found after running the available packages for several hours.An advanced user can increase the default computation time of 2 minutes in the RoOT Excel input file, and in theory, RoOT will eventually obtain an optimal solution.
For practical purposes, we recommend using RoOT with 50 facilities or less to obtain good results within 2 minutes.
The number of vehicles used in the model also impacts computation time.To keep the computation time low we recommend limiting the number of available vehicles to five or less.Based on our experience and discussions with stakeholders, we believe that five is a reasonable number as most provinces and districts in LMICs often do not have more than five vehicles (with accompanying personnel) available to implement simultaneous routes.However, in a situation where more than five vehicles are available for distribution, it is possible to increase the number to more than five vehicles with the understanding that the computation time should be increased from the default of 2 minutes to provide a good solution.
The output generated by RoOT is similar to that of commercial software for relatively small datasets (10 or less health facilities), and RoOT provides a feasible solution faster than commercial software for large datasets (50 health facilities) based on a numerical comparison 25 .This confirms that RoOT provides good results in a timely manner that are correct and represents the information provided in the input files.It is also important to emphasize that unlike commercial software and most of the routing tools on the market, RoOT is opensource and freely available on GitHub (see Software availability section for details), in addition to being easy-to-use 33 .

Operation
RoOT runs on a Windows computer, 64-bit, with Microsoft Excel version 2007 or later.There are no specific RAM requirements, but the RoOT folder needs about 1.1 GB of memory.To check if your computer is 64-bit, go to "Display Settings" and scroll down to find "About" on the left menu.When you click on "About," you can see: "System type: 64-bit operating system."

Challenges and limitations
As with any modeling tool or software, the outputs are only as good as the input data.In many situations, accurate data may not be available, and users must fill data gaps using proxy data or by making assumptions, which puts a limitation on any tool.
Some of the other challenges and limitations specific to RoOT are outlined below.

Troubleshooting when RoOT does not run:
RoOT is a lighttouch Excel-based tool that is easy to run.However, the burden of troubleshooting issues is on the user.For example, if the data in the input file is incomplete, the model will not run or provide results.Unfortunately RoOT can not display any error message indicating to the user to check the input files for errors, nor does it highlight what the error could be.We recommend that the user check each of the seven input sheets to make sure the data entered is correct and that no field is left blank before running the model.

Storage capacity of facilities:
Currently RoOT does not limit distribution to facilities even if the demand exceeds the storage capacity available.This could be viewed as a challenge as the model is allowing distribution even when there are storage constraints.However, in practice, many facilities find ways to store health products beyond the officially designated space for storage, e.g.dry commodities are stored on top of cupboards or in corridors.To mitigate this challenge, we created a "warning" signal that provides users with real-time feedback on storage utilization based on the quantity of health products requested by a facility.This gives logisticians and supervisors insights about storage utilization, and if it exceeds capacity they could discuss with the facility, but the optimization can still be executed.

Using all available vehicles:
RoOT assigns routes to available vehicles, as evenly as possible, even though one vehicle may be more reliable than another one.For example, if two vehicles are available, and one is in good condition while the other is in poor condition, RoOT may assign two routes to the reliable vehicle and one route to the vehicle in poor condition, accomplishing the distribution in two days.If the user wants to explore the option of only using the vehicle in good condition, then the second vehicle should be selected as "unavailable" in the input sheet.

Multi-day routing:
In practice, distributions of health products from provinces and districts to health facilities often take multiple days.However, the current model only allows for trips that can be completed within 24 hours.This limits the practical use of RoOT for distributions in areas which require multi-day routes, especially health facilities that are very far from the distribution warehouse and would take multiple days to reach.However, since the distribution can be done over 24 hours, the user can consider it as 3 days with 8 hours each.To expand RoOT capabilities further for multiday routing, certain additional factors need to be considered such as overnight accommodation locations and maintaining cold chain overnight.These should be considered when expanding RoOT to allow multi-day trips.

Scalability to more health facilities:
Currently RoOT is recommended for use for distributions to 50 facilities or less from a single distribution warehouse.This is a reasonable limitation because distribution is often organized by administrative or distribution boundaries.If there are more than 50 facilities for distribution, the computation time can either be increased or facilities recategorized by geographical proximity.

Matrix of distance between health facilities:
One of the more challenging data to input for RoOT is completing a distance matrix as shown in Figure 3.The user needs to fill in all of the data and cannot leave any cell empty for the model to run properly.This requires the user to look up and enter distances between each health facility, a burden on the user that has a large number of health facilities in the same distribution route.Mapping software, like Open Street Maps, Google maps, or Openrouteservice, may be helpful in creating a distance matrix as well in identifying shared roads, such as a highway or major arterial.However, health facilities in most LMICs are not easily located in mapping software, because either they are not marked, or the facility name may be misspelled in mapping software.This could be mitigated if geo-coordinates were available for health facilities making it easy to locate on mapping software; unfortunately, those are also not readily available.In system design studies using more sophisticated modeling software, we often must search and lookup each health facility; and in circumstances when that information is not available, we often approximate the distance using the district or province's location.While this continues to be a challenge, the user needs to only set up the distance matrix once and logisticians can also estimate the distance based on their local knowledge.
If a road is blocked due to an emergency or some other reason, the user updates it on the "road_condition" input sheet (Figure 6) to 'not accessible'.RoOT will not use this road while creating a route, and determine alternative routes.If a facility is completely inaccessible by existing roads, a make-shift drop-off facility can be created to the point which is accessible by road.

Visualization of outputs:
One of the requirements by users was the ability to visualize the outputs and see the routes on a map, but RoOT is currently limited to generating results in an Excel output file with no visualization.As mentioned above, it is challenging to integrate mapping software (e.g., Open Street Maps or Google maps) with RoOT, but is a possibility in future versions.

Use cases Training logisticians in Mozambique
RoOT is designed to meet the modeling needs of government stakeholders and users, which includes their time availability, resources, access to technology, and skill level with the software.To test if RoOT meets user needs, we trained eight logisticians in Mozambique at both the provincial and district levels.We designed a four-step assessment to measure user feedback on their experience with RoOT: 1. Reaction: Do you like the tool and is it easy to use?This was measured through observation and survey.

Learning: Do trainees leave the training understanding how to use the tool?
This was measured through a practical exercise and survey.

Behavior: Do logisticians use the tool to plan distributions?
4. Results: Does using the tool lead to better outcomes?
The participants of the training described that currently they follow pre-determined routes, starting either with the closest health facility or the largest one.Often, they find out which vehicle is available for distribution at the last minute when it arrives at the distribution facility, and they must quickly adjust the routes accordingly.The participants agreed that the tool was easy-to-use and would help them in distributions as illustrated from these quotes: • " We were creating the routes in an ad-hoc way.We didn't have a platform to guide us to calculate the routes and the quantities per route.This tool can help us by giving us different ways of arriving at the health facility." • " The truth is that we were working in the dark, we first tried something on the ground, then we would know the estimated cost and time, and whether that works or not; but with the tool, we're not in the dark.Calculating the time used in the distribution is one of the hardest things to do, as sometimes we don't know how to calculate whether we'll be able to return on the same day or the next; and this helps us calculate the time.But it should still consider the fact that you sometimes have to come back the next day, not always on the same day."

• "… the tool tells us what is the capacity at each health facility also helps us to visualize. We used to load vaccines according to needs only and not take into account what is the actual capacity at the health facility."
Over 60% of the logisticians in the training rated their confidence to use the tool as skillful, i.e., they could use the tool independently with occasional help from a specialist.Out of the eight participants, seven were confident in being able to use the tool to determine routes and to decide which vehicles to use.The logisticians confirmed that they would be able to use the Portuguese version of the tool for routine and emergency distribution.Unfortunately, due to the COVID-19 pandemic in 2020 and shift in stakeholder priorities, the full deployment of RoOT has been delayed and the behavior and results could not be fully assessed.

Using RoOT for distributions during an emergency
RoOT is designed to meet the needs of government stakeholders for routine as well as emergency distributions.For routine distribution, we anticipate that logisticians would use the tool to determine a number of consistently used routes, updated with current road and vehicle conditions.
RoOT can also be used for emergency situations, like outbreaks or vaccine campaigns when a new health product needs to reach health facilities quickly.As supplies and treatments for COVID-19 become available or a new vaccine is introduced, governments will need to mobilize quickly to make sure the vaccine and health products are getting to the most vulnerable people as quickly as possible.There have been supply shortages as all countries strive to procure COVID-19 vaccines.Hence, countries need to prioritize how many vaccines to deliver and to which health facilities in the fastest way possible, while minimizing risk to vaccine potency.RoOT can be used to quickly determine, and plan routes based on minimizing transit time and risk, to align with government priorities.
To illustrate the use of RoOT in an emergency situation, we used a natural disaster, such as a cyclone as a test scenario.
In an emergency situation, there are several parameters that a logistician may need to assess and modify.In this cyclone test scenario, the primary warehouse was damaged so distributions had to be planned from a different warehouse, several health facilities could not accept supplies, and several roads are not accessible.
Step 1: Changing distribution warehouse location.The user would change the start and return location for distribution in the "parameters" input sheet by selecting another facility from the dropdown menu.As shown in Figure 4, the distribution location has changed from Province A to Center B.
Step 2: Updating demand for health facilities.Given the emergency situation, not all health facilities are intact or have storage for supplies.Hence, health products need to be distributed to a smaller number of health facilities that may see an upsurge in demand as people from nearby areas are also traveling there to seek care.The logistician does this in RoOT by changing the demand to zero for health facilities that are not able to store supplies at this time, and accordingly increases demand for other facilities.As shown in Figure 5, Province A, Center C and Center D have zero demand, and demand has increased for Center B and Center F.
Step 3: Updating road conditions.In case of a natural disaster, like a cyclone, many of the road conditions may change or become completely inaccessible due to flooding or damage, making them unavailable for use.The logistician planning the routes can select the updated road conditions from a list of options in a dropdown menu, as seen in Figure 6, where the road from Center B to Center E is not accessible.The dropdown menu allows the selection from the following options: Fully paved, Partially paved, Dirt road (Good Quality), Dirt road (Rough), and Not accessible.
Step 4: Run RoOT for updated results.After making all the changes to the inputs, the user should save the input file and re-run RoOT.The tool will provide results within 2 minutes, and generate an Excel output file displaying the routes, departure times, and health products that need to be delivered for the emergency situation.Figure 7 illustrates the new routes for the cyclone scenario.There are three routes that start and end at Center B. There are two vehicles available for distribution, the Landcruiser_3PL, and a New Vehicle.As shown in Figure 7, the Landcruiser_3PL leaves Center B at 8am, and visits Centers D, E, and C before returning to Center B. The New Vehicle has two routes, as illustrated in Figure 7. Notice that these three routes never use any roads that are marked "Not Accessible" using the dropdown menu in the input sheet in Figure 6.

Conclusions
In conclusion, RoOT is an easy-to-use optimization tool that enables logisticians to quickly plan and adjust routes for health product distribution accounting for transit time and risk of temperature excursion of sensitive products, such as vaccines.RoOT is designed to • meet the requirements of government stakeholders, and • provide faster results than commercial software As users gain experience with RoOT, they will identify several areas for future improvements.Since the tool is opensource, we invite users to build these capabilities directly, and to reach out to authors for future possible collaborations as well as feedback.One possibility is to integrate RoOT with existing software tools (such as demand projections and cost analyses) to increase consistency of data inputs.At the same time, it is desirable to maintain independent use of RoOT so it can be easily used in many LMICs countries and for many types of health products.Another future extension is to consider multiday routes, where many considerations must be discussed, and appropriate assumptions and constraints developed.Lastly, inclusion of visualization with mapping software will greatly improve the usability of RoOT.

Software availability
Route Optimization Tool (RoOT), with user guide and underlying data is available from: https://github.com/villagereach/RoOT(in English) and https://github.com/villagereach/RoOT-portugues(in Portuguese).Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

○
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?

○
The comparison against other routing options seams to be focused on other commercial options saying "Most of the routing software used in commercial logistics focuses on distribution efficiency to minimize cost and time" -no mention or consideration for other low cost / open source options that are currently available, e.g.https://www.graphhopper.com/or https://openrouteservice.org/.It would be better for the reader to know that there are other options.
I reviewed the reference (21 -Petroianu LPG, Zabinsky ZB, Zameer M, et al.: A light-touch routing optimization tool (RoOT) for vaccine and medical supply distribution in Mozambique.Intl Trans in Op Res.2020) -provided to support the following statement "To understand the impact of number of health facilities on RoOT's computation time, we compared it with other commercially available software.The results showed that RoOT performed very well21".I did not find a table or data that showed computational time between Commercial software and RoOT.I did find against different solvers -Gurobi, CBC and GLPK -how that is not the same as comparison to commercial software.The paper then indicates it did analysis against commercial software: "For 10-20 facilities, the performance of RoOT's indexing method was similar to the best of the commercial software packages and produced an optimal solution within 2 minutes.Hence, the default computation time in RoOT is set to 2 minutes.Additionally, we also tested RoOT for greater number of health facilities to reflect distributions in larger countries.For 50 health facilities, RoOT performed better than the commercial software, providing good results in 2 minutes while the commercial software took much longer 21 .For 100 facilities, RoOT determined a feasible solution within 2 minutes; however, this solution did not perform as well as a solution found after running a commercial software package for several hours."Therefore, since they did this analysis it would be good to include this data within the paper for the reader to see and understand the differences.Additionally, when the default computation time in RoOT is set to 2 minutes, the same setting for computation time could be set for commercial software and then comparison would be on how well the solutions were after 2 mins determined between RoOT and the commercial software.Thus, we don't know if the solution calculated after 2 mins was similar or not between RoOT or the Commercial Software; having this information this would strengthen the evidence.
"Computation time in modeling and routing software is an important factor for users.Commercial software can often take hours or days to obtain an optimal solution."Further context to this statement would be useful to the reader, if you are looking to solve a large routing problem to fully optimal solution then it will take time if using the RoOT or Commerical Software or other open source routing software.Just saying "Commercial software can often take hours or days" without context is misleading.This is then shown when context is then provided in the paper.The paper states for a small problem "For 10-20 facilities, the performance of RoOT's indexing method was similar to the best of the commercial software packages and produced an optimal solution within 2 minutes" thus they take the same time.Then for a slightly larger problem "For 50 health facilities, RoOT performed better than the commercial software, providing good results in 2 minutes while the commercial software took much longer 21 ."This is the "sweet spot" for this tool and thus should emphasized as in the environment this tool is to be deployed in there are a number of distribution points that could meet this criterion.Then for a larger problem "For 100 facilities, RoOT determined a feasible solution within 2 minutes; however, this solution did not perform as well as a solution found after running a commercial software package for several hours", this where you are starting to step out of the "sweet spot", however as described in the paper for practical purposes it is good enough.Thus, the importance for having that table to demonstrate the difference in computation time.
On the Distance Matrix the paper does recognize the challenge in filling this out.Suggestion is to mention such resources to support the distance matrix https://openrouteservice.org/ that provides distance matrix, this could be useful as it is designed to work in this context.
Additionally, "Updating road conditions" is a simplification of the impact and ease of determining which paths between different sites have been affected by a disaster or change in the road conditions.A change in road condition is unlikely to just affect one pair of sites as described in the paper, it would need to be determined if and how affect all paths to a site across matrix.For example if a road is inaccessible "from Center B to Center E is not accessible" it could affect the paths between all sites to B and/or all sites to E. As it is unlikely to have a single path that is only between sites B and E, it is more common that sites will share a road with other paths to sites, thus when a road is affected then it will be across multiple sites.Thus, it would be good to clarify and expand the current clarification on how to update the road conditions and ensure that readers understand what needs to be done when inputting updates to road conditions.

Introduction:
The first paragraph focuses on the efficient and optimized distribution.While budgets are a concern for COVID response, I think these opinions deserve a reference.Are governments more concerned with effective distribution or efficient distribution?It could be (and likely is) both, in which case this paragraph would suggest that both areas need to be addressed in the planning and in the tool.I think the paper goes on to support this idea by having both risk and speed in the objective function, but this sentence tends to focus only on efficiency.

○
The third paragraph warrants a reference for both "one of the most important decisions" and for comments about "no tools", even if this is clarified later in the article, it could reference these points.

○
The fourth paragraph warrants a reference for impacts of Idai on Mozambique's health supply chain.

Methods:
Box 1: is there an article/report/survey/feedback to reference for these questions?
○ Constraints: Is there a need for a constraint on the "driver's day"?For example, that the driver can spend only X hours on the clock?

○
For distances, the time factor may be a more relatable quantification than distance, especially if taking into account transit through urban areas and roads that are known to be in poor condition.

Usability:
In the comment regarding existing tools in the first paragraph and their usability, suggest a reference for the statement or reference back to Landscape Analysis.

○
A table or graphic representing the Nielson heuristics and recognition in the tool would be helpful, as the reader cannot keep track of the 1 st , 5 th , or 6 th referenced heuristic.

Computational Performance
Are the two product categories limited to cold chain and ambient, and is cold chain assumed to be 2-8C?This could be better defined in the text for the reader.
○ This is not my area of expertise, but I imagine that software developers would want to see more data in this section as it pertains to comparisons and the definition of "good results" as it applies to testing against 50 facilities.I listed in my answers above "Partial" on this section because I don't know as much as software developers regarding this section in terms of ability to replicate or analysis.

○
Is there a reference (report, survey, publication) for the "reasonable number of vehicles"?I would agree with the statement, but I don't have proof.

Use Cases:
Training: I didn't see answers to all four posed questions?It looks as if only reaction and learning are answered…what about behavior and results?

General Comments
It would be helpful for the reader to outline the scope of the tool at the top of the article.As I understand, it's for optimizing routing from 1 location to N locations -so the focus is on the location that manages distribution downstream (as opposed to say, an area that has multiple distribution points or a facility that can both drop-off and pick-up goods).Reviewer Expertise: Global health and immunization supply chain planning I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
today with the COVID-19 pandemic, where efficient and effective distribution of the COVID-19 vaccine is needed."Reference added for "one of the most important decision".Sentence changed about "tools" as the landscape and literature review is summarized in the paper under "Landscaping of existing methods and tool".

○
Reference added on impact of Idai.

Methods:
VillageReach is a close supply chain partner with the government in Mozambique, and the questions came from several discussions and meetings with government stakeholders.

○
Yes, there is a need for a constraint on the length of a "driver's day".This can be reflected in the start time and return time of the parameters sheet (figure 1).A sentence has been added to clarify this.

○
Yes, the time factor could be more relatable.However, it might change during the time of the day, especially in urban areas.We decided to use distances between facilities and allow for different vehicles with different speeds, so in this way, we do consider time but assume it is the same throughout the day.

Usability:
Thank you for the suggestion.We added a reference back to the section Landscaping of existing methods and tools.

○
Thank you for the suggestion.A table summarizing the usability heuristics has been added, and language simplified for easier reading.

Computational Performance:
We only consider two product categories, cold chain and ambient.We do not assume a temperature range for the cold chain, or for ambient temperature.When the user identifies the vehicles with cold storage, they can determine an appropriate temperature range.We reworded this in the text as follows, "… aggregating the products into two categories: those requiring cold storage (such as vaccines) and those kept at ambient temperatures.We do not assume a temperature range for the cold chain, or for ambient temperature.When the user identifies the vehicles with cold storage, they can determine an appropriate temperature range that satisfies the health products' requirements (e.g., 2-8 o C for many routine vaccines)."

○
This was also requested by another reviewer.We expanded the paragraph on computation time and referred to Petroianu et al. for detailed results.

○
This number came from experience and discussions with stakeholders, and a phrase is added, "Based on our experience and discussions with stakeholders, …" ○ Use Cases: Unfortunately, due to the COVID-19 pandemic in 2020, the full deployment of RoOT has been delayed, and we are not able to answer all the questions.

General Comments:
It can be used by any facility distributing products -whether it's one big provincial warehouse or multiple large facilities delivery to smaller facilities.We have added a line to clarify this.

○
Paper has been copy edited and cleaned for grammar and easier reading.

Hua Ni
1 USAID GHSC-PSM, Arlington, VA, USA 2 IBM GBS, Falls Church, VA, USA The paper describes the RoOT for distribution of vaccine.This tool addresses a very practice needs of supply chain last mile distribution with dynamic routing capability.I applaud the authors' effort to create an open source solution that factors the operational environment to make the tool more accessible for broader usage in the field of LMIC countries.
Among the questions raised to the reviewer, I answered "Partly" for the following two questions.Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? 1.
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? 2.
Here are some specific comments that I suggest the authors consider, either providing additional details or making some clarifications: Page 4: the comments around existing landscape and in particular on the final two candidates are a bit subjective in the absence of further details.I suggest providing some examples on the complexity, cost, and skills needed to operate.

1.
Page 5: if vehicles have fixed ambient and cold capacity, it seems that the constraint of requiring exactly one vehicle to visit each health facility may be too limiting.It is plausible to result in lower vehicle utilization given either ambient or cold storage could become the bottleneck first.

2.
Page 5: it is not clear whether the transit time refers to the whole route or just the portion that still has vaccine onboard.If we are to reduce spoilage probability, it seems to be the case that we want to minimize the transit time when vaccine is still onboard the vehicle, and the empty return segment is less critical since no vaccine is onboard, and cost doesn't seem to be part of the stated objectives.

3.
Page 5: while the weight used to combine the two objectives is easy to use, it is prone to 4. misuse/misinterpretation.One may think 5 and 5 would put transition time and risk penalty equally important, but without knowing the scale of the individual objective values, the effect is unpredictable.I suggest providing some additional guidance on the nuances here.
Page 5: it is not clear as how risk penalty objective is calculated.I suggest providing some clarification.

5.
Page 6: it is not clear if the vehicles are of different capacity or have the same capacity and just differ by the risk factor.Suggest providing some clarification.

6.
Page 8: while it references the prior page on VeRSA, it may be beneficial to add a high-level recap of the algorithm.

7.
Page 9: when a road is blocked due to emergency, it would have cascade of impacts on the OD matrix.I suggest providing some guidance to the users here.

8.
Page 12: the screenshot seems to indicate that Non-Refrigerated Utilization of Vehicles at 360% which seems to be wrong.The Refrigerated Utilization of Vehicle is at 1.8%.Is it out of the total capacity of the vehicle?I suggest updating the tables.

9.
Page 13: it may be beneficial to cite some solution quality result from the MZ operation to further strengthen the conclusion of the effectiveness of the tool.I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Figure 1 .
Figure 1.Spreadsheet for "parameters" input sheet, including the start and return location and times.

Figure 4 .
Figure 4. Change the distribution warehouse location from Province A to Center B in the "parameters" input sheet.

Figure 6 .
Figure 6.Update the "road_condition" input sheet to reflect current road conditions and accessibility between Centers.

Figure 5 .
Figure 5. Change the demand for health facilities; set demand to zero for facilities that are not able to store health products at this time and adjust demand for other facilities.

Figure 7 .
Figure 7. New routes for emergency cyclone distribution with two available vehicles.
the rationale for developing the new software tool clearly explained?Yes Is the description of the software tool technically sound?Yes Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?Yes Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?Partly Are the conclusions about the tool and its performance adequately supported by the findings presented in the article?Partly Competing Interests: No competing interests were disclosed.I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.Ryan McWhorter UNICEF, Copenhagen, Denmark I enjoyed reading the article and see value in the tool.I look forward to using it in the field.Please see my comments below for the authors' consideration.Abstract: I think the first sentence's sentiment is borrowed from an oft-cited statement that immunization is one of the most cost-effective interventions for children's health.Instead, this statement says that "Delivery…is one of the most effective interventions to ensure availability of supplies…".I think it's obvious that the delivery of a product is one of the most effective interventions to having the product available.I don't think I'd call it an intervention, though.Delivery is just part of the operation, not an intervention.I recommend the authors ○ consider a different intro statement regarding the importance of delivery.

○○
Recommend to the authors to run the text by a copy editor to clean up grammar and word choice.Is the rationale for developing the new software tool clearly explained?YesIs the description of the software tool technically sound?YesAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?PartlyIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?YesAre the conclusions about the tool and its performance adequately supported by the findings presented in the article?Partly Competing Interests: No competing interests were disclosed.

10 .
As I indicated, I applaud the authors' effort, and hope my reviews are beneficial to strengthen this already strong paper.All views expressed here are my personal views and don't represent any of my affiliations.Is the rationale for developing the new software tool clearly explained?YesIs the description of the software tool technically sound?YesAre sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others?PartlyIs sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?PartlyAre the conclusions about the tool and its performance adequately supported by the findings presented in the article?Yes Competing Interests: No competing interests were disclosed.Reviewer Expertise: Not in a research role.Education background in operations research, specializing in mathematical optimization.5+ years of experience in public health supply chain operational analytics, including last-mile distribution.

2018 Assessment report of the Global Vaccine Action Plan. Strategic Advisory Group of Experts on
Immunization.Geneva: World Health Organization; (WHO/IVB/18.11).Licence: CC BY-NC-SA 3.0 IGO.2018:(WHO/IVB/18.11).2018.Reference Source 4. Prosser W, Phillips N, McWhorter R, et al.: A System Design Approach for Comprehensive Immunization Supply Chain Strengthening A Case Study from Mozambique.Seattle and Copenhagen: VillageReach and UNICEF, 2016. Lee BY, Haidari LA, Prosser W, et al.: Re-designing the Mozambique vaccine supply chain to improve access to vaccines.Vaccine.2016; 34(41): 4998-5004.PubMed Abstract | Publisher Full Text | Free Full Text 7. Prosser W, Jaillard P, Assy E, et al.: System

and Challenges for Contraceptive Service Delivery in Cyclone Idai-Affected Areas of Mozambique - Case Study
. New York; 2020.Reference Source 14. USAID Global Health Supply Chain Program: Guidelines: Quick Reference Guide - Responding to and Preparing for Natural Disasters.Coordinated response to Cyclone Idai in Mozambique provides model for supply chain emergency response, 2020.Reference Source 15.Baharmand H, Comes T, Lauras M: Managing in-country transportation risks in humanitarian supply chains by logistics service providers: Insights from the 2015 Nepal earthquake.Int J Disaster Risk Reduct.2017; 24: 549-559.