Limitations to current methods to estimate cause of death: a validation study of a verbal autopsy model

Introduction

Global and disease-specific health statistics are regularly published and constitute an essential tool to define priorities and goals, identify inequalities, and track progress, including the achievement of global targets such as the health-related Sustainable Development Goals (SDGs). Insufficient confidence in the accuracy of estimates, particularly in those related to cause of death (CoD) has been indicated as a constraint to reduce mortality globally^1,2. This lack of precise information on CoD in many low-income settings is largely explained by the significant number of deaths that occur either at home or at poorly resourced health facilities, with significant limitations of both qualified personnel as well as accurate diagnostic methods; but also to the very limited number of diagnostic autopsies performed partly due to the massive shortfall of trained pathologists^1,2. As such, CoD in low-income settings, continues to rely on estimates based on clinical records and verbal autopsies (VAs).

Clinical errors, which are common even in well-equipped hospitals, are more frequent in resource-restricted settings^3–5. On the other hand, VAs remain the most practical and commonly used approach to estimate CoD at the population level in low-income settings⁶. A verbal autopsy consists of a structured interview to witnesses of the death subsequently interpreted and coded by physicians or using computerized methods. The method has shown to provide inconsistent results over time and place⁷. In addition, its diagnostic accuracy depends on the CoD, being high when the disease has a characteristic and well defined set of signs and symptoms, but much lower for conditions with unspecific symptoms, notably, malaria and acute respiratory infection in children, or meningitis in all age groups⁶. This results in frequent misclassifications of the CoD, which in turn leads to inaccurate cause-specific mortality rates⁶. Computerized methods of interpretation of the VA questionnaire have been developed to overcome some of the limitations of the VA technique. These methods are based either on algorithms derived from deaths with a medically confirmed CoD, or on probabilistic analyses⁸.

Computerized VA methods have been validated against physician-certified VA and clinical records^9–11. However, neither computer-coded VA, nor physician-certified VA techniques have been validated against the complete diagnostic autopsy (CDA), the true gold standard for CoD determination. We present herein the results of a validation study of a commonly used computer-coded VA method, the InterVA (Interpreting Verbal Autopsy) model against the CDA in a series of deaths occurring in Maputo, Mozambique.

Methods

Results

The VA model assigned one CoD in 267 (84%) cases and two CoD in 33 (10%) cases. In 16 (5%) cases the model resulted in a non-conclusive diagnosis. The average likelihood of the model in estimating the first CoD was 90% (range 89% to 99%), and for the second CoD it was 38% (range 35% to 46%) (Extended data²⁵: Table S2). Three of the 316 cases (1%) had a non-conclusive diagnosis in the CDA.

Assignment of the CoD at the individual level compared with the CDA

In 168/316 cases (53%) the two methods agreed in the CoD. Most of the agreement was in the first CoD, while only in 8 cases the agreement was in the second CoD with a mean likelihood of 38% [95%CI: (33–43)] (Extended data²⁵: Table S3). In 148/316 cases (47%), there was no agreement in the CoD between the two methods.

Overall, the performance of the VA method in assigning a CoD to individual deaths was low (Table 1). In stillbirths, the sensitivity of the model in identifying infections, fetal growth restriction, and intrapartum and intrauterine hypoxia was 0%. In neonates, the sensitivity was 93% for infectious CoD, while it was 0% and 25% for preterm complications and congenital malformations, respectively. In children, the sensitivity of the model in identifying an infectious disease as CoD was 83%, while it was 0% for the congenital malformations, tumors and other diseases. The sensitivity of the model in identifying maternal mortality causes was low for all conditions except for eclampsia (75%) and obstetric hemorrhage (75%). In other adults, the sensitivity of the model was highest for infectious diseases (68%) and lowest for malignant neoplasms (19%).

Table 1. Performance of the InterVA (Interpreting Verbal Autopsy) model compared to the complete diagnostic autopsy at individual-level prediction by study group and cause of death.

Cause of death (CDA)		n	Classification of cases				Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
			TP	TN	FP	FN
Stillbirths (N=18)	Infections	4	0	14	0	4	0	100	N/A	78
	Fetal growth restriction	7	0	10	1	7	0	91	0	59
	Intrapartum hypoxia	3	0	15	0	3	0	100	N/A	83
	Intrauterine hypoxia	2	0	16	0	2	0	100	N/A	89
	Congenital malformations	0	0	11	7	0	N/A	61	0	100
	Non-conclusive	2	1	7	9	1	50	44	10	88
Neonates (N=41)	Infections	27	25	2	12	2	93	14	68	50
	Congenital malformations	4	1	37	0	3	25	100	100	92
	Preterm complications	5	0	36	0	5	0	100	N/A	88
	Intrapartum complication	3	1	36	2	2	33	95	33	95
	Other diseases	2	0	39	0	2	0	100	N/A	95
	Non-conclusive	0	0	41	0	0	N/A	100	N/A	100
Children (N=54)	Infections	42	35	3	9	7	83	25	80	30
	Congenital malformations	2	0	52	0	2	0	100	N/A	96
	Malignant neoplasms	7	0	47	0	7	0	100	N/A	87
	Other diseases	3	0	44	7	3	0	86	0	94
	Non-conclusive	0	0	51	3	0	N/A	94	0	100
Maternal deaths (N=91)	Infections*	39	11	39	13	28	28	75	46	58
	Abortion	9	0	82	0	9	0	100	N/A	90
	Eclampsia	4	3	64	23	1	75	74	12	98
	Obstetric hemorrhage	16	12	59	16	4	75	79	43	94
	Other obstetric complications	6	0	85	0	6	0	100	N/A	93
	Non-obstetric diseases**	16	4	67	8	12	25	89	33	85
	Non-conclusive	1	0	89	1	1	0	99	0	99
Other adults (N=112)	Infections	80	54	20	12	26	68	62	82	43
	Malignant neoplasms	16	3	95	1	13	19	99	75	88
	Other diseases	16	10	66	30	6	62	69	25	92
	Non-conclusive	0	0	110	2	0	N/A	98	0	100

CDA: complete diagnostic autopsy; n: number of cases; TP: true positives; TN: true negatives; FP: false positives; FN: false negatives; PPV: positive predictive value; NPV: negative predictive value; N/A: not applicable.

* Includes all infections, both obstetric and non-obstetric.

** Non-obstetric diseases do not include infections.

Table 2 shows the measures of overall concordance between the two methods corrected for chance by study group for all CoDs. The CCC ranged between -0.093 and 0.246, and Kappa statistic ranged from -0.030 to 0.232 (lowest in stillbirths and highest in other adults). Figure 1 presents the alluvial diagrams showing the differences in the assignment of individual CoD established in the two methods by study group.

Table 2. Measures of performance of the InterVA (Interpreting Verbal Autopsy) model compared to the complete diagnostic autopsy at individual-level prediction for all causes of death by study group.

Study group	CCC	Kappa
Stillbirths	-0.093	-0.030	(poor)
Neonates	0.119	0.142	(slight)
Children	0.020	0.020	(slight)
Maternal deaths	0.179	0.159	(slight)
Other adults	0.246	0.232	(fair)

CCC: Chance-corrected concordance calculated from 500 Dirichlet draws

Figure 1. Alluvial diagrams of the differences in assignment of individual causes of death established by the Complete Diagnostic Autopsy (CDA) and InterVA (Interpreting Verbal Autopsy) model by study group.

The stacked blocks represent the causes of death (CoDs) determined by the CDA (left) and by the InterVA model (right), and their size as proportional to the cause-specific mortality fractions (CSMFs). The branches between blocks represent differences in the composition of the CoDs between the CDA and the InterVA model, being their thickness proportional to the number of cases contained in both blocks connected by the branch. Each CoD is represented by a different color, which is the same in both diagnostic methods. The color of the branches is determined by the cause of actual death (CDA). The concordant cases between the CDA and the InterVA model are represented by branches connected to blocks of the same color. In contrast, misclassified cases are shown as branches connected to blocks of different color.

Cause of death assignment of the model at the individual level among patients dying of infectious diseases

Table 3 shows the performance of the VA model in assigning CoD among cases who died of an infectious disease according to the CDA in all study groups by infection category. The sensitivity of the model in identifying an infectious disease as CoD was low for all infectious categories, being 0% for tuberculosis, diarrhea, disseminated and other infections. Figure 2 shows the alluvial diagram of the comparison of CoD assigned by both methods.

Table 3. Performance of the InterVA (Interpreting Verbal Autopsy) model at individual-level prediction among all cases who died of infectious diseases according to the complete diagnostic autopsy.

Cause of death (CDA)	n	Classification of cases				Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
		TP	TN	FP	FN
Disseminated infections	51	0	139	2	51	0	99	0	73
Pneumonia	36	13	102	54	23	36	65	19	82
Meningitis	15	3	173	4	12	20	98	43	94
Tuberculosis	7	0	183	2	7	0	99	0	96
Diarrhoea	2	0	186	4	2	0	98	0	99
HIV/AIDS related	57	18	120	15	39	32	89	55	75
Malaria	6	2	181	5	4	33	97	29	98
Other infections	18	0	171	3	18	0	98	0	90

CDA: complete diagnostic autopsy; n: number of cases; TP: true positives; TN: true negatives; FP: false positives; FN: false negatives; PPV: positive predictive value; NPV: negative predictive value; N/A: not applicable

Disseminated infections: bacterial sepsis of newborn (n=21), puerperal sepsis (n=6), streptococcal sepsis (n=5) and other sepsis (n=19)

HIV/AIDS related infections: candidiasis (n=1), congenital viral diseases (n=1), cryptococcosis (n=11), cytomegaloviral disease (n=7), herpes simplex infection (n=1), miliary tuberculosis (n=20), salmonella infections (n=1), pneumocystosis (n=5), pulmonary tuberculosis (n=2), toxoplasmosis (n=7) and tuberculous meningitis (n=1)

Other infections: acute pericarditis (n=2), pyelonephritis (n=2), congenital viral diseases (n=2), chorioamnionitis (n=2), GBS infection (n=2), tetanus (n=1), peritonitis (n=3), rabies (n=3) and zygomycosis (n=1)

Figure 2. Alluvial diagrams of the differences in the individual cause of death as established by the complete diagnostic autopsy (CDA) and the InterVA (Interpreting Verbal Autopsy) model among patients who died of infectious diseases.

The stacked blocks represent the causes of death (CoDs) determined by the CDA (left) and by the InterVA model (right), and their size as proportional to the cause-specific mortality fractions (CSMFs). The branches between blocks represent differences in the composition of the CoDs between the CDA and the InterVA model, being their thickness proportional to the number of cases contained in both blocks connected by the branch. Each CoD is represented by a different color, which is the same in both diagnostic methods. The color of the branches is determined by the cause of actual death (CDA). The concordant cases between the CDA and the InterVA model are represented by branches connected to blocks of the same color. In contrast, misclassification cases are shown as branches connected to blocks of different color. Diseminated infections: bacterial sepsis of the newborn (n=21), puerperal sepsis (n=6), streptococcal sepsis (n=5) and other sepsis (n=19) HIV/AIDS related infections: candidiasis (n=1), congenital viral diseases (n=1), cryptococcosis (n=11), cytomegaloviral disease (n=7), herpes simplex infection (n=1), miliary tuberculosis (n=20), salmonella infection (n=1), pneumocystosis (n=5), respiratory tuberculosis bacteriologically and histologically confirmed (n=2), toxoplasmosis (n=7) and tuberculous meningitis (n=1) Other infections: acute pericarditis (n=2), pyelonephritis (n=2), congenital viral diseases (n=2), chorioamnionitis (n=2), GBS infection (n=2), tetanus (n=1), peritonitis (n=3), rabies (n=3) and zygomycosis (n=1) Non-infectious diseases (by the InterVA model): congenital malformations (n=1), intrapartum complication (n=2), eclampsia (n=12), obstetric haemorrhage (n=10), non-obstetric diseases (n=5)and other diseases (n=29).

Discussion

It is recognized that accurate information on what is causing deaths is essential to reduce mortality. In this study, we have assessed for the first time to our knowledge, the validity of a commonly used VA method in establishing the CoD compared with the gold standard (the CDA) in different age groups of patients dying at a tertiary-level hospital in Maputo, Mozambique. The agreement of the VA model was overall poor across all age groups and conditions, both at the individual and at the population level.

The two reference standards that have been used for validating computer-coded VA, i.e. physician-coded VA methods and health facility medical records, cannot be considered true gold standards. The comparison between computer-coded and physician-coded VA methods lacks an external reference or gold standard comparator^11,31. On the other hand, although health facility-derived information is considered as an appropriate reference standard for VA validation²⁷, reports from both high and low-income countries indicate that this information frequently contains clinical errors^4,5. It seems quite evident that if the main source of input to the VA tool is inaccurate, the output of the VA will not be precise either. Furthermore, if clinical errors are frequent even in well-equipped hospitals, it is expected that their frequency would be higher in VA data.

In this study, the performance of the VA model was overall poor in identifying CoDs in stillbirths. These findings disagree with those of a report from Pakistan using clinical data as reference standard, indicating that a physician-coded VA tool was valid to ascertain causes of stillbirths, specially congenital malformations³². In neonates, the sensitivity of the model in identifying preterm complications as a CoD, was also very low (0%), which may be relevant for pre-term birth prevention programs. In contrast, the performance of the model in identifying infectious diseases as a cause of neonatal death had a high sensitivity, suggesting that it may be an adequate method to identify neonatal sepsis at the community. Among children, the sensitivity of the model was only high in detecting infectious diseases as a CoD but it did not identify deaths due congenital malformations and malignant neoplasms. In maternal deaths, the sensitivity of the model was high in assigning eclampsia as a cause of maternal mortality; however, the probability that a maternal death identified as eclampsia by the model was actually eclampsia was quite low. Although there were 26 maternal deaths assigned to eclampsia as the most probable CoD according to the VA model, only four were actually due to eclampsia (most misdiagnosed cases died of infectious diseases), suggesting a significant overestimation of eclampsia as a cause of maternal mortality by this method. This is in agreement with a previous report where a high frequency of false positive clinical diagnosis of eclampsia compared to the CDA was also found, being most of them deaths from infectious diseases⁴. These findings are of relevance to eclampsia prevention programs, which may fail in reducing maternal mortality due to misdiagnosis. In adults, the sensitivity of the VA model was higher for infectious diseases compared to other CoD, but low in identifying malignant neoplasms as cause of mortality. According to these results, the model would underestimate malignant neoplasms as CoD in adults, which may be important for prevention programs of this condition in high mortality settings.

The performance of the model in identifying the specific infection CoD among patients who died of an infectious disease was overall low. The sensitivity of the model in identifying tuberculosis as a CoD was very low, which may be of public health relevance in high burden countries. Regarding malaria infection, the VA model and the CDA only agreed in two cases, while in the other four cases established by the CDA, the model assigned three of them to a non-infectious disease CoD and one as non-conclusive (Figure 2). Lack of precision at the individual level in assigning malaria infection as a CoD may be important to target malaria control efforts in the community and increasing programme’s effectiveness.

The main use of the VA information is to determine cause-specific mortality and distribution of CoD at the population level³³; for this reason we also estimated the CSMF accuracy between the two methods. Both methods differed in the distribution of the proportion of the deaths assigned to several disease categories. When corrected by chance, the accuracy of the model in predicting in the population the CoD was poor, especially in stillbirths and maternal deaths and imperfect in the other groups.

A possible limitation of our study that might have influenced the predictions of the model, is that the indicators used to estimate the CoD by the VA model were extracted from medical records, since VAs were not done, which relates to the absence in the clinical records of some indicators of the model (43 indicators, 18% of the total). Nevertheless, most of these indicators (n=24) were secondary questions related to the duration of the event and therefore, not pertinent if the event did not occur. On the other hand, the most likely explanation for the lack of registration in the clinical record of the other 19 indicators (8%) is that they were not identified. The fact that the study is based in a large hospital might be seen as a limitation to extrapolate findings to deaths occurring in rural health-facilities or at home, since cause-composition of deaths in the community may be different to that of those occurring in a hospital. However, it is important to remember that this is a validation study and therefore, the objective was not that the deaths included were representative of those occurring in the community, but rather that the comparator of the VA was as true gold standard as possible. Thus, we needed a set of deaths, whose causes were established by the CDA, and therefore they had to occur in a hospital setting with autopsy facilities. On the other hand, to avoid that the cause-composition of deaths in that particular hospital and/or time-period affected the accuracy of the estimates of the VA, we created multiple test datasets with widely varying cause-compositions as it has been suggested²⁸.

As explained in the methods section, we used InterVA version 4 because version 5 was not available at the time of this analysis. Even if the estimated CoD might differ between the two InterVA versions, a change in the group of CoD would not be expected. Otherwise it would mean that the two versions provide different results, requiring a revision of all published information using the previous InterVA model.

The post-2015 Development Agenda expects that high burden countries should have reliable information on number and CoD to reduce their main health problems³⁴. However, this goal cannot continue to rely on imprecise measurement tools. The main shortcoming to achieve the SDGs is the imprecision of the currently used methods to establish CoD. These findings highlight the need of improving the quality and performance of current VA techniques by developing more precise tools for CoD ascertainment.

In conclusion, the “data revolution” of the post-2015 Development Agenda expects that high burden countries should have reliable information on number and causes of death in order to reduce main health problems through evidence-based decision-making, and target and monitor health programs²⁹. However, this goal cannot continue to rely on imprecise measurement tools. The main shortcoming to achieve the SDGs, is not the scarce availability of physicians to carry out death certificates or VA codification, nor the solution is the available automated methods created to overcome some of the physician-coded VA limitations, but rather the imprecision of these methods to reliably establish causes of death. The findings of this study should serve to highlight the need to implement autopsy methods where they may be feasible, but even more importantly to improve the quality and performance of current VA techniques and to develop more precise CoD ascertainment tools.

Data availability