Measuring the Performance of Vaccination Programs Using Cross-Sectional Surveys: A Likelihood Framework and Retrospective Analysis

Justin Lessler, C Jessica E Metcalf, Rebecca F Grais, Francisco J Luquero, Derek A T Cummings and Bryan T Grenfell

PLOS Medicine, 2011, vol. 8, issue 10, 1-10

Abstract: Justin Lessler and colleagues describe a method that estimates the fraction of a population accessible to vaccination activities, and they apply it to measles vaccination in three African countries: Ghana, Madagascar, and Sierra Leone. Background: The performance of routine and supplemental immunization activities is usually measured by the administrative method: dividing the number of doses distributed by the size of the target population. This method leads to coverage estimates that are sometimes impossible (e.g., vaccination of 102% of the target population), and are generally inconsistent with the proportion found to be vaccinated in Demographic and Health Surveys (DHS). We describe a method that estimates the fraction of the population accessible to vaccination activities, as well as within-campaign inefficiencies, thus providing a consistent estimate of vaccination coverage. Methods and Findings: We developed a likelihood framework for estimating the effective coverage of vaccination programs using cross-sectional surveys of vaccine coverage combined with administrative data. We applied our method to measles vaccination in three African countries: Ghana, Madagascar, and Sierra Leone, using data from each country's most recent DHS survey and administrative coverage data reported to the World Health Organization. We estimate that 93% (95% CI: 91, 94) of the population in Ghana was ever covered by any measles vaccination activity, 77% (95% CI: 78, 81) in Madagascar, and 69% (95% CI: 67, 70) in Sierra Leone. “Within-activity” inefficiencies were estimated to be low in Ghana, and higher in Sierra Leone and Madagascar. Our model successfully fits age-specific vaccination coverage levels seen in DHS data, which differ markedly from those predicted by naïve extrapolation from country-reported and World Health Organization–adjusted vaccination coverage. Conclusions: Combining administrative data with survey data substantially improves estimates of vaccination coverage. Estimates of the inefficiency of past vaccination activities and the proportion not covered by any activity allow us to more accurately predict the results of future activities and provide insight into the ways in which vaccination programs are failing to meet their goals. : Please see later in the article for the Editors' Summary Background: Immunization (vaccination) is a proven, cost-effective tool for controlling life-threatening infectious diseases. It provides protection against infectious diseases by priming the human immune system to respond quickly and efficiently to bacteria, viruses, and other disease-causing organisms (pathogens). Whenever the human body is exposed to a pathogen, the immune system—a network of cells, tissues, and organs—mounts an attack against the foreign invader. Importantly, the immune system “learns” from the encounter, and the next time the body is exposed to the same pathogen, the immune system responds much faster to the threat. Immunization exposes the body to a very small amount of a pathogen, thereby safely providing protection against subsequent infection. More than two billion deaths are averted every year through routine childhood immunization and supplemental immunization activities (mass vaccination campaigns designed to increase vaccination coverage where immunization goals have not been reached by routine vaccination). Indeed, these two types of vaccination activities have eliminated smallpox from the world and are close to doing the same for several other infectious diseases. Why Was This Study Done?: To reduce deaths from infectious diseases even further, it is important to know the proportion of the population reached by vaccination activities. At present, countries report vaccination coverage to the World Health Organization (WHO) that is calculated by dividing the number of vaccine doses delivered during the activity by the size of the target population. However, estimates arrived at through this “administrative method” do not account for vaccine doses that were not actually delivered, and can only reflect a single vaccination activity, which prevents us from identifying populations that may be systematically missed by all vaccination activities (for example, children living in remote areas, or children whose parents refuse vaccination). Moreover, estimates of coverage obtained by the administrative method rarely agree with estimates obtained through cross-sectional surveys such as Demographic and Health Surveys (DHS), which are household surveys of family circumstances and health undertaken at a single time point. In this study, the researchers developed a method for measuring the performance of vaccination activities that estimates the fraction of the population accessible to these activities and within-activity inefficiencies. They then tested their method by applying it to measles vaccination in three African countries; before 1980, measles killed about 2.6 million children worldwide every year, but vaccination activities have reduced this death toll to about 164,000 per year. What Did the Researchers Do and Find?: The researchers developed a set of formulae (a “likelihood framework”) to estimate the effective coverage of vaccination activities using data on vaccine coverage from cross-sectional surveys and administrative data. They then applied their method to measles vaccination in Ghana, Madagascar, and Sierra Leone using data obtained in each country's most recent DHS survey and administrative data reported to WHO. The researchers estimate that 93%, 77%, and 65% of the target populations in Ghana, Madagascar, and Sierra Leone, respectively, were ever covered by any vaccination activity, and that inefficiencies within vaccination activities were low for Ghana, but higher for Madagascar and Sierra Leone. Consequently, the researchers' estimates of vaccination activity coverage were substantially lower than the administrative estimates for Madagascar and Sierra Leone but only slightly lower than that for Ghana. Finally, the researchers' estimates of routine vaccination coverage were generally lower than WHO-adjusted estimates but broadly agreed with age-specific vaccination coverage levels from DHS surveys. What Do These Findings Mean?: Although the accuracy of the estimates provided by this likelihood framework depends on the assumptions included in the framework and the quality of the data fed into it, these findings show that, by combining administrative data with survey data, estimates of vaccine coverage can be substantially improved. By providing estimates of both the inefficiency of past vaccination activities and the proportion of the target population inaccessible to any vaccination activity, this method should help public health experts predict the results of future activities and help them understand why some vaccination programs fail to meet their goals. Importantly, knowing both the size of the inaccessible population and the inefficiency level of past programs makes it possible to estimate the effect of providing additional doses of vaccine on vaccination coverage. Finally, the application of this new method might help individual countries understand how susceptibility to specific infectious diseases is building up in their population and enable them to avoid outbreaks similar to the measles outbreaks that have recently occurred in several African countries. Additional Information: Please access these Web sites via the online version of this summary at http://dx.doi.org/10.1371/journal.pmed.1001080.

Date: 2011
References: View complete reference list from CitEc
Citations:

Downloads: (external link)
https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001110 (text/html)
https://journals.plos.org/plosmedicine/article/fil ... 01110&type=printable (application/pdf)

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:plo:pmed00:1001110

DOI: 10.1371/journal.pmed.1001110

Access Statistics for this article

More articles in PLOS Medicine from Public Library of Science
Bibliographic data for series maintained by plosmedicine ().