Coverage Confusion! Trying to Make Sense of It

The Topic

Vaccination coverage rates are the most commonly used indicators of immunization program performance. Have you ever encountered different vaccination coverage estimates for the same country? In this issue of Snap Shots, we explain where these different coverage estimates come from and how they should and should not be used. We also point you to other references that we hope will help to clear up the coverage confusion.

DTP3 and Routine Immunization

Although coverage is calcuated for all vaccines, DTP3 coverage by one year of age is now widely accepted as the performance indicator for routine immunization. This is because DTP vaccine is most often delivered in routine immunization sessions versus campaigns. Also, completion of the DTP series before the first birthday has been a serious problem in some countries, so adoption of the DTP3 indicator is helping to focus global attention on this problem. We use DTP3 coverage throughout this issue to describe coverage trends.

Regional Coverage Trends

After increasing rapidly in the 1980s, vaccination coverage in many countries held steady or decreased during the 1990s. Since 2000, coverage has slowly begun to increase again. Globally, WHO/UNICEF estimate that DTP3 coverage increased from 75% in 2000 to 78% in 2004. The Africa region, where coverage fell sharply after 1990, experienced the most dramatic gains (54% in 2000 to 66% in 2004). Coverage in the Eastern Mediterranean region also improved (81% in 2000 to 86% in 2004), but in Southeast Asia coverage remained low (69% in 2004). The fact that it has not changed significantly since 1995 is also cause for concern.

Common Sources of Coverage Data

WHO/UNICEF's estimates of DTP3 coverage are used in Figure 1 above. Other common sources of coverage data are administrative reports and household surveys. How do these three sources compare?

• Administrative reports: National coverage rates are usually based on administrative data collected during vaccination sessions. Administrative data are used monthly, quarterly and annually to estimate coverage, calculate left-outs and drop-outs, and monitor trends. Administrative data overestimate coverage(1) when vaccinations given to children after their first birthdays are reported, census figures (coverage denominators) are too low, or reports are falsified. On the other hand, they underestimate coverage when census figures are too high or reporting is incomplete.
  

• Household surveys: Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS) produce national coverage estimates, but they are conducted infrequently and do not, in most cases, produce district coverage estimates. The CORE Group's Knowledge, Practice and Coverage (KPC) surveys produce reliable coverage estimates and are often used with smaller populations. Immunization Coverage Cluster Surveys, which are appropriate for use at both district and national level, provide a wealth of data not only about immunization coverage but also about service quality.
  

WHO/UNICEF Joint Reporting Form Key Performance Indicators Official country estimates of coverage for all EPI antigens WHO/UNICEF estimates of coverage for most EPI antigens DTP1-DTP3 drop-out rates - national rate and % of districts with drop-out >10% Proportion of districts reporting DTP3 coverage of >90%, 80%-89%, 50%-79%, >50% Proportion of districts reporting and with data included in the JRF Line items in the national budget for vaccines, for injection supplies Proportion of vaccine spending funded by Government (routine program only) Proportion of immunization financed by Government

• WHO/UNICEF estimates of coverage: In April each year, WHO and UNICEF collect coverage estimates and other information from individual countries on a Joint Reporting Form, or JRF. The JRF contains the official country estimates of coverage, which are most often based on administrative reports and census data. WHO and UNICEF then adjust the official country rates to reflect findings of population-based surveys and known data-quality and/or denominator issues.(2) The WHO/UNICEF coverage estimates are now used at global level to track country performance and measure progress toward global and regional immunization goals.
  
  

Where to Find Country Coverage Estimates

Both the official country estimates and the WHO/UNICEF estimates of vaccination coverage can be accessed through WHO's Vaccine Preventable Diseases Monitoring System.(3) This interactive web page contains country coverage estimates since 1980. It also makes available many other items of information from the annual JRF submissions that may be useful in assessing country performance (see box to the right).

Interpreting Coverage Estimates

When interpreting immunization coverage estimates, keep the following points in mind...

1. Coverage rates are averages. High coverage rates almost always mask pockets of much lower coverage. Coverage rates also tell us little on their own about disparities among socioeconomic groups.(4) To truly understand a country's coverage situation, one must have access to subnational coverage data. To determine whether the poor and other marginalized groups are being reached with immunization services, whenever possible, household surveys should be analyzed by wealth quintile.
   
2. Like stocks and mutual funds, even very high coverage this year does not guarantee the same coverage next year. In one African country, national DTP3 coverage has been over 80% for all but three of the past 10 years, but only 11% of districts have reported 80% coverage or higher in each of the last three years. This suggests frailty in the immunization system and demonstrates the need to consider more than one year's data at any given time.
   
3. A coverage estimate is only reliable if the data behind it are of reasonable quality. GAVI Data Quality Audits (DQA) are helping improve data collection and reporting in many countries. Nonetheless, different sources of immunization data often yield different coverage estimates, and all coverage data have certain limitations. For this reason, one must always look at the source of a coverage estimate to figure out what it really means and how it might be used.

Deciding Which Coverage Estimates to Use

Coverage estimates from different sources are often used for different purposes. Administrative data are most readily available at each level of a health system and, in lieu of more reliable data, they are routinely used by program managers and donors to monitor trends in coverage and pinpoint problems. Surveys generate more reliable estimates of coverage, but they are carried out less frequently and cannot be used to monitor trends on a monthly or even an annual basis. Also, national surveys do not generally permit district-level estimates, which, as mentioned above, are critical to understanding country performance.

Whichever data source or sources are chosen, it is always important NOT to attempt to compare coverage estimates that are NOT comparable. In Figure 2 above, clearly it would be meaningless to use the DHS data from 1997 and the administrative data from 1999 to describe a coverage trend. A quick list of the questions that program planners and managers should ask themselves before selecting and attempting to compare coverage estimates is shown below.

What Can USAID Missions and Projects Do to Reduce the Coverage Confusion?

• Use available coverage rates, but also question them. Look behind the figures to understand how they were calculated and what they mean.
  
• Seek information on coverage over time and in smaller geographical areas. This will give you the information you need to better direct scarce resources toward low-performing areas.
  
• When commissioning or conducting household surveys, make sure that samples permit coverage estimates by socioeconomic status and other units of interest. This will help in identifying special-needs populations, even in countries and regions with relatively high immunization coverage.
  
• Support health systems initiatives to improve the quality of immunization data and teach health personnel to interpret and use their own data to improve program performance.
  
• Visit the WHO Vaccine Preventable Disesases Monitoring System web page and become familiar with the information it contains. This site makes both official country estimates and the WHO/UNICEF estimates of immunization coverage since 1980 available in an interactive format.
  
  

References:

(1) Murray CJ, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren M. Validity of reported vaccination coverage in 45 countries. Lancet, 2003 Sep 27;362(9389):1022-7

(2) WHO/UNICEF Estimates of National Immunization Coverage, 1980-2004: Methods

(3) WHO Data, Statistics and Graphics and WHO Vaccine Preventable Diseases Monitoring System web pages

(4) Gwatkin DR."The need for equity-oriented health sector reforms." International Journal of Epidemiology 2001;30:720-723

(5) Immunization Essentials: A Practical Field Guide, USAID, 2003 Chapter 4, pp.80-81.