Board of Scientific Counselors – Presentations

September 17-18, 2007

Back to September 2007 Presentations

NIS & SLAITS Program Overview: National Immunization Survey

PowerPoint Presentation [PDF – 13 MB]

Slide 1

NIS & SLAITS Program Overview:
National Immunization Survey

NCHS BSC Meeting
September 17, 2007
James A. Singleton, M.S.
Chief, Assessment Branch
Immunization Services Division

Picture of slide 1 as described above, which includes pictures of a telephone and a doctor holding a clipboard.

Slide 2


  • History of the NIS
  • Purpose
  • Methods
  • Resources
  • Information Dissemination
  • Proposed objectives of program review
Picture of slide 2 as described above

Slide 3

Development of the U.S. Immunization Program

  • Vaccination Assistance Act, 1962
    • Established Section 317 grant program
  • Childhood Immunization Initiative, 1977
    • Focus on enacting school entry laws
  • Vaccine coverage monitoring, 1957-91
    • U.S. Immunization Survey, CPS
      • Parental report of vaccinations
      • Dropped after 1985; cost, validity concerns
    • Retrospective school surveys, 1985-91
Picture of slide 3 as described above

Slide 4

Polio, DTP, Measles-Containing Vaccination Coverage Levels: 1959-1990, U.S. Immunization Survey

Picture of slide 4 as described above, which includes a picture of a line graph

Slide 5

U.S. Measles Resurgence

  • Cases 55,622
  • Age group affected < 5 yrs
  • Hospitalizations > 11,000
  • Deaths 132
  • Direct medical costs > $150 million
Picture of slide 5 as described above

Slide 6

Childhood Immunization Initiative (CII), 1993

  • Achieve 90% coverage for preschool children
  • Vaccines for Children Program
    • Entitlement program funded by Medicaid
    • ACIP recommendations translated into appropriations
  • Increase in Section 317 funding
  • Funding for the National Immunization Survey (NIS)
    • Created in 1994 to measure coverage goals of the CII by state and selected urban areas
  • NHIS NIPRCS 1994-1999
    • National provider-reported data on vaccination of children 19-35 mo
    • Used in NIS weighting adjustments for households without telephones
Picture of slide 6 as described above

Slide 7


  • National Center for Immunization and Respiratory Diseases (NCIRD)
    • Prevent disease, disability and death through immunization and by control of respiratory and related diseases
  • Immunization Services Division
    • Protects individuals and communities from vaccine-preventable diseases through provision of federal funds and contracts to purchase vaccine, the provision of technical and financial support of immunization programs, provider and public education, evaluation and research
Picture of slide 7 as described above

Slide 8

Picture of slide 1 as described above

Slide 9

What gets measured – gets done

Notes: The National Commission on Prevention Priorities guided an update to 2001 ranking of clinical preventive services. Rankings were based on two measures: clinically preventable burden and cost-effectiveness. The top three ranked preventive services included the childhood immunization series.

Picture of slide 9 as described above

Slide 10

Purpose of CDC Vaccine Assessment

  • Overarching goal – facilitate program improvement and behavior change leading to increased vaccination levels, thus reducing health and societal impact of vaccine-preventable diseases (VPD’s)
  • Need vaccine assessment system for immunization programs
    • Children < 3, 4-6, 11-18 years
    • Adults
Picture of slide 10 as described above

Slide 11

Purpose of CDC Vaccine Assessment

  • Evaluate effectiveness of immunization grant programs over time
  • Help with allocation of Vaccines for Children (VFC) program resources
  • Monitor progress toward national Healthy People objectives
  • Build and maintain support for national & state immunization programs
Picture of slide 11 as described above

Slide 12

Purpose of CDC Vaccine Assessment

  • Identify subgroups at higher VPD risk
  • Identify facilitators & barriers to vaccination to improve interventions
  • Evaluate implementation of vaccine recommendations from the Advisory Committee on Immunization Practices (ACIP)
  • Assess differential impact of vaccine shortages
  • Evaluate uptake of new vaccines
Picture of slide 12 as described above

Slide 13

Purpose of CDC Vaccine Assessment

  • Assist in evaluating health impact of vaccination
    • Proxy for immunity
    • Ecologic analysis of trends
    • Vaccine effectiveness studies
    • Vaccine safety studies
  • Emergency preparedness
    • Influenza pandemic monitoring
Picture of slide 13 as described above

Slide 14

Designing a Vaccine Coverage Assessment System

  • Target population
  • Sampling method
  • Vaccines assessed
  • Auxiliary data collected
  • Geographic specificity
  • Periodicity of data collection and reporting
  • Timeliness (vaccination to data availability)
  • Comparability across areas and over time
  • Representativeness
  • Accuracy
  • Resources
Picture of slide 14 as described above

Slide 15

Overview of the National Immunization Survey (NIS)

Picture of slide 15 as described above

Slide 16

Target Population

  • Noninstitutionalized children aged 19-35 months at time of telephone interview
Picture of slide 16 as described above

Slide 17

Sample Design

  • Stratified, two phase survey:
    • List-assisted, random-digit-dialing survey to identify age-eligible children
      • Spanish interviewers & CATI version
        • 9% of 2005 interviewers conducted in Spanish
      • Language Line Services used for 187 interviews (0.67%) in 2005
    • Mailed survey to providers identified during telephone interview to collect provider-reported vaccination histories
Picture of slide 17 as described above

Slide 18

Picture of slide 18 as described above

Slide 19


  • All ACIP recommended vaccines that children should have received by 19 months of age
Picture of slide 19 as described above

Slide 20

Picture of slide 20 as described above

Slide 21

Demographics and Other Data Collected

  • Household
    • Race/ethnicity of mother, child
    • Mother’s age, education, marital status
    • Family income
    • Health insurance
    • WIC participation
    • Shot card available
    • Breastfeeding
  • Providers
    • # physicians at practice
    • Facility type
    • VFC provider
Picture of slide 20 as described above

Slide 22

NIS Special Purpose Supplements (Topical Modules)

  • Subsamples of NIS survey can be asked special-purpose questions
    • Insurance Status (2001-02; 2006+)
    • Day Care and Breast Feeding (2001-02)
    • Attitudes and Beliefs (2001-02)
    • Vaccine Safety (2003-04)
    • Vaccine Shortage (2003-04)
    • Childhood Influenza (2004)
    • SES (planned 2008)
    • Parental concerns (planned 2008)
Picture of slide 22 as described above

Slide 23

New NIS Modules

  • Develop 2007, collect data 2008
  • SES Module (national)
    • Barriers to immunization
    • Factors associated with racial/ethnic and income-related coverage disparities
    • Add key questions to NIS core later
  • Parental Concerns Module
    • Early warning system for parental concerns about vaccination
    • Developed with NVAC Subcommittee on Public Engagement
    • Consider periodic state-specific estimates

Notes: Past NIS modules have addressed parent safety concerns, deferral of vaccination in a shortage, influenza vaccination of young children. We are also interested in ideas for future modules for the 2009 NIS.

Picture of slide 23 as described above

Slide 24

Geographic Specificity

  • National level
  • State level
  • City level
    • Six cities receiving Section 317 immunization grant funding
  • Other city/county areas
    • chosen/funded by state grantees
    • Eight areas chosen for 2007
Picture of slide 24 as described above

Slide 25

NIS is More Than One Survey…

Picture of slide 25 as described above

Slide 26


  • Conducted continuously based on quarterly samples
  • Reported biannually based on data delivered in June and December
  • Focus on data reporting each summer using calendar year data
Picture of slide 26 as described above

Slide 27


  • Average time from vaccination to delivery of calendar year data (months):
    • 38m for HepB birth dose (range 25-52m).
      • (19m in Dec + 6m till data deliv. = 25)
      • (34m in Jan + 12m data col. y + 6m till data deliv. = 52)
    • 32m for 3rd Rota (range 19-46m).
    • 26m for MMR1, VAR1 (range 13-40m).
    • 19m for DTaP4 (range 6-33m).
    • 17m from end of flu vacc. Period (Jan)
Picture of slide 27 as described above

Slide 28


  • Same methods and contractor conduct survey in all sampling areas
  • Sample size chosen to achieve effective sample size of 180 children with adequate provider data in each area
  • 95% CI half-width of 7.5% or less
  • Comparability of methodology means estimates are comparable among states and urban areas over time
Picture of slide 28 as described above

Slide 29


  • Data weighted to account for households without landline phones and for nonresponse
  • In Q1/Q2 of 2006, 10.4% of children in US lived in HH w/o landline phones
  • Research underway to possibly improve the noncoverage adjustments
Picture of slide 29 as described above

Slide 30

NIS 2005 Sample

Total sample released – 4,465,261
Resolved sample – 3,721,224 (83%)
Households identified – 1,085,040
Households screened – 1,006,435 (93%)
Eligible households – 31,909 (3.2%)
Complete interviews – 26,867 (84%) 27,627 children
Children with adequate Provider data – 17,448 (63%)

Picture of slide 30 as described above

Slide 31

Wireless Substitution: Preliminary Data from the 2006 National Health Interview Survey

Picture of slide 31 as described above

Slide 32


  • Parental report of childhood vaccination has been shown to be unreliable
  • For all NIS estimates, only provider-reported vaccinations are used for estimation of vaccine coverage
Picture of slide 32 as described above

Slide 33

Picture of slide 33 as described above

Slide 34


  • Children aged 13-17 years at time of telephone interview with parent
  • Vaccination histories from providers
  • Measures vaccinations ages > = 11 years
  • Conducted national survey Q4 2006
    • 5,483 completed interviews
  • Repeat Q4 2007
  • Expansion to state/grantee level survey high priority for 2008
Picture of slide 34 as described above

Slide 35

Strengths of the NIS

  • States and selected city/county areas use estimates for program improvement
  • Comparability of methodology means estimates are comparable among states and urban areas over time
  • Identifies areas, populations and factors associated with under-immunization
  • Provider-verified results
  • Provides sampling frame for other health surveys (SLAITS)
Picture of slide 35 as described above

Slide 36

Challenges for NIS

  • Declining telephone survey response rates – 87% (1994) to 64.5% (2006)
  • Wireless-only households
  • Some non-coverage and non-response bias may remain after weighting adjustments
  • Increasing costs
Picture of slide 36 as described above

Slide 37

NIS Operational/Methods Research

  • Research to decrease bias, increase cost-efficiency, and guide future NIS methodology
    • Use of IIS sample frame & age-targeted phone lists, 2+ phone banks, redesign questionnaire, advance letter, calling rules, answering machine messages, sensitivity analysis model to evaluate potential bias
  • Research to address wireless only and phoneless households
    • Analysis of NHIS by phone status
    • Pilot studies calling cell phones
Picture of slide 37 as described above

Slide 38

NIS and Immunization Information Systems (IIS)

  • “The quality and completeness of the registry data must be improved and must be comparable across all states before consideration may be given to supplement or replace the provider-reported data in the NIS.”
    – Khare et al., Arch Pediatr Adolesc Med 2006
  • NIS provider question on use of IIS to obtain vaccination histories (2006)
  • Offering service to grantees to evaluate IIS via match with NIS sample
Picture of slide 38 as described above

Slide 39

NIS and Immunization Information Systems (IIS)

  • Conducting NIS survey & provider record check on IIS sample in 2 states
    • Dual frame approach may reduce cost, increase validity
    • Evaluate use of provider record check on IIS sample to estimate local vaccination coverage
    • Evaluate non-response bias in the RDD frame using IIS data as “gold” standard
  • Encourage use of IIS for local assessment
Picture of slide 39 as described above

Slide 40

Vaccine Assessment Funding Sources (FY07)

  • Total $23.3m
    • < 1% of annual $2.5b VFC vaccine purchase
  • $12.8m PHS Evaluation
  • $ 7.7m Vaccines for Children Program
    • Additional VFC $ for NIS-Teen, FY08
  • $ 1.4m state Section 317 grants
    • NIS oversampling in areas chosen by state
  • $ 0.9m Division (ISD) research funds
  • $ 0.6m influenza pandemic planning
Picture of slide 40 as described above

Slide 41

Vaccine Assessment Funding Allocation (FY07)

  • $20.5m NIS contract (NORC)
    • $12.2m core NIS
    • $ 2.2m NIS-Teen
    • $ 2.7m modules (adult, SES, Concerns)
    • $ 1.5m operational/methods research
    • $ 1.4m NIS oversampling in selected areas
    • $ 0.6m influenza pandemic survey module
  • $ 1.1m IAA’s with NCHS
    • Support of NHIS immunization questions
    • NCHS staff support of NIS (~1.5 fte)
  • $ 1.7m other assessment activities
Picture of slide 41 as described above

Slide 42

Vaccine Assessment Staff Resources

  • Assessment Branch (AB), ISD/NCIRD
    • 14 FTE’s, 3 contractors
      • Statisticians, epidemiologists, health scientists
      • ~ 9 FTE-equivalents devoted to NIS
  • NCHS (~1.5 FTE)
    • NIS contract originally managed by NCHS
    • Effective 2005, co-project officers established
      • AB/ISD responsible for NIS
      • NCHS responsible for SLAITS
        • Also handles NIS IRB
    • Statistician
      • Methods research, public use file development
Picture of slide 42 as described above

Slide 43

Information Dissemination

  • Annual MMWR articles
    • Briefings with states/local grantees
  • Post detailed tables online twice yearly
    • Q3-4 2005 + Q1-2 2006 data ~ Jan. 2007
    • Q1-4 2006 data ~ summer 2007
  • Public use file available annually
  • In-house analysis
    • 13 NIS papers published in 2006
  • Ad hoc requests from states, CDC, other
  • Developing online data query system
Picture of slide 43 as described above

Slide 44

Uses of NIS Data

  • State immunization program improvements
  • Assist in management of the Vaccines for Children (VFC) entitlement program (health insurance)
  • Monitoring Healthy People 2010 objectives
  • Uptake of new vaccines
  • Factors associated with coverage
  • Racial/ethnic disparities
  • Parental vaccine safety concerns
  • Quality performance measures
    • timeliness, age-appropriate & validly-spaced doses
  • Use of combination vaccines
  • County-level coverage estimation
  • Coverage trends by birth cohort
  • Cost-effectiveness analysis
  • Vaccine-effectiveness studies
Picture of slide 44 as described above

Slide 45

Strategic Planning

  • April 2007 External Review of Intramural Research
    • Prioritization of vaccination assessment
      • “What should we be assessing?”
  • CDC vaccination assessment work group
  • Feedback on priorities
    • National Vaccine Advisory Committee
    • Immunization stakeholders
  • October 2007 CCID BSC NCIRD subcommittee
    • Issues in data analysis and reporting
Picture of slide 45 as described above

Slide 46

NCHS Program Review Proposed Questions

  • What could be done to improve the validity and efficiency of the NIS?
  • Are we conducting the appropriate types of methods research to guide improvement of the NIS?
  • How should we deal with increasing prevalence of households with only cell phones?
  • What alternative sampling designs should be considered?
Picture of slide 46 as described above

Slide 47

Additional Slides

Picture of slide 47 as described above

Slide 48

Vaccine-Specific Coverage: Preschool-Aged Children

Picture of slide 48 as described above

Slide 49

Trends in Vaccine Coverage in Vermont, NIS 2000-2006

Picture of slide 49 as described above

Slide 50

Estimated PCV7 Coverage (3+ doses), by State, National Immunization Survey, U.S., 2004

Picture of slide 50 as described above

Slide 51

Data Analysis & Reporting

  • Reporting coverage for vaccine series
  • Defining vaccination outcome
  • Other measures of coverage
    • Timeliness
    • Cumulative % vaccinated by age
  • Trends by birth cohort vs. data collection year
  • Small area estimation
    • Estimates for 181 counties using data from 2004-05 combined
Picture of slide 51 as described above