Adding Households with Cell Phone Service to the National Immunization Survey (NIS), 2011

The Centers for Disease Control and Prevention (CDC) uses the National Immunization Survey (NIS) to monitor national, state, and selected local area vaccination coverage among US children, specifically children age 19-35 months (NIS) and 13-17 years (NIS-Teen). From 1994 to 2010, the surveys used a random digit dial (RDD) list-assisted landline phone sample frame. Over time, landline phone use decreased while cell phone use increased. By the end of 2011, 38.1% of children in the United States lived in households with only cell phone service[1]. Poverty has been associated with both cell phone only service and not receiving vaccines, which can affect accurate measurement of vaccination coverage [1-3]. In 2011, the NIS sampling frame was expanded from sampling landline phones to sampling landline and cell phones, creating a dual frame sample. Differences between the dual-frame and landline data were documented. Using the dual-frame sampling, including cell phone only survey participants, assures that the survey base reflects the US population.

Key Findings

• Adding cell phones made few changes in the final 2011 NIS (19-35 month old) and NIS-Teen national estimates of vaccination coverage, overall and by poverty status (See NIS Difference Tablepdf icon and NIS-Teen Difference Tablepdf icon).
  • NIS national dual-frame and landline frame differences for specific vaccines were small and ranged from -0.8 to +0.4 percentage points (See NIS Difference Tablepdf icon).
    • Differences between dual-frame and landline frame estimates for specific vaccines at the state and local level were larger, with a range of -6.2 to +6.1 percentage points for the 4:3:1:-:3:1:4 series (4 or more doses of DTaP, 3 or more doses of poliovirus vaccine, 1 or more doses of any MMR vaccine, 3 or more doses of HepB vaccine, 1 or more doses of varicella vaccine, and 4 or more doses of PCV. Hib vaccine is excluded.) (See NIS Difference Tablepdf icon.)
  • NIS-Teen national dual-frame and landline frame differences for specific vaccines ranged from -1.2 to 2.8 percentage points (NIS-Teen Difference Tablepdf icon).
    • Differences between dual-frame and landline frameestimates for specific vaccines at the state and local level were larger, with a range of -3.4 to +3.3 percentage points for Tdap (tetanus toxoid, reduced diptheria toxoid, and acellular pertussis on or after age 10 years) and -6.1 to +3.4 percentage points for MenACWY (≥1 dose of meningococcal conjugate vaccine or meningococcal vaccine) (NIS-Teen Difference Tablepdf icon).
• The change to a dual-frame made the NIS sample frame look more like the US population.
  • Telephone surveys need to include households with cell phones [4].
  • Since a dual-frame sample has lower response rates, validity of the NIS dual-frame sample will be monitored annually.

 Top of Page

Change to Dual Frame

• Differences in Results
• Guide to Interpretation
• Evaluation Plan

The NIS began in 1994 to monitor vaccination coverage among children age 19-35 months using a list-assisted landline RDD telephone sample frame with a follow-up mail survey of vaccination providers for those NIS eligible children whose parent/guardian gave consent. The NIS-Teen was added in 2006 using the same methods, and surveying households for adolescents age 13-17 years. The RDD household survey and provider record check methods have been described [5]. The sample of telephone numbers must be large to identify a sufficient number of households with children in the target age ranges and allow estimation of vaccination coverage with the desired level of precision for each state and selected local area sampled. With decreasing response rates over time, larger telephone samples were needed to ensure the required numbers of children with adequate provider data are reached, increasing data collection costs. Results from the National Health Interview Survey (NHIS) show that since 2003, the proportion of households without phone service remained stable (2%); however, the proportion of children living in households with cell phone only service has increased from 2.9% in 2003 to 38.1% in the second half of 2011 [1]. At least one factor, poverty, has been associated both with having cell phone only service and lower vaccination coverage, increasing the potential for bias in surveys reaching only landline phones because of lack of a representative sampling frame [1-3]. This combination of a large and growing cell phone only population and lower child vaccination coverage among households living in poverty can lead to bias in the estimation of vaccination coverage if survey weights developed to account for probability of selection and minimize bias from incomplete sample frame and nonresponse do not adequately compensate. In the fourth quarter of 2010 the NIS piloted a national dual landline and cell phone sample frame.

In 2011, the NIS changed from a landline sample to a dual landline and cell phone sample. In the first half of the year, the dual-frame sample was implemented at the national level. In the second half of the year, the cell phone component of the dual-frame sample was increased to reach state and local areas and allow vaccination coverage estimates across all areas. This online update summarizes the effect of the change on vaccination coverage estimates, provides some guides to interpretation of the dual sampling frame results, describes the weighting methods, and identifies additional changes that were made to facilitate implementation of the sampling frame.

Differences in Results

Adding cell phones made few changes in the final 2011 NIS (19-35 month old) and NIS-Teen national estimates of vaccination coverage, overall and by poverty status (See NIS Difference Tablepdf icon and NIS-Teen Difference Tablepdf icon). The NIS national dual-frame and landline frame differences for specific vaccines were small and ranged from -0.8 to +0.4 percentage points (See NIS Difference Tablepdf icon); however, differences between dual-frame and landline frame  estimates for specific vaccines at the state and local level were larger, with a range of -6.2 to +6.1 percentage points for the 4:3:1:-:3:1:4 series (4 or more doses of DTaP, 3 or more doses of poliovirus vaccine, 1 or more doses of any MMR vaccine, 3 or more doses of HepB vaccine, 1 or more doses of varicella vaccine, and 4 or more doses of PCV. Hib vaccine is excluded.) (See NIS Difference Tablepdf icon). NIS-Teen national dual-frame and landline frame differences for specific vaccines ranged from -1.2 to +2.8 percentage points (NIS-Teen Difference Tablepdf icon); however, differences between dual-frame and landline frame  estimates for specific vaccines at the state and local level were larger, with a range of -3.4 to +3.3 percentage points for Tdap (Tetanus toxoid, reduced diptheria toxoid, and acellular pertussis on or after age 10 years) and -6.1 to +3.4 percentage points for MenACWY (≥1 dose of meningococcal conjugate vaccine or meningococcal vaccine).

For national, state, and local areas, differences between the dual-frame and landline estimates were generally larger when stratified by poverty level (See NIS Difference Tablepdf icon and NIS-Teen Difference Tablepdf icon).

Guide to Interpretation

It is necessary to account for potential biases that may be built into survey methods [6]. For example, poverty is one known factor associated with both cell phone only status and low vaccination coverage. This could be a risk to accurate vaccination coverage estimates. The change in the NIS and NIS-Teen sample frames from a landline frame in 2010 to dual-frame in 2011 was made to address potential bias that could exist if vaccination coverage was different between children in landline and cell phone only households. Because the cost of adding a cellular phone sample is greater than a landline phone sample, an optimum sample size could not always be added during each survey quarter in each sampling area. Therefore, compared to differences at the national level, it is harder to interpret the meaning of the differences seen at the state or local level given the wider confidence intervals, most likely due to smaller final cell sample size at the state and local level. While it is possible that the expansion to a dual landline and cellular telephone sampling frame could account for at least some of the differences in state or local level vaccination coverage estimates between 2010 and 2011, some of the variation could be due to variation in the actual sample or in vaccination coverage. Immunization programs are encouraged to review the differences between the 2010 and 2011 estimates with caution and look at programmatic as well as sampling changes that may be related to the change in sampling frame for those differences that are statistically significant.

Evaluation Plan

Adding the cell phone sampling frame increased the NIS representativeness compared to the general population, but it is not obvious if this necessarily improved the validity of the vaccination coverage estimates since the cell phone response rates were lower than the landline phone response rates. For the 2011 NIS, the landline Council of American Survey Research Organizations (CASRO) response rate was 61.7% for children in the landline sample and 25.2% for the cell phone sample. Providers returned adequate vaccination records for 71.6% of children with completed household interviews. For the 2011 NIS-Teen, the CASRO was 61.5% of adolescents in the landline and 22.4% for the cell phone sample [7,8]. Providers returned adequate vaccination records for 57.2% of adolescents with completed household interviews. The CASRO response rate is the product of three other rates:

1. the resolution rate, which is the proportion of telephone numbers that can be identified as either for a business or residence;
2. the screening rate, which is the proportion of qualified households that complete the screening process; and
3. the cooperation rate, which is the proportion of contacted eligible households for which a completed interview is obtained.

The NIS will evaluate the 2011 dual sampling frame results using data from the National Health Interview Survey (NHIS) Provider Record Check (PRC) completed in the same year, evaluating the total survey error, the validity of the dual sampling frame estimates, and changes in nonresponse and total survey error. The NHIS PRC has been continued in 2012 for ongoing evaluation of the NIS dual sampling frame.

 Top of Page

Weighting Methods

The 2011 NIS included a total of 19,144 completed household interviews with adequate provider data with 16,919 in the landline sample and 2,225 in the cell phone sample, of which 1,445 were cell phone only households. At the state and local area level, a median number of 312 children had completed household interviews with adequate provider data with 279 in the landline sample and 33 in the cell phone sample, of which 22 were cell phone only households. The 2011 NIS-Teen included a total of 23,564 completed household interviews with adequate provider data with 20,848 in the landline sample and 2,716 in the cell phone sample, of which 1,060 were cell phone only households. At the state and local area level, a median number of 403.5 children had completed household interviews with adequate provider data with 360.5 in the landline sample and 41.5 in the cell phone sample, of which 16 were cell phone only households.

The 2011 NIS and NIS-Teen dual-frame sample design used independent random digit dial (RDD) samples from landline and cell phone sampling frames. While the dual sampling frames provide nearly complete coverage of the population, the landline and cell phone samples overlap in their coverage of household with both a landline and cell phone. Weighting for both the landline and cell phone frames must account for phone numbers determined to be not associated with a household, non-response at multiple points during the survey, multiple phone lines, and for variations in the volume of calls by survey quarter. Additionally, the final estimates must take into account the small phoneless population not included in the dual-frame. These issues are managed through a dual-frame weighting strategy to combine the landline and cell phone samples, account for overlap between the two samples, and minimize the mean squared error (MSE) for domains subject to high sampling variance. Overlap between the landline and cell phone samples results from cell mainly households, cell phone users who responded that they were unlikely to answer a call on their landline phone, from the cell sample in quarter one of 2011, and from dual landline/cell phone households from the cell phove sample beginning quarter two of 2011 when cell phone use was no longer screened and all cell phone respondents were considered eligible. Weighting addressed overlap within telephone status domains by compositing the landline and cell sample weights so as to minimize the variance associated with the dual sample frame. The independent landline and cell samples are weighted separately during initial steps through the creation of annual weights using methods previously described (See Weighting Table pdf icon[1 page] [9,10]. The landline and the cell phone samples were then combined and the remaining weighting adjustment steps carried out on the combined sample to produce estimates that are based on appropriate contribution from each sample and to ensure agreement with socio-demographic population controls.

Due to higher costs per call, the cell phone frame was under-sampled compared to the landline frame and the smaller relative size increased the weights for the cell sample compared to the landline sample and increased the variability of the cell phone sample. As a result, the variability of the survey weights increased, and the precision of the total population estimates decreased. To minimize this effect the cell sample weights were attenuated. For the cell phone sample where a landline was reported, the overlap of the cell phone and landline samples was adjusted on predefined variables (See Weighting Table pdf icon[1 page]) within each estimation area to minimize standard error and attenuate the effect of the small cell-phone sample. For the cell phone only sample attenuation was completed by combining the cell phone only sample with landline sample cases deemed “similar” to cell phone only sample cases, as determined using a logistic regression model for predicting cell only status run on the full set of dual-frame sample cases, and including both socio-demographic characteristics and vaccination status as explanatory variables. Landline sample cases predicted as having a relatively large probability  of being “similar” to cell phone only cases served as “proxy“ cell phone only sample cases and were weighted in conjunction with true cell phone only sample cases to represent the total cell phone only population. The composite factors were truncated to avoid extreme values and minimize the potential bias. The potential improvement in validity was weighed against changes in precision and bias which were managed through attenuating the cell phone sample with the landline sample.

 Top of Page

Additional Changes

• Age Eligibility
• 2012 Survey Design

Age Eligibility

Starting in 2011, the costs of building a more representative NIS sample frame were partially offset by an expansion of the age eligibility definition. Traditionally, the NIS defined a child as age eligible if they were 19-35 months on the day of the call. In 2011, this was changed to define a child as age eligible if they were 19-35 months on any day during the survey quarter. While the child was included in the NIS if the age definition was met, the provider report inclusion criteria remained 19-35 months and the provider report was restricted to those vaccines reported received by the time the child was 19-35 months if age of child at screener/interview was 17-18 months or 36-37 months. The change in age definition increased the number of eligible households, decreased the required number of calls, and decreased the total survey costs.

2012 Survey Design

In 2012, a streamlined NIS questionnaire was implemented [11], including questions on whether a child had ever received a vaccine and influenza vaccination and removing all other specific vaccine questions from the phone household questionnaire. The mailed provider record check was retained. No data were lost from the survey; however, the survey estimated time decreased from a maximum of 20 minutes to 10 minutes resulting in decreased burden on the respondent, higher completion rates, decreased number of required calls, and decreased survey costs [12]. The NIS-Teen questionnaire will be reviewed and a simplified version of the questionnaire will be tested to assess changes in data and costs. If the quality of the data is not affected and the simplified questionnaire is found to produce cost savings similar to the NIS simplified questionnaire, it may be implemented in a future NIS Teen survey.

Also in 2012, the NIS used an optimal cell phone sample design, reducing the number of landlines and increasing the number of cell phones, to reach the optimal balance between the two samples to minimize variance and bias at a fixed cost. The resulting share of cell phone sample in the dual-frame was closer to the share of cell phones in the actual population. In the same year, the NIS-Teen dual-frame sample included fewer landlines and more cell phones, however, it was not an optimum sample due to cost constraints.

Expansion of the NIS-Teen to an optimal cell phone sample design is being explored.

 Top of Page

Authors:

References

1. Blumberg, S. and J. Luke. Wireless substitution: Early release of estimates from the National Health Interview Survey, July–December 2010. June, 2011.
2. CDC Vaccination Coverage Among Children in Kindergarten — United States, 2009-10 School Year. MMWR, 2011. 60(21): p. 700-704.
3. CDC. National and State Vaccination Coverage Among Children Aged 19–35 Months — United States, 2010. MMWR, 2011. 60(34): p. 1157-1163.
4. CDC. Methodologic Changes in the Behavioral Risk Factor Surveillance System in 2011 and Potential Effects on Prevalence Estimates. MMWR, 2012. 61(22): p. 410-413.
5. CDC. Vital and Health Statistics, Statistical Methodology of the National Immunization Survey, 1994–2002 pdf icon[2.63MB, 64 pages]. Series 2 March, 2005; Number 138.
6. Simpson, D.M., T.M. Ezzati-Rice, and E.R. Zell, Forty Years and Four Surveys How Does our Measuring Measure Up? Am J Prev Med, 2001. 20(4S).
7. CDC. National and state vaccination coverage among adolescents aged 13 through 17 years—United States, 2011. In press.
8. CDC. National and state vaccination coverage among children aged 19–35 months—United States, 2011. MMWR 2012. In press.
9. CDC. National Immunization Survey, A User’s Guide for the 2010 Public-Use Data File pdf icon[1.51MB, 176 pages]. August 2011.
10. CDC. National Immunization Survey-Teen A User’s Guide for the 2010 Public-Use Data File pdf icon[1.32MB, 154 pages]. October 2011;
11. CDC. NIS-Child Hard Copy Questionnaire. 2012 pdf icon[62 pages].
12. Welch, B., et al., It’s About Time: Examining the Effect of Interviewer-Quoted Survey Completion Time Estimates on Nonresponse, in American Association for Public Opinion Research Annual Conference 2012: Orlando, FL.

• Go to NIS-Teen data for past years

 Top of Page