Comparison of COVID-19 vaccination coverage estimates from the Household Pulse Survey, Omnibus Panel Surveys, and COVID-19 vaccine administration data, United States, March 2021

Summary

COVID-19 vaccination coverage data can be used to evaluate the effectiveness of state and national COVID-19 vaccination programs, ensure equitable distribution of vaccine, and inform interventions to decrease disease burden. Since December 2020 when the first COVID-19 vaccines were authorized for use under an Emergency Use Authorization (EUA) in the United States, the Centers for Disease Control and Prevention (CDC) has collected data on COVID-19 vaccine administration through immunization information systems (IIS), the Vaccine Administration Management System (VAMS), and direct data submission from federal agencies and pharmacies (1). These COVID-19 vaccine administration data have been used to assess COVID-19 vaccination coverage nationally, by state and county, and by select sociodemographic characteristics. They have also been used to calculate vaccination coverage estimates presented in CDC’s COVID Tracker (2–5). To supplement data on vaccine doses administered, starting in December 2020, CDC sponsored questions concerning COVID-19 vaccination and factors related to vaccine confidence (6) appearing in two nationally representative household omnibus panel surveys (Ipsos Knowledge Networks and NORC AmeriSpeak) (7). CDC also worked with the Census Bureau to include questions about COVID-19 vaccination on the Household Pulse Survey (HPS) starting in January 2021 (8).

In this report, CDC compared estimates of the percent of respondents reporting receipt of at least one dose of COVID-19 vaccine among adults aged ≥18 years across three data sources: vaccine administration data that included the cumulative number of people who received at least one vaccine dose from the start of the vaccination program through each day from March 3 to March 15, 2021 (1); omnibus surveys from responses submitted March 5–8 (Ipsos) and March 11–15 (NORC); and HPS from responses submitted March 3–15. Overall, estimated COVID-19 vaccination coverage with at least one dose was 7.7 percentage points (95% confidence interval [CI]: 7.2–8.1) higher in HPS than in the vaccine administration data from March 3–15 (34.0% versus 26.3%, respectively). By state, differences in coverage between HPS and vaccine administration data ranged from 3.4 to 13.8 percentage points. State-level estimates of vaccination coverage from HPS were positively associated with estimates from the vaccine administration data (correlation coefficient=0.83 [95% CI: 0.72–0.90], p<0.05), with HPS estimates slightly higher than the vaccine administration data. Ratios of coverage by sex and by age group were similar in HPS and vaccine administration data.  Estimates of COVID-19 vaccination coverage from the NORC AmeriSpeak survey were 6.7 percentage points (95% CI: 3.1–10.2) higher compared to the vaccine administration data. Similar estimates of COVID-19 vaccination coverage were found between the Ipsos Knowledge Panel survey and the vaccine administration data (difference=0.7, 95% CI: -2.1–3.6).

Differences were found between estimates of COVID-19 vaccination coverage from HPS, omnibus surveys, and vaccine administration data. These results suggest that while HPS may overestimate COVID-19 vaccination coverage, estimates by sex, age, and state exhibited similar patterns as the vaccine administration data. Using national surveys to assess COVID-19 vaccination coverage may be a useful tool to supplement the vaccine administration data. First, HPS provides additional information about COVID-19 vaccination coverage by race/ethnicity, given that racial/ethnic status is missing for a large proportion of vaccine recipients in the vaccine administration data (40% as of May 15, 2021) (5).  In HPS, only 2.5% of race/ethnicity information is missing. Second, the vaccine administration data do not include information on occupation/industry, comorbidities, or intention to receive a COVID-19 vaccine/reasons for choosing not to receive a COVID-19 vaccine, which surveys can provide. In addition to estimates of vaccination coverage, information on knowledge, attitudes and beliefs toward COVID-19 vaccines provided by surveys can help federal, state, and local jurisdictions and public health partners develop messaging and strategies to increase confidence and vaccine uptake. As COVID-19 vaccines continue to be distributed and available to all Americans, monitoring vaccination coverage using the vaccine administration data and national surveys is important for addressing gaps and disparities in coverage, developing strategies and messages to prevent the spread of COVID-19, and bringing an end to the pandemic.

Methods

We assessed and compared the proportion of respondents who received at least one dose of COVID-19 vaccine from vaccine administration data, HPS data, and Omnibus data nationally overall and by age group and sex. For the vaccine administration data and HPS, we also assessed ≥1 dose COVID-19 vaccination receipt by state. We compared national and state estimates from HPS and Omnibus surveys to those from the vaccine administration data.

Vaccine administration data

COVID-19 vaccine administration data are reported to CDC by multiple sources, including IISs, VAMS, and by direct data submission from federal entities and pharmacies, including administration data from the Federal Long-Term Care Program (1,9). IISs are centralized data repositories for vaccination information specific to each jurisdiction. VAMS is a new, online tool used by public health jurisdictions, healthcare providers, and employers/organizations to manage vaccine administration from the time the vaccine arrives at a clinic until it is administered to a recipient. VAMS connects with IISs via data reporting systems to send vaccination data through the Immunization Data Lake, which is a cloud-hosted data repository to receive, store, manage, and analyze deidentified COVID-19 vaccination data.  Data for this analysis were retrieved from the Immunization Data Lake and included de-duplicated reports of COVID-19 vaccines administered from all sources, for all states except Texas, which reports only aggregate COVID-19 vaccination data. The data included vaccine administrations from March 3 to March 15 that were retrieved by the CDC on March 24, 2021 to allow for lag in reporting.  Providers are required to report administration records to the state IIS within 72 hours, and additional days of observation were included to account for delays in reporting and transmission of records to CDC.

Omnibus Surveys

Starting on December 18, 2020, CDC sponsored questions on two nationally representative household panel omnibus surveys (Ipsos KnowledgePanel and NORC AmeriSpeak),  each with a sample size of approximately 1,000 U.S. adult panelists aged ≥18 years, to assess COVID-19 vaccination coverage and intention to receive a COVID-19 vaccine (10,11). These surveys are fielded twice a month for approximately 3 to 10 days. Internet-enabled devices and Internet access were provided to KnowledgePanel members lacking necessary equipment to complete the survey. AmeriSpeak interviews were conducted by telephone if panel members did not have Internet access. Vaccination coverage was assessed between March 5–8 for Ipsos and between March 11–15 for NORC. The cooperation rates were 38.0% for Ipsos and 22.0% for NORC. Data were weighted to ensure representativeness of the U.S. population using demographic benchmarks from the 2020 Current Population Survey (for age, sex, race/ethnicity, Census region, residence in a Metropolitan Statistical Area [MSA], level of education, and household income) and the 2018 American Community Survey (for language proficiency [English or Spanish] among Hispanic respondents).

Household Pulse Survey

The Household Pulse Survey is a large, nationally representative online survey of over 60,000 participants conducted by the U.S. Census Bureau to examine how the coronavirus pandemic is impacting households across the country from a social and economic perspective. The survey asks questions about how education, employment, food security, health, housing, social security benefits, household spending, consumer spending associated with stimulus payments, and transportation have been affected by the ongoing crisis. CDC sponsored questions on receipt of and intention to receive a COVID-19 vaccination, which were added to the survey beginning in January 2021. This survey is fielded for approximately 13 days twice a month. Data for these analyses were collected during March 3–15, with a response rate of 7.4% (12). HPS utilizes the Census Bureau’s Master Address File (MAF) as the source to select a very large sample, one sufficient in size to accommodate anticipated lower response rates and still produce estimates at the state level and for 15 MSAs. To enable HPS’s use of a rapid deployment internet and telephone interview system, email and mobile telephone numbers from the Census Bureau Contact Frame were paired with addresses in the MAF, for which there were 80% matches. Respondents’ birth year, sex, race, and categorization of Hispanic ethnicity were used to align the survey estimates with population controls based on 2020 estimates. The number of people in the household, as well as the number of children (<18 years) were used to align the weighted estimates with the adult population distribution. The final weighted distribution was also aligned using educational status adjusted by age and sex to match estimates from the 2018 American Community Survey.

Survey Questions

The omnibus surveys and HPS included the same question to assess respondents’ COVID-19 vaccination status (“Have you received a COVID-19 vaccine?”).

Analysis

For comparison to HPS estimates, the vaccine administration data included the cumulative number of people who received at least one vaccine dose from the start of the vaccination program through each day from March 3 to March 15, 2021. COVID-19 vaccination coverage was estimated overall, by age group, sex, and state of residence using cumulative vaccination doses as the numerator and the estimate of the population in each respective group based on the weighted sample of the HPS during the same data collection period as the denominator. Vaccination coverage from the vaccine administration data was compared with the HPS and omnibus survey estimates. For each of the Omnibus surveys (Ipsos and NORC), estimates for vaccination receipt were analyzed by dividing the number of respondents who received ≥1 doses of a COVID-19 vaccine by the weighted sample size of each survey. For comparison to HPS, a weighted average of the cumulative daily coverage estimates from the March 3–15, 2021 vaccine administration data was used, with the weights derived from the weighted distribution of the dates participants submitted their answers for the HPS online questionnaire. This accounted for the non-uniform dates of HPS questionnaire submissions, with a peak during the first few days of survey launch followed by a second peak around the middle of the data collection period (Figure 1). These weighted averages of the coverage estimates from the vaccine administration data used the distributions of HPS questionnaire submissions by sex, by age group, and by state. Coverage estimates from the omnibus surveys were compared with coverage estimates from the vaccine administration data using the median date of submission from the omnibus surveys. The simple differences and their respective 95% CIs in coverage estimates between the vaccine administration data, the HPS, and the omnibus surveys were determined. For the comparison between the vaccine administration data and HPS, differences in coverage estimates were assessed overall, and by age group, sex, and state of residence. For the comparison with the omnibus survey estimates, differences were assessed overall, and by age group and sex. Estimates for Texas were removed from all analyses in all data sources because Texas does not report sex-specific and age-specific dose number information to CDC. All analyses were conducted using SAS callable SUDAAN (Cary, NC). All point estimates and 95% CIs presented are weighted. This activity was reviewed by CDC and was conducted consistently with applicable federal law and CDC policy.^*

Results

Comparison of vaccine administration data and HPS

Overall, COVID-19 vaccination coverage was 7.7 percentage points (95% CI: 7.2–8.1) higher in HPS compared to the vaccine administration data (34.0% compared to 26.3%) (Table 1). By age group and sex, the differences in coverage ranged from 6.7 to 9.0 percentage points. Both data sources indicated higher coverage for females than males, and higher coverage with increasing age group. By state, differences in coverage ranged from 3.4 percentage points (Oregon) to 13.8 percentage points (Missouri).

Based on correlation analysis of state level COVID-19 vaccination coverage among adults aged ≥18 years between estimates from HPS and administration data,  the correlation coefficient was 0.83 (95% CI: 0.72–0.90; p<0.05), indicating a strong positive association between these estimates, with HPS having slightly higher coverage than the administration data for each state (Figure 2).

Comparison of vaccine administration data and Omnibus Surveys

COVID-19 vaccination coverage estimates from the Ipsos KnowledgePanel omnibus survey were similar to estimates based on the vaccine administration data overall (difference 0.7 percentage points, 95% CI: -2.1–3.6), by sex (Male: difference=0.5, 95% CI: -3.2–4.2; Female: difference=1.6, 95% CI: -2.6–5.9), and by age group (18–49 years: difference=1.4, 95% CI: -1.8–4.7; 50–64 years: difference=-1.4, 95% CI: -6.1–3.3; ≥65 years: difference=1.4, 95% CI: -3.2–10.4) (Table 2). Vaccination coverage from the NORC AmeriSpeak omnibus survey was higher than estimates based on the vaccine administration data overall (difference=6.7 percentage points, 95% CI: 3.1–10.2), by sex (Male: difference=6.0, 95% CI: 0.5–11.5; Female: difference=8.1, 95% CI: 2.8–13.4), and by age group (18–49 years: difference=8.6, 95% CI: 3.7–13.5; 50–64 years: difference=2.0, 95% CI: -5.8–9.9; ≥65 years: difference=7.4, 95% CI: 1.3–13.5) (Table 3).

Limitations

These findings are subject to at least six limitations. Although we cannot determine the exact reasons for the higher estimates from HPS and NORC AmeriSpeak survey compared to those from the vaccine administration data, there are several possible explanations. For the survey data, COVID-19 vaccination status was based on self-report, which may not accurately reflect actual vaccination status for some respondents. The HPS is subject to bias from non-response and from exclusion of households with neither e-mail address nor cellular telephone number identified, and non-response bias. Although panel recruitment methodology and data weighting were designed to produce nationally representative results, the response rates were low across all three surveys and respondents may not be fully representative of the general U.S. adult population (12). There is evidence of nonresponse bias in the HPS estimates, some of which appears to be mitigated by weighting adjustments (13). Second, Texas was not included in the analyses because data on doses administered were not available by sex and age group. Residents of other states who received their vaccinations in Texas were also excluded from the vaccine administration data, though they could not be identified and excluded from the survey data. This could explain some of the larger differences seen between the Immunization Data Lake and HPS data in states such as Arkansas, Louisiana, and Oklahoma, which border Texas. Third, the vaccine administration data included persons vaccinated in long-term care facilities, while the survey data excluded the non-institutionalized population, so differences between the vaccine administration data and the survey data may be larger than presented in this report. Fourth, comparison by race/ethnicity was unavailable due to only two states having ≥90% complete data on race and ethnicity in the vaccine administration data. Fifth, there may be COVID-19 vaccinations occurring by March 15, 2021 that had not been reported by March 24, 2021, which would artificially lower the coverage estimates based on vaccine administration data, although this may be a very small number. Finally, these findings are for March data only, and any bias may have changed subsequently.

Discussion

In these results, coverage estimates from the HPS and NORC AmeriSpeak omnibus survey are higher than that seen in vaccine administration data by 7–8 percentage points. However, the HPS estimates showed similar patterns in relative coverage by sex and age group compared to the vaccine administration data, and state estimates showed high correlation between the vaccine administration data and the HPS. While the vaccine administration data continue to provide the most direct estimates of COVID-19 vaccination coverage, the survey data may provide useful additional information on relative vaccination coverage among subpopulations not assessed with the vaccine administration data and factors associated with vaccination. National survey data are useful in the assessment of COVID-19 vaccination coverage, which is important because these data include information on race/ethnicity, and other information such as socioeconomic characteristics and knowledge, attitudes, and behaviors that may be associated with COVID-19 vaccination. National survey data are thus complementary to administration data, providing complete or additional information to help answer questions on population characteristics and perceptions toward vaccination coverage, thus helping to monitor disparities in vaccine uptake and guide communication efforts to increase confidence in vaccines.

^* See e.g., 45 C.F.R. part 46, 21 C.F.R. part 56; 42 U.S.C. §241(d); 5 U.S.C. §552a; 44 U.S.C. §3501 et seq.

Disclaimer

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

References

1. COVID-19 Vaccine IT overview. Atlanta, GA: US Department of Health and Human Services, CDC; 2021. www.cdc.gov/vaccines/covid-19/reporting/overview/IT-systems.html
2. Hughes MM, Wang A, Grossman MK, et al. County-Level COVID-19 vaccination coverage and social vulnerability — United States, December 14, 2020–March 1, 2021. MMWR Morb Mortal Wkly Rep 2021;70:431–6. DOI: http://dx.doi.org/10.15585/mmwr.mm7012e1
3. Painter EM, Ussery EN, Patel A, et al. Demographic characteristics of persons vaccinated during the first month of the COVID-19 vaccination program — United States, December 14, 2020–January 14, 2021. MMWR Morb Mortal Wkly Rep 2021;70:174–7. DOI: http://dx.doi.org/10.15585/mmwr.mm7005e1
4. Kriss JL, Reynolds LE, Wang A, et al. COVID-19 vaccine second-dose completion and interval between first and second doses among vaccinated persons — United States, December 14, 2020−February 14, 2021. MMWR Morb Mortal Wkly Rep 2021;70:389–95. DOI: http://dx.doi.org/10.15585/mmwr.mm7011e2
5. CDC. COVID data tracker: COVID-19 vaccinations in the United States. Atlanta, GA: US Department of Health and Human Services, CDC; 2021. https://covid.cdc.gov/covid-data-tracker/#vaccinations

6. CDC. Building confidence in COVID-19 vaccines. Atlanta, GA: US Department of Health and Human Services, CDC; 2021. www.cdc.gov/vaccines/covid-19/vaccinate-with-confidence.html
7. Nguyen KH, Srivastav A, Razzaghi H, et al. COVID-19 vaccination intent, perceptions, and reasons for not vaccinating among groups prioritized for early vaccination — United States, September and December 2020. MMWR Morb Mortal Wkly Rep 2021;70:217–22. DOI: http://dx.doi.org/10.15585/mmwr.mm7006e3
8. United States Census Bureau. Measuring household experiences during the coronavirus pandemic. Suitland, MD: United States Census Bureau; 2021. https://www.census.gov/data/experimental-data-products/household-pulse-survey.html
9. CDC COVID-19 vaccination program provider requirements and support. Atlanta, GA: US Department of Health and Human Services, CDC; 2021. www.cdc.gov/vaccines/covid-19/vaccination-provider-support.html
10. Ipsos. Knowledge Panel Omnibus services. New York, NY: Ipsos; 2021. https://www.ipsos.com/en-us/solutions/public-affairs/knowledgepanel-omnibus
11. Amerispeak. Amerispeak Omnibus. Chicago, IL: AmeriSpeak, NORC; 2021. https://amerispeak.norc.org/us/en/amerispeak/our-capabilities/amerispeak-omnibus.html
12. United States Census Bureau. Source of the data and accuracy of the estimates for the Household Pulse Survey – Phase 3. Suitland, MD: United States Census Bureau; 2021. https://www2.census.gov/programs-surveys/demo/technical-documentation/hhp/Phase3_Source_and_Accuracy_Week_24.pdf
13. United States Census Bureau. Nonresponse Bias Report for the 2020 Household Pulse Survey. Suitland, MD: United States Census Bureau; 2021. https://www2.census.gov/programs-surveys/demo/technical-documentation/hhp/2020_HPS_NR_Bias_Report-final.pdf

Figure 1. Proportion of HPS questionnaire submissions by date, Household Pulse Survey, March 3–March 15, 2021

Table 1. Comparison of COVID-19 vaccination coverage from the Household Pulse Survey (HPS) and vaccine administration data (IIS), overall, and by sex, age group, and state, United States, March 3–15, 2021

Abbreviations: CI=confidence interval, HPS=Household Pulse Survey, IIS=immunization information system.

* Based on the COVID-19 vaccine administration data (1), with vaccination coverage estimates using same denominator as the HPS estimates and using the weighted average of the cumulative daily vaccination coverage from March 3–15, 2021, with weights derived from the HPS weighted distribution of the day online questionnaires were submitted during the HPS data collection period of March 3–15, 2021.

Figure 2. Comparison of COVID-19 vaccination coverage estimates* from the Household Pulse Survey (HPS) and vaccine administration data (IIS)^† by state, United States, March 3–15, 2021

Abbreviations: HPS=Household Pulse Survey, IIS=immunization information system.

*Correlation coefficient=0.83 (95% CI: 0.72–0.90, p<0.05)

^†Based on the COVID-19 vaccine administration data (1), with vaccination coverage estimates using same denominator as the HPS estimates and using the weighted average of the cumulative daily vaccination coverage from March 3–15, 2021, with weights derived from the HPS weighted distribution of the day online questionnaires were submitted during the HPS data collection period of March 3–15, 2021.

Table 2. Comparison of COVID-19 vaccination coverage between Ipsos KnowledgePanel and vaccine administration data (IIS) nationally, and by sex and age group, United States, March 5–8, 2021

Abbreviations: CI=confidence interval, IIS=immunization information system.

^* Based on the COVID-19 vaccine administration data (1), with vaccination coverage estimates using same denominator as the HPS estimates and using the median vaccination coverage from March 5–8, 2021.

Table 3. Comparison of COVID-19 vaccination coverage between NORC AmeriSpeak and vaccine administration data (IIS), overall, by sex and age group, United States, March 11–15, 2021

Abbreviations: CI=confidence interval, IIS=immunization information system.

^* Based on the COVID-19 vaccine administration data (1), with vaccination coverage estimates using same denominator as the HPS estimates and using the median vaccination coverage from March 11-15, 2021.