How Flu Vaccine Effectiveness and Efficacy are Measured
Questions & Answers
- How do we measure how well flu vaccines work?
- How do vaccine effectiveness studies differ from vaccine efficacy studies?
- What factors can affect the results of flu vaccine efficacy and effectiveness studies?
- Why are there so many different outcomes for vaccine effectiveness studies?
- Can you describe potential biases that should be considered in observational studies measuring vaccine effectiveness?
- What are vaccine effectiveness point estimates and confidence intervals?
- Why are confidence intervals important for understanding flu vaccine effectiveness?
- How does CDC monitor vaccine effectiveness?
How do we measure how well flu vaccines work?
Two main types of studies are used to determine how well flu vaccines work: randomized controlled trials and observational studies. These study designs are described below.
Randomized controlled trials (RCTs)
A randomized controlled trial (or “RCT”) compares outcomes in two groups of volunteers who are randomly assigned to get either a vaccine or a placebo (a placebo looks like vaccine but does not contain any vaccine—often, a shot of saline solution is used). RCTs measure vaccine efficacy. Vaccine efficacy in this case refers to the percent reduction in how often flu illness occurs among vaccinated people compared to people given placebo (i.e., unvaccinated people). RCTs are usually conducted under ideal conditions where vaccine storage and delivery are monitored, and participants are usually in good health or selected for a specific health status. The RCT study design minimizes bias that could lead to invalid study results. Bias is an unintended systematic error in a study, which can include the way researchers select study participants, measure outcomes, or analyze data that can lead to inaccurate results. An RCT, is usually conducted using double-blinded methods, which means neither the study volunteers nor the researchers know whether study participants have been given vaccine or placebo. National regulatory authorities, such as the Food and Drug Administration (FDA) in the United States, require RCTs to be conducted and to show the protective benefits of a new vaccine before the vaccine is licensed for regular use.
There are several types of observational studies, including cohort and case-control studies. Observational studies measure flu vaccine effectiveness. Vaccine effectiveness is a measure of how well flu vaccines work among different groups of people, in different settings, and in different real-world conditions (as opposed to RCTs or “clinical trials”). Flu vaccine effectiveness is measured by comparing how often people in the vaccinated and unvaccinated groups get flu.
How do vaccine effectiveness studies differ from vaccine efficacy studies?
Vaccine efficacy, defined above, is determined in RCTs, usually clinical trials. Vaccine effectiveness measures how well a vaccine works in real-world conditions. Differences in real-world conditions compared to the controlled conditions in clinical trials can influence how well a vaccine works. Vaccine effectiveness studies can:
- Be used to determine if people at increased risk of severe flu illness (often excluded from clinical trials/RCTs) respond differently to vaccination.
- Determine if different flu viruses that are circulating and evolving in real-world conditions affect vaccine performance.
- Be conducted to account for how potential differences in vaccine dosing schedules or storage and handling may affect vaccine performance since vaccine dosing schedules or vaccine storage and handling requirements may not be followed as closely in the real world as in clinical trials.
Results from vaccine effectiveness studies are potentially subject to biases that are much less likely to occur in vaccine efficacy studies, like selection bias and confounding, which is why licensure of vaccines depends upon data collected in RCTs.
Once a flu vaccine has been licensed by FDA, recommendations for its “routine” (regular) use are typically made by CDC’s Advisory Committee for Immunization Practices (ACIP). For example, ACIP recommends annual flu vaccination for all U.S. residents 6 months and older, with rare exception. So-called “universal vaccine recommendations” introduce ethical challenges in performing RCTs that assign people to a placebo group, which could place them at elevated risk for serious complications from flu. Also, observational studies often are the only option to measure vaccine effectiveness against more severe, less common flu outcomes, such as hospitalization.
What factors can affect the results of flu vaccine efficacy and effectiveness studies?
The measurement of flu vaccine efficacy and effectiveness can be affected by virus and host factors as well as the type of study used. Therefore, vaccine efficacy/effectiveness point estimates have varied among published studies.
Virus factors refer to the similarity between the vaccine virus and circulating viruses. (More information is available at How the flu virus can change: Drift and Shift.) Protection from vaccination can be lower when circulating viruses are very different from vaccine viruses. However, in recent seasons when this has happened, vaccine has offered about 30% or less reduction in risk [1-6]. When circulating flu viruses are mildly or moderately drifted in comparison to the vaccine, people may still receive some protective benefit from vaccination; and if other circulating flu viruses are well matched, the vaccine could still provide protective benefits overall.
Host factors refer to characteristics of the person being vaccinated, including, for example, their age, underlying medical conditions, history of prior flu illness, and prior flu vaccinations. All of these factors can affect how well vaccines work.
Study Design Factors
RCTs provide the most reliable results because they are less susceptible to biases, including selection bias and confounding. However, as stated above, RCTs may be difficult to conduct when vaccination is recommended in a population or for more severe outcomes that are less common, given the large numbers of people that would have to be randomized. There are several observational study designs; however, many flu vaccine evaluation programs currently use the test-negative design. In the test-negative design, people who seek care for an acute respiratory illness are enrolled at care settings (such as outpatient clinics, urgent care clinics, emergency departments, or in-patient settings), and information is collected about the patients’ flu vaccination status. All participants in a test-negative design study are tested for flu using a highly specific and sensitive test for flu virus infection, such as reverse transcription polymerase chain reaction (RT-PCR). The ratio of vaccinated to unvaccinated persons (i.e., the odds of flu vaccination) is compared among patients with and without laboratory-confirmed flu. In this way, a test-negative design study estimates VE by comparing vaccination rates among persons with confirmed flu illness (also called “cases”) versus persons with similar illness who do not have flu (also called “controls”) based on laboratory tests. The test-negative design reduces selection bias due to health care seeking behaviors. Other observational study designs have also been used to estimate flu vaccine effectiveness.
Factors Related to Measuring Specific versus Non-Specific Outcomes
For both RCTs and observational studies, the specificity of the outcome measured in the study is important. Non-specific outcomes, such as pneumonia hospitalizations or influenza-like illness (ILI) can be caused by flu virus infections or infections with other viruses and bacteria. Vaccine efficacy/effectiveness estimates for non-specific outcomes are generally lower than estimates made for more specific outcomes, depending on what proportion of the outcome measured is attributable to flu. For example, a study among healthy adults found that inactivated flu vaccine (i.e., a flu shot) was 86% effective against laboratory-confirmed flu, but only 10% effective against all respiratory illnesses in the same population and season . Laboratory-confirmed flu virus infections, by RT-PCR or viral culture, are generally the most specific outcomes for vaccine efficacy/effectiveness studies.
Why are there so many different outcomes for vaccine effectiveness studies?
Vaccine effectiveness studies that measure different outcomes are conducted to better understand the different kinds of benefits provided by vaccination. Ideally, public health researchers want to evaluate the benefits of vaccination against illness of varying severity. To do this, they assess how well flu vaccines work to prevent illness resulting in a doctor visit, or illness resulting in hospitalization, ICU admission, and even death associated with flu. Because estimates of vaccine effectiveness may vary based on the outcome measured (in addition to season, population studied, and other factors), results should be compared between studies that used the same outcome for estimating vaccine effectiveness.
Can you describe potential biases that should be considered in observational studies measuring vaccine effectiveness?
Results from observational studies are more likely to be affected by various forms of bias (see above for definition) than are results from RCT studies. Therefore, results from observational studies can be more difficult to interpret. Bias can be reduced through careful study designs and analyses of data collected. Observational studies of flu vaccine effectiveness are subject to at least three forms of bias: confounding, selection bias, and information bias.
Confounding is when the effect of vaccination on the risk of the outcome being measured (e.g., flu-related hospitalizations confirmed by testing) is distorted by another factor associated both with vaccination (the exposure) and the outcome. In RCTs, factors associated with exposure and outcomes can be evenly distributed between vaccinated and unvaccinated groups. This is not always true in observational studies. For example, chronic medical conditions can confound the association between flu vaccination and hospitalization with flu in observational studies. Chronic medical conditions increase the risk of flu-related hospitalization and vaccination often is more common among people with chronic medical conditions. Therefore, the presence of a chronic medical condition in a study participant is a potential confounding factor that should be considered in analysis. This is an example of confounding by indication because those at greatest risk for the outcome being measured (i.e., flu-related hospitalization) are targeted for vaccination, and therefore, they are more likely than those without a chronic medical condition to receive a flu vaccine. Not adjusting for confounders can bias the vaccine effectiveness estimate higher or lower than the true estimate. In the example given, the vaccine effectiveness estimate could be biased lower, or towards lower effectiveness.
Selection bias occurs when people with the outcome being measured by the study (e.g., flu virus infection) differ from people who do not have the outcome. In observational studies of flu vaccine effectiveness, people with and without flu may have different likelihoods of being vaccinated, and this can bias the estimate of vaccine effectiveness. For example, people who visit their health care provider in outpatient settings (e.g., clinics and urgent care) may be more likely to be vaccinated than people who do not go to a provider for care as often. If controls are selected from a different population than the cases (e.g., cases are from a health clinic and controls from a community sample) with different health care seeking behaviors, selection bias related to health care seeking (and the likelihood to be vaccinated) may be introduced. The test-negative study design minimizes selection bias related to health care seeking by enrolling patients who seek care for a respiratory illness. This study design is used by many researchers globally, including CDC-funded networks that measure vaccine effectiveness.
Information bias occurs if exposures or outcomes are based on different sources of information for people with and without the disease of interest. For example, if researchers obtain information on vaccination for children with flu from vaccination records but ask parents of children without flu if the child was vaccinated, this difference in data collection procedures could bias the results of the study.
What are vaccine effectiveness point estimates and confidence intervals?
CDC typically presents flu vaccine effectiveness (VE) as a single point estimate: for example, 60%. This point estimate represents the reduction in risk provided by a flu vaccine. CDC vaccine effectiveness studies measure different outcomes. For example, outcomes measured can include laboratory-confirmed flu illness (that results in a doctor’s visit), hospitalizations or intensive care unit (ICU) admissions. For these outcomes, a VE point estimate of 60% means that on average the flu vaccine reduces a person’s risk of that flu outcome by 60%.
In addition to the VE point estimate, CDC also provides a “confidence interval” (CI) for this point estimate, for example, 60% (95% CI: 50%-70%). The confidence interval provides a lower boundary for the VE estimate (e.g., 50%) as well as an upper boundary (e.g., 70%). One way to interpret a 95% confidence interval is that if CDC were to repeat this study 100 times, 95 times out of 100, the confidence interval would contain the true VE value. Another way to look at this is that there is a 95% chance that the true VE lies within the range described by the confidence interval. This means there is still the possibility that five times out of 100 (a 5% chance) the true VE value could fall outside of the 95% confidence interval.
Why are confidence intervals important for understanding flu vaccine effectiveness?
Confidence intervals are important because they provide context for understanding the precision or exactness of a VE point estimate. The wider the confidence interval, the less exact the point value estimate of vaccine effectiveness becomes. Take, for example, a VE point estimate of 60%. If the confidence interval of this point estimate is 50%-70%, then we can have greater certainty that the true protective effect of the flu vaccine is near 60% than if the confidence interval were between 10% and 90%. Furthermore, if a confidence interval includes zero, for example, (-20% to 60%), then the point value estimate of VE provided is considered “not statistically significant.” People should be cautious when interpreting VE estimates that are not statistically significant because such results cannot rule out the possibility of zero VE (i.e., no protective benefit). The width of a confidence interval is related in part to the number of participants in the study, and so studies that provide more precise estimates of VE (and consequently, have a narrower confidence interval) typically include a larger number of participants. When vaccine effectiveness is low, a larger sample size is needed to detect a statistically significant estimate.
How does CDC monitor vaccine effectiveness?
CDC has been working with researchers at universities and hospitals since the 2003-2004 flu season to estimate how well flu vaccines work through observational studies using laboratory-confirmed flu as the outcome. CDC’s studies are conducted in sites located across the United States and among different age groups to gather more representative data.
Over the past few years, CDC has conducted VE studies using multiple vaccine effectiveness networks. More information on CDC’s vaccine effectiveness networks and studies is available at CDC’s Influenza Vaccine Effectiveness Networks.
- Price AM, Flannery B, Talbot HK, et al. Influenza vaccine effectiveness against influenza a(h3n2)-related illness in the United States during the 2021–2022 influenza season. Clinical Infectious Diseases. 2022. doi:10.1093/cid/ciac941
- Tenforde MW, Kondor RJG, Chung JR, et al. Effect of Antigenic Drift on Influenza Vaccine Effectiveness in the United States-2019-2020. Clin Infect Dis. 2021;73(11):e4244-e4250. doi:10.1093/cid/ciaa1884
- Tenforde MW, Talbot HK, Trabue CH, et al. Influenza Vaccine Effectiveness Against Hospitalization in the United States, 2019-2020. J Infect Dis. 2021;224(5):813-820. doi:10.1093/infdis/jiaa800
- Campbell AP, Ogokeh C, Weinberg GA, et al. Effect of Vaccination on Preventing Influenza-Associated Hospitalizations Among Children During a Severe Season Associated With B/Victoria Viruses, 2019-2020. Clin Infect Dis. 2021;73(4):e947-e954. doi:10.1093/cid/ciab060
- Flannery B, Kondor RJG, Chung JR, et al. Spread of Antigenically Drifted Influenza A(H3N2) Viruses and Vaccine Effectiveness in the United States During the 2018-2019 Season. J Infect Dis. 2020;221(1):8-15. doi:10.1093/infdis/jiz543
- Kim SS, Naioti EA, Halasa NB, et al. Vaccine effectiveness against influenza hospitalization and emergency department visits in 2 A(H3N2) dominant influenza seasons among children <18 years old-new vaccine surveillance network 2016-2017 and 2017-2018. J Infect Dis. 2022;226(1):91-96. doi:10.1093/infdis/jiab624
- Bridges CB, Thompson WW, Meltzer MI, Reeve GR, Talamonti WJ, Cox NJ, Lilac HA, Hall H, Klimov A, Fukuda K. Effectiveness and cost-benefit of influenza vaccination of healthy working adults: A randomized controlled trial. JAMA. 2000;284(13):1655-63