Population-Level Risk-Benefit Analysis

Print
Related Pages

After use of the Janssen (Johnson & Johnson) COVID-19 vaccine was paused in the U.S., a population-level risk-benefit assessment was conducted to quantify the age-specific risks and benefits of resuming vaccination with this product, including the risk of thrombosis with thrombocytopenia syndrome (TTS), and the benefits of preventing COVID-19-related hospitalizations, intensive care unit (ICU) admissions, and deaths.

The objectives of the population-level risk-benefit analysis were to:

  • Quantify COVID-19 associated hospitalizations, ICU admissions, and deaths averted under different assumptions about the future use of the Janssen COVID-19 Vaccine use.
  • Quantify population level, age-specific benefits and harms of resuming vaccination with the Janssen COVID-19 vaccine.

A compartmental model previously used by the Advisory Committee on Immunization Practices (ACIP) was used for this analysis. This model is stratified by age, essential worker status, and underlying medical condition status and was calibrated to past COVID-19 incidence data through spring 2021 (incidence of COVID-19 cases and hospitalizations available here: https://covid.cdc.gov/covid-data-tracker/#new-hospital-admissions and https://covid.cdc.gov/covid-data-tracker/#trends_dailytrendscases). Future activity was based on the ensemble projection from Round 4 of the multiple model COVID Scenario Modeling Hub (https://covid19scenariomodelinghub.org/index.html and https://github.com/midas-network/covid19-scenario-modeling-hub ) The “Low SARS-CoV-2 transmission” was calibrated to the Modeling Hub Round 4 Scenario with high vaccination and moderate future compliance to non-pharmaceutical interventions and “Moderate SARS-CoV-2 transmission” was calibrated to the Modeling Hub Round 4 scenario with high vaccination and low future compliance to non-pharmaceutical interventions.

Two types of vaccines were modeled (an mRNA COVID-19 vaccine and the Janssen COVID-19 vaccine); vaccine efficacy estimates were informed by clinical trials1-3 and efficacy was assumed to develop 14 days after administration of each dose (Table 1).

Table 1
Vaccine Vaccine efficacy against infection Vaccine efficacy against hospitalization or death
mRNA (First Dose) 70% 75%
mRNA (Both Doses) 85% 95%
J&J/Janssen 65% 90%

Additional assumptions in the model included no loss to follow-up with mRNA second doses, no delays in administration of the second dose in the vaccination series, and no waning of immunity from natural infection or vaccination over the 6-month time horizon. Age-group-specific COVID-19 vaccination coverage (https://covid.cdc.gov/covid-data-tracker/#vaccination-demographics-trends ) and intention data through April 2021 (https://www.census.gov/data/tables/2021/demo/hhp/hhp27.html)  were used to project future vaccination coverage. Administration rates for mRNA and Janssen vaccines were based on data from April 6–April 12, 2021 (https://covid.cdc.gov/covid-data-tracker/#vaccination-trends ). ICU admissions were estimated by multiplying the number of hospitalizations estimated by the model by the estimated proportion of hospitalized patients who were admitted to the ICU from the COVID-19 pandemic planning scenarios, available here: www.cdc.gov/coronavirus/2019-ncov/hcp/planning-scenarios.html. Benefit calculations were evaluated over a 6-month time frame.

Expected numbers of TTS cases were based on confirmed TTS cases, as of April 23rd (Table 2).

Table 2. Confirmed TTS cases, Janssen COVID-19 doses administered, and reporting rate, by age and sex
Females Males
Age group Cases Doses admin Reporting rate Cases Doses admin Reporting rate
18–49 years old 13 1,866,294 7.0 per million 0 1,977,330 0 per million
50+ years old 2 2,125,239 0.9 per million 0 2,010,144 0 per million

Five different vaccination scenarios were evaluated, with all scenarios assuming the continued use of mRNA vaccines:

  • Janssen COVID-19 vaccination not resumed; mRNA vaccine only
  • Janssen COVID-19 vaccination resumed on April 24 for all adults aged ≥18 years, at 50% of the pre-pause administration rate
  • Janssen COVID-19 vaccination resumed on April 24 for all adults aged ≥18 years, at 100% of the pre-pause administration rate
  • Janssen COVID-19 vaccination resumed on April 24 for all adults aged ≥50 years, at 50% of the pre-pause administration rate
  • Janssen COVID-19 vaccination resumed on April 24 for all adults aged ≥50 years, at 100% of the pre-pause administration rate

Table 3 details the results from the population-level benefits and harms scenarios, 6 months post-Janssen COVID-19 vaccine pause, by resumption strategy.  With this approach to assessing risks and benefits, the benefits of vaccination apply to the entire U.S. adult population over a 6-month period, and result from both direct and indirect effects. Indirect effects are also known as ‘herd protection’ and are based on the assumption that the vaccine can block transmission to unvaccinated persons as well. However, the risks only pertain to vaccinated persons.

TABLE 3. Population-level estimated number of and percent decrease in COVID-19–related hospitalizations, intensive care unit (ICU) admissions, and deaths after resuming use of Janssen COVID-19 vaccine for 6 months* and number of expected cases of thrombosis with thrombocytopenia syndrome (TTS), by age group and SARS-CoV-2 transmission level† — United States, 2021
Benefits and harms from resuming vaccination Resumption strategy
Recommended for adults aged ≥18 yrs Recommended for adults aged ≥50 yrs
Low SARS-CoV-2 transmission Moderate SARS-CoV-2 transmission Low SARS-CoV-2 transmission Moderate SARS-CoV-2 transmission
50% of prepause administration rate
No. of persons expected to receive the Janssen COVID-19 vaccine 9.8 million 3.6 million
Benefits, no. (% decrease§)
Hospitalizations prevented 3,926 (1.4) 9,395 (1.6) 1,361 (0.5) 3,532 (0.6)
ICU admissions prevented 928 (1.4) 2,236 (1.5) 295 (0.4) 799 (0.5)
Deaths prevented 586 (1.6) 1,435 (1.8) 54 (0.1) 257 (0.3)
Harms
No. of TTS cases expected 26 26 2 2
100% of prepause administration rate
No. of persons expected to receive the Janssen COVID-19 vaccine 17.5 Million 5.9 Million
Benefits, no. (% decrease§)
Hospitalizations prevented 5,987 (2.2%) 14,771 (2.5%) 1,947 (0.7%) 4,522 (0.7%)
ICU admissions prevented 1,418 (2.1%) 3,517 (2.4%) 396 (0.6%) 953 (0.6%)
Deaths prevented 882 (2.4%) 2,226 (2.8%) 43 (0.1%) 208 (0.3%)
Harms
No. of TTS cases expected 45 45 3 3
Abbreviations: ICU = intensive care unit; TTS = thrombosis with thrombocytopenia syndrome.
* Resumption of vaccination after a 10-day pause that commenced on April 13, 2021 https://emergency.cdc.gov/han/2021/han00442.asp
† This model evaluated the direct and indirect effects of resuming 50% or 100% of Janssen COVID-19 administration rates (compared with rate before use was paused) among all adults aged ≥18 years or only among adults ≥50 years compared with not resuming vaccination. The model was also calibrated to both low and moderate COVID-19 transmission levels based on varying assumptions about nonpharmaceutical interventions during the modeled time period.
§ Compared with no resumption of Janssen vaccination.

Individual-Level Risk-Benefit Analysis

An individual-level risk-benefit assessment was conducted to quantify age and sex-specific benefits and harms, per million Janssen vaccine doses.

The objectives of the individual-level risk-benefit analysis were to:

  • Quantify COVID-19-associated hospitalizations, ICU admissions, and deaths averted among vaccinated persons per million doses of the Janssen COVID-19 vaccine used, by age group and sex, and
  • Compare these benefits to the expected numbers of TTS cases in the same population groups.

Calculations were based on:

  • Recent age-specific incidence of hospitalizations for COVID-19 from COVID-NET
    • Week ending March 27, 2021
    • Age groups 18–29, 30–49 (averaged from 30–39 and 40–49), 50–64, ≥65 years
    • Overall hospitalization rates, (weekly incidence of 7.9 per 100,000 population), one location with higher weekly incidence (17.8 per 100,000), and one location with lower weekly incidence (2.6 per 100,000)
  • Proportions of hospitalized COVID-19 patients admitted to ICU or who died
  • Number of persons already vaccinated as of April 21, 2021
  • Expected TTS cases, based on estimated rates of TTS as of April 24, 2021, Table 1. One case of TTS in a male clinical trial participant in the 18–49 year age group was included4. The denominator was not adjusted for doses received during Phase III trial because they were not available in the specified age groups; however, the numbers are small relative to the numbers administered post authorization.
  • 90% efficacy of Janssen COVID-19 vaccine to prevent hospitalizations from COVID-193.
  • Additional simplifying assumptions: All unvaccinated persons have equal age-specific risk of hospitalization due to COVID-19 and all incident cases were considered to be in unvaccinated persons.

Details on COVID-NET methods are available here. COVID-NET is a population-based surveillance system that collects data on laboratory-confirmed COVID-19-associated hospitalizations among children and adults through a network of over 250 acute-care hospitals in 14 states. Hospitalization rates are calculated by the number of residents of a defined area who are hospitalized with a positive SARS-CoV-2 laboratory test divided by the total population within that defined area. Calculations were based on data available through April 20, 2021.

The calculations assume a 30-day period at risk as a result of delay in receiving an alternative COVID-19 vaccine, if the Janssen COVID-19 vaccine was not available, compared to not receiving any COVID-19 vaccine for this period of time. Analyses were also conducted to evaluate 60-day and 120-day delays in vaccination, and lower and higher incidence.

The individual-level risk-benefit analysis could be adapted to different incidence situations and with different assumptions about the lag in vaccination if the Janssen COVID-19 vaccine is not available. Table 4 details the results from the individual-level benefits and harms scenarios, 30, 60, and 120 days post-Janssen COVID-19 vaccine pause, by age and sex.  With this approach to assessing risks and benefits, the benefits and risks of vaccination apply during the time period where there may be a delay in receipt of a COVID-19 vaccine, if a Janssen COVID-19 vaccine is not available. In addition, the risks and benefits both apply directly to the vaccinated person.

TABLE 4. Individual-level estimated number of COVID-19–related hospitalizations, ICU admissions, and deaths prevented after resuming use of Janssen COVID-19 vaccine* and number of expected cases of thrombosis with thrombocytopenia syndrome per 1 million vaccine doses, by sex and age group, with varying incidence and timing scenarios† — United States, 2021
Benefits and harms from resuming vaccination No. per million vaccine doses administered§
Females Males
18–49 yrs ≥50 yrs 18–49 yrs¶ ≥50 yrs
Harms
TTS cases expected 7 1 1 0
Benefits, 30d, overall incidence‖
Hospitalizations prevented 297 2,454 272 2,821
ICU admissions prevented 56 661 51 760
Deaths prevented 6 394 6 471
Benefits, 30d, low U.S. state incidence‖
Hospitalizations prevented 106 698 97 786
ICU admissions prevented 20 188 18 212
Deaths prevented 2 107 2 127
Benefits, 30d, high U.S. state incidence‖
Hospitalizations prevented 580 5766 530 6644
ICU admissions prevented 110 1554 100 1790
Deaths prevented 12 929 11 1113
Benefits, 60d, overall incidence‖
Hospitalizations prevented 594 4908 544 5642
ICU admissions prevented 112 1323 103 1520
Deaths prevented 12 787 11 942
Benefits, 120d, overall incidence‖
Hospitalizations prevented 1189 9816 1087 11285
ICU admissions prevented 225 2645 206 3040
Deaths prevented 25 1574 23 1884
Abbreviations: ICU = intensive care unit; TTS = thrombosis with thrombocytopenia syndrome.
* Resumption of vaccination after a 10-day pause that began on April 13, 2021. https://emergency.cdc.gov/han/2021/han00442.asp
† This analysis evaluated direct benefits and harms, per 1 million Janssen COVID-19 vaccine doses, over 30, 60, or 120 days.
§ Compared with no resumption of Janssen vaccination.
¶ Analyses incorporated one TTS case that occurred in the Phase III trial in a man in the 18–49 years age group.
‖ Incidence of hospitalization for week ending March 27, 2021, obtained from https://gis.cdc.gov/grasp/COVIDNet/COVID19_3.html

Summary

The population- and individual-level approaches to risk-benefit analyses offer complementary information on the risks and benefits of resuming vaccination, and in which age groups.

The population approach takes into account direct and indirect (herd) effects of vaccination, incorporates availability of different vaccines, simulates hospitalizations, ICU admissions and deaths under different transmission scenarios, and makes estimations of all outcomes over a 6-month time horizon. This approach shows a large population benefit of Janssen vaccination relative to rare occurrence of TTS.

The direct individual level approach estimates benefits and risks to vaccinated individuals, and only considers getting Janssen vaccine compared to not getting any vaccine over a short, 30-day time horizon. The analysis showed a positive balance of benefits and risks for all age and sex groups. However, the relative balance of risks and benefits for individuals varies by age and sex because cases of TTS were primarily identified among women aged 18–49 years. Benefits are higher when incidence (i.e., risk of exposure) is higher and when the assumed delay in vaccination is greater.

References:

  1. Oliver S, Gargano J, Marin M, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for Use of Pfizer-BioNTech COVID-19 Vaccine — United States, December 2020. MMWR Morb Mortal Wkly Rep 2020;69:1922-1924. DOI: http://dx.doi.org/10.15585/mmwr.mm6950e2
  2. Oliver S, Gargano J, Marin M, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for Use of Moderna COVID-19 Vaccine — United States, December 2020. MMWR Morb Mortal Wkly Rep 2021;69:1653-1656. DOI: http://dx.doi.org/10.15585/mmwr.mm695152e1
  3. Oliver SE, Gargano JW, Scobie H, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for Use of Janssen COVID-19 Vaccine — United States, February 2021. MMWR Morb Mortal Wkly Rep 2021;70:329–332. DOI: http://dx.doi.org/10.15585/mmwr.mm7009e4
  4. Vaccines and Related Biological Products Advisory Committee. Vaccines and Related Biological Products Advisory Committee February 26, 2021, meeting: FDA briefing document. Silver Spring, MD: U.S. Department of Health and Human Services, Food and Drug Administration; 2021. https://www.fda.gov/media/146217/download
Top of Page