Population-Level Risk-Benefit Analysis
After use of the Janssen (Johnson & Johnson) COVID-19 vaccine was paused in the US, a population-level risk-benefit assessment was conducted to quantify the age-specific risks and benefits of resuming vaccination with this product, including the risk of thrombosis with thrombocytopenia syndrome (TTS), and the benefits of preventing COVID-19-related hospitalizations, intensive care unit (ICU) admissions, and deaths.
The objectives of the population-level risk-benefit analysis were to:
- Quantify COVID-19 associated hospitalizations, ICU admissions, and deaths averted under different assumptions about the future use of the Janssen COVID-19 Vaccine use.
- Quantify population level, age-specific benefits and harms of resuming vaccination with the Janssen COVID-19 vaccine.
A compartmental model previously used by the Advisory Committee on Immunization Practices (ACIP) was used for this analysis. This model is stratified by age, essential worker status, and underlying medical condition status and was calibrated to past COVID-19 incidence data through spring 2021 (incidence of COVID-19 cases and hospitalizations available here: https://covid.cdc.gov/covid-data-tracker/#new-hospital-admissions and https://covid.cdc.gov/covid-data-tracker/#trends_dailytrendscases). Future activity was based on the ensemble projection from Round 4 of the multiple model COVID Scenario Modeling Hub (https://covid19scenariomodelinghub.org/index.htmlexternal icon and https://github.com/midas-network/covid19-scenario-modeling-hubexternal icon ) The “Low SARS-CoV-2 transmission” was calibrated to the Modeling Hub Round 4 Scenario with high vaccination and moderate future compliance to non-pharmaceutical interventions and “Moderate SARS-CoV-2 transmission” was calibrated to the Modeling Hub Round 4 scenario with high vaccination and low future compliance to non-pharmaceutical interventions.
Two types of vaccines were modeled (an mRNA COVID-19 vaccine and the Janssen COVID-19 vaccine); vaccine efficacy estimates were informed by clinical trials1-3 and efficacy was assumed to develop 14 days after administration of each dose (Table 1).
|Vaccine||Vaccine efficacy against infection||Vaccine efficacy against hospitalization or death|
|mRNA (First Dose)||70%||75%|
|mRNA (Both Doses)||85%||95%|
Additional assumptions in the model included no loss to follow-up with mRNA second doses, no delays in administration of the second dose in the vaccination series, and no waning of immunity from natural infection or vaccination over the 6-month time horizon. Age-group-specific COVID-19 vaccination coverage (https://covid.cdc.gov/covid-data-tracker/#vaccination-demographics-trends ) and intention data through April 2021 (https://www.census.gov/data/tables/2021/demo/hhp/hhp27.htmlexternal icon) were used to project future vaccination coverage. Administration rates for mRNA and Janssen vaccines were based on data from April 6–April 12, 2021 (https://covid.cdc.gov/covid-data-tracker/#vaccination-trends ). ICU admissions were estimated by multiplying the number of hospitalizations estimated by the model by the estimated proportion of hospitalized patients who were admitted to the ICU from the COVID-19 pandemic planning scenarios, available here: https://www.cdc.gov/coronavirus/2019-ncov/hcp/planning-scenarios.html. Benefit calculations were evaluated over a 6-month time frame.
Expected numbers of TTS cases were based on confirmed TTS cases, as of April 23rd (Table 2).
|Age group||Cases||Doses admin||Reporting rate||Cases||Doses admin||Reporting rate|
|18–49 years old||13||1,866,294||7.0 per million||0||1,977,330||0 per million|
|50+ years old||2||2,125,239||0.9 per million||0||2,010,144||0 per million|
Five different vaccination scenarios were evaluated, with all scenarios assuming the continued use of mRNA vaccines:
- Janssen COVID-19 vaccination not resumed; mRNA vaccine only
- Janssen COVID-19 vaccination resumed on April 24 for all adults aged ≥18 years, at 50% of the pre-pause administration rate
- Janssen COVID-19 vaccination resumed on April 24 for all adults aged ≥18 years, at 100% of the pre-pause administration rate
- Janssen COVID-19 vaccination resumed on April 24 for all adults aged ≥50 years, at 50% of the pre-pause administration rate
- Janssen COVID-19 vaccination resumed on April 24 for all adults aged ≥50 years, at 100% of the pre-pause administration rate
Table 3 details the results from the population-level benefits and harms scenarios, 6 months post-Janssen COVID-19 vaccine pause, by resumption strategy. With this approach to assessing risks and benefits, the benefits of vaccination apply to the entire U.S. adult population over a 6-month period, and result from both direct and indirect effects. Indirect effects are also known as ‘herd protection’ and are based on the assumption that the vaccine can block transmission to unvaccinated persons as well. However, the risks only pertain to vaccinated persons.
|Benefits and harms from resuming vaccination||Resumption strategy|
|Recommended for adults aged ≥18 yrs||Recommended for adults aged ≥50 yrs|
|Low SARS-CoV-2 transmission||Moderate SARS-CoV-2 transmission||Low SARS-CoV-2 transmission||Moderate SARS-CoV-2 transmission|
|50% of prepause administration rate|
|No. of persons expected to receive the Janssen COVID-19 vaccine||9.8 million||3.6 million|
|Benefits, no. (% decrease§)|
|Hospitalizations prevented||3,926 (1.4)||9,395 (1.6)||1,361 (0.5)||3,532 (0.6)|
|ICU admissions prevented||928 (1.4)||2,236 (1.5)||295 (0.4)||799 (0.5)|
|Deaths prevented||586 (1.6)||1,435 (1.8)||54 (0.1)||257 (0.3)|
|No. of TTS cases expected||26||26||2||2|
|100% of prepause administration rate|
|No. of persons expected to receive the Janssen COVID-19 vaccine||17.5 Million||5.9 Million|
|Benefits, no. (% decrease§)|
|Hospitalizations prevented||5,987 (2.2%)||14,771 (2.5%)||1,947 (0.7%)||4,522 (0.7%)|
|ICU admissions prevented||1,418 (2.1%)||3,517 (2.4%)||396 (0.6%)||953 (0.6%)|
|Deaths prevented||882 (2.4%)||2,226 (2.8%)||43 (0.1%)||208 (0.3%)|
|No. of TTS cases expected||45||45||3||3|
* Resumption of vaccination after a 10-day pause that commenced on April 13, 2021 https://emergency.cdc.gov/han/2021/han00442.asp
† This model evaluated the direct and indirect effects of resuming 50% or 100% of Janssen COVID-19 administration rates (compared with rate before use was paused) among all adults aged ≥18 years or only among adults ≥50 years compared with not resuming vaccination. The model was also calibrated to both low and moderate COVID-19 transmission levels based on varying assumptions about nonpharmaceutical interventions during the modeled time period.
§ Compared with no resumption of Janssen vaccination.
Individual-Level Risk-Benefit Analysis
An individual-level risk-benefit assessment was conducted to quantify age and sex-specific benefits and harms, per million Janssen vaccine doses.
The objectives of the individual-level risk-benefit analysis were to:
- Quantify COVID-19-associated hospitalizations, ICU admissions, and deaths averted among vaccinated persons per million doses of the Janssen COVID-19 vaccine used, by age group and sex, and
- Compare these benefits to the expected numbers of TTS cases in the same population groups.
Calculations were based on:
- Recent age-specific incidence of hospitalizations for COVID-19 from COVID-NET
- Week ending March 27, 2021
- Age groups 18–29, 30–49 (averaged from 30–39 and 40–49), 50–64, ≥65 years
- Overall hospitalization rates, (weekly incidence of 7.9 per 100,000 population), one location with higher weekly incidence (17.8 per 100,000), and one location with lower weekly incidence (2.6 per 100,000)
- Proportions of hospitalized COVID-19 patients admitted to ICU or who died
- Number of persons already vaccinated as of April 21, 2021
- Expected TTS cases, based on estimated rates of TTS as of April 24, 2021, Table 1. One case of TTS in a male clinical trial participant in the 18–49 year age group was included4. The denominator was not adjusted for doses received during Phase III trial because they were not available in the specified age groups; however, the numbers are small relative to the numbers administered post authorization.
- 90% efficacy of Janssen COVID-19 vaccine to prevent hospitalizations from COVID-193.
- Additional simplifying assumptions: All unvaccinated persons have equal age-specific risk of hospitalization due to COVID-19 and all incident cases were considered to be in unvaccinated persons.
Details on COVID-NET methods are available here. COVID-NET is a population-based surveillance system that collects data on laboratory-confirmed COVID-19-associated hospitalizations among children and adults through a network of over 250 acute-care hospitals in 14 states. Hospitalization rates are calculated by the number of residents of a defined area who are hospitalized with a positive SARS-CoV-2 laboratory test divided by the total population within that defined area. Calculations were based on data available through April 20, 2021.
The calculations assume a 30-day period at risk as a result of delay in receiving an alternative COVID-19 vaccine, if the Janssen COVID-19 vaccine was not available, compared to not receiving any COVID-19 vaccine for this period of time. Analyses were also conducted to evaluate 60-day and 120-day delays in vaccination, and lower and higher incidence.
The individual-level risk-benefit analysis could be adapted to different incidence situations and with different assumptions about the lag in vaccination if the Janssen COVID-19 vaccine is not available. Table 4 details the results from the individual-level benefits and harms scenarios, 30, 60, and 120 days post-Janssen COVID-19 vaccine pause, by age and sex. With this approach to assessing risks and benefits, the benefits and risks of vaccination apply during the time period where there may be a delay in receipt of a COVID-19 vaccine, if a Janssen COVID-19 vaccine is not available. In addition, the risks and benefits both apply directly to the vaccinated person.
|Benefits and harms from resuming vaccination||No. per million vaccine doses administered§|
|18–49 yrs||≥50 yrs||18–49 yrs¶||≥50 yrs|
|TTS cases expected||7||1||1||0|
|Benefits, 30d, overall incidence‖|
|ICU admissions prevented||56||661||51||760|
|Benefits, 30d, low US state incidence‖|
|ICU admissions prevented||20||188||18||212|
|Benefits, 30d, high US state incidence‖|
|ICU admissions prevented||110||1554||100||1790|
|Benefits, 60d, overall incidence‖|
|ICU admissions prevented||112||1323||103||1520|
|Benefits, 120d, overall incidence‖|
|ICU admissions prevented||225||2645||206||3040|
* Resumption of vaccination after a 10-day pause that began on April 13, 2021. https://emergency.cdc.gov/han/2021/han00442.asp
† This analysis evaluated direct benefits and harms, per 1 million Janssen COVID-19 vaccine doses, over 30, 60, or 120 days.
§ Compared with no resumption of Janssen vaccination.
¶ Analyses incorporated one TTS case that occurred in the Phase III trial in a man in the 18–49 years age group.
‖ Incidence of hospitalization for week ending March 27, 2021, obtained from https://gis.cdc.gov/grasp/COVIDNet/COVID19_3.html
The population- and individual-level approaches to risk-benefit analyses offer complementary information on the risks and benefits of resuming vaccination, and in which age groups.
The population approach takes into account direct and indirect (herd) effects of vaccination, incorporates availability of different vaccines, simulates hospitalizations, ICU admissions and deaths under different transmission scenarios, and makes estimations of all outcomes over a 6-month time horizon. This approach shows a large population benefit of Janssen vaccination relative to rare occurrence of TTS.
The direct individual level approach estimates benefits and risks to vaccinated individuals, and only considers getting Janssen vaccine compared to not getting any vaccine over a short, 30-day time horizon. The analysis showed a positive balance of benefits and risks for all age and sex groups. However, the relative balance of risks and benefits for individuals varies by age and sex because cases of TTS were primarily identified among women aged 18–49 years. Benefits are higher when incidence (i.e., risk of exposure) is higher and when the assumed delay in vaccination is greater.
- Oliver S, Gargano J, Marin M, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for Use of Pfizer-BioNTech COVID-19 Vaccine — United States, December 2020. MMWR Morb Mortal Wkly Rep 2020;69:1922-1924. DOI: http://dx.doi.org/10.15585/mmwr.mm6950e2external icon
- Oliver S, Gargano J, Marin M, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for Use of Moderna COVID-19 Vaccine — United States, December 2020. MMWR Morb Mortal Wkly Rep 2021;69:1653-1656. DOI: http://dx.doi.org/10.15585/mmwr.mm695152e1external icon
- Oliver SE, Gargano JW, Scobie H, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for Use of Janssen COVID-19 Vaccine — United States, February 2021. MMWR Morb Mortal Wkly Rep 2021;70:329–332. DOI: http://dx.doi.org/10.15585/mmwr.mm7009e4external icon
- Vaccines and Related Biological Products Advisory Committee. Vaccines and Related Biological Products Advisory Committee February 26, 2021, meeting: FDA briefing document. Silver Spring, MD: US Department of Health and Human Services, Food and Drug Administration; 2021. https://www.fda.gov/media/146217/downloadexternal icon