Grading of Recommendations, Assessment, Development, and Evaluation (GRADE): Janssen COVID-19 Vaccine

Overview

A Grading of Recommendations, Assessment, Development and Evaluation (GRADE) review of the evidence for benefits and harms for Janssen coronavirus disease 2019 (COVID-19) vaccine was presented to the Advisory Committee for Immunization Practices (ACIP) on February 28, 2021. GRADE evidence type indicates the certainty in estimates from the available body of evidence. Evidence certainty ranges from type 1 (high certainty) to type 4 (very low certainty) [1].

The policy question was, “Should vaccination with Janssen COVID-19 vaccine be recommended for persons 18 years of age and older during an Emergency Use Authorization?” The potential benefits pre-specified by the ACIP COVID-19 Vaccines Work Group were prevention of symptomatic laboratory-confirmed COVID-19 (critical), hospitalization due to COVID-19 (critical), all-cause death (important), SARS-CoV-2 seroconversion (important), and asymptomatic SARS-CoV-2 infection (important). The two pre-specified harms were serious adverse events (critical) and reactogenicity grade ≥3 (important).

A systematic review of evidence on the efficacy and safety of a one-dose regimen of Janssen COVID-19 vaccine among persons aged 18 years and older was conducted. The quality of evidence from one Phase I/II randomized controlled trial, one Phase II randomized controlled trial, and one Phase III randomized controlled trial were assessed using a modified GRADE approach [2-5].

The Phase III randomized controlled trial of the Janssen COVID-19 vaccine was conducted on three continents during a time of high COVID-19 incidence while viral variants were emerging. The trial exceeded the target numbers of cases to meet its primary endpoints by the time the required median 2-month follow-up was complete. The vaccine efficacy estimates for the Janssen COVID-19 vaccine were 66% for symptomatic laboratory-confirmed COVID-19, 74% for asymptomatic seroconversion, 93% for hospitalization due to COVID-19, and 75% for all-cause death. No deaths due to COVID-19 were identified among vaccine recipients, and 7 deaths due to COVID-19 were identified among placebo recipients. No serious safety concerns were identified in the randomized controlled trials, with balanced reports of serious adverse events between arms (0.4% each).  Grade ≥3 local or systemic reactions were more common among vaccine than placebo recipients, and were reported by <3% of vaccinated subjects. The certainty in the estimates for all critical benefits and harms was type 2 (moderate), with certainty impacted by concern for indirectness because of the short duration of follow-up in the available body of evidence.

Introduction

As of December 19, 2020, two vaccines had been recommended for prevention of COVID-19, caused by the SARS-CoV-2 virus which emerged in late 2019 [6,7].

On February 27, 2021, the U.S. Food and Drug Administration (FDA) issued an Emergency Use Authorization (EUA) for Janssen COVID-19 (Ad26.COV2.S) vaccine for prevention of symptomatic COVID-19 for persons aged ≥18 years [8]. As part of the process employed by the ACIP, a systematic review and GRADE evaluation of the evidence for Janssen COVID-19 vaccine was conducted and presented to ACIP. There were no conflicts of interest reported by CDC and ACIP COVID-19 Vaccines Work Group members involved in the GRADE analysis.

The ACIP adopted a modified GRADE approach in 2010 as the framework for evaluating the scientific evidence that informs recommendations for vaccine use, including those for the Janssen COVID-19 vaccine. Evidence of benefits and harms were reviewed based on the GRADE approach [1].

The primary policy question was, “Should vaccination with Janssen COVID-19 vaccine be recommended for persons 18 years of age and older during an Emergency Use Authorization?” (Table 1).

Methods

We conducted a systematic review of evidence on the efficacy and safety of a single-dose regimen of Janssen COVID-19 vaccine. We assessed outcomes and evaluated the quality of evidence using the GRADE approach.

We identified studies in Medline, Embase, and Cochrane Library, written in English, and limited to studies published from 2020 to February 8, 2021. Search terms included coronavirus, COVID-19, SARS-CoV-2, respiratory (symptom, disease, illness, condition), vaccine, immunization, trial, double blind, single blind, placebo, comparative study, phase I, phase II, phase III, immunogenicity, efficacy, effective, adverse, evidence, and variations on these terms (see Appendix 2 for details).

Articles were included if they provided data on vaccination with the Janssen COVID-19 vaccine and 1) involved human subjects; 2) reported primary data; 3) included adults (ages 18 and older) at risk for SARS-CoV-2 infection; 4) included data relevant to the efficacy and safety outcomes being measured; and 5) included data for the specific vaccine formulation, dosage, and schedule being recommended (Ad26.COV2.S, 5×10¹⁰ viral particles IM, single dose).  In addition, efforts were made to obtain unpublished and other relevant data by hand-searching reference lists, and consulting with vaccine manufacturers and subject matter experts. Titles and abstracts were screened independently and in duplicate by two separate reviewers. Characteristics of the included studies are shown in Appendix 1.

Patient-important outcomes (including benefits and harms) for assessment were selected by the Work Group during Work Group calls and via online surveys where members were asked to rate and rank the importance of relevant outcomes. The GRADE assessment across the body of evidence for each outcome was presented in an evidence profile.

Relative risks (RR) were calculated from numerators and denominators available in the body of evidence. Vaccine efficacy estimates were defined as 100% x (1-RR). When multiple studies were available, pooled estimates were calculated using RevMan software.

Results

The results of the GRADE assessment were presented to ACIP on February 28, 2021.

After title and abstract screening of 3,999 records, 29 studies were identified as eligible for full-text review. Of these, 26 were excluded because they assessed a different vaccine. This left 3 studies for the evidence synthesis and GRADE evidence assessment [2-3]. Characteristics of the included studies are shown in Appendix 1.

Outcomes of interest included individual benefits and harms. Benefits of interest deemed critical were prevention of symptomatic laboratory-confirmed COVID-19 and prevention of hospitalization due to COVID-19 (Table 2). Other important outcomes included prevention of all-cause death; SARS-CoV-2 seroconversion to a non-spike protein; and asymptomatic SARS-CoV-2 infection. The critical harm of interest was serious adverse events, including vaccine-associated enhanced disease; reactogenicity grade ≥3 was deemed an important harm. Asymptomatic infection was not included in the evidence profile because no data were available from serial PCR assessments.

Three studies were reviewed that provided data on outcomes specified for GRADE (Appendix 1). Data were reviewed from one published Phase I/II randomized control trial, with additional data provided by the sponsor [2,]. Data were reviewed from one unpublished Phase II randomized controlled trial and one Phase III randomized controlled trial using data provided by the sponsor and FDA [2-5, 9, 10].

The Janssen COVID-19 vaccine reduced symptomatic laboratory-confirmed COVID-19 when compared to no COVID-19 vaccination (vaccine efficacy: 66%; 95% CI: 60%, 71%) (Table 3a). For hospitalization due to COVID-19 in cases with onset ≥14 days post vaccination, 31 events were documented, 2 in the placebo group and 29 in the vaccine group. Vaccine efficacy against hospitalization due to COVID-19 was 93% (95% CI: 71%, 98%) (Table 3b). All-cause deaths were less common, 5 in the vaccine group and 20 in the placebo group (VE: 75%; 95% CI: 33%, 91%) (Table 3c).

Preliminary data from serum collected at day 71 suggested a lower risk of seroconversion to a non-spike protein, i.e., marker of natural infection (vaccine efficacy 74%, 95% CI: 48%, 87%) (Table 3d).

For evaluation of potential harms, data were pooled from the Phase I/II, II, and Phase III randomized controlled trials. Proportions with serious adverse events were comparable between the vaccine group and the placebo group when pooled across the three studies (pooled RR: 0.85; 95% CI: 0.63, 1.13); there were no cases of vaccine-associated enhanced disease or vaccine-related deaths (Table 4 (pooled), Table 3e (individual trial data)). Grade ≥3, or severe, local or systemic reactions 7 days following vaccination, were reported by 2.5% of vaccine recipients and 0.7% of placebo recipients (Table 4 (pooled), Table 3f (individual trial data)). No grade 4 reactions were reported.

GRADE Summary

The initial GRADE evidence level was type 1 (high) for each outcome because the body of evidence was from randomized controlled trials (Table 4). In terms of critical benefits, the available data indicated that the vaccine was effective for preventing symptomatic COVID-19 and for preventing hospitalization due to COVID-19; these were each downgraded one point for serious concern of indirectness related to the median two months follow-up (type 2, moderate). The certainty in the effect estimate for the important outcome of all-cause death was downgraded one point for serious concern of indirectness related to the median two months follow-up (type 2, moderate). The certainty in the estimate for seroconversion was downgraded two points due to very serious concern of indirectness because the efficacy against seroconversion based on day 71 serology may not be a direct measure of efficacy over a relevant period of time for an emergency use authorization and the subset of 6.8% with serology data likely differed substantially from the total population of interest (type 3, low). The certainty in the estimate of the effect for serious adverse events was downgraded one point due to serious concern of indirectness related to the median two months follow-up (type 2, moderate). No serious concerns impacted the certainty in the estimate of reactogenicity (type 1, high) (Table 4).

References

1. Ahmed F. U.S. Advisory Committee on Immunization Practices (ACIP) Handbook for Developing Evidence-based Recommendationspdf icon
2. Sadoff J, Le Gars M, Shukarev G et al. Interim Results of a Phase 1–2a Trial of Ad26.COV2.S Covid-19 Vaccine. NEJM. 2021. DOI: 10.1056/NEJMoa2034201.
3. Janssen, 2021 personal communication, February 5 – 17, 2021.
4. Food and Drug Administration (FDA). FDA Briefing Document – Sponsor: Janssen COVID-19 Vaccineexternal icon. Accessed February 24, 2021.
5. Food and Drug Administration (FDA). FDA Briefing Document Addendum – Sponsor: Janssen COVID-19 Vaccineexternal icon. Accessed February 24, 2021.
6. Oliver S, Gargano J, Marin M, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for Use of Pfizer-BioNTech COVID-19 Vaccine — United States, December 2020. MMWR Morb Mortal Wkly Rep. ePub: 13 December 2020.
7. Oliver S, Gargano J, Marin M, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for Use of Moderna COVID-19 Vaccine — United States, December 2020. MMWR Morb Mortal Wkly Rep. ePub: 20 December 2020.
8. Food and Drug Administration. Janssen COVID-19 Vaccine Emergency Use Authorization.  Janssen COVID-19 Vaccine EUA Letter of Authorization (fda.gov)external icon. Accessed February 27, 2021.
9. Food and Drug Administration (FDA). FDA Briefing Document Janssen COVID-19 vaccine. https://www.fda.gov/media/146217/downloadexternal icon. Accessed February 24, 2021.
10. Food and Drug Administration (FDA). FDA Review of Efficacy and Safety of Janssen COVID-19 Vaccine EUA. https://www.fda.gov/media/146267/downloadexternal icon. Accessed February 26, 2021.

Table 1: Policy Question and PICO

Table 1: Policy Question and PICO

Policy question:	Should vaccination with Janssen COVID-19 vaccine (single-doses, IM) be recommended for persons 18 years of age and older under an Emergency Use Authorization?
Population	Persons aged ≥18 years
Intervention	Janssen COVID-19 vaccine Ad26.COV2.S (5×1010 viral particles, single-dose IM)
Comparison	No COVID-19 vaccine
Outcomes	Symptomatic laboratory-confirmed COVID-19 Hospitalization due to COVID-19 All-cause death SARS-CoV-2 seroconversion to a non-spike protein Asymptomatic SARS-CoV-2 infectiona Serious adverse events Reactogenicity grade ≥3

Abbreviations: IM = intramuscular.

^aAssessed through serial PCR testing.

Table 2: Outcomes and Rankings

^aData from serial PCR assessments to inform an evaluation of asymptomic SARS-CoV-2 infection were not available in studies identified in the review of evidence.

Table 3a: Summary of Studies Reporting Symptomatic Laboratory-confirmed COVID-19^a

Table 3b: Summary of Studies Reporting Hospitalization due to COVID-19^a

Table 3c: Summary of Studies Reporting All-cause Death^a

Table 3d: Summary of Studies Reporting Asymptomatic SARS-CoV-2 Infection^a

Abbreviations: RR = relative risk; CI = confidence interval.

^aAsymptomatic SARS-CoV-2 infection is defined as (1) positive serology (non-S protein), and (2) no prior SARS-CoV-2 positive PCR or COVID-19 symptoms during the study. Seroconversion to a non-spike protein can distinguish between natural infection and vaccine-induced immunity.

Table 3e: Summary of Studies Reporting Serious Adverse Events^a,b

Abbreviations: RR = relative risk; CI = confidence interval; RCT = randomized controlled trial.

^aDeath, life-threatening event, hospitalization, incapacity to perform normal life functions, medically important event, or congenital anomaly/birth defect

^bExcludes serious adverse events due to COVID-19

^c9 participants (7 in the vaccine and 2 in the placebo group) were deemed by blinded investigators to have serious adverse events related or possibly related to vaccination. Among the 7 vaccine participants, these included: pericarditis, facial paralysis, injection site pain, Guillain-Barre Syndrome, systemic reactogenicity, and hypersensitivity. Through further investigation by the FDA, only 3 were classified as likely related to vaccination: injection site pain, hypersensitivity, and systemic reactogenicity.

Table 3f: Summary of Studies Reporting Reactogenicity^a

Table 4. Grade Summary of Findings Table

Abbreviations: CI = confidence interval; RR = relative risk; COVID-19 = coronavirus disease 2019; RCT = randomized controlled trial.

1. Risk of bias related to blinding of participants and personnel may have been present. Although participants and study staff were blinded to intervention assignments, they may have inferred receipt of vaccine or placebo based on reactogenicity. This was deemed unlikely to overestimate efficacy or underestimate risk of serious adverse events, therefore the risk of bias was rated as not serious.
2. Serious concern for indirectness was noted due to the short duration of observation in the available body of evidence. Vaccine efficacy or adverse events observed at a median 2-month follow-up may differ from those observed with ongoing follow-up.
3. The effects noted are from an analysis of the per protocol population with outcomes assessed at least 14 days post vaccination, who had no evidence of prior SARS-CoV-2 infection, and counting cases who met the case definition with symptoms for moderate to severe COVID-19 and were PCR positive but not necessarily molecularly confirmed at the central laboratory. In an interim analysis using the full analysis set (persons with or without evidence of prior SARS-CoV-2 infection), there were 267 cases among 21,895 persons in the vaccine arm and 621 cases among 21,888 persons in the placebo arm (RR = 0.43 (0.37 to 0.50)).
4. The RCT excluded persons with prior COVID-19 diagnosis, pregnant women or women planning to become pregnant within 3 months of vaccination, and persons who were immunocompromised due to conditions and/or treatments (participants with stable/well-controlled HIV infection were not excluded). The population included in the RCT may not represent all persons aged >=18 years.
5. Absolute risk was calculated using the observed outcomes in the placebo arm during the available clinical trial follow-up. Absolute risk estimates should be interpreted in this context.
6. The effects noted are from an analysis of the per protocol population with outcomes assessed at least 14 days post vaccination among persons who had no evidence of prior SARS-CoV-2 infection.
7. Includes 15 hospitalized cases in the placebo arm identified from the Serious Adverse Events form rather than from the Medical Resource Utilization form.
8. Very serious concern for indirectness was noted. Efficacy against seroconversion based on day 71 serology may not be a direct measure of efficacy over a relevant period of time for an emergency use authorization. Additionally, serology data were only available for a subset of 7% of the per protocol population, likely not representing all ages, comorbidities, geographies, and exposures to circulating variants, raising additional concern for indirectness.
9. Data were pooled from one Phase III trial, one Phase I/II trial, and one Phase II trial.
10. Differences in demographic composition were noted between the full analysis set (FAS) and the safety subset used for evaluation of reactogenicity. Notably, compared to the FAS, the safety subset included a higher proportion of White race (83.4% vs. 58.7%), a higher proportion from Brazil (38.5% vs. 16.6%), and a lower proportion who were seropositive for SARS-CoV-2 at baseline (4.5% vs. 9.6%).

Appendix 1. Studies Included in the Review of Evidence

Appendix 1. Studies Included in the Review of Evidence

Last name first author, Publication year	Study design	Country (or more detail, if needed)	Population	Total population	N Intervention	N comparison	Outcomes	Funding source
Janssen, 2021 [2-5, 9, 10]	Phase III RCT	USA	Persons aged ≥18 years	43783	21895	21888	Symptomatic laboratory-confirmed COVID-19 Hospitalization due to COVID-19 All-cause death Seroconversion Serious adverse events Reactogenicity	Government, Industry
Janssen, 2021 [3]	Phase II RCT	USA	Aged 18-55 years Aged ≥65 years	354	276	78	Serious Adverse Events Reactogenicity	Government, Industry
Sadoff, 2021 [2]	Phase I/II RCT	USA	Aged 18-55 years Aged ≥65 years	486	323	163	Serious Adverse Events Reactogenicity	Government, Industry

Abbreviations: SD = standard deviation; RCT = randomized controlled trial; COVID-19 = coronavirus disease 2019.

Appendix 2. Databases and strategies used for systematic review

Appendix 2. Databases and strategies used for systematic review

Database	Strategya	Records
Medline (OVID) 1946- And Embase (OVID) 1988-	exp coronavirus/ OR ((corona* or corono*) adj1 (virus* or viral* or virinae*)).ti,ab,kw OR (coronavirus* or coronovirus* or coronavirinae* or Coronavirus* or Coronovirus* or Wuhan* or Hubei* or Huanan or “2019-nCoV” or 2019nCoV or nCoV2019 or “nCoV-2019” or “COVID-19” or COVID19 or “CORVID-19” or CORVID19 or “WN-CoV” or WNCoV or “HCoV-19” or HCoV19 or CoV or “2019 novel*” or Ncov or “n-cov” or “SARS-CoV-2” or “SARSCoV-2” or “SARSCoV2” or “SARS-CoV2” or SARSCov19 or “SARS-Cov19” or “SARSCov-19” or “SARS-Cov-19” or Ncovor or Ncorona* or Ncorono* or NcovWuhan* or NcovHubei* or NcovChina* or NcovChinese*).ti,ab,kw OR (((respiratory* adj2 (symptom* or disease* or illness* or condition*)) or “seafood market*” or “food market*”) adj10 (Wuhan* or Hubei* or China* or Chinese* or Huanan*)).ti,ab,kw OR ((outbreak* or wildlife* or pandemic* or epidemic*) adj1 (China* or Chinese* or Huanan*)).ti,ab,kw. OR “severe acute respiratory syndrome*”.ti,ab,kw. OR exp Coronavirus Infections/ AND Vaccin* OR immunization* AND Trial* OR rct* OR randomi?ed OR double blind OR single blind OR clinical stud* OR comparative stud* OR placebo* OR phase 3 OR phase III OR safe* OR immunogenicity OR efficacy OR effective* OR adverse OR evidence Limit 2020 – current
Cochrane Library	“novel coronavir*” OR “novel corona virus*” OR “2019 coronavirus” OR “coronavirus disease” OR “coronavirus 2019” OR covid19 OR “covid 19” OR nCoV OR “novel CoV” OR “CoV 2” OR CoV2 OR sarscov2 OR sars-cov* OR sarscov OR 2019nCoV OR 2019-nCoV AND Vaccin* OR immunization*	– duplicates = unique items

1. Most recent search conducted February 8, 2021.