Intervention Description

• Clear description of key aspects of the intervention

Quality–Study design

• Prospective study design
• Appropriate and concurrent comparison arm, or historical comparison (provided it is similar to intervention arm with respect to population, setting, time frame in the epidemic, follow-up time, recall period, and outcome measures)
  • Post hoc selection of control is allowed
• ≥ 1 community per arm
  • 1 community per arm is acceptable only if the following conditions are met: (1) there is a significant pre- and post-intervention change in the relevant outcome for the intervention arm, and (2) the significant pre- and post-intervention change is based on appropriate subject-level analysis or repeated-measures analysis.
• Select similar communities (units) for assignment
  • To minimize selection bias before assignment regardless of assignment methods (randomization or not); used methods such as systematic, a-priori approaches to choose intervention and control communities that are similar (e.g., matching or stratification on factors related to important/appropriate community characteristics)

Quality–Study implementation and analysis

• Sample individuals from assigned communities in acceptable ways (e.g., random, systematic) and use identical methods and eligibility criteria for selecting participants in each community, study arm, and data collection wave
  • If demographic differences are identified a priori, differential selection (e.g., over-sampling based on demographics) may be used to achieve equivalence between study arms on those factors.
• Follow-up assessment ≥ 1 month post completion of entire time-specific CLI or post full implementation of on-going CLI with recall not referring to pre-intervention period
  • Post full implementation of an on-going CLI means after all components of the CLI have been started or put in place in communities
• If cohort, at least 60% retention rate at a single follow-up assessment for each study arm
  • If cohort chart review, ≥ 60% success rate in matching medical records
• Comparison between intervention arm and an appropriate comparison arm
• Analysis of communities (units) as originally assigned, or communities may be excluded due to contamination or logistic/implementation issues only if dropping no more than one community per study arm AND retaining at least two thirds of intended communities
• Analysis of individuals within the communities (units) as originally assigned, or contaminated individuals may be excluded if numbers are small, but they may not be reassigned for analytic purposes
• Analysis of communities (units) regardless of community level of intervention exposure
• Analysis of individuals within the communities (units) may be based on intervention exposure, where dropping individuals who were not exposed to any intervention component (e.g., have not heard of or recognized intervention materials) as long as retaining at least 60% of total sample
• Cluster-level analyses may be provided, but is not required
• Analysis must be based on postintervention levels or among pre-post changes in measures
  • For pre-post changes used in analysis, measures must be identical, including identical recall period
• Analysis based on an α =.05 and either a 2-sided test or 1-sided test if an a-priori direction is hypothesized
• Either no statistical differences in baseline levels of the outcome exist or baseline differences are controlled for in the analysis, regardless of allocation method (e.g., randomization, non-randomization)
  • No differences on baseline levels of the outcome means reporting no significant difference between groups on BL relevant outcomes or match/stratify/statistically adjust participant data by using propensity scores or relevant outcome covariates (regardless of assignment methods - RCT or non-RCT)

Strength of Evidence–Significant positive intervention effects

• Positive and statistically significant (p ≤ .05) intervention effect for ≥ 1 relevant outcome measure
  • A positive intervention effect is defined as a greater reduction in HIV/STD incidence or risk behaviors or a greater increase in HIV protective behaviors in the intervention arm relative to the comparison arm
  • A relevant outcome is defined as a behavior (e.g., abstinence, mutual monogamy, number of sex partners, negotiating safer sex, condom use, injection drug use) that directly impacts HIV risk or a biologic measure indicating HIV or STD infection (i.e., HIV or STD incidence)
• Effect at the follow-up and based on the analyses that meet study implementation and analysis criteria

Strength of Evidence–Negative intervention effects

•  No negative and statistically significant (p < .05) intervention effect for any relevant outcome
  • A negative intervention effect is defined as a greater increase in HIV/STD incidence or risk behaviors or a greater decrease in HIV protective behaviors in the intervention arm relative to the comparison arm
• No other statistically significant harmful intervention effect
• For an intervention with a replication evaluation, no significant negative intervention effects in the replication study

Additional Limitations to Evaluate:

• No evidence that additional limitations resulted in a fatal flaw:
  • A fatal flaw has occurred when the overall evaluation of limitations resulted in considerable bias, thus substantially reducing the confidence of the findings
  • Examples of limitations to check for possible fatal flaw:
    • Group non-equivalence in baseline measures of important demographics or risk factors
    • Differential Retention (for cohort studies): (1) association between study arms and characteristics related to retention or attrition; OR (2) more than minimal rate of differential retention (> 10%)
    • Differential Refusal: At baseline for cohort studies; by wave for serial cross-sectional studies: (1) association between study arms and characteristics related to refusal; OR (2) more than minimal rate of differential refusal rate (> 10%)
    • Intervention activities did not match with the intervention concepts or guiding theories intended to produce the desired outcomes
    • Did not clearly describe issues related to generalizability
    • Effects only found within a potentially biased subset analyses
    • Substantial missing data (> 10% or missing data plus loss to attrition does not exceed acceptable limits for retention alone)
    • Too many post hoc analyses (even with Bonferrroni corrections)
    • Pilot study or very small sample size per study arm (< 40)

Go to top

Back to Efficacy Criteria