Intervention Description

• Clear description of key aspects of the intervention

Quality–Study design

• Prospective or quasi-prospective study design
• Appropriate and concurrent comparison arm, or historical comparison (provided it is similar to intervention arm with respect to population, setting, and time frame in the epidemic, and identical with respect to follow-up interval, recall period, and outcome measures)
• Random, minimally biased, or moderately biased allocation of participants to study arms, allowing for selection bias unrelated to the intervention or HIV risk. Assignment may be based on pre-established groups or selection into something other than the intervention, provided neither are directly related to HIV risk.

Quality–Study implementation and analysis

• Follow-up assessment > 1-month post completion of intervention for each study arm with recall not referring to pre-intervention period
• At least a 60% retention rate (or medical chart recovery) at a single follow-up for each study arm
• Comparison between intervention arm and an appropriate comparison arm
• Analysis of participants in study arms as originally allocated, or contaminated participants may be excluded if numbers are small, but participants may not be re-assigned for analytic purposes
• Analysis of participants may be based on intervention exposure, where participants exposed to < 50% of the entire intended intervention may be excluded
• If participants excluded due to contamination or low exposure (as described above), retention rate must include these participants at each follow-up they were assessed
• Analysis must be based on postintervention levels or on pre-post changes in measures
  • For pre-post changes used in analysis, measures must be identical, including identical recall period
• Analysis based on an α =.05 and either a 2-sided test or 1-sided test if an a-priori direction is hypothesized
• With nonrandomized assignment, either no statistical differences exist in baseline levels of the outcome measure, or baseline differences must be controlled for in the analysis. If moderately-biased assignment or historical comparison was used, differences in baseline demographics also must be controlled for in the analysis.
• Analytic sample of > 40 participants per study arm

Strength of Evidence–Significant positive intervention effects

• Positive and statistically significant (p ≤ .05) intervention effect for ≥ 1 relevant outcome measure
• A positive intervention effect is defined as a greater reduction in HIV/STD incidence or risk behaviors or a greater increase in HIV protective behaviors in the intervention arm relative to the comparison arm
• A relevant outcome is defined as a behavior (e.g., abstinence, mutual monogamy, number of sex partners, negotiation of safer sex, condom use, injection drug use) that directly impacts HIV risk or a biologic measure indicating HIV or STD infection (i.e., HIV or STD incidence)
• Effect at the follow-up and based on the analyses that meet study implementation and analysis criteria

Strength of Evidence–Negative intervention effects

• No negative and statistically significant (p < .05) intervention effect for any relevant outcome
• A negative intervention effect is defined as a greater increase in HIV/STD incidence or risk behaviors or a greater decrease in HIV protective behaviors in the intervention arm relative to the comparison arm.
• No other statistically significant harmful intervention effect
• For an intervention with a replication evaluation, no significant negative intervention effects in the replication study

Additional Limitations to Evaluate:

• No evidence that additional limitations resulted in a fatal flaw:
• A fatal flaw has occurred when the overall evaluation of limitations indicates they resulted in considerable bias, thus substantially reducing the confidence of the findings.
• Examples of limitations to check for possible fatal flaw:
  • Effects only found within potentially biased subset analyses
  • Substantial missing data:  Missing data plus loss to attrition exceeds acceptable limits for retention alone (>  40%)
  • Study arm non-equivalence: statistically significant differences between arms in important baseline demographics or risk factors
  • Differential Retention: (1) association between study arms and characteristics related to retention or attrition; OR (2) more than minimal rate of differential retention (> 10%)
  • Intervention activities did not match with the intervention concepts or guiding theories intended to produce the desired outcomes
  • Did not clearly describe issues related to generalizability
  • Too many post hoc analyses (even with Bonferroni corrections)
  • Inconsistent findings

All criteria must be satisfied for an intervention to be considered as a good -evidence individual-level or group-level intervention.
Source: Lyles et al. (2006) and Lyles at al. (2007). 

Go to top

Back to Efficacy Criteria