Intervention Description

• Clear description of key aspects of the intervention

Quality–Study design

• Prospective study design
• Appropriate and concurrent comparison arm
• Random or minimally biased assignment of subjects to study arms

Quality–Study implementation and analysis

• Follow-up assessment > 3-months post completion of intervention for each study arm with recall not referring to pre-intervention period)
• At least a 70% retention rate at a single follow-up assessment for each study arm
• Comparison between intervention arm and an appropriate comparison arm
• Analysis of participants subjects in study arms as originally allocated regardless of contamination or logistic/implementation issues
• Analysis of participants regardless of the level of intervention exposure
• Use of appropriate cluster-level analyses if assigned to study arms by cluster or group
• Analysis must be based on post-intervention levels or on pre-post changes in measures
  • For pre-post changes used in analysis, measures must be identical, including identical recall period
• Analysis based on an α =.05 (or more stringent) and a 2-sided test
• With nonrandomized assignment, either no statistical differences in baseline levels of the outcome exist or baseline differences are controlled for in the analysis
• Analytic sample > 50 participants per study arm

Strength of Evidence–Significant positive intervention effects

• Positive and statistically significant (p ≤ .05) intervention effect for ≥ 1 relevant outcome measure
  • A positive intervention effect is defined as a greater reduction in HIV/STD incidence or risk behaviors or a greater increase in HIV protective behaviors in the intervention arm relative to the comparison arm
  • A relevant outcome is defined as a behavior (e.g., abstinence, mutual monogamy, number of sex partners, negotiation of safer sex, condom use, injection drug use) that directly impacts HIV risk or a biologic measure indicating HIV or STD infection (i.e., HIV or STD incidence)
• Effect at the follow-up and based on the analyses that meet study implementation and analysis criteria

Strength of Evidence–Negative intervention effects

• No negative and statistically significant (p < .05) intervention effect for any relevant outcome
  • A negative intervention effect is defined as a greater increase in HIV/STD incidence or risk behaviors or a greater decrease in HIV protective behaviors in the intervention arm relative to the comparison arm.
• No other statistically significant harmful intervention effect
• For an intervention with a replication evaluation, no significant negative intervention effects in the replication study

Additional Limitations to Evaluate:

• No evidence that additional limitations resulted in a fatal flaw:
  • A fatal flaw has occurred when the overall evaluation of limitations indicates they resulted in considerable bias, thus substantially reducing the confidence of the findings.
  • Examples of item limitations to check for possible fatal flaw:
    • Effects only found within potentially biased subset analyses;
    • Substantial missing data. Missing data plus loss to attrition exceeds acceptable limits for retention alone (> 40%)
    • Study arm non-equivalence: statistically significant differences between arms in important baseline demographics or risk factors
    • Differential retention: (1) significant difference between study arms in characteristics among retained or attritted participants; OR (2) more than minimal rate of differential retention (> 10%)
    • Intervention activities did not match with the intervention concepts or guiding theories intended to produce the desired outcomes § Did not clearly describe issues related to generalizability
    • Too many post hoc analyses (even with Bonferroni corrections)
    • Inconsistent findings

Source: Lyles et al. (2006) and Lyles at al. (2007).

Go to top

Back to Efficacy Criteria