Intervention Description

• Clear description of key aspects of the intervention

Quality–Study design

•  Prospective study design
• Appropriate and concurrent control/comparison arm
• ≥ 4 communities per arm or appropriate power analysis indicating that a smaller number of communities was adequate (i.e., 2 or 3 communities per arm)
• Select similar communities (units) for assignment
  • To minimize selection bias before assignment regardless of assignment methods (randomization or not); use methods such as systematic, a-priori approaches to choose intervention and control communities that are similar (e.g., matching or stratification on factors related to important/appropriate community characteristics)

Quality–Study implementation and analysis

• Sample individuals from assigned communities in acceptable ways (e.g., random, systematic) and use identical methods and eligibility criteria for selecting participants in each community, study arm, and data collection wave
  • If demographic differences are identified a priori, differential selection (e.g., over-sampling based on demographics) may be used to achieve equivalence between study arms on those factors
• Follow-up assessment ≥ 3 months post completion of entire time-specific CLI or post full implementation of on-going CLI with recall not referring to pre-intervention period
  • “Post full implementation of an on-going CLI” means after all components of the CLI have been started or put in place in communities
• If cohort, at least 70% retention rate at a single follow-up assessment for each study arm
  • If cohort chart review, ≥ 70% success rate in matching medical records
• Comparison between intervention arm and an appropriate comparison arm
• Analysis of communities (units) and analysis of individuals within the communities as originally assigned regardless of contamination or logistic/implementation issues
• Analysis of communities (units) regardless of community level of intervention exposure
• Analysis of individuals within the communities (units) regardless of individual level of intervention exposure
• Use of appropriate cluster-level analyses, e.g., adjusting for ICC
• Analysis must be based on postintervention levels or among pre-post changes in measures
  • For pre-post changes used in analysis, measures must be identical, including identical recall period
• Analysis based on an a=.05 (or more stringent) and a 2-sided test
• Either no statistical differences in baseline levels of the outcome exist or baseline differences are controlled for in the analysis, regardless of allocation method (e.g., randomization, non-randomization)
  • No differences on baseline levels of the outcome means reporting no significant difference between groups on BL relevant outcomes or match/stratify/statistically adjust participant data by using propensity scores or relevant outcome covariates (regardless of assignment methods - RCT or non-RCT)

Strength of Evidence–Significant positive intervention effects

• Positive and statistically significant (p ≤ .05) intervention effect for ≥ 1 relevant outcome measure
  • A positive intervention effect is defined as a greater reduction in HIV/STD incidence or risk behaviors or a greater increase in HIV protective behaviors in the intervention arm relative to the comparison arm.
  • A relevant outcome is defined as a behavior (e.g., abstinence, mutual monogamy, number of sex partners, negotiating safer sex, condom use, injection drug use) that directly impacts HIV risk or a biologic measure indicating HIV or STD infection (i.e., HIV or STD incidence)
• Effect at the follow-up and based on the analyses that meet study implementation and analysis criteria

Strength of Evidence–Negative intervention effects

• No negative and statistically significant (p < .05) intervention effect for any relevant outcome
  • A negative intervention effect is defined as a greater increase in HIV/STD incidence or risk behaviors or a greater decrease in HIV protective behaviors in the intervention arm relative to the comparison arm.
• No other statistically significant harmful intervention effect
• For an intervention with a replication evaluation, no significant negative intervention effects in the replication study

Additional Limitations to Evaluate:

•  No evidence that additional limitations resulted in a fatal flaw:
  • A fatal flaw has occurred when the overall evaluation of limitations resulted in considerable bias, thus substantially reducing the confidence of the findings
  • Examples of limitations to check for possible fatal flaw:
    • Group non-equivalence in baseline measures of important demographics or risk factors
    • Differential Retention (for cohort studies): (1) association between study arms and characteristics related to retention or attrition; OR (2) more than minimal rate of differential retention (> 10%)
    • Differential Refusal: At baseline for cohort studies; by wave for serial cross-sectional studies: (1) association between study arms and characteristics related to refusal; OR (2) more than minimal rate of differential refusal rate (> 100)
    • Intervention activities did not match with the intervention concepts or guiding theories intended to produce the desired outcomes
    • Did not clearly describe issues related to generalizability
    • Effects only found within a potentially biased subset analyses
    • Substantial missing data (> 10% or missing data plus loss to attrition does not exceed acceptable limits for retention alone)
    • Too many post hoc analyses (even with Bonferrroni corrections)
    • Pilot study or very small sample size per study arm (< 50)

Go to top

Back to Efficacy Criteria