Outcome Monitoring and Evaluation Reporting Requirements
Selecting an Intervention for Outcome Monitoring and Evaluation
Developing Outcomes Appropriate for Evaluation
Designing Outcome Monitoring
Designing Outcome Evaluation
Institutional Review Boards

This chapter:

• Reviews outcome monitoring and outcome evaluation reporting requirements,
• Distinguishes outcome monitoring and outcome evaluation,
• Presents criteria for selecting an intervention for outcome monitoring or outcome evaluation,
• Provides examples of outcomes appropriate for evaluation,
• Describes the basic design for outcome monitoring,
• Describes an evaluation design that avoids common concerns about outcome evaluation, and
• Explains the purpose of an Institutional Review Board.

Outcome Monitoring and Evaluation Reporting Requirements

Health departments with at least $1 million in cooperative agreement funding from CDC are required to collect and report outcome data for either an outcome monitoring or outcome evaluation project during the cooperative agreement period. Health departments may choose whether to conduct outcome evaluation or outcome monitoring. The specific reporting requirements for these two types of evaluation are described below.

Outcome Monitoring Reporting Requirements

Health departments that choose to conduct outcome monitoring are required, for the year 2002, to conduct this evaluation with at least 10 percent of their contractors who are implementing interventions appropriate for outcome monitoring. These data are to be reported in April 2003. For the year 2003, health departments are required to conduct outcome monitoring with 20 percent of their contractors and report their findings in April 2004. It is up to each health department and its contractors to decide which interventions to evaluate. It may be preferable to conduct outcome monitoring with a variety of interventions rather than with the same intervention across several contractors.

Reports to CDC on outcome monitoring projects should contain the following information:

• names and affiliations of evaluators conducting the outcome monitoring
• intervention type
• intervention goals
• target population
• evidence and justification for the intervention
• copy of instruments and data collection tools
• methods of data collection and statistical analysis
• appropriate descriptive statistics, including client demographics
• summary of findings; and
• how results will be used for program improvement.

Outcome Evaluation Reporting Requirements

Health departments that choose to conduct outcome evaluation are required to evaluate at least one distinct HIV prevention intervention or set of integrated interventions. The intervention should be of sufficient design and maturity of development to warrant a rigorous evaluation. The evaluation design should be quasi-experimental, using a non-equivalent comparison group or multiple measurements before and after the intervention. When feasible, health departments may use an experimental design with random assignment of clients to treatment and control groups. Any experimental-type design (e.g., assignment of clients to treatment and control groups or comparison of outcomes between clients in standard and enhanced interventions) must undergo local Institutional Review Board (IRB) approval. No contact with "human subjects" in an experimental-type design may take place without local IRB approval. (See p. 52 for more information about IRBs.)

One outcome evaluation report is due to CDC in September 2003 with health departments' applications for funding. The report should contain the following information:

• names and affiliations of evaluators conducting the outcome evaluation;
• intervention type;
• intervention goals;
• target population;
• evidence and justification for the intervention;
• evaluation design and methods;
• sample sizes for treatment and comparison groups and numbers of participants lost to attrition (as appropriate);
• copy of instruments and data collection tools;
• methods of data collection and statistical analyses;
• appropriate descriptive statistics, including client demographics;
• summary of findings (e.g., attrition, overall outcomes, and any subgroup analyses of differences due to demographics, features of the intervention, or other variables); and
• how results will be used for program improvement.

Distinguishing Outcome Monitoring and Outcome Evaluation

Outcome monitoring and outcome evaluation both involve collecting data about client outcomes before and after the intervention. Outcome evaluation, however, also collects data from people not participating in the intervention or, in some cases, collects data from clients at several points in time both before and after the intervention. This difference in how data is collected underlies an important difference in what can be learned from these two types of evaluation.

Outcome monitoring tells you if the expected outcomes occurred.

Outcome evaluation tells you if the intervention caused the expected outcomes.

The difference between outcome monitoring and outcome evaluation is illustrated with the following example. A contractor implemented a GLI consisting of four small group sessions with heterosexual African American women who are partners of IDUs. One of the stated outcomes is to increase condom use by 25%. For outcome monitoring, a questionnaire was used to measure condom use before and after the intervention. When the intervention was complete, program staff found that condom use increased by 35%.

The increase in condom use may be the direct result of the intervention or there may be other explanations for why condom use increased. Perhaps some women knew that prevention staff wanted them to use condoms, and they were not truthful on the final questionnaire, reporting that they used condoms more often than they really did. Maybe outreach workers from another intervention were recently working in this neighborhood distributing condoms, resulting in increased condom use. It is possible that some of the increase in condom use is the result of the intervention and some is due to these or other factors.

In this scenario, outcome monitoring would show that the stated outcome of increasing condom use by 25% was exceeded. But outcome monitoring cannot rule out other factors that might be responsible for this change. If outcome evaluation where conducted, then the same questionnaire would also be used to measure condom use among women not participating in the intervention.

By comparing changes in condom use among women participating and not participating in the intervention, outcome evaluation can better assess how much of the change among participants was caused by the intervention and how much was the result of other factors. For a more comprehensive discussion of the distinction between outcome monitoring and outcome evaluation, see the Guidance, volume 2, chapters 6 and 7.

Selecting an Intervention for Outcome Monitoring and Evaluation

Regardless of whether health departments choose to conduct outcome monitoring or outcome evaluation, there are important criteria to consider when selecting an intervention to evaluate. Outcome evaluation and, to a lesser extent, outcome monitoring are more complex and resource intensive than other evaluation activities required by the Guidance. To ensure the effective use of evaluation resources, health departments are encouraged to carefully select interventions for evaluation that will produce valid findings useful to the health department, contractors, and CDC.

The criteria listed below can be used to screen interventions being considered for outcome monitoring and outcome evaluation and to identify one or more good candidates. Health departments are free to consider additional criteria relevant to their jurisdiction. If no currently funded intervention meets these criteria, health departments can use the criteria to guide efforts to strengthen an intervention as a prelude to evaluating.

SMART outcomes: Intervention outcomes should be Specific, Measurable, Appropriate, Realistic and Time-based. If the expected outcomes of the intervention are not clearly stated, outcome monitoring cannot assess if the outcomes occurred and outcome evaluation can not determine if the intervention caused these outcomes. Interventions with unclear outcomes should not be selected for evaluation. (For more information about creating outcome objectives, see the Guidance, volume 2, chapter 3.)

Defined intervention plan: The intervention should have a strong basis in formal or informal theory and clearly explain how intervention activities will lead to the outcomes (i.e., sufficient evidence and justification). Without a clear intervention plan it will be difficult to know why the expected intervention outcomes did or did not occur.

Fidelity to the intervention plan: The intervention should be implemented consistent with the intervention plan. For a variety of reasons, an intervention is not always implemented as intended. Process monitoring and process evaluation data can be used to assess consistency with the intervention plan and to help identify interventions appropriate for evaluation. Variation from the plan will make it difficult to know what "version" of the intervention caused the outcomes. (For a more complete discussion of the relationship between implementation and outcomes, see the Guidance, volume 2, chapters 6 and 7.)

Stability over time: The intervention should not be changed during the evaluation. Changes to the intervention will confound understanding of which aspects of the intervention achieved, or caused, the stated outcomes. Health departments should consider the organizational strength of the contractor implementing the intervention, reliability of funding for the intervention, compatibility of the intervention with local laws and ordinances, and other factors that may impact the stability of the intervention over time.

Sufficient reach: Interventions should be considered for outcome monitoring or outcome evaluation if they reach a sufficiently large number of clients (i.e., sample size) to apply statistical tests necessary for data analysis. The number of clients needed depends on several factors, including the evaluation design, the intended outcomes, and the intensity of the intervention. (For more information about sample size, see the Guidance, volume 2, chapter 7.)

Sufficient dosage: Clients should have sufficient exposure to the intervention to result in the intended outcomes. Interventions with limited client contact are less likely to result in measurable outcomes as compared with interventions that provide more in-depth intervention with clients.

Obtainable data: Interventions should be considered for outcome monitoring or outcome evaluation if the data needed to measure outcomes are reasonable and accessible. Health departments should avoid attempting outcome monitoring and outcome evaluation on interventions that may have difficulty following up clients to collect post intervention data.

Contractor capacity: The contractor implementing the intervention should have the capacity and motivation to partner with the health department to conduct outcome monitoring or outcome evaluation. These evaluation activities may place an additional burden on the contractor in terms of resources, staff training, intervention monitoring, and data collection. The contractor should be well informed about roles and responsibilities in this endeavor and be a willing participant.

Utility of findings: An intervention should be selected for which outcome monitoring or outcome evaluation will produce findings useful to the health department and its contractors and clients. In choosing an intervention to evaluate, health departments should seek to address gaps in understanding about interventions within their jurisdiction.

Developing Outcomes Appropriate for Evaluation

An important objective of HIV prevention is to reduce HIV incidence by changing risky behaviors. Measures of behavior change are preferred for outcome monitoring and outcome evaluation. Predictors of behavior change, such as changes in knowledge, attitudes, beliefs, skills, behavioral intentions, or other domains, are acceptable for evaluation purposes and are preferable if post-intervention data are collected before there is an opportunity for behavior change to occur (e.g., immediately after the intervention). When predictors of behavior change are used, the evaluation plan should describe the formal or informal theory that explains how changes in these domains will lead to behavior change.

Outcomes should be stated in clear and measurable terms and be appropriate for the intervention to enable a good evaluation. For example, "reduce high-risk sexual behavior" may be the stated outcome for a given intervention. The meaning of "high-risk" and "sexual behavior" must be defined by asking questions such as: Does it include oral sex? and /or Does it include intercourse with a long-term partner? Maybe the only behavior addressed in the intervention is vaginal intercourse with an injection drug-using partner. How much these behaviors will be reduced must also be considered. Does the intervention intend to entirely eliminate sexual risk behaviors for all clients receiving the intervention or just for some? Similarly, one must consider if the outcome is appropriate for the intervention. Perhaps the intervention focuses primarily on needle use and does not have sufficient emphasis on sexual risk behaviors to result in the desired behavior change. Clarification of the intended outcomes of an intervention is an important step in preparing to evaluate. (For more information about creating outcome objectives, see the Guidance, volume 2, chapter 3).

Examples are provided below to illustrate the different types of outcomes that may be used for outcome monitoring or outcome evaluation and the level of specificity appropriate for their description. Health departments may identify other outcomes for the interventions they choose to evaluate. These examples are not meant to be a comprehensive list nor are the percentages or time frames meant to suggest CDC's expectations for changes in these particular indicators. Health departments should collaborate with their contractors to develop outcome objectives appropriate for each intervention and targeted population. (For more information about different types of outcome objectives, see the Guidance, volume 2, chapter 6).

Designing Outcome Monitoring

Outcome monitoring requires, at a minimum, the collection of outcome data at least once before and once after the intervention (commonly known as the one-group pretest and posttest design). If feasible, health departments are encouraged to collect a second set of follow-up data after the intervention. This second set of data helps determine the extent to which changes among clients are sustained over time.

"We're going to look at how many of our contractors are interested in doing outcome monitoring and start to move them in that direction, making it a real collaborative process. They've already been saying that they'd like to know if they're making a difference and changing behaviors. So we are taking their lead on this, and they'll get lots of technical assistance from us to develop an evaluation plan and the necessary tools." Health Department Staff Member

Health departments are also encouraged to collect data over time on multiple clients participating in the intervention. Combining outcome data from multiple clients participating in the intervention yields a larger sample size, enabling statistical analyses that produce more robust findings and can, therefore, be more useful to program improvement.

Interventions appropriate for outcome monitoring include ILI, GLI, PCM, and client-centered counseling in the context of HIV counseling, testing, and referral. It may not be feasible to carry out outcome monitoring for street outreach and HC/PI because of the difficulty of collecting follow-up data after the intervention is complete. (For more information about designing outcome monitoring studies, see the Guidance, volume 2, chapter 6.)

Designing Outcome Evaluation

The Guidance states that health departments may use a quasi-experimental design with a non-equivalent comparison group or multiple measurements before and after the intervention. An experimental design with random assignment of clients to treatment and control groups may also be used, when feasible. This type of design requires local Institutional Review Board (IRB) approval. (For a more complete discussion of evaluation design, see the Guidance, volume 2, chapter 7.)

Health departments and contractors may find it undesirable to use a comparison or control group in their evaluation. There may not be sufficient numbers of clients to deliver the intervention to some and have others serve as a comparison or control group. Even if there are enough clients, ethical concerns may prevent withholding the intervention from some clients.

" We cannot get a control group here. We're not in a place where the epidemic is horrible, so anybody that we could identify, there's a space for them in an intervention. So sort of prolonging them getting into the intervention, that's just not morally and ethically the right thing that we want to be doing." Health Department Staff Member

These concerns can be minimized by comparing a basic and enhanced intervention for outcome evaluation. This evaluation design is discussed below.

Comparing a Basic and Enhanced Intervention

Comparing a basic and an enhanced version of the same intervention is an experimental design and requires local IRB approval. It does not, however, withhold an intervention from clients and, therefore, avoids some of the concerns associated with the typical use of comparison and control groups. In this design, two versions of the same intervention are delivered to two comparable groups of clients and each group serves as a comparison for the other.

In this scenario, the participants in one intervention can serve as a comparison group for the other. The relative effectiveness of both interventions can be compared by assessing the extent to which they each achieved a common set of outcomes (e.g., changes in knowledge, attitudes, beliefs, behaviors, and skills). However, because the evaluation lacks a comparison or control group that does not receive an intervention, it is not possible to entirely control for other factors that may affect intervention outcomes (e.g., clients receiving another intervention at the same time they participate in the intervention being evaluated).

This evaluation design addresses the need to identify effective alternatives to ineffective interventions and avoids ethical concerns about denying client access to the intervention. However, if the jurisdiction already has credible evidence that the enhanced intervention is more effective that the basic intervention, then the enhanced intervention should not be denied to any client and a different basic and enhanced intervention set should be evaluated. Many jurisdictions usually do fund some type of basic intervention, and this evaluation design can be used to help determine if the basic intervention can be improved. Comparing basic and enhanced interventions can be a pragmatic approach to conducting outcome evaluation that meets Guidance requirements, avoids some of the concerns related to other designs, and is compatible with the array of intervention types being conducted in many jurisdictions.

Institutional Review Boards

An IRB is a group established to protect the welfare of human subjects recruited to participate in biomedical or behavioral research. IRBs are responsible for reviewing and supervising proposed research to make sure they are in compliance with minimum standards for protection of human subjects.

Health departments must receive IRB approval when conducting outcome evaluation using an experimental-type design, such as assignment of clients to treatment and control groups or comparison of outcomes between clients in basic and enhanced interventions. Contact with human subjects in an experimental design is only permitted with local IRB approval. IRB approval pertains to the health department that funds the intervention undergoing outcome evaluation as well as to the contractor(s) implementing the intervention, known as the "performance site" for IRB purposes. Depending on the jurisdiction, local IRB review and approval may be required for evaluation designs other than experimental-type designs. Health departments should check with their IRB before starting outcome evaluations.

Health departments conducting outcome evaluation using an experimental-type design will need to submit to CDC a copy of their application to their local IRB, including the evaluation protocol and a copy of the IRB's response. Because CDC is not a co-investigator in the outcome evaluation, health departments do not need to apply for approval from the CDC's IRB.

IRBs review research protocols to ensure that they comply with standards for protection of human subjects as described in the Federal Policy (also know as the Common Rule). Health departments and contractors conducting outcome evaluation with an experimental-type design must enter into a binding commitment to the Common Rule before research begins. The document containing this binding commitment is called an "assurance." There are several types of assurances including: Multiple Project Assurance, Federal-wide Assurance, Inter-Institutional Amendment, Cooperative Amendment, Single Project Assurance, and Cooperative Project Assurance. Each type of assurance is appropriate for different circumstances.

IRBs are registered with the Office for Human Research Protections (OHRP), US Department of Health and Human Services. An OHRP website maintains extensive information on IRBs and the different types of assurances (http://www.hhs.gov/ohrp/). The following website can be used to identify local IRBs: http://www.hhs.gov/ohrp/assurances/assurances_index.html.