Overview
Development of Tools to Collect Outcome Monitoring Data
When to Monitor Outcomes
Managing Outcome Monitoring Data
Analyzing Outcome Monitoring Data
References and Resources
Appendix A: CDC Draft Outcome Monitoring Questions
Appendix B

Overview

Throughout this guidance document, there has been an emphasis on the importance of evaluation at each step in the process of HIV prevention programming. This has included evaluation of the community planning process, linkages between the plan and proposed interventions, the comprehensiveness and integrity of intervention plans, and implementation of interventions. These evaluation activities help build a high quality foundation for the most important goal of interventions: reduced HIV risk behaviors. This foundation also allows the monitoring (and possible evaluation) of programs’ outcomes (see Figure 6.1).

Figure 6.1: Good intervention plans and implementation provide a foundation for prevention outcomes.

HIV Prevention Intervention Plan

HIV Prevention Intervention Implementation

Behavioral Risk Reduction for HIV Prevention

Purposes of the Chapter

Chapter 4 of this document (Monitoring and Evaluating the Implementation of HIV Prevention Interventions) addressed the question of what services were provided, to whom, and to what extent. This chapter describes ways of monitoring the achievement of outcome objectives for each counseling intervention undertaken with health department funds. Figure 6.2 illustrates the relationship between process and outcome monitoring.

Figure 6.2: The relationships between process and outcome monitoring

Develop the Intervention Plan State the process objectives State the outcome objectives
Monitor Process	Monitor Outcomes
Count number of individuals participating in the intervention Determine risk and demographic characteristics of each individual served
Keep a record of each intervention protocol component delivered	Keep a record of each intervention protocol component delivered
Keep a record of staffing and other relevant intervention activities (see process monitoring variables)	Measure each participant’s desired outcomes following the intervention
Compare process monitoring data to process objectives stated in the intervention plan	Determine if there are differences between clients completing both measures and those completing only the pre-intervention measure (because of attrition)
	Determine if the average change among all clients is at least at the level stated in the outcome objective
	Determine if there are differences between clients for whom the intervention worked and those for whom it did not.

Data derived from monitoring intervention outcomes are needed to determine whether the program’s objectives are being met. For example, an agency might state an outcome objective as: “The intervention will result in at least 20% increase in the frequency of condom use for each participant receiving at least three of the four sessions.” Outcome monitoring is necessary to determine whether the intervention is meeting its own objectives. In contrast, the next chapter will address evaluation methods that can be used to determine whether an intervention works better than another (or better than doing nothing at all).

In particular, this chapter will discuss 1) the differences between outcome monitoring and outcome evaluation, 2) the development of tools to collect outcome monitoring data, 3) when to collect outcome data, and 4) issues in analyzing the data.

Distinguishing Outcome Monitoring And Outcome Evaluation

Prevention interventions are implemented with the expectation that they will reduce risk behaviors. Therefore, it is important to examine whether intervention participants change their behavior. As noted in the chapter on intervention plans, each intervention should have specific and measurable outcomes. We will discuss two ways to assess the attainment of these objectives: outcome monitoring (the focus of this chapter) and outcome evaluation (the focus of the next chapter). The primary difference between the two is that a rigorous evaluation design is essential to outcome evaluations.

Outcome Evaluation. Outcome evaluation entails the application of rigorous methods to assess whether the prevention program has an effect on the predetermined set of goals; the use of rigorous methods allows one to rule out factors that might otherwise appear responsible for the changes seen. Rigorous methods usually refer to experimental and quasi-experimental designs (e.g., see reference to Cook and Campbell, 1979). An example of a “rigorous method” is a randomized experiment in which some clients are randomly assigned to a treatment group receiving an intervention and others are assigned to a control group receiving no intervention. The use of this or other rigorous designs is the only way to make the claim that any changes in outcomes in the treatment group were due to your intervention.

The application of these evaluation designs requires the use of well-developed principles of scientific inquiry to provide credible, defensible information on intervention effectiveness. For these reasons, conducting an outcome evaluation is more complex and resource-intensive than simple outcome monitoring.

Outcome Monitoring. In this document, outcome monitoring refers to efforts to track the progress of clients or a program based upon outcome measures set forth in program goals. These measurements assess the effects of counseling interventions on client outcomes (i.e. knowledge, attitudes, beliefs, and behavior).

Outcome monitoring information should be collected from each participant in these interventions at least once prior to the intervention and at least once following it. The purposes of outcome monitoring are

• understand clients’ progress toward behavioral goals and objectives,
• understand differential progress within subgroups of clients (e.g., young clients make more progress than older ones), and
• understand if particular aspects of implementation contribute to or hinder clients’ progress.

Care must be taken when interpreting outcome monitoring findings. The monitoring described here is intended for the accountability and improvement of a particular intervention. With some exceptions (see reference to Chen, 1990), monitoring is not intended to produce evidence that can be compared with findings related to another intervention. This is especially true of comparing data from outcome monitoring to data from outcome evaluations that provide more conclusive information about the cause-and-effect relationship between the intervention and client outcomes.

As defined for this document, outcome monitoring requires the collection of outcome data at least once before and once after the intervention. In the evaluation literature, this is commonly called the one-group pretest and posttest design (Campbell and Stanley, 1963; Cook and Campbell, 1979).

While this approach does not have adequate rigor to be used for outcome evaluation, it has three particular benefits in the context of HIV prevention programming that strongly recommend it for assessing clients’ progress for program improvement.

First, the pretest/posttest approach can provide timely information about stakeholders’ immediate concerns such as whether clients are generally moving in the right direction. If monitoring data indicate that outcomes are not improving, this is sufficient information to suggest that program managers need to modify or refine the intervention. There is no need to carry out a rigorous and time-consuming outcome evaluation to verify an intervention’s ineffectiveness. However, if the data from monitoring are promising, then one might consider carrying out a rigorous outcome evaluation to verify the strength of the intervention so that it might be recommended to others.

Second, outcome monitoring is easier to carry out, less expensive, and less intrusive than outcome evaluation. Monitoring outcomes before and after an intervention is much more feasible than randomly assigning clients into various groups and ensuring that one group receives the intervention and the other does not.

Third, it takes time and experience for an organization to build evaluation capacity for conducting rigorous outcome evaluations. However, outcome monitoring—the collection of pre- and post-intervention data from the people receiving the intervention—is one of the crucial elements in outcome evaluation. As will be discussed in the next chapter, the rigorous designs in outcome evaluation usually build on or expand on the pretest and posttest design. Thus, one of the additional benefits for an organization engaging in outcome monitoring is that it develops a solid foundation for future outcome evaluations.

Types of Prevention Interventions Suitable for Outcome Monitoring. Unlike process evaluation, which can be applied to every prevention intervention, outcome monitoring is not equally feasible with all types of prevention interventions. For example, street outreach interventions may encounter a particular client only once, and it may be difficult (if not impossible) to do follow-up for many clients. Under this condition, it is difficult to do outcome monitoring. Similarly, it is difficult to obtain outcome data on public information campaigns.

Therefore, this guidance recommends that outcome monitoring be applied only October to programs whose clients are accessible for a follow-up measure of program outcomes. Individual-level counseling and group-level counseling are typical of interventions that meet this criterion. Outcome monitoring is also feasible for prevention case management and counseling done in the context of testing for HIV. PCM meets the basic requirements for outcome monitoring in that a particular client is provided a known set of services and is followed over time. This allows an agency to 1) know what intervention “package” is being assessed for a given client and 2) obtain both pre- and post-PCM measures of the risk behaviors of interest for that client. Similarly, outcomes can be assessed for clients who receive counseling before and after testing to determine the extent of changes over that time period. Jurisdictions are encouraged to assess outcomes in as many other types of interventions as they are capable of and find feasible.

Back to top

Development of Tools to Collect Outcome Monitoring Data

As stated earlier, outcome monitoring is the measurement of progress in achieving the goals set forth in the intervention plan (as described in Chapter 3). The goals and measurable outcomes that have been articulated in an intervention plan generally address the client outcomes or the expected changes in the target group(s) as a result of participation in the intervention. To determine whether these behavior change objectives have been achieved, some type of measurement must be made at at least two points in time. This section of the chapter will help program planners and managers understand the kinds of issues that need to be considered in designing or selecting an appropriate measurement tool for monitoring client outcomes.¹

Domains to Be Addressed in Outcome Monitoring

Generally, the types of client outcomes that need to be measured to assess the effects of individual or group health education or risk-reduction interventions are in the domains of knowledge, attitudes, beliefs, and behaviors (KABB). Most interventions involve activities that encourage reduction of clients’ HIV risk behaviors. However, according to behavioral theorists, interventions may also target the determinants of risk behaviors (Fishbein, Bandura, Triandis, Kanfer, Becker, and Middlestadt, 1991). Such determinants can include HIV/AIDS-related knowledge, attitudes, beliefs, and other domains.

Most importantly, the measurements must reflect the outcomes that are believed to result from the intervention. These typically are stated as the outcome objectives. For example, a group counseling intervention may focus on increasing clients’ self-efficacy (confidence) and skills in correctly and consistently using condoms; the intervention’s outcome objectives would include statements about the extent to which the intervention would increase clients’ self-efficacy and skills related to condom use as well as increase the frequency with which clients use condoms consistently. In this example, the domains that need to be included in a tool for monitoring outcome objectives would be measures of “self-efficacy,” “condom use skills,” and “consistent condom use.”

On the other hand, it would make little sense to measure behaviors or determinants that the intervention does not attempt to modify. In the example above, for instance, measuring knowledge of HIV transmission routes would be irrelevant to the example above, since the intervention makes no effort to modify knowledge. Measuring domains that are not addressed in the intervention and its outcome objectives wastes the clients’ time answering unnecessary questions and program staff’s time analyzing and interpreting data that will not be used for program planning or management.

Tables 6.1, 6.2, and 6.3 show examples of some common behavioral domains and determinants that have been proposed to be relevant to HIV prevention.

How to Collect the Data

Once program managers and evaluators have determined the domains that need to be addressed in a measurement tool, they will need to identify how to best collect the pre- and post-test data. Assessment of the selected domains is often conducted through administration of a brief survey before and after the intervention. Surveys can be self-administered (e.g., a paper-and-pencil version) or administered to an individual or group by a staff member.

There are advantages and disadvantages to each type of method. Self- and group-administered surveys are inexpensive and generally quick to administer but are inappropriate for populations who have poor reading skills or are uncomfortable completing questionnaires. When personnel and funds are adequate, the use of an interviewer can address those barriers; an interviewer can also probe and clarify clients’ responses and identify subtleties in responses that might not be detected in pencil-and-paper inventories.

A draft set of questions has been developed by CDC staff that can be used or modified for use by an interviewer for the client to self-administer. These questions were derived from the Core Items for HIV/STD Behavioral Surveillance being developed at CDC (CDC, National Center for HIV, STD and TB Prevention, Behavioral Surveillance Workgroup). The questions can be found in Appendix A at the end of this chapter. In this appendix, the reader can find 1) a set of questions that deal with condom use and sexual risk behaviors and 2) another set dealing with injection drug use and other drug-use risks associated with HIV transmission. In the left-hand column of each set are the questions designed to be used prior to the start of an intervention. The right-hand column contains the related question re-worded for use after the end of the intervention.

Table 6.1

Table 6.2

Table 6.3

Both self-administered surveys and interviewer-assisted questionnaires can be structured or semi-structured. A structured survey or questionnaire includes questions that are predetermined and standardized. They include closed-ended responses that are easily quantifiable and typically pre-coded to facilitate the transfer of data to the computer.

Semi-structured questionnaires combine structured questions with open-ended questions. Open-ended questions are those in which client responses are not limited to a set of alternatives chosen by the evaluators. This allows for standardized collection of a core set of variables and an opportunity to examine client responses in greater depth.

Question Format. When designing instruments to monitor client outcomes, program managers and evaluators need to consider not only methods that will be used to gather data and the domains to be investigated, but the response format as well. The types of question structure and response categories that are available on typical KABB instruments include 1) true-false, 2) multiple-choice, 3) fill in the blanks, 4) Likert-type scales, and 5) frequency indicators.

True-false items typically measure knowledge. They can be scored easily and lend themselves to computer analysis. However, when the same item is used with a client at different times (e.g., before and after the intervention), it increases the possibility that the client has learned the correct answers from the initial test.

Multiple-choice items are those in which an incomplete statement, or item root, is presented and the client selects from three or four responses that would best complete it. As with true-false items, multiple-choice items are easily administered and scored and are ready for computer analysis. There is difficulty in constructing good responses, however. Each possible response needs to appear equally correct to an uninformed client.

A fill-in-the-blank item Drug injection equipment should only be used _____ time(s) to avoid risk of HIV infection.

Fill-in the blank items are ones where key words or phrases are missing and the client is required to complete the statement correctly. The advantages of this response format are the same as those for true-false and multiple choice items. In addition, fill-in the blank items allow you to know what information is readily available to the client, because they must recall–not just recognize–the correct response.

Likert-type scales are often used for many quantitative attitudinal or behavioral measures. For example, the intention item used in the box shown here uses a Likert-type scale for responding. A Likert-type scale has a set of intervals assumed to be equal, with extremes anchored by opposites (e.g., strongly disagree/strongly agree).

The use of Likert-type scales requires a decision about the number of points—called intervals—to include in the scale. In an odd-interval scale (e.g., 1 to 5) the center response category (in this case, “3”) is a noncommittal response such as “unsure” or “don’t know.” This is often considered by researchers to be useless information because the client has not committed to either the positive or negative side of the scale. With even-internal scales (e.g., 1 to 4), the client is forced to choose one side of the scale. Even-interval scales are sometimes referred to as forced-choice scales because clients are forced to choose either the positive or negative end of the scale.

There are advantages and limitations to using Likert-type scales. Generally they are 1) easy to quantify and construct and 2) easily and inexpensively administered. However, the labels attached to the response choices may be confusing to some clients. This can happen when a client has to choose among multiple responses such as “strongly disagree, disagree, and somewhat disagree.” Also, when a series of items uses a Likert-type scale, a client may fall into a pattern of responses rather than reading each item and responding to it separately.

Frequency indicators are used to measure how often behaviors occur. They can be in the form of frequency counts, Likert-type scales, and proportional indicators. The usefulness of frequency indicators depends on an intervention’s data needs and the “client’s capacity to respond to more sophisticated items. If frequency counts require long-term recall (e.g., activities that occurred in the last 6 months), clients may not be able to provide accurate counts. In addition, some clients may have difficulty with the concept of “proportion” outside of 100% or 0%, which can lead to unreliable estimates of frequency.

Other Measurement Issues to Consider for Outcome Monitoring

Good tools to measure client outcomes are based on clear concepts to be measured and the construction of appropriate items to measure these concepts. Such a tool ensures that managers and evaluators can have confidence in the evaluation results.

In addition to the topics that have been addressed, several other issues need to be considered when developing or utilizing a measurement instrument. These include comprehensibility and language, cultural sensitivity, specificity of measures, clarity, and content validity.

Monitoring tools need to include language-appropriate and culturally sensitive items that closely match the reading level of the clients. This is an important consideration when deciding whether to use previously developed KABB instruments. Such instruments may have been designed for target groups of different racial, ethnic, and socioeconomic backgrounds as well as with different reading considerations than clients in the current intervention. Words and terms common to one group may not be readily comprehended by others. For instance, terminology that is used in the gay male community or among injection drug users and included on a measurement tool may be inappropriate for other audiences. The appropriateness of language may also vary from one part of the country to another. For example, among gay men in New York bars, informal terms for sexual behaviors are more commonly used and clearer to them, while in Ohio, gay men are more comfortable with clinically-oriented terms (Mantell, DiVittis, Kochems, and Ostfield, 1989).

As a general rule of thumb, KABB items that are specific are better predictors of outcomes than more general items. For example, if one predicts that partner communication positively affects condom use, then the items should reflect communication related to condom use rather than general communication. Similarly, items related to condom use should specify the type of sexual activity (e.g., vaginal intercourse with a condom, anal intercourse with a condom) rather than any sexual activity in general.

In this same vein, a single item should measure a single thing. Items that combine multiple terms are ambiguous and are often referred to as “double-barreled.” An example is, “Do you believe that you could enjoy sex using a condom with your main partner or a casual partner?” If a client answers “no” it is unclear whether the response is indicating that he/she could not enjoy sex with a condom at all or if enjoyment of sex with only one of those types of partners would be impaired.

A number of the issues that have already been reviewed in this section relate to the importance of establishing the validity of the measurement instrument— that is, the degree to which it measures what it is supposed to measure. In particular, content validity refers to the degree to which the intended domains of the instrument are adequately covered. For example, take the case where the outcome objectives of an intervention include self-efficacy and condom use skills as well as consistent condom use. To establish the content validity, each of these factors (e.g., self-efficacy, skills, etc.) should be measured with items that closely relate to it, and with enough items to represent the full range of the concept. For example, if “condom use skills” include both “how to correctly use a condom” and “how and where to purchase condoms,” then both of these concepts should be represented in items measuring the domain of “skills.”

Pretesting the Monitoring Tool. An important step in developing measurement tools is pretesting them before they are used. This involves administering the tool to a small number of clients who represent the target group. This administration is often followed by discussion groups or individual interviews to obtain feedback on how clear the items were and the degree to which the instrument retrieves information from a particular item. These steps allow for immediate feedback about whether the instrument needs to be revised.

Human Subjects Issues. The measurement tools to assess client outcomes often include items related to highly sensitive topics such as sexual behaviors, drug use, and HIV status. This requires that everyone engaged in the monitoring process take steps to protect the confidentiality of clients. At minimum, the following steps are important (National AIDS Fund, 1995):

• Names, addresses, and other information that identify the clients should not be collected unless the information is essential for re-contacting clients for follow-up purposes.
• identifying information is collected, no one other than project personnel should have access to it.
• soon as possible after data have been collected, identifying information should be destroyed or removed from materials that contain client answers to sensitive questions. If the identifying information must be retained, as in the case of follow-up, the surveys can be assigned unique client identifier numbers. The list linking these numbers to client names and addresses should be stored in a locked place with highly restricted access. Furthermore, the surveys should be stored separately.
• All members of the assessment team should be carefully trained in the importance of confidentiality and their responsibility to comply with the procedures that are in place for clients’ protection.

Borrowing From Instruments Used in the Field. Research projects evaluating the effectiveness of HIV prevention interventions are good sources of questionnaires that contain behavioral measures that are typically administered at least once before and once after an intervention with a client. Several federal agencies have funded such projects; these include CDC, the National Institute on Drug Abuse (NIDA), and the National Institute of Mental Health (NIMH). Examples of the questions used on some of the projects they have funded are shown on the following page (Hasin, 1994; Joe & Simpson, 1993; Metzger, 1995; NIDA, 1993). Appendix B contains information for contacting the investigators for more information about these questionnaires.

Condom Use with Steady Partners

Condom Use with Casual Partners or with Any Partners

Adapted from NIDA, National AIDS Research Project

Clean Needle and Works Use

Several states and HIV prevention programs are conducting HIV prevention outcome monitoring using instruments developed in the field. Two locales that have formally developed outcome monitoring procedures are San Francisco and Colorado. The San Francisco project was part of a 5-year Strategic Evaluation Plan developed by the San Francisco HIV Prevention Planning Council (HPPC). This plan outlines specific objectives for conducting a proficient Behavioral Risk Assessment with intervention clients. The assessment instrument includes standard demographic and risk behavior variables approved by the HPPC as well as site-specific variables.

In Colorado, an evaluation project is assessing the impact of HIV prevention program delivery on clients’ risk behaviors and intentions to change risk behaviors. This project targets individuals who utilized prevention services in the Denver metropolitan area and the more rural northeast quadrant of Colorado as well as those who did not. In this circumstance, the assessment instrument is not being used for pre- and post-intervention measurement. Instead, it will be used more as a population-based survey to assess changes throughout these areas over a 4-year period.

For more information on these evaluation projects and the instruments used, please see Critical Issues in HIV Prevention Evaluation (AED, September 1997).

Back to top

When to Monitor Outcomes

As noted earlier, determining whether outcomes have occurred requires at least one measure taken before the intervention occurs and another taken after it. Assuming that the behavior of each person receiving individual or group counseling warrants some improvement (that is, each person really needs the counseling), the pre-intervention measurement describes the behaviors, behavioral intentions, knowledge, attitudes, and beliefs that need to be enhanced. This is the baseline or reference point against which improvement can be measured. The post-intervention measurement is the source of data about the extent to which the behaviors, intentions, etc., have changed since initial data collection. The difference between the pre- and post-intervention measures is the amount of change that occurred during the intervention period.²

Pre-Intervention Measures

It is important that a pre-intervention measure reflects the client’s characteristics right before receiving the intervention. Remember that the purpose of the pre-intervention measure is to establish a reference point for assessing behavior change that program staff believes is due to the intervention (see Footnote 2 for a caution concerning this interpretation). As the time between the pre-intervention measurement and the intervention increases (e.g., as it gets to be 1 or 2 weeks), one can have increasingly less confidence that the measurement accurately reflects the client’s characteristics when he or she began the intervention.

In practice, ensuring that pre-intervention data are current usually means collecting these data just prior to the first or only session of the intervention. Depending on the type of instrument and method of administration chosen, this may take anywhere from 10 to 30 minutes before the intervention begins. In many cases, clients will not be available much before (or after) the intervention. Therefore, at sites where this is the case, outcome monitoring needs to be conducted while clients are on-site.

Follow-up Measures

One post-intervention measurement is a minimum expectation for outcome monitoring. However, there are advantages to collecting data at two or more points following the intervention. Generally, however, only one post-intervention measurement will be feasible. It is important to understand the implications of single post-intervention measurements as well as the benefits of and barriers to multiple measurements.

Single Post-intervention Measures. When single post-intervention measurements are used, the timing of their administration determines 1) what change can be reasonably expected in that time frame and 2) what interpretation can be made about the findings. In general, the less time between the end of the intervention and the post-intervention measurement, the less opportunity there will be for behavior change to occur. Therefore, one must balance the interest in actual behavior change with the reality of clients’ availability to respond to questionnaires after the intervention and the ability of the provider to locate clients.

For example, when the post-intervention measurement is made at the end of a one-session counseling intervention, there is no opportunity for the client to have engaged in a behavior— safely or unsafely. Therefore, it may be more reasonable to ask clients about their intentions to change their behavior. However, if the counseling consisted of multiple sessions over a longer time period, it might be reasonable to ask about behavior change. It is still appropriate in this situation to ask about intentions as well as about actual behavior change.

The following timeline is an example of an intervention with single pre- and post-intervention measures administered on the same days as the intervention activity.

Multiple or Delayed Post-intervention Measurements. The goal of an intervention is change, and change often takes time, particularly when it involves overcoming well-developed habits. It may also take time to observe a behavior change because the person may have limited opportunities to engage in the target behavior. Also, the specific objectives that have been stated for a particular intervention may be long-term objectives that are not expected to be achieved until 6 or more months after the end of the intervention. Therefore, a critical aspect of outcome monitoring is ensuring that there has been an opportunity for behavior to occur before attempting to measure a change in it.

There are two basic options for capturing information about behavior occurring after the intervention. The first is to administer multiple post-intervention measurements; the second is to administer a single post-intervention measurement after some time has passed. If multiple measurements are chosen, one of them is typically administered after the last session. The others (usually only one or two additional measurements) can be implemented at various times, depending on the logistics of the program and what information is sought. For instance, to know about the immediate or short-term effects of the program, one may want to have a post-intervention measurement at 2 weeks after the last session of the intervention. One option is to administer the next measurement after 1 month to give more time for one or more instances of the behavior to occur. A 6-month measurement can indicate whether the effects of the intervention last over time. These same time frame considerations hold true for delayed single post-intervention measurements.

The timelines on the following page give examples of some of these alternatives.

Back to top

Managing Outcome Monitoring Data

For outcome monitoring data to be useful to program managers, it must be accessible and useful. As with process data, this entails the development and maintenance of simple record-keeping systems. Also, in Chapter 8, Evaluating Outcomes and Monitoring Impact of HIV Prevention Programs, the importance of data systems becomes even more critical; development and implementation of basic systems prior to contemplating full-scale outcome evaluations will facilitate response to this need.

Keeping the data in a simple, standardized format makes it easier to enter and use. The commercial database and statistical software packages available make it very easy to create a computerized database; most have tutorials that can quickly walk a person through the steps needed to create a database for the minimal number of items expected for outcome monitoring. Also, having the data in one of these computerized formats will make the simple data analysis needed quick and painless.

Preparing the Data

To do the analysis, data will need to be effectively entered into the database. To facilitate data entry, it is best to precode the instrument used. Precoding means that a certain number or letter is printed on the instrument near each response that clients can choose. Using an example shown earlier, consider the following question:

There are five possible responses. The numbers next to each verbal response are the precodes. These numbers allow for easy, accurate transfer of data from the instrument into the database.

Data cleaning is another important step that should be completed before data are entered into the computer. This entails reviewing the completed instruments to determine the extent to which data are missing, whether there is a response pattern, or if the responses are illegible. In the case of missing data or illegibility of responses, there is a possibility that the client can be re-contacted to complete a section of the instrument or clarify something that was illegible. However, if this is not possible and there are extensive problems with the completed instrument, it may need to be thrown out.

After the data have been entered it is best to conduct a frequency analysis. The output from this analysis will list each of the variables (i.e. each question asked) and the number of each response to that variable. If there were 20 clients who responded to a 5-point Likert scale item, the frequency distribution of their responses might look like this:

This frequency distribution shows how many chose each of the possible choices on the scale. Note also that, despite the fact that the only legitimate responses to the question were 1, 2, 3, 4, or 5, one “6” and one “9” were entered for this question. This may mean that the respondents chose an incorrect response or that the person responsible for data entry mis-entered the numbers. In either case, this will need to be resolved before the data are analyzed.

Collection of Client Social and Demographic Data

It is necessary to collect some social and demographic data in the context of process monitoring (e.g., age, gender, race/ethnicity); by doing so, these data will be available for analysis in conjunction with outcome monitoring data for each client. For instance, typical social and demographic variables that can help in the interpretation of outcome monitoring include marital status, number of children, educational attainment, prior exposure to prevention services, and other related risk behaviors. Linking clients’ characteristics and outcome measures provides information that may help program staff determine which subgroups are better served by particular interventions (e.g., younger clients may be more motivated than older clients by an intervention based on social norms).

Back to top

Analyzing Outcome Monitoring Data

Once pre- and post-intervention data have been gathered on client outcomes and, perhaps, on the determinants of these outcomes, these data will need to be analyzed. The results of data analysis should allow program staff and program evaluators to answer the question: “Have we achieved our outcome objectives?” There are three steps in the analysis of this question:

• Compare the characteristics of those who completed the post-intervention measure with those who did not (assessing drop-out or attrition);
• Compare the combined pre-intervention scores for everyone receiving the intervention with the combined post-intervention measures to determine if, overall, the intervention is reaching its goals; and
• Determine whether particular characteristics of the clients or the program are associated with different levels of outcomes (efficiency analysis).

Analysis of Drop-Outs

In the context of outcome monitoring analysis, the primary analyses will be conducted for those people who have completed both pre- and post-intervention outcome measures. However, some participants will not complete the intervention, while others, for various reasons, might not complete the post-intervention measure; this last group is the one to focus on first.

It is critical that some assessment be made to determine if the group of clients for whom there are at least two outcome measures differs in any significant way from the group of clients who do not have follow-up measures. If an agency reports on only those people from whom they have both pre- and post-intervention measures, they may be misrepresenting the “true” effectiveness of the intervention. For example, consider a four-session individual counseling intervention for female sex partners of IDUs. After the second session, 25 of the 100 women decide that the risks they are taking by having unprotected sex with their partners are worth it (to feel like their relationship is intimate and special, to keep the man from leaving, etc.) and they stop participating in the intervention. Of the 75 women who complete the intervention, 50 report significant changes in their behaviors. If data from only those 75 women were analyzed, the intervention may seem very effective. However, if there were data from the 25 who left after the second session, the analysis and interpretation of the data may be very different (50 changed, 50 did not). If comparisons of the 25 and 75 were made using pre-intervention data, they may show that the 25 who left had engaged in more high-risk behaviors than those who stayed. Or, the comparison might suggest that the women who stayed with the intervention had fewer children that their partner was helping support. Each of these findings suggest ways to maximize retention rates.

In these cases, one might conclude that the intervention was effective for some women whose partners are IDUs and not others. Another conclusion may be the need to tailor the content to address the drop-outs’ concerns or to identify strategies to retain them in the intervention as configured. This information is very important to program managers and stakeholders, as well as to community planning groups attempting to reduce the risk of HIV infection in their communities.

Analysis of Outcome Monitoring Data

The primary purpose of analyzing outcome monitoring data is for program staff and evaluators to answer the question: “Do clients make progress toward their goals and outcomes after receiving the intervention?” This involves a comparison of the mean (i.e. average) pre-intervention scores for everyone receiving the intervention to the mean post-intervention scores for that same group. This comparison allows one to determine if, overall, the intervention is reaching its goals.

A simple data analysis for monitoring generally involves the following steps:

1. Select clients who receive the intervention and complete the measurement instrument before and after the intervention.
2. Calculate the mean scores for pre-intervention and the mean scores for post-intervention.
3. Conduct a paired (or matched) t-test to determine if the post-intervention scores are significantly different (i.e. improved) after receiving the intervention. The paired t-test is a simple statistical test that uses two scores from the same individual, as when collecting pre- and post-intervention data in outcome monitoring.

If the data analysis shows that there are no significant differences between pre- and post-intervention scores, this is sufficient information to suggest that the program needs some changes to improve its effectiveness. There is no need to carry out a rigorous outcome evaluation to reconfirm the ineffectiveness.

However, there is a problem in interpreting the results when this analysis shows that the post-intervention mean score is significantly greater than the pre-intervention score. This is an encouraging finding and its prudent use is warranted, especially when the intervention has a strong scientific basis and experience and context supports its continued use. However, one cannot confidently attribute the changes to the intervention without the use of a more rigorous design that controls for other possible sources of improvement (e.g., participation in other interventions, maturation, etc.). This issue will be discussed in greater detail in the next chapter.

Efficiency Analysis

When demographic, social, and other contextual data are available, the analysis of outcome monitoring data can be taken one step further. Such data allow a program to assess which sub-groups receiving the intervention do better (or, conversely, which need special help in attaining the goals of the intervention). An efficiency analysis would follow steps similar to those described below.

1. Select social demographic variables such as sex, ethnicity, and age that program staff or other stakeholders are interested in.
2. Divide the intervention group participants into two or more groups based on social or demographic variables (e.g., younger than 25, 26 to 34 years old, 35 years old and older).
3. Calculate mean pre-intervention and post-intervention outcome scores for each subgroup.
4. Use statistical techniques such as the t-test, F-test, or covariance analysis to analyze group differences and determine whether the mean differences among groups are statistically significant.
5. Examine the difference between the mean score before and after the intervention to determine whether the particular group is improving.

Back to top

References and Resources

Academy for Educational Development. Critical Issues in HIV Prevention Evaluation. Washington, DC: Academy for Educational Development, 1997.

Campbell, D. T., & Stanley, J. C. Experimental and Quasi-experimental Designs for Research. Chicago: Rand McNally, 1963.

National Center for HIV, STD and TB Prevention, Behavioral Surveillance Workgroup. Core Items for HIV/STD Behavioral Surveillance. Atlanta, GA: Centers for Disease Control and Prevention, April 12, 1999.

Chen, H-T. Theory-Driven Evaluations. Newbury Park, CA: Sage, 1990.

Cook, T.D., Campbell, D. T. Quasi-experimentation: Design and Analysis Issues for Field Settings. Chicago: Rand McNally, 1979.

Coyle, S., Boruch, R., Turner, C. (eds.). Evaluating AIDS Prevention Programs. Washington, DC: National Academy Press, 1991.

Fishbein, M., Bandura, A., Triandis, H., Kanfer, F., Becker, M., & Middlestadt, S. Factors influencing behavior and behavior change. Final Report–Theorist’s Workshop. Washington, DC: National Institutes of Mental Health, 1991.

Mantell, J. E., DiVittis, A.T., Auerbach, M.I. Evaluating HIV Prevention Interventions. Plenum Press: New York and London, 1997.

Prochaska, J. O., DiClemente, C. C. Stages of change in the modification of problem behaviors. Progress in Behavior Modification 1992;28:183-218.

Schalock, R. Outcome-Based Evaluation. Plenum Publishing: New York and London, 1995.

Back to top

Appendix A: CDC Draft Outcome Monitoring Questions

Condom Use and Sexual Behavior Risks
Injection Drug Use and Other Drug-Related Risks

Sample Demographic Items to Be Used with Outcome Monitoring Questions

CDC Draft Outcome Monitoring Questions

Appendix B

Joe, G. W., & Simpson, D. D. (1993). “Needle use risks: Composite measures and comparisons.” In B. S. Borwn, G. M. Beschner, & the National AIDS Research Consortium (Eds.), Handbook on risk of AIDS: Injection drug users and sexual partners (pp. 137-154). Westport, CT: Greenwood Press.

Dwayne Simpson, Ph.D.
Institute of Behavioral Research, Texas Christian University
http://www.ibr.tcu.edu/pubs/datacoll/AIDSRisk.html

Richard Needle, Ph.D.
National AIDS Research Project
National Institute on Drug Abuse
Community Research Branch

Deborah S. Hasin
Associate Professor of Clinical Public Health (Epidemiology) in Psychiatry
NYSPI Unit #123
1051 Riverside Drive
NY, NY
dsh2@columbia.edu

Dave Metzger, Ph.D.
Director of Opiates and AIDS Research Division
University of Pennsylvania Center for Studies of Addiction
Philadelphia, PA 19104
metzger@research.trc.upenn.edu

Back to top

Go to Evaluating Outcomes and Monitoring Impact of HIV Prevention Programs