Reasons for Conducting Process Evaluation
Types of Process Data to Collect
Collecting and Analyzing Process Monitoring Data
References and Resources
Appendix A: Example Forms to Summarize Process Data
Appendix B

In the last chapter, we suggested that an intervention plan is like a blueprint for a house: a clear and logical plan is an essential guide for building a functional and sturdy structure. However, unless the carpenters, plumbers, and electricians follow the blueprint carefully, they will not build the house expected.

The same is true for the processes that comprise the implementation of HIV prevention services. Because HIV prevention programs are complex, there are many reasons they may not be implemented as designed. The purpose of implementation evaluation, also referred to as process evaluation, is to objectively examine the qualities of intervention implementation and improve it if necessary.

Purpose of this Chapter

This chapter supplements Announcement 99004, which states that health departments should engage in “ongoing data collection and monitoring regarding the implementation of health department and health department-funded program activities” (CDC, 1998). To that end, this chapter will provide an understanding of the usefulness of process evaluation and guidance for conducting process evaluations of health department-implemented and -funded HIV prevention interventions. The chapter 1) discusses reasons for conducting process evaluation and types of data needed, 2) provides information about collecting and reporting process evaluation data, including the core elements to be reported in the aggregate to CDC for assessments of national progress in HIV prevention and determination of ongoing technical assistance needs, and 3) provides sample data collection forms for use by health departments as they collect information about HIV prevention interventions that they and their grantees implement. A typical set of steps in the preparation, collection, management, and analysis of process monitoring data is shown in Figure 4.1.

For the purposes of this guidance, the term “monitoring” is used to refer to the routine documentation of characteristics describing the target population served, the services that were provided, and the resources that were used to deliver those services. Monitoring is contrasted here with “process evaluation” in which the data collected through process monitoring are used to answer more detailed questions about implementation (e.g., If the intervention did not conform to its design, what factors contributed to that difference?), the clients served (e.g., Did we reach the group at most risk? How could we improve our coverage of them?), and resources used in the delivery of the program.

Health Department Receives CDC 99004 Award
HD funds subcontractors to provide HIV prevention services (e.g., through an RFP)	HD uses funds to implement interventions by HD staff
As part of the planning for the implementation of each intervention, data collection, management, analysis, and feedback systems are established to meet the provider’s and health department’s needs
Interventions are implemented
Data are collected consistently throughout the budget year
Data are analyzed and used by the provider to monitor and improve service provision
At end of budget year, each provider submits the year’s process data for each intervention to the HD	At end of budget year, the HD compiles the year’s process data for each intervention its staff implements
The health department submits process monitoring data to CDC

Assessment of implementation can be evaluated on its own merits as well as compared with the intervention plan, which describes the intended objectives and steps of the intervention. Process evaluation data do not address the extent to which an intervention has achieved its desired outcomes (e.g., less risky behavior or attitudes and beliefs that support those behaviors). The evaluation of an HIV prevention intervention’s outcomes will be the focus of the next two chapters.

Typical Issues Addressed by Process Evaluations

• Appropriateness of the program for the intended participants
  • What needs to be in place for the intervention to work?
• Type and numbers of treatments and services provided
  • How is the intervention actually implemented?
  • How much effort was needed to achieve a given outcome (labor hours, materials distributed, shots administered, etc.)?
• Means of optimizing access to the intervention, including location and physical facilities of the service delivery site
• Participant retention, referral, and follow-up efforts
• Qualifications and competencies of staff

Back to top

Reasons for Conducting Process Evaluation

The Links Between Intervention Plans and Outcomes

In the last chapter we suggested that it was reasonable to expect a good intervention plan to lead to desired outcomes (e.g., fewer risk behaviors) and impacts (e.g., less HIV transmission) if that design is implemented as it was proposed. Until such implementation occurs, the relationship between the intervention plan and outcomes is only hypothetical. This hypothetical model was described in Chapter 3 and is shown again in Figure 4.2.

Figure 4.2. The relationship between the intervention plan and HIV prevention results is only hypothetical.

HIV Prevention Intervention Plan

Behavioral Risk Reduction for HIV Prevention

The real test of the intervention plan— Will the intervention achieve its objectives?— is in the “nuts-and-bolts” of intervention implementation. This issue can be framed simply by the question, “Did we really do what we said we were going to do?” Figure 4.3 illustrates this mediating role. Process evaluation is the way to assess the mediating role of intervention implementation. This is crucial, because focusing only on the intervention plan and outcomes can result in misleading interpretations of how certain outcomes (be they promising or disappointing) came to be.

Figure 4.3. Mediating role of intervention implementation.

HIV Prevention Intervention Plan

HIV Prevention Intervention Implementation

Behavioral Risk Reduction for HIV Prevention

Benefits of Process Evaluation

In general, process evaluation provides health departments, service providers, and CDC with information about whether the program has reached its intended audience, the level or extent of services provided, and what resources were required to support the prevention effort made. Table 4.1 outlines several distinct benefits that process evaluation generates for local, state, and federal stakeholders. The characterization of benefits of process data suggest three important purposes of process evaluation:

1. Providing information for improving intervention implementation
2. Providing a context for understanding intervention effectiveness
3. Meeting accountability needs

Purpose 1—Providing Information for Improving Intervention Implementation. Process evaluation consists of a set of procedures that can provide timely information for improving implementation by identifying a program’s strengths and weaknesses. For example, process data may indicate that an outreach program is failing to reach its target population because the times that outreach specialists are in the neighborhood are the same times that target population members are working, buying drugs, socializing, etc. A program manager can use this information to identify times of day during which potential clients would be willing to talk and to schedule her staff’s outreach activities accordingly.

It is essential to keep in mind that the driving force behind all evaluation is to optimize the effectiveness of HIV prevention services. It is hard to imagine an HIV prevention program whose developers and staff do not believe that they are offering helpful assistance to people with behaviors that put them at risk for contracting HIV. It also is reasonable to assume that people who want to help will also want to continually improve their capacity to help. However, even the best interventions require oversight and monitoring so that they can provide the most efficacious services to the most people and use resources efficiently.

Purpose 2—Providing a Context for Understanding Intervention Effectiveness. One underlying assumption of HIV prevention community planning is that resources will be directed to interventions with known effectiveness. For instance, one commonly hears about “looking to the scientific literature” for interventions with proven track records for achieving desired outcomes. However, for this approach to help a new provider, an intervention chosen for its known efficacy must be implemented the way its developers intended. That is, “effectiveness” can be claimed only when one knows which intervention was intended to be delivered and which intervention actually was delivered.

Process evaluation data allow evaluators to distinguish an ineffective intervention from one that is ineffectively implemented. This is a critical distinction when assessing how best to serve a population and when allocating resources for various interventions. By ensuring that interventions are carried out in keeping with their design, a health department can have increased confidence attributing changes in HIV transmission in its jurisdiction to interventions it funded.

Despite the known benefits of process evaluation, many decision makers continue to believe that the only valuable measure of a program is to conduct outcome evaluation to determine its effectiveness in achieving outcome objectives. In fact, as is discussed in the next chapter, some HIV prevention programs are unsuitable for outcome evaluation. For these programs, process evaluation may be the only— and therefore the most critical— means of ensuring accountability. In light of such considerations, Chen (1994) contends that asking “How does the program achieve its goals?” is almost as important as asking “Does the program achieve its goals?”

Purpose 3— Meeting Accountability Needs. It is important to note that process evaluation serves an accountability function. Because the HIV prevention efforts discussed in this document utilize public funds, CDC, health departments, and service providers have obligations to provide stakeholders with answers to basic questions about programs. A variety of federal policymakers in the Executive and Legislative branches and numerous state and local officials regularly demand information about what HIV prevention services are being provided. Process data serves stakeholders at all levels by informing them about the nature of HIV prevention efforts and documenting whether the efforts are heading in desired directions.

HIV prevention interventions must be accountable to their stakeholders in terms of two aspects: 1) the quality of implementation (measured through process evaluation) and 2) the effectiveness of the intervention (measured through outcome monitoring and outcome evaluation). One devaluing myth about process evaluation is that it is a “kinder and gentler” type of evaluation that yields no information for making difficult judgements about a program. However, process evaluation addresses management and operational issues that are critical to program managers, administrators, and funders (Chen, 1994).

Back to top

Types of Process Data to Collect

In this and the previous chapter, the range of typical HIV prevention interventions has been discussed. Because process monitoring and evaluation assess the specific activities that comprise each type of intervention, the relevant process evaluation questions will vary by the kind of intervention in question. Similarly, the specific data elements needed are determined based on the nature of the process evaluation question.

The following tables show illustrative process evaluation questions, client-level data elements, and program-level data elements for the following types of interventions:

Individual- and Group-Level Interventions

Outreach

Prevention Case Management

Counseling, Testing, Referral, & Partner Counseling & Referral Services

Mass Media

Hotlines

Clearinghouses

Media Advocacy

Materials Distribution

Needle Exchange

Back to top

Collecting and Analyzing Process Monitoring Data

This chapter has articulated some of the value and benefits that can accrue through the collection and use of process data for HIV prevention interventions. In conjunction with the goals and objectives set forth in intervention plans, process data can help providers and funders assess achievement of desired results, pinpoint areas for refinement or improvement, and convey to stakeholders the successes that have been achieved (as well as the need for additional resources to meet remaining needs).

The previous chapter, Designing and Evaluating Intervention Plans, contained the beginning of a discussion of data collection, management, and analysis systems needed by health departments and local HIV prevention providers. Health departments need management information systems that will allow them to understand the technical assistance needs of their subcontractors, use data for program improvement throughout the jurisdiction, and use data for quality assurance and accountability purposes. Local providers also need data systems that help them derive information that can be used for these purposes; however, local providers must consider the logistics of data collection “on the front lines.”

Some front-line providers may have reservations about expending resources to collect data when the resources could be used to offer more prevention services. Similar reservations may be held by health department staff. Two questions seem to underlie their concerns and require responses to increase buy-in regarding evaluation:

1. Will the data be used and useful?
2. How much effort will it really take to collect these data?

First, program staff, health department staff, and technical assistance providers should demonstrate that the data being requested will, in fact, be put to use for important purposes. “Important” purposes for staff may mean that the data will be used and beneficial to them, their work, and the people they are trying to serve. An important message to convey is that it is by examining data in a systematic way that providers can ensure that prevention efforts are really helping the people they want to help. Without such data, providers may “feel good” about their efforts but cannot be assured that they are making a difference.

The message to staff about the importance of using data is also conveyed by deed. Putting a process in place to review and interpret data and to make needed changes is the concrete manifestation of the message. Once staff members see that the agency is serious about using data, their concerns about collecting “data for data’s sake” may diminish.

Second, the effort required to collect data should be gauged with respect for the situations, settings, and provider skills present with each intervention. Virtually every institution that is involved in data collection has been guilty at one time or another of asking for more data than it will eventually use. In this context, health departments and local providers should be mindful of the anticipated uses— and users— of the data they ask for. Users include providers, the health department, and CDC.  An assessment should be made of the common and the unique needs of each of these users. For instance, health departments may need additional data that would not be as useful at the national level, but which may be critical for their jurisdiction. Additionally, local providers may have needs for more detailed information about their staff’s activities that goes beyond what the health department may be interested in.

Data collection must be also be feasible for the setting in which the intervention occurs. For instance, the structured settings in which most individual- or group-level counseling interventions occur (e.g., clinics or agency offices) may allow time for data collection each time an intervention occurs.  Similarly, the very nature of prevention case management seems to require record keeping to allow tracking of the service plan for each client and the types of services that each client receives. With respect to outreach, the dynamic, unstructured nature of some street or community outreach may preclude the recording of extensive data about each encounter. Instead, the agency may decide to focus on a subset of critical variables that meet some threshold of importance and to forego other less important— albeit, interesting— data. On the other hand, many providers have found it manageable to collect a great deal of data about outreach encounters; consultation with peer agencies may help a provider determine what is feasible for staff members with similar experience and training.

Appendix A contains example forms that may be used to summarize process data from each of the following types of interventions:

• Individual-level interventions
• Group-level interventions
• Outreach
• Prevention Case Management
• Partner Counseling and Referral Services
• Health Communications and Public Information Activities
• Community-level interventions

Appendix B contains the standard HIV Counseling and Testing Report Form that has long been used for collecting and reporting data about counseling and testing. This appendix also contains the guidelines to be used when filling out this report form.

Back to top

References and Resources

Centers for Disease Control and Prevention. Planning and Evaluating HIV/AIDS Prevention Programs in State and Local Health Departments: A Companion to Program Announcement #300. Atlanta: Centers for Disease Control and Prevention, 1993.

Centers for Disease Control and Prevention. Planning and Conducting Street Outreach Process Evaluation. Atlanta: Centers for Disease Control and Prevention, 1994.

Centers for Disease Control and Prevention. Guidelines for Health Education and Risk Reduction Activities. Atlanta: Centers for Disease Control and Prevention, 1995.

Centers for Disease Control and Prevention. HIV Prevention Cases Management: Guidance. Atlanta: Centers for Disease Control and Prevention, 1997.

Chen, H.T. Current trends and future directions in program evaluation. Evaluation Practice, 1994;15(3):229-238.

Chen, H.T. A comprehensive typology for program evaluation. Evaluation Practice, 1996;17(2):121-130.

Corby, N. H., and Wolitski, R. J., eds. Community HIV Prevention: The Long Beach AIDS Community Demonstration Project. Long Beach: The University Press, California State University, 1997.

The Health Communication Unit, Centre for Health Promotion, University of Toronto. Evaluating Health Promotion Programs, no date.

Mantell, J. E., DiVittis, A. T., and Auerbach, M. I. Evaluating HIV Prevention Interventions. New York: Plenum Press, 1997.

National Community AIDS Partnership. Evaluating Prevention Programs in Community-Based Organizations, 1993.

National Minority AIDS Council. The Program Development Puzzle: How to Make the Pieces Fit, 1997.

National Research Council. Evaluating AIDS Prevention Programs, Expanded Edition. Washington, D.C.: National Academy Press, 1991.

U.S. Department of Health and Human Services. Making Health Communication Programs Work: A Planner’s Guide. NIH Publication No. 92-1493. Rockville, MD: US Department of Health and Human Services, 1992.

Back to top

APPENDIX A: Example Forms to Summarize Process Data

• Individual-Level Interventions
• Group-Level Interventions
• Outreach
• Prevention Case Management
• Partner Counseling and Referral Services
• Health Communications and Public Information Activities
• Community-Level Interventions
• Counseling and Testing

Back to top

APPENDIX B

• HIV Counseling and Testing Report Form
• Guidelines for Completing Report Form

Back to top

Go to Evaluating Linkages between the Comprehensive HIV Prevention Plan and Resource Allocation