Background
Conceptual Framework for HIV Prevention and Evaluation
Contents of This Document
References and Resources

Background

Increasingly, federal, state, and local agencies involved in HIV prevention are recognizing the importance of evaluation for two primary purposes: 1) to determine the extent to which HIV prevention efforts have contributed to a reduction in HIV transmission and 2) to be accountable to stakeholders by informing them of progress made in HIV prevention nationwide. In response to this, CDC has identified the types of evaluation data needed to be accountable for use of federal funds and to conduct systematic analysis of HIV prevention to improve HIV prevention policies and programs. Evaluation data that are needed include the types and quality of HIV prevention interventions provided by CDC health department grantees and their grantees, the characteristics of clients targeted and reached by the interventions, and the effects of interventions on client behavior and HIV transmission.

These data needs correspond to CDC’s conceptual framework for HIV prevention and evaluation (Figure 1.1), which sets forth the six evaluation activities that health departments receiving CDC funding for HIV prevention might implement to achieve a comprehensive understanding of the status of HIV prevention in their jurisdictions.

The purpose of this document is to provide CDC health department grantees with resources for implementing all six types of evaluation. While only some of these have been put forth as requirements CDC (as outlined in Volume 1: Guidance and requested in CDC Announcement 99004), all of the activities are beneficial to the successful assessment of progress in HIV prevention. In fact, grantees are encouraged to supplement the activities described herein with additional evaluation activities tailored to local needs.

Variation in Evaluation Capacity Across Jurisdictions

This document takes into account the diversity of health departments’ capacities and experience in evaluation and seeks to provide assistance and references useful across the spectrum of grantee experience. For instance, health departments that already have designed and engaged in many of the evaluation activities may find the guidance useful in determining how to use existing data mechanisms to report the minimum set of data asked for by CDC rather than duplicate efforts by creating a new system. For health departments initiating some or all of the evaluation activities for the first time, this guidance provides basic evaluation information, recommendations for developing evaluation systems, and references to other evaluation texts and resources. CDC also is developing mechanisms for providing technical assistance, training, and other support to grantees as they implement evaluation activities.

Figure 1.1 CDC’s Conceptual Framework for HIV Prevention and Evaluation

Participation in Evaluation Activities Not Required by CDC

CDC health department grantees are strongly encouraged to engage in evaluation activities that exceed the expectations of this Guidance. Staff members from the Program Evaluation Research Branch are available to discuss ideas and projects. In some cases, special studies might be arranged in collaboration with the Division of HIV/AIDS Prevention staff. For further information or to initiate discussions about such projects, please direct your correspondence to:

Chief
Program Evaluation Research Branch
Division of HIV/AIDS Prevention--Intervention, Research, and Support
1600 Clifton Rd.
Mailstop E-59
Atlanta, GA 30333

Limitations of this Document

By design, this document describes evaluation activities that are presented primarily for the purposes of collecting and reporting data that correspond to CDC’s conceptual framework for HIV prevention evaluation. Following are several limitations of the information provided in this document:

• This document is not intended to be a comprehensive text on how to conduct evaluation; rather, it is designed to assist grantees in responding to CDC recommendations and requirements regarding the evaluation of HIV prevention interventions supported with CDC funds.
• The methods suggested herein are sufficient for all CDC accountability purposes and for some program improvement, but they are not sufficient for use by health departments in the thorough evaluation of all their HIV prevention activities. Grantees are strongly encouraged to supplement the data with data gleaned through other evaluation activities.
• This document does not explain how health department grantees should use the data for program improvement; grantees should consult evaluation texts and experts for assistance in this area.
• This document provides guidelines for evaluating most but not all types of HIV prevention interventions.

Development of this Document

This document was developed by the Program Evaluation Research Branch in CDC’s National Center for HIV, STD, and TB Prevention, Division of HIV/AIDS Prevention—Intervention Research and Support, with the assistance of Macro International, Inc. The branch conducted literature reviews and solicited materials from more than 50 individuals representing health departments, CBOs, evaluation consulting firms, and academic institutions. Through this process, CDC determined that the underlying principles of this document would be the following:

• Evaluation activities discussed should reflect the critical steps in HIV prevention planning, implementation, and assessment of effectiveness.
• Data collected through implementation of the activities outlined in this document will supplement qualitative data collected through case studies and other local evaluation activities.
• Recommended data to be collected and methods for collecting them should meet the requirements of scientific rigor, usefulness, and practicality.
• Evaluation activities should consist of a balance between process and outcome evaluation.
• The evaluation activities and discussions about them should set forth the minimum amount of evaluation recommended rather than be the gold standard for a comprehensive evaluation of HIV prevention.

The first draft of the guidance was provided to more than 20 CDC staff members, state AIDS directors, state and local health department staff members, HIV prevention provider staff members, and expert evaluators. Reviewers provided comments to CDC in writing as well as during a meeting on June 4-5, 1998. Written and verbal comments from all reviewers were incorporated into a revised version of the draft, which was then provided to state AIDS directors before they met with CDC on July 20-21, 1998, to discuss the content and format. Their comments led to additional changes, which were discussed at a NASTAD-wide conference on August 3, 1998. A meeting with CBO representatives on October 5, 1998, contributed to efforts by shedding light on the issues affecting health departments’ grantees. A final meeting with NASTAD representatives on May 6, 1999, led to the decision to employ this document as a resource book that complements requirements and activities featured in CDC’s companion document, Evaluating CDC-funded Health Department HIV Prevention Programs—Volume 1: Guidance and CDC Announcement 99004.

Comments received during and following all meetings guided the finalization of this document and completion of forms for collecting HIV prevention community planning, process evaluation, and outcome monitoring data. The forms were then pilot-tested by six CDC-grantee health departments, whose feedback was invaluable in the improvement of the forms and data collection instructions. Additionally, five states provided CDC with in-depth information about their evaluation data systems; this provided further insight regarding current evaluation activities and implementation issues.

Back to top

Conceptual Framework for HIV Prevention and Evaluation

The HIV prevention community planning initiative is a participatory planning process intended to document jurisdictions’ HIV prevention needs, prioritize the needs, and identify ways for meeting them. The overarching intent of the initiative is to lay a foundation for the funding and implementation of interventions that are expected to reduce HIV risk behaviors among target populations and, eventually, to reduce HIV transmission in each jurisdiction.

The framework for evaluation (see Figure 1.1) is based upon a conceptual framework referred to as the theory-driven evaluation perspective (Bickman 1990; Chen 1990; Chen and Rossi 1992; Weiss 1997), which holds that a successful evaluation of an initiative or program needs to systematically examine the crucial components and the linkages between them.

The eight components of HIV prevention programming shown in the framework comprise the theoretical progression of events necessary to create a comprehensive HIV prevention program. Each oval represents an evaluation activity designed to evaluate one or more of the components (as delineated by the arrows). The components are described in the following paragraphs, while the evaluation activities are summarized in Table 1.1 and in the chapter descriptions at the end of this chapter.

Component 1. All state, territorial, and city health departments that receive funding from CDC use the monies to form and maintain one or more HIV prevention community planning groups (CPGs) as required by CDC’s HIV prevention community planning initiative, introduced in 1994. The CPG is to be representative of the epidemic in the health department’s jurisdiction and, according to the requirements of the initiative, is to implement five core objectives (Table 1.2). The CPG draws on an epidemiologic profile, a needs assessment, behavioral science, community norms, and other data to determine HIV prevention needs and identify interventions for responding to them.

Component 2. The CPG develops a comprehensive HIV prevention plan based on prioritized needs and interventions.

Component 3. The health department incorporates elements of the plan into its annual application for CDC funding. CDC reviews the application and awards funds to the health department, which draws on the funds for HIV prevention planning activities as well as for HIV prevention interventions.

Table 1.1

Table 1.2

Component 4. The health department, CBOs, and other provider agencies develop intervention plans that respond to needs identified in the comprehensive HIV prevention plan as well as those identified by the health department.

Component 5. The health department reviews proposals (which contain intervention plans¹; see definition below) submitted in response to requests for proposals (RFPs) and other funding mechanisms and determines which interventions to fund. The health department works with providers to ensure the interventions selected will be sound and feasible.

Component 6. The health department and its grantees implement HIV prevention interventions.

Component 7. The implemented interventions are expected to lead to changes in risk determinants among intervention participants.

Component 8. The changes in risk determinants among intervention participants is expected to lead to a reduction in HIV transmission in the health department grantee’s jurisdiction.

Ideally, to ensure that the components of HIV prevention are implemented with the highest quality and contribute effectively to reducing HIV transmission, each component would be evaluated and the findings used for program and policy improvement as well as assessment of local and national progress. Table 1.2 lists the eight components and the six types of evaluation activities that correspond to them. Each evaluation activity will be discussed in a chapter of this document. Table 1.3 highlights the benefits of evaluation for various stakeholders.

Table 1.3

Back to top

Contents of This Document

Each chapter in this guidance addresses one type of evaluation activity that corresponds to CDC’s conceptual framework. Although the format of chapters varies slightly to accommodate variation in topics and emphasis, all the chapters include an overview describing the purpose of the chapter, a discussion of reasons for conducting the particular type of evaluation, an outline of data elements that could be collected, a list of references and resources, and, when applicable, methodology and example forms for collecting and analyzing relevant data.

The order of the chapters does not correspond directly to the sequence of HIV prevention components featured in Table 1.1. Instead, the chapters are ordered to achieve a logical flow based on the types of data expected for each type of evaluation in the event that grantees conduct each type. Consequently, the chapter on developing an evaluation plan comprising all types of evaluation is provided at the end of the guidance so that grantees will understand the evaluation activities they may be conducting before incorporating them into a plan.

Chapter 2, Evaluating the HIV Prevention Community Planning Process, addresses the importance of evaluating implementation of the HIV prevention community planning initiative as the first component in the HIV prevention continuum. The chapter outlines the core objectives of the initiative and the steps involved in fulfilling the objectives. It then discusses measures for determining fulfillment of each objective and makes recommendations for using evaluation findings. The appendix features example surveys containing core questions for CPG co-chairs and members as they evaluate the process.

Chapter 3, Designing and Evaluating Intervention Plans, highlights steps health departments could take when selecting interventions and ensuring that they are designed to be as effective as possible. To that end, the chapter describes desirable elements of an intervention plan. As described in Footnote 1, an intervention plan is a description of the intended intervention that justifies the elements chosen to achieve desired outcomes and ensures that resources and logistics are adequate to meet the needs of the proposed intervention. It contains information about the target population(s), process and outcome objectives, protocol, budget, and staffing. It might be contained in a proposal for funding or other document describing a planned intervention.

The chapter also provides recommendations for evaluating and improving the relevance and scientific soundness of each element so that health departments can assist providers in revising intervention plans as needed and ensuring that they correspond to recommendations set forth in the comprehensive HIV prevention plan for the jurisdiction.

The chapter concludes with a set of variables for describing each intervention that will be funded by the health department and for determining the extent to which the interventions correspond to strategies outlined in the comprehensive HIV prevention plan. Example forms for collecting intervention plan data are in the chapter’s appendix.

Chapter 4, Monitoring and Evaluating the Implementation of HIV Prevention Programs, is intended to provide health departments with a working knowledge of process monitoring and evaluation that will enable them to evaluate implementation of their own interventions as well as to assist health department grantees in the evaluation of their interventions. For purposes here, process monitoring and evaluation is the assessment of each program’s conformity to its design, program implementation, or the extent to which it reaches its intended audience. This includes routine documentation of characteristics of the people served and the services that were provided. A description of implementation should be evaluated on its own merits as well as compared with the intervention plan, which describes the intended objectives of the intervention.

The chapter takes into account the importance of evaluating implementation in general and of referring to intervention plans to determine if interventions are implemented as intended. To facilitate both efforts, the appendix features example data collection forms that guide providers in the collection of monitoring data as well as in the comparison of actual activities to intended ones. Ideally, one form would be completed by the health department or its grantees for each intervention.

Chapter 5, Evaluating Linkages Between the Comprehensive HIV Prevention Plan and Resource Allocation, responds to the need for evaluating fulfillment of HIV prevention community planning core objective 5. To this end, the chapter describes the ways in which health departments and CPGs can assess whether there are clear and substantial linkages between the comprehensive HIV prevention plan, the CDC funding application, and allocation of resources for HIV prevention interventions. The recommended methodology includes several steps that yield pertinent data that can be used to:

• identify HIV prevention activities in the health department’s jurisdiction;
• determine which strategies in the plan are included in the annual CDC funding application;
• determine which strategies in the plan are being enacted in the jurisdiction and which are not; and
• identify interventions that are being implemented but do not correspond to the plan.

To facilitate this process, the chapter’s appendix includes several example worksheets.

Chapter 6, Monitoring Outcomes of Health Education/Risk Reduction Individual- and Group- Level HIV Prevention Interventions, describes ways of monitoring the achievement of outcome objectives for individual- and group-level counseling interventions. Outcome monitoring involves the measurement of progress in achieving the goals and objectives set forth in intervention plans for each intervention. These measurements assess the effects of counseling interventions on client outcomes (i.e., knowledge, attitudes, beliefs, and behavior).

The chapter discusses the differences between outcome monitoring and outcome evaluation (discussed in detail in Chapter 7), the development of tools to collect outcome monitoring data, when to collect the data, and issues in analyzing the data.

Chapter 7, Evaluating Outcomes and Monitoring Impact of HIV Prevention Programs, examines the important and complex subject of program effectiveness and balances the need to be “scientific” with outcome evaluation recommendations that are straightforward and feasible. To that end, the chapter proposes ways to select interventions for outcome evaluation and outlines steps for conducting outcome evaluation. It also describes the advantages and disadvantages of various research designs and discusses incorporation of process data (discussed in Chapter 4) into outcome evaluation. Throughout this document, outcome evaluation is defined as the use of rigorous methods and designs to assess the effects of an intervention.

Chapter 7 also briefly discusses impact monitoring and evaluation, which assesses the cumulative effects of all HIV prevention activities in a jurisdiction.

Chapter 8, Developing an Evaluation Plan, focuses on the importance of developing a concrete, comprehensive evaluation plan before embarking on evaluation activities. The chapter points out that creating a plan requires the identification of local capacity, experience, resources, and technical assistance needs in an effort to set a realistic and worthwhile course for evaluation that addresses both local issues and national requirements. The chapter also describes the six types of evaluation that ideally would be included in an evaluation plan and lists basic elements of a plan. Furthermore, the chapter strongly encourages participation of stakeholders throughout the evaluation process, beginning with creation of the evaluation plan.

Back to top

References and Resources

Bickman, L., ed. Advances in program theory. San Francisco: Jossey-Bass, 1990.

Centers for Disease Control and Prevention. Cooperative agreements for human immunodeficiency virus (HIV) prevention projects program announcement and availability of funds for fiscal year 1998 (Announcement 99004). Atlanta: Centers for Disease Control and Prevention, 1998.

Chen, H.T. Theory-driven evaluations. Newbury Park, CA: Sage, 1990.

Chen, H.T. and Rossi, P.H., eds. Using theory to improve program and policy evaluation. Westport, CT: Greenwood, 1992.

Weiss, C.H. How can theory-based evaluation make greater headway? Evaluation Review 1997; 21(4):501-524.

Back to top

Go to Evaluating the HIV Prevention Community Planning Process