Background
Use of Evaluation Data
Purposes of this Document
Format and Content of the Guidance and Supplemental Handbook
Limitations of this Document
Timeline for Submission of Evaluation Data

Background

Increasingly, federal, state, and local agencies involved in HIV prevention are recognizing the importance of evaluation for three primary purposes: 1) to determine the extent to which HIV prevention efforts have contributed to a reduction in HIV transmission, 2) to improve programs to better meet that goal, and 3) to be accountable to stakeholders by informing them of progress made in HIV prevention nationwide. In response to this, CDC has identified the types of standardized evaluation data it needs to be accountable for its use of federal funds and to conduct systematic analysis of HIV prevention to improve HIV prevention policies and programs. Evaluation data that are needed (but not yet available at the national level) include the types and quality of HIV prevention interventions provided by CDC health department grantees and their grantees, the characteristics of clients targeted and reached by the interventions, and the effects of interventions on client behavior and HIV transmission.

These data needs guided the development of the proposed evaluation requirements in CDC’s Announcement 99004, which sets forth the evaluation activities that health departments receiving CDC funding for HIV prevention are expected to implement. The evaluation activities are:

• Evaluating HIV Prevention Community Planning
• Designing and Evaluating Intervention Plans
• Monitoring and Evaluating the Implementation of HIV Prevention Programs
• Evaluating Linkages Between the Comprehensive HIV Prevention Plan, CDC Funding Application, and Resource Allocation
• Evaluating Outcomes of HIV Prevention Programs
• Developing an Evaluation Plan

In addition, CDC encourages grantees to conduct another type of evaluation that would provide a benefit to HIV prevention planning and programs. It entails monitoring the outcome objectives that are set for each intervention by its developers.

Throughout the period covered by the announcement, health departments are to report on evaluation activities conducted during the previous year in their annual CDC funding applications in order to contribute to a data system for use at the national level. (See the table at the end of this chapter for the timeline regarding evaluation data submission.) It is important to note that data are to be collected only for HIV prevention activities supported with CDC funds, not all activities within a jurisdiction. Similarly, the requirement applies only to CDC’s health department grantees and their grantees, not to community-based organizations or other prevention providers receiving funds directly from CDC.

The data provided by health departments will be used for three purposes:

• To report to federal, state, and local stakeholders (including communities, health departments, local and national organizations, and federal policymakers) progress made through HIV prevention programs supported by CDC funds
• To improve national policies regarding HIV prevention
• To identify ways to improve HIV prevention programs nationwide

Back to top

Use of Evaluation Data

CDC acknowledges that some health departments and providers have significant concerns about evaluation and the potential punitive implications of “negative” evaluation findings. It is important to note that the purpose of evaluation data collection and analysis is to assess progress and improve HIV prevention activities.

Moreover, CDC’s primary interest in the data will be in the aggregate for identification of national trends and issues. Health departments, for their part, are encouraged to analyze data for individual interventions (and, when possible, to aggregate the data to improve HIV prevention activities across their jurisdictions). One underlying intent of Announcement 99004 and this guidance is that new evaluation activities and reporting will open up communication about HIV prevention evaluation so that stakeholders will find it easier to discuss the strengths and weaknesses of their efforts to improve HIV prevention and will benefit from their and others’ lessons learned.

Back to top

Purposes of this Document

This guidance has two primary purposes:

• To describe each type of evaluation as it applies to CDC requirements
• To explain types of data to be collected and mechanisms for reporting the data to CDC

Back to top

Format and Content of the Guidance and Supplemental Handbook

Evaluating CDC-funded Health Department HIV Prevention Programs–Volume 1: Guidance

CDC determined that the most efficient way to assist grantees in fulfilling CDC evaluation requirements and in meeting their own evaluation needs is to provide guidance and supplemental resource information in two separate documents.

Evaluating CDC-funded Health Department HIV Prevention Programs–Volume 1: Guidance is the first document. It focuses solely on data collection and reporting required by CDC (see purposes in the following section). To this end, each chapter addresses one type of evaluation activity. All of the activities, except for outcome monitoring (Chapter 6), are required, and the chapters are structured to respond directly to the proposed CDC reporting requirements. Each chapter includes an introduction describing the type of evaluation, a summary of CDC reporting requirements, a discussion of potential methods for collecting required data, and, when appropriate, an appendix containing example data reporting forms. It is important to note that, whereas the development of an evaluation plan should take place before implementation of the other activities described in this document, the discussion of the plan is in the last chapter so that grantees become familiar with all the evaluation activities before determining how to formulate the plan.

Evaluating CDC-funded Health Department HIV Prevention Programs–Volume 2: Supplemental Handbook

The second document is entitled Evaluating CDC-funded Health Department HIV Prevention Programs–Volume 2: Supplemental Handbook. It provides extensive information and suggestions for obtaining the minimum data requested by CDC as well as for conducting additional evaluation activities. Importantly, each chapter in the Supplemental Handbook (Volume 2) corresponds to a chapter in the guidance document (Volume 1); however, because outcome monitoring is recommended but not required, suggested outcome monitoring activities are discussed in much more detail in Chapter 6 of the supplemental handbook than in the guidance. Grantees are strongly encouraged to consult both documents when designing and implementing HIV prevention evaluation activities.

Back to top

Limitations of this Document

By design, this document describes evaluation activities as they relate to collecting and reporting data in keeping with CDC Announcement 99004. Following are several limitations of the required data and the information provided in this document:

• This guidance is not intended to provide information about how to conduct evaluation; rather, it is designed to assist grantees in responding to CDC requirements regarding the evaluation of HIV prevention interventions supported with CDC funds.
• The data that will be collected through implementation of Announcement 99004 comprise the minimum data set that CDC and its partners have agreed upon as sufficient for national evaluation purposes. Thus, that set of data does not constitute a comprehensive evaluation of all HIV prevention activities. Grantees are strongly encouraged to supplement the data with data gleaned through other evaluation activities.
• This document does not explain how health department grantees should use the data for program improvement; grantees should consult evaluation staff, other experts, and evaluation texts.
• This document provides guidelines for evaluating the basic characteristics of most, but not all, types of HIV prevention interventions. The types of interventions that are specified (e.g., individual-level interventions, outreach and peer opinion leader models, prevention case management, health communications) are believed to account for the vast majority of HIV prevention interventions that are being implemented throughout the U.S. There are suggestions for capturing basic data related to other types of interventions supported by Health Departments’ HIV prevention programs (e.g., community-level interventions or approaches related to STD treatment).

In addition, because forms already exist for monitoring HIV antibody counseling and testing activities, the forms and guidance here do not address evaluation of those activities. However, the currently required counseling and testing forms are included for reference purposes.

Back to top

Timeline for Submission of Evaluation Data

Below is a table depicting when data for each type of required evaluation are due to CDC. The data should be included with each health department’s annual application for CDC HIV prevention funds. Grantees who wish to begin submitting data earlier than the submission dates shown below or who wish to submit data that are not required are strongly encouraged to do so.

Back to top

Go to Designing and Evaluating Intervention Plans