Purpose of the Guidance
The CBO Guidance
Guidance Requirements
Schedule for Reporting Guidance Data to CDC
CDC Framework for HIV Prevention
Limitations of the Guidance

This chapter:

• Describes the purpose of the Guidance;
• Provides a brief overview of Guidance requirements;
• Summarizes the schedule for reporting Guidance data to CDC;
• Presents the CDC framework for HIV prevention planning, implementation, and results and shows its relationship to Guidance requirements; and
• Explains the limitations of the Guidance.

Purpose of the Guidance

In December 1999, CDC released the Final Draft of the Guidance for Evaluating CDC-Funded Health Department HIV Prevention Programs (i.e., the Guidance). The Guidance was subsequently approved by the U.S. Department of Health and Human Services, Office of Management and Budget, and was released in its final form in June 2001. The Guidance specifies the types of evaluation data to be reported to CDC by health departments about their CDC-funded health department HIV prevention programs. Prior to the release of the Guidance, there were no systematic, standardized approaches to documenting and assessing the effects of HIV prevention efforts. As a result, evaluation findings were not comparable among health departments and not generalizable to the national level. With the Guidance in place, health departments and CDC can now employ common strategies and measures to document and interpret the varied and numerous programs they have funded and implemented. Specifically, the data reported to CDC will be used to report to federal, state, and local stakeholders (including community representatives, health departments, local and national organizations, and federal policymakers) progress made through HIV prevention programs supported by CDC funds; improve national policies regarding HIV prevention; and identify ways to improve HIV prevention programs.

It is important to note that the Guidance pertains only to HIV prevention activities supported with CDC funds and not all HIV prevention activities conducted within a jurisdiction. Similarly, the requirement applies only to CDC's health department grantees and their contractors, not to community-based organizations (CBOs) or other agencies receiving funds directly from CDC. A related Guidance has been created for these directly funded agencies.

The CBO Guidance

CDC is currently developing guidance for directly funded CBOs that are conducting HIV prevention interventions (i.e., CBO Guidance). The CBO Guidance is similar to the Guidance for Evaluating CDC-Funded Health Department HIV Prevention Programs, but applies specifically to community-based organizations or other agencies receiving funds directly from CDC. Similar to the Guidance for health departments, the CBO Guidance specifies intervention plan and process monitoring data to be reported to CDC. These data describe the CBO's and other agency's HIV prevention interventions and the characteristics of clients reached by those services.

There is one important difference between the two Guidance documents regarding how data are reported to CDC. The CBO Guidance directs agencies to report to CDC about their CDC directly funded interventions. The heath department Guidance directs health departments to report to CDC about interventions funded by the health department with CDC funds. Health departments aggregate data within the jurisdiction for reporting to CDC.

Within a jurisdiction, some contractors may receive direct funding from CDC as well as CDC funding through their health department. To avoid complications of dual reporting systems, the CBO Guidance was modeled on the health department Guidance so that data collection and reporting requirements are the same. For contractors receiving dual-funding for an intervention, data reported to their health department can also be reported to CDC to satisfy requirements of the CBO Guidance. However, if a contractor is receiving CDC funding for one intervention and health department funding for another, data on each intervention would be reported separately to each respective funding source. Health departments and directly funded CBOs within a jurisdiction are encouraged to share data with each other to improve the local planning and delivery of prevention services.

Guidance Requirements

CDC divided the Guidance into two documents, Volume 1: Guidance, and Volume 2: Supplemental Handbook, to assist grantees in meeting both CDC's evaluation requirements and their own evaluation needs. Grantees are strongly encouraged to consult both documents when designing and implementing HIV prevention evaluation activities.

Volume 1: Guidance focuses solely on data collection and reporting required by CDC. Each chapter addresses one type of evaluation activity, including a description of the type of evaluation, a summary of CDC reporting requirements, a discussion of potential methods for collecting required data, and, when appropriate, an appendix containing sample data reporting forms.

Volume 2: Supplemental Handbook provides extensive information and suggestions for obtaining the minimum data requested by CDC and for conducting additional evaluation activities. Each chapter in the Supplemental Handbook corresponds to a chapter in the Guidance.

The Guidance describes health department evaluation and reporting activities in the following seven areas. These seven areas represent the minimum data required for collection and reporting to CDC. Health departments are encouraged to collect and use evaluation data in excess of these minimum requirements.

Community Planning: Evaluation of community planning does not specify any new reporting requirements beyond those discussed in CDC's Program Announcement 99004 for health department HIV prevention funding. These requirements include documenting implementation of the community planning process; completing the Profile of Community Planning Group Members to describe group member characteristics; and using the Table of Estimated Expenditures for HIV Prevention to describe HIV prevention allocations by intervention, population, and race/ethnicity. (For a comprehensive description of Community Planning evaluation reporting requirements see the Guidance, volume 1, chapter 2.)

Intervention Plans: An intervention plan sets forth the goals, expectations, and implementation procedures for an intervention and is often part of a proposal for funding. Intervention plans require that a core set of data elements be reported by the health department to CDC in the aggregate by type of intervention and risk population including: type of agency; number of clients to be reached, categorized by race/ethnicity and gender (except for health communication/public information [HC/PI] interventions); evidence or theory basis for the intervention; justification of the intervention for application to the target population and setting; and sufficiency of the service plan for implementing the intervention (For a comprehensive description of Intervention Plan reporting requirements see the Guidance, volume 1, chapter 3.)

Process Monitoring: Process Monitoring is the routine documentation of data that describe the characteristics of risk populations served, the services provided, and the resources used to deliver those services. Process monitoring requires that a core set of data elements be reported by the health department to CDC in the aggregate by type of intervention and risk population including: type of agency; number of clients served, categorized by race/ethnicity and gender (except for HC/PI interventions); number of full-time equivalent (FTE) staff used to provide the intervention; and expenditures for the intervention. Some intervention-specific implementation data are also required. (For a comprehensive description of Process Monitoring reporting requirements see the Guidance, volume 1, chapter 4.)

Linkages: Linkages examine the extent of congruence between the health department's comprehensive HIV prevention plan and its application to CDC for prevention funding, as well as the relationship between the comprehensive plan and the allocation of resources. Data to be reported include which recommended interventions in the plan are and are not included in the application and which funded interventions match and do not match the recommended populations and interventions in the plan. (For a comprehensive description of Linkages reporting requirements see the Guidance, volume 1, chapter 5)

Outcome Monitoring: Outcome monitoring assesses the extent to which an intervention achieved the expected outcomes. It does not, however, establish a causal relationship between the intervention and these outcomes. Health departments with at least $1 million in cooperative agreement funding from CDC are required to collect and report outcome data during the cooperative agreement for either an outcome monitoring or outcome evaluation project. Health departments that choose to conduct outcome monitoring are required, for the year 2002, to conduct this evaluation with at least 10 percent of their contractors who are implementing interventions appropriate for outcome monitoring. These data are to be reported in April 2003. For the year 2003, health departments are required to conduct outcome monitoring with 20 percent of their contractors and report their findings in April 2004. Data to be reported include: names and affiliations of evaluators conducting the outcome monitoring; intervention type and goals; target population; evidence and justification for the intervention; copies of instruments and data collection tools; methods of data collection and statistical analysis; appropriate descriptive statistics, including client demographics; summary of findings; and how results will be used for program improvement. (For a comprehensive description of Outcomes Monitoring reporting requirements see the Guidance, volume 1, chapter 6.)

Outcome Evaluation: Outcome evaluation assesses intervention effectiveness in producing the desired cognitive, belief, skill, and behavioral outcomes within a defined at-risk population. Health departments with at least $1 million in cooperative agreement funding from CDC are required to collect and report outcome data during the cooperative agreement for either an outcome monitoring or outcome evaluation project. Health departments that choose to conduct outcome evaluation are required to evaluate at least one distinct HIV prevention intervention or set of integrated interventions by September 2003. Data to be reported include: names and affiliations of evaluators conducting the outcome evaluation; intervention type and goals; target population; evidence and justification for the intervention; evaluation design and methods; sample sizes for treatment and comparison groups and numbers of participants lost to attrition, as appropriate; copy of instruments and data collection tools; methods of data collection and statistical analyses; descriptive statistics, including client demographics; summary of findings (e.g., attrition, overall outcomes, and any subgroup analyses of differences due to demographics, features of the intervention, or other variables); and how results will be used for program improvement. (For a comprehensive description of Outcomes Evaluation reporting requirements see the Guidance, volume 1, chapter 7)

Evaluation Plan: An evaluation plan describes how the health department will implement the activities described in the Guidance. Information to be reported includes a description of how each of the Guidance reporting requirements will be met; how evaluation data will be collected, managed, and used; and the evaluation technical assistance needs for the jurisdiction. (For a comprehensive description of reporting Evaluation Plan reporting requirements see the Guidance, volume 1, chapter 8)

The schedule below has been approved by the Division of HIV/AIDS Prevention - Intervention Research and Support. Each type of evaluation activity is reported annually, with the exception of outcome monitoring and outcome evaluation. Outcome monitoring data are due April 2003 and April 2004. Outcome evaluation is a single effort due in September 2003. Note that some annual due dates report data for the prior calendar year (i.e., retrospective) and others report projections for the next calendar year (i.e., prospective).

The Evaluation Guidance applies to HIV/AIDS prevention community planning and HIV/AIDS prevention programming carried out, in whole or in part, under Program Announcement 99004. Problems, issues, and concerns regarding time lines, due dates, and data submission should be discussed with CDC project officers.

CDC Framework for HIV Prevention

The evaluation activities addressed by the Guidance relate to CDC's conceptual framework for HIV prevention evaluation. This framework shows the relationship between HIV prevention planning, implementation of interventions, and the results of HIV prevention services. The framework is presented below, illustrating the relationship to each of the evaluation activities described in the Guidance.

Limitations of the Guidance

The Guidance describes evaluation activities as they relate to collecting and reporting data in keeping with CDC's Program Announcement 99004. Following are limitations of the required data and the information provided in the Guidance:

• The Guidance is not intended to provide information about how to conduct evaluation; rather, it is designed to assist grantees in responding to CDC requirements regarding the evaluation of HIV prevention interventions supported with CDC funds.
• The data that are collected through the implementation of Program Announcement 99004 comprise the minimum data set that CDC and its partners have agreed upon as sufficient for national evaluation purposes. Thus, that set of data does not constitute a comprehensive evaluation of all HIV prevention activities.
• The Guidance does not explain how health department grantees should use the data for program improvement; grantees should consult prevention program and evaluation staff, behavioral scientists and other experts, and evaluation texts for assistance.
• The Guidance provides guidelines for evaluating the basic characteristics of most, but not all, types of HIV prevention interventions. The types of interventions that are specified (see the Guidance, volume 1, chapter 3) are believed to account for the vast majority of HIV prevention interventions implemented throughout the United States.