Overview
Reasons for Creating an Evaluation Plan
HIV Prevention Evaluation Activities to Be Addressed in Evaluation Plans
Steps for Developing an Evaluation Plan
References and Resources
Appendix

Overview

The previous seven chapters of this resource book provide important information about HIV evaluation activities that correspond with CDC’s conceptual framework for HIV prevention. This information is necessary for the foundation of successful evaluation: development of an evaluation plan for implementing each type of evaluation. Ideally, such a plan would describe when and how all types of existing and proposed evaluation activities will be implemented by the health department and its grantees during a particular time period.

Purposes of the Chapter

The purpose of this chapter is to provide guidance to health departments in the development of evaluation plans that are realistic, feasible, and take into account health departments’ unique needs, resources, capabilities, and priorities. The goal is for health departments to create a plan that will guide the collection of data for improving HIV prevention efforts and informing stakeholders of the progress made in HIV prevention.

This chapter addresses several factors pertaining to the development of an evaluation plan: 1) reasons for developing an evaluation plan, 2) types of evaluation activities, 3) steps for developing an evaluation plan, and 5) a sample evaluation plan outline that can be used as a format guide if desired.

Back to top

Reasons for Creating an Evaluation Plan

Because of the complexity and broad scope of HIV prevention programs, creating an evaluation plan can assist health departments in determining their evaluation needs and setting a realistic and concrete course for meeting them. An evaluation plan has many significant benefits, such as enabling health departments to:

• Assess their jurisdictions’ evaluation resources, needs, capabilities, and priorities
• Explain how the health department and its grantees will be accountable for the expenditure of HIV prevention funds
• Take stock of the evaluation data they already have and are expected to receive through current evaluation efforts and identify appropriate evaluation strategies for supplementing them as needed
• Employ the plan as a long-term guide for making decisions about evaluation activities that will be implemented by health departments, community planning groups, CBOs, and other providers
• Employ the plan as a reference point when seeking resources for HIV prevention and evaluation activities

Back to top

HIV Prevention Evaluation Activities to Be Addressed in Evaluation Plans

Chapter 1 of this document discusses the primary components of CDC’s conceptual framework for HIV prevention as well as the seven evaluation activities designed for collecting data to inform stakeholders of progress and improve HIV prevention programs. The components and evaluation activities are featured in Table 8.1 (also in Chapter 1). Ideally, each evaluation activity would be addressed in the evaluation plan; the sample evaluation plan in the appendix of this chapter lists suggested information to be included for each type of evaluation.

Back to top

Steps for Developing an Evaluation Plan

(An evaluation) design is a plan for allocating investigative resources. In evaluation–as in basic science–the designer’s task is to produce maximally useful evidence within a specified budget of dollars, a specified number of person-years from evaluation staff and of person-hours from informants, and other such constraints. ‘Maximally useful’ is a key phrase. (Cronbach, 1982)

Although Cronbach is speaking of specific evaluation activities, development of a broad evaluation plan encompassing many evaluation activities also requires a realistic assessment of what can be done given available resources. With this in mind, this section outlines the common steps for developing a comprehensive evaluation plan that would include required as well as supplemental evaluation activities. The steps are general enough to account for the variation in activities and needs across jurisdictions and to acknowledge that many grantees will have already implemented some or all of the steps for evaluation or other purposes.

• Step 1: Identify goals. Determine the evaluation goals in the jurisdiction and the ways in which they will be met through development of a plan. It is extremely important to involve health department staff, community planning group members, community members, local evaluation experts, and others in this and all other steps in the evaluation plan development process. Their buy-in and expertise are essential to the successful development and implementation of evaluation plans and activities.

Table 8.1

• Step 2: Assess health department evaluation resources and capacity. Health departments should identify potential sources of evaluation funding, expertise, and capacity in the jurisdiction. Evaluation expertise refers to individuals with training in and knowledge about evaluation methodology and theory; potential experts might include health department staff members, providers, consultants, and representatives of academic institutions. Evaluation capacity refers to resources available to organizations; these include staff, existing data, data collection systems, computer software, and systems linking computers within an organization or jurisdiction.
• Step 3: Describe past, current, and planned evaluations of HIV prevention activities supported by the health department (and, when possible, those supported by other agencies). It is extremely useful to take into account existing evaluation resources in the jurisdiction such as funding for evaluation, evaluation activities built into health department programs, evaluation data reported by health department-funded providers, and (when possible) evaluation activities in the jurisdiction that are not funded by the health department.
• Step 4: Identify evaluation activities required by CDC that are not adequately covered by existing evaluation efforts. This can be accomplished by comparing the health department’s past, current, and planned evaluation activities with types of evaluation required by CDC. Decisions about how to proceed then can be made according to the health department resources, capabilities, needs, and priorities identified during Step 2.
• Step 5: Determine when each type of evaluation will be implemented. Evaluation activities that are required need to be instituted well before the data are required so that the data will be available in time for submission. Using the timeline as a guide, health departments can balance their evaluation needs with their resources and capabilities to determine which evaluation activities need to be conducted in addition to existing evaluation ones.
• Step 6: Determine how data will be collected, managed, and monitored for each type of evaluation. Drawing on knowledge about evaluation expertise available and the type of data expected for each evaluation activity, identify practical ways for obtaining and keeping track of it so that it is accessible and can be applied for program improvement and other purposes.

Back to top

References and Resources

Academy for Educational Development. Critical Issues in HIV Prevention Evaluation: A Forum for Peer Exchange. Washington, DC: Academy for Educational Development, 1997.

Battelle. Development of a National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP) Evaluation for Decision Making Strategy. Arlington, VA: Battelle, 1993.

Centers for Disease Control and Prevention. Planning and Evaluating HIV/AIDS Prevention Programs in State and Local Health Departments: A Companion to Program Announcement #300. Atlanta, GA: Centers for Disease Control and Prevention, 1994.

Cronbach, L.J. Designing Evaluations of Educational and Social Programs. San Francisco, CA: Jossey-Bass, 1982.

Rossi, P.H. and Freeman, H.E. Evaluation. Newbury Park, CA: Sage, 1993.

Back to top

Appendix: Example Outline of an Evaluation Plan

[NOTE: Every section addresses information needed for conducting one type of evaluation corresponding to the CDC conceptual framework for HIV prevention and evaluation. This document can be used as a checklist to ensure the information is covered in a plan; alternatively, this document can be used as a format for developing the evaluation plan.]

1. Evaluating the HIV Prevention Community Planning Process
   1. Describe, in general terms, ways in which this type of evaluation is (or is already planned to be) implemented.
      For instance, some health departments and community planning groups may conduct exit interviews with departing members, circulate evaluation forms after each meeting, or draw on existing tools such as the Academy for Educational Development’s self-assessment tool.
   2. Given what is already implemented and planned, list what else needs to be done.
      Chapter 2 in this document contains surveys that can be completed by community planning group members and co-chairs.
   3. List resources available to the health department and community planning group(s) to carry out this type of evaluation.
      This type of evaluation does not require extensive resources. Staff with evaluation expertise may be useful, however, if the health department decides to implement innovative evaluation activities.
   4. List who will be involved in implementing the evaluation and identify their roles.
      This may include health department staff members, community planning group members, community members, and evaluation consultants. Roles include collecting, reporting, entering, and analyzing data as well as employing data for improving the planning process and providing feedback to stakeholders.
   5. Discuss the scope of the data that will be collected.
      This refers to the amount of data that will be collected and the sources from which it will be collected. Community planning sources may include health department staff members, community planning group members, and community members involved in HIV prevention.
   6. Discuss how data will be collected, managed, and monitored over time.
      This refers to how often the data will be collected, the methodology that will be used, and the extent to which it will be housed in a management information system.
2. Designing and Evaluating Intervention Plans
   1. Describe, in general terms, ways in which this type of evaluation is (or is already planned to be) implemented.
      Many health departments review proposals and assess plans for implementing proposed interventions in order to improve the interventions before they are implemented.
   2. Given what is already implemented and planned, list what else needs to be done to meet CDC evaluation requirements for this type of evaluation.
   3. List resources available to the health department to carry out this type of evaluation.
      Reviewing intervention plans is not as resource-intensive as other evaluation activities. Resources may include staff to review the intervention plans and data experts to design a system for managing information about proposed interventions.
   4. List who will be involved in implementing the evaluation and identify their roles.
      This may include health department staff members, community planning group members, community members, and representatives of HIV prevention providers. Roles include collecting, reporting, entering, and analyzing data as well as employing data for program improvement and providing feedback to stakeholders.
   5. Discuss how data will be collected, managed, and monitored over time.
      This refers to how often intervention plans will be received and reviewed, the methodology that will be used to review them, and the extent to which information about proposed interventions will be housed in a management information system.
3. Monitoring and Evaluating the Implementation of HIV Prevention Programs
   1. Describe, in general terms, ways in which this type of evaluation is (or is already planned to be) implemented.
      Many health departments require their grantees to submit progress reports, which may contain process evaluation data. It is expected that different types of HIV prevention interventions may be evaluated in different ways at different intervals.
   2. Given what is already implemented and planned, list what else needs to be done to meet CDC evaluation requirements for this type of evaluation.
   3. List resources available to the health department to carry out this type of evaluation.
      Resources may include funding, staff with evaluation expertise, existing evaluation activities, management information systems, and staff to provide evaluation training to providers.
   4. List who will be involved in implementing the evaluation and identify their roles.
      This may include health department staff members, providers, and evaluation consultants. Roles include collecting, reporting, entering, and analyzing data as well as employing data for program improvement and providing feedback to stakeholders.
   5. Discuss how data will be collected, managed, and monitored over time.
      This refers to how often the data will be collected, the methodology that will be used, and the extent to which it will be housed in a management information system.
4. Evaluating Linkages between the Comprehensive HIV Prevention Plan and Resource Allocation
   1. Describe, in general terms, ways in which this type of evaluation is (or is already planned to be) implemented.
      Some health departments have developed mechanisms for ensuring their comprehensive HIV prevention plans are being implemented as intended.
   2. Given what is already implemented and planned, list what else needs to be done to meet CDC evaluation requirements for this type of evaluation.
   3. List resources available to the health department and community planning groups to carry out this type of evaluation.
      Resources may include funding, staff with evaluation expertise, and a resource inventory listing providers and interventions in the jurisdiction.
   4. List who will be involved in implementing the evaluation and identify their roles.
      This may include health department staff members, providers, and evaluation consultants. Roles include collecting, reporting, entering, and analyzing data as well as employing data for program improvement and providing feedback to stakeholders.
   5. Discuss the scope of the data that will be collected.
      For assessing linkages between the plan, application, and resource allocation, grantees may choose to focus only on health department-funded interventions or they may decide to collect information about all interventions within the jurisdiction. Grantees will also need to determine how often they will collect data about interventions.
   6. Discuss how data will be collected, managed, and monitored over time.
      This refers to how often the data will be collected, the methodology that will be used, and the extent to which it will be housed in a management information system.
5. Monitoring Outcomes of Health Education/Risk Reduction Individual- And Group-Level HIV Prevention Interventions
   1. Describe, in general terms, ways in which this type of evaluation is (or is already planned to be) implemented.
      Some health departments may require their grantees to provide progress reports on the effects of interventions on their clients.
   2. Given what is already implemented and planned, list what else needs to be done to meet CDC evaluation requirements for this type of evaluation.
   3. List resources available to the health department to carry out this type of evaluation.
      Resources may include funding, staff who could train providers in outcome monitoring, existing evaluation activities, and management information systems.
   4. List who will be involved in implementing the evaluation and identify their roles.
      This may include providers, evaluation consultants, and others. Roles include collecting, reporting, and analyzing data as well as employing data for program improvement and providing feedback to stakeholders.
   5. Discuss how data will be collected, managed, and monitored over time.
      This refers to how often outcome monitoring data will be collected and submitted to the health department, the methodology that will be used, and the extent to which it will be housed in a management information system.
6. Evaluating Outcomes of HIV Prevention Programs
   1. Describe, in general terms, ways in which this type of evaluation is (or is already planned to be) implemented.
      Some health departments may already require their grantees to conduct outcome evaluation. It is also possible that one or more providers in the jurisdiction receive funding from sources other than the health department to conduct outcome evaluation.
   2. Given what is already implemented and planned, list what else needs to be done to meet CDC evaluation requirements for this type of evaluation.
   3. List resources available to the health department to carry out this type of evaluation.
      Resources may include funding, staff to train providers, existing evaluation activities, and management information systems.
   4. List who will be involved in implementing the evaluation and identify their roles.
      This may include health department staff members, providers, evaluation consultants, and others. Roles include collecting, analyzing, and reporting data as well as employing data for program improvement and providing feedback to stakeholders.
   5. Discuss the scope of the data that will be collected.
      How will the provider(s) that will participate in the outcome evaluation be identified?
   6. Discuss how data will be collected, managed, and monitored over time.
      What methodology will be used?
7. Monitoring the Impact of HIV Prevention Programs
   1. In general terms, list ways in which this type of evaluation is (or is already planned to be) implemented.
      Many health departments regularly use jurisdiction-wide epidemiological and behavioral data. Similarly, the Youth Risk Behavior Survey, the Behavioral Risk Factor Surveillance System, and locally-specific surveys are used in many areas. In addition, social and political indicators are often considered during the planning and budgeting process.
   2. Given what is already implemented and planned, list what else needs to be done.
   3. List resources available to the health department to carry out this type of evaluation.
      Resources may include funding, existing evaluation activities, and management information systems.
   4. List who will be involved in implementing the evaluation and identify their roles.
      This would most likely be the responsibility of health department staff members. Roles include collecting, reporting, and analyzing data as well as employing data for program improvement and providing feedback to stakeholders.
   5. Discuss the scope of the data that will be collected.
   6. Discuss how data will be collected, managed, and monitored over time.

Back to top