Evaluation Framework

Section VI: Program Evaluation

An important component of any program is the evaluation of its effectiveness from the earliest stages of implementation. Program evaluation is a systematic way to account for public health actions and to provide data for program improvement. The purpose of CVH State Program evaluation is to document that participating state programs are achieving their goals and progressing toward their intended long–term outcomes.

Evaluation Goals for CDC's Heart Disease and Stroke Prevention Program

• Document changes in state capacity to address CVH.
   
• Systematically document CVD burden using surveillance data.
   
• Document changes in CVH policies and environmental factors that support CVH.
   
• Document the process of implementing interventions and the impact of interventions at the state and local level, in particular settings, and in priority populations.

Evaluation methodology for the State Heart Disease and Stroke Prevention Program involves separate evaluations of capacity building, surveillance, and policy and environmental interventions. Evaluation does not have to include comparison communities or quasi–experimental designs but should rely upon existing data systems for comparison data. States are encouraged to use process evaluation methods to (1) evaluate how policy and environmental strategies were implemented, (2) evaluate the extent to which their program is being implemented as intended, and (3) determine whether their program is appropriately focusing its CVH efforts, especially toward priority populations.

Capacity Building Programs

Purpose
To determine whether state health departments have increased their capacity to perform tasks needed to address heart disease and stroke in a comprehensive manner and to reach the long–term goals of the Cardiovascular Health State Program.

Evaluation Question(s)
What progress has been made in addressing the eight components of capacity building?

Expectations for Capacity Building and Basic Implementation States
Demonstrate an increasing ability over time to perform the eight core capacity building activities, as measured by the semi–annual report.

Data Collection
CDC has developed a suggested semiannual reporting form (available upon request) that states can use to track their capacity building. The reporting form includes information on the eight capacity building activities discussed in the program description.

Surveillance

Purpose

1. To collect epidemiologic data from the BRFSS, mortality and morbidity reports, hospital discharge data, and other state–based data sources so changes in a population's CVD burden and related risk factors and conditions can be tracked.
2. To aggregate years of BRFSS data for priority populations to determine whether CVD rates have changed or if CVD disparities have been reduced.
3. To collect data on existing policies and environmental changes across states using established indicators.
4. To monitor use of secondary prevention strategies (through Peer Reviewed Organizations data and other appropriate data sources).

Evaluation Questions

1. What changes are occurring in the state population's CVD burden and risk factors over time?
2. What changes are occurring specifically in priority populations over time?
3. What policy and environmental changes have taken place over time?
4. What changes are occurring in the use of secondary prevention strategies over time?

Expectations for Capacity Building

1. Demonstrate the scientific capacity to define the cardiovascular disease burden for their state.
2. Demonstrate the ability to track the following trends in CVD in the general population and priority populations over time: CVD mortality, morbidity, disability, and risk factors; patients' age at onset of CVD, and the disparity in these factors between general and priority populations. States should collect cardiovascular–related data using the protocols and time line. We recommend that states collect data using the BRFSS modules on hypertension awareness, cholesterol awareness, and cardiovascular disease. We also recommend that funded states collect data using the BRFSS Module on heart attack and stroke signs and symptoms at least every four years or, if possible, every two years.
3. Publish a document describing the state CVD burden every 5 years and collect burden data at least every 2 years or as needed for program planning.

Expectations for Basic Implementation States
Basic Implementation states should meet the three expectations for core states plus the following:

1. Demonstrate that they have collected and analyzed indicators of CVH–related policies and environmental supports for CVH.
2. Demonstrate that they can collect data on secondary prevention strategies at least every two years or as needed for program planning.

Data Collection
The following are the main variables to consider when measuring a populations' CVD burden:

• Race/ethnicity
• Age
• Gender
• Socioeconomic status (SES)
• Deaths due to heart disease and stroke
• CVD prevalence and average age of CVD patients at disease onset
• CVD disability rates
• Prevalence of CVD risk factors:
  • High blood pressure
  • High blood cholesterol
  • Tobacco use
  • Poor nutrition
  • Physical inactivity
  • CVD-related conditions:
    • Obesity
    • Diabetes
• Knowledge of signs and symptoms
• Secondary Prevention

Program Intervention

Purpose
To monitor the implementation and outcomes of the program interventions.

Evaluation Questions

• Did CVH program interventions influence policy or environmental supports?
• Did educational interventions increase public awareness of CVD (e.g., its signs and symptoms)?
• Were interventions implemented as expected?
• Were program evaluation results used for program improvement and to identify "models that work?"
• Were interventions conducted in priority populations using culturally appropriate strategies?

Expectations for Capacity Building States
Capacity Building states are not expected to implement major population–based interventions. If Capacity Building states choose to conduct pilot interventions or receive supplemental funds for interventions, the interventions should be evaluated.

Expectations for Basic Implementation States

1. Develop and implement population–based intervention strategies for general and priority populations.
2. Show that interventions result in policy and environmental changes. Educational interventions should increase public awareness of CVH and CVD issues, increase support for policy and environmental changes to improve people's CVH, and increase public knowledge about the signs and symptoms of CVD. Over time, states should address policy and environmental changes at the state level, in all four settings, in the general population, and in all priority populations. In addition, they should document anticipated and unanticipated outcomes, lessons learned, and "models that work" and use these findings for program improvement.

Data Collection
Basic Implementation states should provide process and outcome data and other information regarding setting– and state–level interventions. Information to be provided includes the following:

• A brief description of the intervention
• Program objective(s)
• Documentation of whether the objective was met
• Demographic characteristics of the population served by the intervention
• Setting(s) for the intervention (i.e., community, school, worksite, health care facility)
• The geographic region in which the intervention was conducted
• Materials developed
• The target disease (e.g., heart disease, stoke)
• Risk factors addressed (e.g., hypertension, high cholesterol, tobacco use, obesity, nutrition)
• Healthy People 2010 objectives addressed
• Policy changes achieved
• Environmental changes achieved
• Outcome measures to be used
• Lessons learned
• The intervention’s impact on participants
• The intervention’s impact on the setting
• The theoretical model used for the intervention

Table 1: Summary of CVH Program Components and Related Activities

^† Evaluation results from selected CVH interventions will be reviewed and summarized.

Date last reviewed: 05/12/2006
Content source: Division for Heart Disease and Stroke Prevention, National Center for Chronic Disease Prevention and Health Promotion