Overview of ECHPP Evaluation Activities
CDC’s programmatic and structural shift in the planning and delivery of HIV-related services requires a systems-level evaluation approach to assess the impact of this shift on the HIV epidemic. ECHPP is using this systems-level evaluation approach to account for all HIV prevention, care, and treatment activities supported by a health department in a specific metropolitan statistical area (MSA), regardless of funding source (public or private). In addition to assessing the extent to which ECHPP affects the HIV epidemic in the 12 MSAs, evaluation activities will include monitoring of jurisdiction-level progress toward NHAS goals and objectives. Process, outcome, and impact indicators were selected based on the NHAS goals; data sources were identified based on these indicators. Process data will be collected directly from grantees and from other federal agencies where feasible; outcome and impact data will be collected from existing CDC disease and behavioral surveillance systems. Data that represent the time periods before (2008-2010), during (2011-2013), and after (2014-2015) ECHPP implementation will be collected.
Major evaluation activities are described below along with the purpose of each activity, key monitoring and evaluation questions, and data that will be collected to answer the questions.
Process Monitoring and Evaluation
PURPOSE: Determine whether programs were implemented as intended and whether target populations are reached
DATA SOURCES: Programmatic data submitted by ECHPP grantees to CDC and other federal agencies
Key Questions
- Which interventions and programs were proposed through ECHPP?
- Which interventions and programs were provided?
- Which populations were reached?
- Were local ECHPP programmatic objectives met?
- What were the program-, agency-, community-, and structural-level (including policies and regulations) successes and challenges related to implementation?
- What was the total amount of health department funding for HIV prevention (all sources of funding)?
- How was HIV prevention funding allocated across major funding activities, by funding source?
- How was HIV prevention funding allocated across target populations?
Indicators and Other Data Collected
- Jurisdictional plans for providing optimal combinations of HIV prevention, care, and treatment services
- Service coverage (i.e., extent to which programs were delivered)
- Reach of services (i.e., extent to which programs contact target populations or programs are received by those populations)
- Qualitative and contextual information associated with program implementation (e.g., facilitators and barriers)
- Resource allocation data
Outcome Monitoring and Evaluation
PURPOSE: Determine if community-level changes are observed in HIV risk behaviors, service access, and health outcomes over time among priority populations
DATA SOURCES: CDC supplemental surveillance systems (Medical Monitoring Project, National HIV Behavioral Surveillance System)
Key Questions
- Was there a reduction in HIV risk behaviors among HIV-positive individuals and high-risk, HIV-negative/HIV-unknown individuals from 2008 to 2015?
- Was there an increase in service access among HIV-positive individuals and high-risk, HIV-negative/HIV-unknown individuals from 2008 to 2015?
- Was there an increase in overall positive health outcomes for HIV-positive individuals from 2008 to 2015?
Indicators and Other Data Collected
- HIV-related drug and sexual risk behaviors
- HIV testing
- Service access
- Exposure to HIV prevention messages and programs
- Screening for TB, hepatitis, STDs
- Clinical outcomes for HIV-positive individuals
Impact Monitoring and Evaluation
PURPOSE: Monitor progress toward NHAS goals and assess whether changes in community-level outcomes are sustained over time
DATA SOURCES: CDC’s National HIV Surveillance System
Key Questions
- Was there a reduction in HIV incidence or indicators of HIV risk from 2008 to 2015?
- Was there an increase in linkage to, and impact of, prevention and care services for people living with HIV/AIDS from 2008 to 2015?
- Was there a reduction in HIV-related health disparities from 2008 to 2015?
Indicators and Other Data Collected
- HIV/AIDS diagnoses and prevalence
- HIV diagnosis at early stage (before AIDS)
- HIV incidence
- Health disparities among newly infected individuals
- HIV transmission rate
- Linkage to care for newly identified HIV-positive individuals
- Viral load
System-level Monitoring and Evaluation
PURPOSE: Overall assessment of whether ECHPP had a systems-level effect on (or contribution to) the HIV epidemic in the 12 MSAs
DATA SOURCES: Programmatic data submitted by ECHPP grantees to CDC and other federal agencies, CDC supplemental surveillance systems (Medical Monitoring Project, National HIV Behavioral Surveillance System), CDC’s National HIV Surveillance System
Key Questions
- To what extent did ECHPP have an effect on the HIV/AIDS epidemic in the 12 MSAs?
- What combinations of HIV prevention, care, and treatment programs contribute to positive outcomes among priority populations?
Indicators and Other Data Collected
- All process, outcome, and impact data described above
Where possible, indicator data will be analyzed and reported separately for the priority populations listed below:
- African Americans
- Latinos
- High-risk heterosexuals
- Injection drug users (IDU)
- Men who have sex with men (MSM)
- High-risk, HIV-negative or HIV-unknown individuals
- HIV-positive individuals
Data Sharing Across Federal Agencies
In order to obtain a comprehensive picture of HIV prevention, care, and treatment activities in each jurisdiction, CDC is collaborating with other federal agencies (e.g., Substance Abuse and Mental Health Services Administration, Health Resources and Services Administration, Department of Housing and Urban Development, and Department of Veterans Affairs) to identify program domains where data may be shared, both to inform the ECHPP evaluation as well as to inform 12 Citiesexternal icon initiatives implemented by other federal agencies (see ECHPP & 12 Cities Project for more information). Effective federal collaboration will also ultimately reduce the data-reporting burden on health department grantees that should allow them to focus their efforts on service delivery.
Data Synthesis and Triangulation
CDC will use a data triangulation approach to review, synthesize, and interpret primary and secondary (quantitative and qualitative) data from many different sources for the ECHPP evaluation. Data describing contextual factors (e.g., unemployment, homelessness, insurance coverage, poverty) will be analyzed given their potential effect on the HIV epidemic in the 12 MSAs. This approach to data analysis and interpretation reflects the Division of HIV/AIDS Prevention’s new evaluation framework for assessing the impact of HIV-related programs on the epidemic.
Evaluation Reporting
The ECHPP evaluation team will routinely produce jurisdiction-specific and cross-jurisdiction reports that describe overall programmatic progress and summarize key findings to date. These reports are described below.
- Annual ECHPP Monitoring and Evaluation Report
✔ Purpose: Summarize yearly core process data* submitted by grantees, progress made by grantees toward local ECHPP objectives, and overall successes and challenges experienced; jurisdiction-specific data will be described
✔ Frequency: Produced annually during ECHPP implementation (2011 to 2013)
✔ Dissemination: To ECHPP grantees and CDC staff - Annual ECHPP Findings
✔ Purpose: Publicize cross-jurisdiction, key indicators and successes and challenges associated with program implementation, including highlights from the ECHPP Monitoring and Evaluation Report
✔ Frequency: Produced annually during ECHPP implementation (2011 to 2013)
✔ Dissemination: On ECHPP website - Final ECHPP Evaluation Report
✔ Purpose: Communicate final ECHPP evaluation findings
✔ Frequency: At end of evaluation period. Report will represent 2008 to 2015 time period; a preliminary report will be released in 2013/2014, depending on data availability
✔ Includes descriptive reporting of process, outcome, and impact indicators as well as contextual data associated with the HIV epidemic in the 12 MSAs
✔ Data triangulation and synthesis methods will be used across all quantitative and qualitative data sources to assess impact of ECHPP
✔ Where appropriate, statistical tests will be used to assess change in outcomes over time (e.g., trend analysis, modeling)
✔ Dissemination: To ECHPP grantees and CDC staff
*Core process data are common data elements reported by all grantees during ECHPP project period.
ECHPP & 12 Cities Project
The Department of Health and Human Services (HHS) 12 Cities Project builds on CDC’s ECHPP initiative by addressing the fourth goal of the National HIVAIDS Strategy: achieve a more coordinated national response to HIV, particularly across federally funded HIV programs and services. A separate evaluation effort associated with the 12 Cities Project is being led by HHS and will assess activities aimed at enhancing horizontal and vertical services integration; data sharing across organizational lines; cross-agency training; implementation of common performance metrics; and, where possible, use of braided funding streams. The 12 Cities evaluation relies upon key informant interviews with federal officials, and site visits with state and local health officials, providers and consumers. In addition, the 12 Cities evaluation seeks to identify lessons learned that can be applied to other jurisdictions.