Developing Evaluation Buy-In
Using Evaluation Data

This chapter:

• Describes why health departments and contractors may not "buy-in" to the Guidance,
• Suggests strategies for developing support for evaluation, and
• Lists ways Guidance data can be used to improve prevention efforts in the jurisdiction.

Developing Evaluation Buy-In

Implementation of Guidance activities is facilitated when health department staff and contractors see the benefit of collecting, managing, and using evaluation data. Health departments who have achieved some successes in Guidance implementation report that developing evaluation "buy-in," both internally and with their contractors, was an essential step in their process. For a variety of reasons, however, contractors and health department staff may resist the Guidance and its reporting requirements.

Challenges to Getting Buy-in

Contractors and health department staff may be concerned about the time and resources necessary to collect and report Guidance data. In general, they consider service delivery to be their first priority, and the Guidance may not be valued unless they feel the data can be used to improve their prevention programs. Some may consider data collection activities to be potentially disruptive to service delivery and damaging to client trust and rapport. In addition to these concerns, some may fear that evaluation results will suggest that interventions are not successful and will negatively affect funding. Although many jurisdictions had data collection and reporting systems in place prior to the Guidance, the increased emphasis on intervention plans and process monitoring may amplify any existing concerns that interventions will be deemed ineffective.

"There's a big fear that evaluation means we're going to find out something bad about their intervention and the next year their money's going to be gone. A lot of it just has to be education that that's not what this process is about, that it is about making sure we're delivering the most appropriate services." Health Department Staff Member

Strategies for Getting Buy-in

Contractors and health department staff are motivated to implement evaluation systems that yield useful data. Health departments are encouraged to consider how data will be used to improve prevention efforts within their jurisdiction as they plan for implementation of Guidance activities. The Guidance represents only a minimal data set and, therefore, health departments may want to consider additional data needs within the jurisdiction that can be addressed by evaluation systems established to meet Guidance requirements.

Some health departments collect data in excess of Guidance requirements to address local evaluation needs, including:

• Client state of residence, county, and zip code;
• Client STD history and HIV status;
• Client knowledge, attitudes, and beliefs related to HIV risk;
• Behavioral and situational co-factors for HIV risk;
• Topics and skills addressed in the intervention; and
• Contractor demographics and training relevant to the intervention.

Health department are encouraged to present the Guidance to contractors as an opportunity to gather data to improve programs locally, and not just as a CDC requirement. Using this approach, contractors are more likely to consider how they can use the data themselves and, perhaps gather additional data to address local evaluation needs.

"It's worth the time to put in place a substantive data collection process locally that responds to immediate and longer term needs. Really spend the time before you get into the technical aspects of data collection to ask the questions you really want answered, then apply the technical analysis to what is possible. A lot of us shoot ourselves in the foot by constructing data systems to meet minimal requirements, which end up being a lot of work and a lot of time and a lot of burden on the contractor where a little more thinking would have gotten you a lot more useful data." Health Department Staff Member

Regardless of whether data collection is limited to or exceeds the Guidance requirements, health departments should consider collaborating with their contractors to design procedures for data collection and reporting. Engaging contractors in the process of developing data collection instruments, deciding how evaluation data will be used, and planning other aspects of the evaluation system can help address concerns about evaluation's impact on service delivery, foster ownership of the evaluation process, and develop buy-in for evaluation activities. Consistent with the community development approach used in some HIV prevention interventions, involving the contractor "community" in creating and deploying the jurisdiction's evaluation system can enhance evaluation behavior.

"The contractors who are generating the data are one of your end users and so their needs have to get met. Having them generate the broad questions as well as working on the implementation steps really saves you not only a lot of political headache but actually a lot of practical headache, because they can tell you what won't work, and they always come up with stuff that you would never think of from your desk in the main office. They really have their finger on the pulse of what their staff are capable of." Health Department Staff Member

Although Guidance requirements may increase the data collection burden on contractors, evaluation systems can be designed to reduce reporting burden. Health departments are encouraged to eliminate redundancies between quantitative data required by the Guidance and the qualitative narrative reports some health departments require of their contractors. Narrative reports to the health department may provide important information about interventions and should be maintained at the discretion of the jurisdiction. However, health departments should continue to identify and eliminate areas of overlap in their reporting procedures. Contractors have and will welcome these improvements.

The magnitude of contractor evaluation responsibilities can also be reduced when the health department manages contractor data. When contractors are allowed to send client-level data to the health department for data entry the contractor no longer has to tabulate and report aggregate data, eliminating a time-consuming task that many contractors are happy to avoid. (See p. 33 for more information about client-level data.) Some health departments have reduced the burden on contractors by allowing them to use a portion of their prevention funds to support evaluation tasks or by allocating additional resources for this purpose.

Using Evaluation Data

An important goal of the Guidance is to provide information to improve prevention services. Although the Guidance is still early in implementation and health departments have not yet had an opportunity to fully explore all the ways the data may be used, several suggestions for using the data have emerged. These ideas are listed below. Health departments are encouraged to explore these and other opportunities to use evaluation Guidance data to strengthen their prevention efforts.

"The contractors are excited about actually receiving feedback reports about what they did. Contractors will submit process and outcome data and we'll develop standardized reports so they can monitor their own progress. We'll provide feedback at the agency-level and the health department will probably look at this across agencies within intervention types." Health Department Staff Member

Planning Interventions: Intervention plan and process monitoring data can help increase contractor awareness of the range of possible interventions; highlight important distinctions between different intervention types; and improve the quality of interventions through consideration of evidence, justification, and sufficiency of the service plan. These data can also prompt contractors to be more specific about the risk behaviors they want to change and the rationale for how they would conduct an intervention to achieve these changes. Process monitoring data in particular can inform subsequent intervention plans, especially estimates of the number, demographics, and risk behaviors of clients to be served. Using past performance to inform future plans provides a basis for contractors and health department staff to identify realistic expectations for intervention performance.

"One thing we discovered was how many agencies are doing AIDS 101, which may have been considered by some to be a group-level intervention. But it's not, and the Guidance was useful in helping people to understand what an intervention is and is not." Health Department Staff Member

Monitoring Interventions: Intervention plan and process monitoring data can be compared to assess the congruence between the intentions of an intervention and its actual performance. This assessment can occur during implementation to identify opportunities to strengthen the intervention during the funding cycle and at the end of a contract to inform decisions about future resource allocation and ways to improve interventions for the next funding cycle.

"One of our contractors who worked with gay youth had done some group-level interventions and had planned a three- or four-session group. By tracking that they found that the implementation was really very shaky because the youth would go to one session but not commit to attending three or four. And it was through looking at the process measures that they saw this." Health Department Staff Member

Identifying Gaps: Process monitoring data can be used by community planning groups to document the extent to which interventions are reaching high-priority populations with priority interventions. This information is an important component of a resource inventory -- a requirement for community planning groups -- and can help identify current gaps between prevention priorities and actual performance. Identification of gaps in prevention services can guide future efforts to better reach priority populations with priority interventions.

"We've talked about being able to do some mapping, something real visual where people could actually see where the GLIs are located and map that on top of where our HIV rates were high and be able to see holes." Health Department Staff Member

Focusing Evaluation: Process monitoring data help select interventions that are appropriate for more in-depth study using outcome monitoring and outcome evaluation and, in this way, ensure the effective use of evaluation resources. These data can be used to identify interventions that demonstrate fidelity to the intended program model, reach a sufficient number of clients, and meet other criteria necessary for an intervention to be evaluated. (See p. 47 for more information about selecting interventions for evaluation.)

Securing Funds: Evaluation help document intervention success in reaching risk populations. This information can be used to support funding applications submitted to health department and non-health department sources to continue the intervention. Documentation of intervention success is increasingly expected from contractors by a variety of funders, and these data can support efforts to expand the resources available to support prevention services.

Improving Communication: Intervention plan and process monitoring terms used in the Guidance establish a common language for planning and evaluation in general, and for describing populations and interventions specifically. This helps to facilitate communication among contractors, between contractors and the health department, and between the health department and CDC. Improved communication contributes overall to the use of evaluation data and the improvement of prevention services.