Intervention Plan Reporting Requirements
Collecting and Managing Intervention Plan Data
Estimating Clients to be Served by the Intervention
Determining Evidence or Theory Basis for the Intervention
Determining Justification of the Intervention for the Target Population and Setting
Using Logic Models
Collecting Data on Evidence and Justification

This chapter:

• Reviews intervention plan reporting requirements;
• Describes methods for collecting and managing intervention plan data; and
• Presents strategies for reporting data on clients to be served by the intervention, evidence or theory basis for an intervention, and justification of the intervention for the target population and setting.

Intervention Plan Reporting Requirements

An intervention plan describes the goals, expectations, and implementation procedures for an intervention. For the purposes of the Guidance, an intervention is distinct from a program.

An intervention is a specific activity (or set of related activities) intended to bring about HIV risk reduction in a particular target population using a common strategy for delivering the prevention messages. An intervention has distinct process and outcome objectives and a protocol outlining the steps for implementation.

A program is a distinction often used by an agency to describe an organized effort to design and implement one or more interventions to achieve a set of predetermined goals.

Intervention plans describe the services contractors are funded to deliver and should reflect final agreements between the health department and contractors after contract negotiations are complete. For the purposes of the Guidance, intervention plans require a core set of data elements to be reported by the health department to CDC in the aggregate by type of intervention and risk population, including:

• Type of agency;
• Number of clients to be reached, categorized by race/ethnicity and gender 2 (except HC/PI);
• Evidence or theory basis for the intervention;
• Justification of the intervention for application to the target population and setting; and
• Sufficiency of the service plan for implementing the intervention.

For a comprehensive description of intervention plan reporting requirements see the Guidance, volume 1, chapter 3. Additional information about designing and evaluating intervention plans is provided in the Guidance, volume 2, chapter 3.

Collecting and Managing Intervention Plan Data

Health departments usually gather some or all of the required intervention plan data from proposals, workplans, and contract amendments from the contractors they fund. Unfortunately, some data required for intervention plan reporting may not be included in these documents or are presented in a manner that requires some interpretation to meet Guidance reporting requirements.

To facilitate collection of intervention plan data, some health departments have modified their requests for proposals (e.g., requests for applications, invitations to negotiate) to elicit information needed for intervention plan reporting to CDC. For example, health departments may ask their contractors to use the Guidance population and intervention definitions in their proposals, to describe the evidence or theory basis for the intervention, and provide justification of the intervention for application to the target population and setting. Reporting of intervention plan data can be simplified by developing worksheets for contractors to complete as part of their proposals. Examples of intervention plan worksheets from Colorado, Virginia, and Wisconsin are included in the Appendix, p. 80-82.

Intervention plan data reported by contractors to the health department are aggregated and then reported to CDC. Health departments may tabulate this data manually or use data management software to enter and aggregate this information. If intervention plan data are already part of an existing management information system within the health department, these data may be combined with additional intervention plan data gathered from proposals or worksheets and aggregated for reporting to CDC.

Regardless of the methods used to collect and manage intervention plan data, health departments are frequently challenged by requirements to report three types of data:

• Clients to be served by the intervention,
• Evidence or theory basis for the intervention, and
• Justification of the intervention for the target population and setting.

Estimating Clients to be Served by the Intervention

Health departments are required to report to CDC the aggregate number and demographic characteristics of clients to be served by intervention type and population. At a minimum, client demographics should include race, ethnicity, and gender. Reporting age is encouraged but not required. Ideally, the number of unduplicated clients to be served would be reported. However, duplicate counts of the number of clients to be served are acceptable for reporting to CDC because of the difficulty of estimating unduplicated clients for some interventions (e.g., outreach).

Contractors may have difficulty with intervention plan reporting because of differences in how the number of clients to be served is estimated. Also, estimates may be compromised when a jurisdiction lacks a common understanding of how to define "served." Some contractors may count everyone who will receive a pamphlet at a community health fair; others may count only clients with whom they will have a face-to-face interaction.

Contractors may also tend to inflate estimates because they associate large numbers with success. Some may also believe that their funding organizations have this same view, highlighting concerns that funding will decrease if they do not propose to serve a large number of clients. Similarly, contractors may be inclined to propose interventions that reach a large number of clients quickly (e.g., one-time presentations) rather than those that reach fewer clients with greater depth and effectiveness (e.g., GLI).

The need to estimate clients to be served precedes the Guidance in most jurisdictions; however, the Guidance does place greater emphasis on this aspect of intervention planning and process monitoring. Three strategies are suggested for improving estimates of clients to be served in intervention plans:

• Define "served,"
• Accept smaller numbers, and
• Use past performance.

Define "Served"

Health departments and their contractors can collaborate to define how to count clients served. This may involve establishing standards for the duration of contact with a client for them to be counted as served and defining other aspects of service delivery related to how clients are counted (e.g., clients reached through health fairs are not counted as GLI clients). See the Appendix, p. 83, for an example of how Wisconsin distinguishes between a client "contact" and an "interaction" for different intervention types.

Some health departments may choose to establish uniform standards for counting clients served. Others may ask contractors to first develop their own standards and then negotiate to reach agreement. Initial estimates of the number of clients to be served can then be revised. This approach can help the contractor better understand the concept of "served" and allows flexibility in using standards for counting clients depending on the intervention. The ability to compare data across interventions should be considered when determining how clients will be counted.

Accept Smaller Numbers

Health departments are encouraged to assure contractors that funding will not be affected if reductions in the proposed number of clients to be served are the result of their plans to implement more effective interventions, with greater dosage, and in adherence with intervention standards established in the jurisdiction. This message is particularly important for contractors as a jurisdiction begins to establish uniform procedures for estimating clients.

Use Past Performance

Process monitoring data can help in estimating the number of clients to be served in the future by the same or similar interventions. Estimates may be based on the past number of clients reached by the intervention or an analysis of cost per client served. The Guidance may help to improve process monitoring data systems and, therefore, increase the utility of these data in estimating clients to be served in intervention plans.

Determining Evidence or Theory Basis for the Intervention

Health department staff are required to decide if intervention plans are supported by sufficient scientific evidence or theory (i.e., evidence). Multiple types of evidence can be used to support an intervention. In this section, the following four types of evidence are discussed:

• Evaluation of the same intervention,
• Evaluation of a similar intervention,
• Theory from the scientific literature, and
• Informal theory.

Evaluation of the Same Intervention

With this type of evidence, the proposed intervention is identical to one that has already been evaluated and shown to be effective. Congruence must exist between the proposed intervention and the evaluated intervention with regard to the population served, intervention setting, and core elements of the intervention. Though core elements may vary, for two interventions to be considered the same, contractors are encouraged to use the same content, format, and method of delivering the intervention and to deliver the same number and length of intervention sessions.

The financial resources available may challenge the feasibility of replicating exactly a previously evaluated intervention (e.g., the same level of funding is not available with a jurisdiction). If this occurs, "evaluation of a similar intervention" may be the best choice.

Evaluation of a Similar Intervention

With this type of evidence, the proposed intervention is similar, though not identical, to an intervention that has already been evaluated. Although modifying a previously evaluated intervention may compromise its effectiveness, it may be necessary if available resources cannot support full implementation of the evaluated intervention or if the intervention needs to be adapted to be culturally appropriate for a different population and setting.

Generally, "evaluation of a similar intervention" means that there are differences between the proposed intervention and the previously evaluated intervention in one or more of the following areas: population served; intervention setting, content, and format; method of delivering the intervention; and the number and length of interventions session. If differences are too significant between the proposed and the previously evaluated intervention, the prior evaluation may no longer provide sufficient evidence to support using the proposed intervention.

Theory from the Scientific Literature

With this type of evidence, the proposed intervention is based on formal behavioral science theory, social science theory, or some other theory that is published in the scientific literature. The theory is divided into component parts (e.g., skills, self-efficacy) and corresponding intervention elements are then developed (e.g., intervention activities to develop condom use skills and increase self-efficacy to use condoms). When using this approach, the intervention plan cannot simply mention a theory. It must explain how the theory is integrated into the content, format, and delivery of the intervention.

"We've had a few problems with whether the intervention was science-based or not, because people would talk about one theory in their application, but maybe not really use it. So we came up with a list of the different behavior theories, a definition of each, and did training with them about the theories in general. And now we're asking them to check the theory they are using." Health Department Staff Representative

A brief summary of behavioral science theories is included in the Appendix, p. 84. Another resource that describes behavioral science theories and their application to health programs is Theory at a Glance, A Guide for Health Promotion Practice, National Institutes of Health (NIH), September 1997 (NIH publication number 97-3896).

Informal Theory

With this type of evidence, the proposed intervention is based on a theory that is not described in conventional theoretical language and is not published in the scientific literature. The distinction between an informal and formal theory is subtle. Informal theory usually describes a contractor's "practice wisdom" (i.e., knowledge that comes from working with or being a member of a population) and is explained in lay terms. For example, the concept of "self-efficacy" from the behavioral science literature on Social Learning Theory may be stated as "confidence to use condoms" by someone not familiar with the formal language of behavioral science. Health departments are encouraged to work with their contractors to ensure that informal theory provides a logical explanation of why the population is at risk and to help them describe how the theory is integrated into the content, format, and delivery of an intervention that will address that risk.

Summary

Health departments may use any of the four types of evidence to help them judge whether intervention plans are supported by sufficient evidence. Two examples are provided below to further illustrate the difference between interventions that do and do not have sufficient evidence.

The Compendium

The Behavioral Intervention Research Branch in CDC's Division of HIV/AIDS Prevention has compiled a review of interventions with effectiveness determined through empirical research. This review will help guide health departments and contractors in selecting interventions. Interventions are described in the Compendium of HIV Prevention Interventions With Evidence of Effectiveness, November 1999. http://www.cdc.gov/hiv/topics/prev_prog/rep/index.htm

Determining Justification of the Intervention for the Target Population and Setting

Health department staff are required to decide if intervention plans provide sufficient justification of the intervention for the target population and setting (i.e., justification). Sufficient justification is provided when the plan clearly explains how the intervention will lead to the specified outcomes in the specific population and in the contractor's specific setting. Justification is different from evidence. Evidence supports the rationale for the proposed intervention; justification provides greater detail about how and why the intervention will result in the stated outcomes with the specified target population and in the particular setting in which the intervention is conducted (e.g., clinic, bars, prison). Health departments are encouraged to request from contractors logic models (see below) or other descriptions of program theory that the health department can use to assess justification for the proposed intervention. Following is an example of an intervention with and without sufficient justification.

Using Logic Models

A logic model describes the main elements of an intervention and how they work together to prevent HIV in a specific population. This model is often displayed in a flow chart, map, or table to show the steps leading to intervention outcomes. Elements that are connected within a logical model vary, but generally include inputs; activities; outputs; immediate and intermediate outcomes, and long-term impacts. A problem statement may be included to provide context for the logic model. Definitions and examples of each logic model component are presented below.

Logic models do take time to develop and often require the contractor to anticipate the flow of complex, dynamic processes. However, logic models are a good tool for summarizing information for justification and can also help:

• Make explicit the intended outcomes of the intervention,
• Help planners recognize when intended outcomes are unrealistic,
• Show the internal logical consistency of the intervention,
• Help identify gaps in the plan,
• Reveal assumptions about how the intervention leads to outcomes,
• Help contractors be more deliberate about what they are doing,
• Reveal when resources are not sufficient to achieve intervention outcomes,
• Help monitor progress by providing a clear plan for tracking changes to the intervention so that successes can be replicated and mistakes avoided,
• Promote communication about the intervention among contractors, funders, community members, and other stakeholders, and
• Focus evaluation of the intervention by revealing appropriate evaluation questions and relevant data needs.

See the Appendix, p. 85, for materials from Maryland that can be used to train health department staff and contractors on logic models. To receive the US-Mexico Boarder Health Association training curriculum, Outcomes Based Evaluation Using the Logic Model, July 2000, call the Association at 915-833-6450. This curriculum focuses on substance abuse, but could be adapted for application to HIV prevention. Additional information about logic models can be found at the CDC Evaluation Working Group website at www.cdc.gov/eval/resources.htm#logicmodel.

Collecting Data on Evidence and Justification

Contractors need to be able to describe evidence and justification in their intervention plans. Some contractors may have difficulty reporting these data because they lack knowledge of the evaluation literature, are unfamiliar with the formal language of behavioral science and evaluation, and do not have experience in using logic models and other planning frameworks for linking behavioral theory and intervention design. In light of these challenges, the following four strategies can be used, individually or in combination, to collect intervention plan data for evidence and for justification:

• Request for proposals,
• Prescribed interventions,
• Intervention standards, and
• Community planning.

Requests for Proposals

Requests for proposals, contract amendments, and workplans may be used to elicit intervention plan data from contractors. Health departments can use this information to judge the sufficiency of evidence and justification provided for each proposed intervention. This approach encourages contractors to explicitly consider the rationale for their proposed interventions which may result in more thoughtful planning and more effective interventions.

Contractors may need technical assistance to improve their skills in describing evidence and justification. Health department technical assistance to contractors has included training them to use logic models, providing descriptions of evaluated interventions, and distributing summaries of behavioral theories. Building contractor capacity takes time and resources and should be considered a long-term strategy.

See the Appendix, p. 84, for a summary of behavioral science theories. See Using Logic Models, p. 25, for additional information about logic models. For information about interventions with evidence of effectiveness, see The Compendium, p. 24.

"We tried to root our RFP on the concepts of scientific basis and justification. As part of the application, we asked that every program that was to be funded needed to provide some kind of theoretical basis or a logical framework for their intervention. They couldn't just say we're going to do this, they had to say why they were proposing to do it that way. And while we didn't restrict our funding to proven effective interventions because we wanted to fund some innovation, I felt that every one of our programs gave some rationale for their work. So I can say, across the board, that all our programs have some theoretical basis for their interventions." Health Department Staff Member

Prescribed Interventions

Interventions may be developed and disseminated by health departments as a way to assure that contractors are planning interventions that are supported by evidence and justification. These prescribed interventions may specify the objectives, content, and format for the intervention and include curricula and other intervention materials. Prescribed interventions may be developed through collaboration between contractors, program planners, and behavioral and social scientists. Evidence and justification reporting requirements will likely be met when health departments fund contractors to implement these prescribed interventions. Contractors may be allowed to deviate from prescribed interventions if they can provide sufficient evidence and justification to support their proposed changes.

Intervention Standards

Standards for intervention implementation may be developed, based on science and theory, to describe intervention elements required for specific populations. Intervention standards may describe the content and format of the intervention, duration of contact with the client, method of delivering the intervention, and other aspects of the intervention considered essential for it to be effective. Evidence and justification reporting requirements will likely be met when a health department funds interventions that are implemented according to these standards. Intervention standards tend to be less specific than the prescribed interventions described above and, therefore, may not ensure sufficient data for justification. Supplemental information may be requested from contractors to ensure that intervention plan reporting requirements are fully met. See the Appendix, p. 86, for an example of intervention standards from Colorado.

Community Planning

HIV prevention community planning groups should use information about behavioral science theory and evaluation to prioritize interventions. For those prioritized interventions that are being funded by the health department, the minimum requirements for evidence will be met and health departments may report those intervention plans to CDC as having sufficient evidence. To help ensure specificity in reporting requirements for justification, intervention plans submitted by contractors should include descriptions of how the proposed intervention will result in the specified outcomes in the specific population and in the contractor's specific setting. The health department can then decide if the intervention plan did or did not provide sufficient justification.

Summary

The four strategies for collecting intervention plan data on evidence and justification may be combined to ensure the most effective interventions possible and to maximize the quality of the intervention plan data. A health department with intervention standards may still require a contractor to submit a proposal describing its adherence to the standards so as to ensure that the contractor fully understands the importance of these elements to deliver an effective intervention. A contractor may also be asked to articulate how the standards will be applied to a particular population and setting. Similarly, a contractor may be asked to describe evidence and justification in the proposals even if the proposed intervention was prioritized by the community planning group based on evaluation research and behavioral science theory.