spacer

CDC HomeHIV/AIDS > Topics > Evaluation > Evaluating CDC-Funded Health Department HIV Prevention Programs > Supplemental Handbook

space Evaluating CDC-Funded Health Department HIV Prevention Programs: Supplemental Handbook
space
arrow Introduction
space
arrow Evaluating Community Planning
space
arrow Designing and Evaluating Intervention Plans
space
arrow Monitoring Implementation
space
arrow Evaluating Linkages with the Comprehensive Plan
space
arrow Monitoring Outcomes
space
arrow Evaluating Outcomes
space
arrow Developing an Evaluation Plan
space
LEGEND:
PDF Icon   Link to a PDF document
Non-CDC Web Link   Link to non-governmental site and does not necessarily represent the views of the CDC
Adobe Acrobat (TM) Reader needs to be installed on your computer in order to read documents in PDF format. Download the Reader.
spacer spacer
spacer
Skip Nav spacer  
Designing and Evaluating Intervention Plans
spacer
spacer

Introduction
Developing and Evaluating an Intervention Plan
Elements of an Intervention Plan
References and Resources
Appendix A: Interventions With Scientific Evidence of Effectiveness
Appendix B: Example Forms to Summarize Intervention Plan Data

Introduction

An intervention plan is the blueprint for implementing an HIV prevention intervention. For instance, one would not build a house without planning its intended features, assessing whether those combined features would produce the house desired, and ensuring that needed resources were available to build it. Likewise, HIV prevention services should not be undertaken without an explicit plan that describes the intended intervention, justifies the elements chosen to achieve desired outcomes, and ensures that staffing and other resources are adequate for implementation. Careful development and review of intervention plans are concrete steps to ensure that HIV prevention interventions are relevant to the community, scientifically sound, feasible, and meet standards established by health departments.

Intervention

An intervention is a specific activity (or set of related activities) intended to bring about HIV risk reduction in a particular target population using a common method of delivering the prevention messages. An intervention has distinct process and outcome objectives and a protocol outlining the steps for implementation.

Example: An individual-level counseling intervention may be comprised of four related sessions, but they are all provided in a clinic through one-on-one interaction.

Program:

A program is a distinction often used by an agency to describe a related set of interventions serving a particular population.

Example: The Men's Education Network (MEN) program consists of an individual-level counseling intervention, a social marketing campaign, and outreach in bars based on the peer opinion

The intervention plan serves as the reference point for the provider proposing the intervention and for the health department. Often, it is the heart of a proposal for funding and refined during negotiations with the health department until an agreed-upon version becomes incorporated into contract requirements. For the provider, the intervention plan is the template for organizing and deploying resources, for determining the content of work, and, eventually, for assessing whether the interventions have met their goals. Many CBOs that provide excellent service and save many lives may not be adept at documenting their accomplishments. The intervention plan is a first step to help them get credit for their work and become even more effective.

For the health department, the plan serves as the implementation standard for which the provider is accountable. It establishes some of the criteria for contract monitoring and alerts the health department to the provider’s potential technical assistance needs.

Because these are such vital functions, CDC Announcement 99004 emphasizes the importance of evaluating intervention plans as a necessary foundation for the funding and implementation of HIV prevention interventions. Examination of the strengths and weaknesses of crucial elements in the intervention plan allows health departments to guide providers in the improvement of programs and increases accountability to all stakeholders in the HIV prevention community. In theory, if the intervention has a sound intervention plan, is science-based, and is implemented as intended, it is likely to lead to reductions in clients’ risk behaviors. Figure 3.1 illustrates this hypothetical relationship; Chapter 4 will address the evaluation of program implementation that can help increase the likelihood that desired results will be achieved.

Figure 3.1. The intervention plan is the initial reference point for implementation and subsequent prevention outcomes.

HIV Prevention Intervention Plan
HIV Prevention Intervention Implementation
Behavioral Risk Reduction for HIV Prevention

To maximize the soundness of intervention plans, a standardized evaluation of intervention plans should:

  • Provide explicit criteria—agreed upon within the jurisdiction—to be used when making decisions about the quality of proposed interventions and programs.
  • Allow reviewers to apply those criteria consistently when assessing proposals.
  • Offer a systematic means for providing feedback for improvement of intervention plans.
  • Ensure that the intended characteristics of the intervention are clearly expressed so that they can be referenced during the assessment of intervention implementation (This is described in Chapter 4).

Purposes of the Chapter

The intervention plan is produced through a dynamic process of development, review, and refinement. The steps in Figure 3.2 comprise typical cycles for an intervention plan in the context of funding by the health department.

Following these assumptions, this chapter provides guidance on 1) the basic components of an intervention plan, 2) the features of high quality interventions, and 3) the use of intervention plan evaluations for improving intervention designs and as a reference point during process and outcome evaluation.

Figure 3.2

Health Department Receives CDC 99004 Award
HD offers funds to subcontractors
(e.g., through an RFP)
A provider develops and submits a proposal (which contains an intervention plan describing the proposed intervention) in response to a health department’s request
The health department reviews and evaluates the proposal and plan according to specific, pre-established criteria
If the health department is interested in funding the proposed intervention, both parties negotiate changes to the plan
The provider submits a revised plan to the health department that includes the negotiated changes
HD uses funds to implement interventions by HD staff
HD staff develop an intervention plan describing the proposed intervention

The provider and health department refer to the intervention plan when evaluating

  • intervention implementation (process evaluation) and
  • outcomes (outcome monitoring and evaluation)

Back to top

Developing and Evaluating an Intervention Plan

Framework of an Intervention Plan

An intervention plan serves three primary functions: it provides a rationale, a description, and an operational plan for a specific HIV prevention intervention. The ultimate objectives of each intervention are to affect various determinants of risky behavior (e.g., to increase use of condoms or to increase awareness of the risks of sharing injection works) and to reduce the transmission of HIV within the intervention’s target population. To achieve these objectives, each intervention—whether an individual counseling model, a street outreach program, or a media campaign—must be designed with attention to specific characteristics that comprise an intervention plan. Under the framework of theory-driven evaluation (Chen, 1990), these intervention characteristics can be grouped into the following six categories. This chapter will describe each category and its corresponding elements.

  • Specifying the target populations
  • Choosing interventions
  • Establishing intervention goals and outcome objectives
  • Developing an implementation strategy and process objectives
  • Assessing characteristics of the implementing organization
  • Describing the data system

Evaluation Criteria for Each Category

For each of these categories of intervention characteristics, there are various ways of assessing the merits of the specific characteristics included in the intervention plan. First, there are two broad criteria that should be considered:

Relevance. Relevance refers to the extent to which an intervention plan addresses the needs of affected populations in the jurisdiction and of other community stakeholders. The community’s needs—determined during the needs assessment component of HIV prevention community planning—should be reflected as discrete interventions for specific populations. Thus, consistency between proposed interventions and the jurisdiction’s comprehensive HIV prevention plan is one primary aspect of relevance.

However, as will be noted in the discussions of each category, while consistency with the comprehensive HIV prevention plan is necessary, it may not be sufficient without assurance that the intervention design and implementation address other needs. For instance, the particular implementation of an intervention must be culturally competent for, and accessible to, its intended audience. In some cases, the needs assessment used for a comprehensive plan may not have addressed the specific needs of that population within a particular area of the jurisdiction. In this scenario, the intervention plan should address this greater level of specificity. For instance, “MSM” may be a priority population noted in the comprehensive HIV prevention plan, but young gay men of color may be the primary subset of MSM who need services. An intervention plan should describe the particular needs of this population and the ways in which the proposed intervention will meet those needs.

Scientific Soundness. This criterion considers the scientific foundation of each characteristic covered in the intervention plan. As used here, “science” does not refer to an academic study of the characteristics; rather, it is used to emphasize the need for clear and logical evidence to support the inclusion of a specific characteristic, strategy, or approach in the design and implementation of the intervention. Such an approach usually assumes that some type of systematized knowledge is applied in the conception, development, and choice of intervention components. That knowledge may be based on traditional scientific sources of information (e.g., looking to the scientific literature for “what works”), but it also includes the use of systematic operational data (e.g., staffing patterns or steps in implementing the intervention) maintained by an agency and used as evidence to support the continued use or refinement of particular aspects of the intervention.

Scientific soundness also refers to the application of behavioral and social science theories developed or adapted by the provider agency. For purposes here, a theory is a statement of the hypothesized relationships between what a provider proposes to do and how those activities will affect HIV risk behaviors in the service area. The section on Choosing Interventions provides more discussion on the use of formal and informal theory. In summary, a theory describes the projected relationships between a problem or need, an intervention, the hypothesized effects of the intervention, and desired outcomes. An intervention plan that specifies how an intervention will affect risk behaviors through these relationships is more scientific than one that does not address how the intervention is believed to work. Similarly, an intervention plan that clearly states the steps that will be followed to implement the intervention is more scientifically sound than one that does not provide an adequate level of detail about its implementation.

Both relevance and scientific soundness criteria should be regarded as integral parts of a high quality intervention plan. On the one hand, an intervention plan without relevance is useless to stakeholders and may lead to an inappropriate allocation of limited resources. On the other hand, a relevant intervention that is not carefully specified and based on scientific evidence will not be as likely to yield positive benefits for the population it is intended to serve.

Back to top

Elements of an Intervention Plan

Specifying the Target Population

An intervention plan should contain a description of the target population for whom the intervention is intended. One of the foundations of HIV prevention community planning is the setting of priorities among target populations in a jurisdiction. There is an assumption that a substantial portion of health department resources will be allocated to these high priority populations. In some cases, the comprehensive HIV prevention plan may provide detailed outlines of target populations (e.g., “gay youth of color residing in the 5-Points area”). In others, the comprehensive HIV prevention plan will be more general, requiring an intervention plan to define more narrowly who will be served by specific activities. Issues to consider when specifying the target population in an intervention plan are shown in Table 3.1.

Table 3.1

Target Population Specification
  • Correspondence to a high priority population noted in the comprehensive HIV prevention plan
  • Proportion of target population that engages in specific risk behaviors (especially if population is defined by race, ethnicity, or other non-risk related identifier)
  • Culture and norms
  • Predominant languages
  • Education and literacy
  • Competing economic or social needs
  • Predominant media channels used

Demographics. A description of the target population needs to include the risk factors and demographics of the target population as well as the extent of the population that will be reached by the intervention (often referred to as coverage). The basic demographics of age, race, ethnicity, and sex can provide insight into developmental, cultural, and sex-specific issues that the intervention will need to account for. The intervention description can also include other relevant details about the audience that inform the tailoring of the intervention. For example, the plan might discuss the languages and social or behavioral norms that are common to the service area. Consideration of education and literacy is critical, especially when written materials are proposed as part of the intervention approach.

Risk Factors. It is critical to identify the specific risk factors that affect the audience. Most important is the clarification of the route of transmission of HIV that they are exposed to. A simple classification for these risks is based on the system used for HIV and AIDS surveillance:

Table 3.2

Risk Population Exposure Route and Risk Behaviors
Men who have sex with men Unprotected sex between men that results in exposure to semen or blood
Injection drug users Use of needles, syringes, or preparation materials by two or more people
MSM and IDU Risks through both sex with other men and injection drug use
Women who are risk for or infected with HIV who are pregnant or who may become pregnant Transmission to the baby prenatally, during delivery, or through breast-feeding
Heterosexual sex with someone at risk for or
infected with HIV
Unprotected vaginal or anal sex between a man and woman that results in exposure to semen or blood
Other
  • Tattooing
  • Sex toy sharing between women who have sex with other women
General Public No specific risk for HIV, but often the target of broad prevention or education efforts to increase awareness or change community norms

The specific audience to be served may also have economic or social needs that are different from the general audience described in the comprehensive HIV prevention plan. For instance, the comprehensive HIV prevention plan may list “injection drug users” as a high priority population, yet in a particular city, young methamphetamine users may be the majority of IDUs. Among these methamphetamine users, there may be low employment and high IDU-on-IDU crime. These unique issues should be taken into account in the intervention plan.

Coverage

The extent to which an intervention is reaching its intended target population

Audience Coverage. Another consideration in developing an intervention is determining the relationship of how much of the target population will be reached. For instance, a provider may believe that there are 300 injection drug users in her jurisdiction, but that she can only reasonably expect to reach 50 of them with case management services during one fiscal year. Specification of the expected coverage provides a goal to which the provider and her funders can refer when determining if the intervention reached the intended number and types of individuals.

The relevance and scientific soundness criteria for evaluating choice of target populations are reviewed in Table 3.3.

Table 3.3

Risk Population Exposure Route and Risk Behaviors
Relevance The comprehensive HIV prevention plan is the primary basis for measuring target population relevance. Relevance, in this case, is primarily the extent to which the population targeted in the intervention plan is consistent with the target population in the comprehensive HIV prevention plan.
Scientific Soundness In the context of target populations, scientific soundness refers to the extent to which the intended target population is clearly defined. In addition, discussion and accommodation of the cultural and environmental issues specific to the intervention’s target audience increases the scientific soundness of the specification of the population. Furthermore, the strategies to reach the target population must be well organized and feasible.

Choosing Interventions

Types of Interventions. An HIV prevention intervention is an organized activity designed to bring about changes in behaviors, knowledge, attitudes, and beliefs that put people at risk for HIV infection. CDC’s Announcement 99004 classified three broad categories of interventions and it is expected that most interventions funded by health departments will fall into one of them. The broad intervention categories and the most typical examples of prevention activities are shown in Table 3.4.

Table 3.4

Intervention Category Specific Types of Interventions Within the Category
Health Education/Risk Reduction (HE/RR)
  • Individual-Level Intervention
  • Group-Level Intervention
  • Community-Level Intervention
  • Outreach
  • Prevention Case Management
Health communications/Public Information
(HC/PI)
  • Mass & Other Media
  • Hotlines
  • Clearinghouses
Counseling, Testing, Referral, & Partner Counseling and Referral Services
  • HIV Antibody Counseling & Testing
  • Partner Counseling and Referral Services

It is important to note that, for purposes here, an HIV prevention program implemented by a provider may consist of either a single intervention or two or more interventions serving a particular population. An intervention is a specific activity (or set of related activities) intended to bring about HIV risk reduction in a particular population using a common method of delivering the prevention messages. The evaluations discussed here require assessing each of the interventions that comprise a program, making the intervention the unit of analysis. Figure 3.3 shows a hypothetical example of an array of services that distinguishes between intervention and program as used here.

For example, an individual counseling intervention may consist of four sessions of related activities, but they are all provided in a clinic through one-on-one interaction. Only one intervention plan would be needed for this counseling intervention. Conversely, a program that contains both street outreach and a media campaign should have two distinct intervention plans. To assess the intervention plan quality, one should independently evaluate each of these plans.

Choosing Interventions on Their Scientific Basis. Several factors will determine the choice of interventions, but the principles of HIV prevention community planning assume that the priorities established in the comprehensive HIV prevention plan will be a driving force in selecting intervention strategies to fund and implement. The guidance for community planning suggests that some of the critical factors to consider include the efficacy of the intervention, its behavioral or social science basis, and its cost effectiveness. The attention given to science in the guidance for community planning is not intended to minimize the role of providers’ experience with their communities, their constituents, and their services. It is intended to highlight the importance of increasing the extent to which prevention funds are used for interventions whose effectiveness is known or strongly supported.

Figure 3.3 Distinctions between programs and interventions.

Health Department’s HIV Prevention Program

Strategy

An intervention approach recommended by the community planning group in the comprehensive plan for a specific target population.

An intervention (i.e. an implemented prevention service) may or may not correspond to a strategy listed in the comprehensive plan.

Comprehensive HIV prevention plans should include scientific evidence to support the intervention strategies that are proposed for each target population. Evidence of scientific support can be in the form of prior evaluation data, behavioral and social science theories, and logic models or similar descriptions of the proposed means by which the intervention is expected to affect outcomes. However, comprehensive HIV prevention plans provide varying levels of detail about the evidence supporting the strategies rated as high priorities; in some cases little or no scientific evidence is cited to support them. Therefore, provider agencies often need to include documentation about the scientific basis for the specific interventions they are implementing to ensure that their interpretation of the strategy is scientifically supported. Table 3.5 summarizes some of the types of evidence that might be used to support the choice of an intervention.

Table 3.5

Types of Scientific Evidence That Can Support a Choice of Interventions
The proposed intervention has
  • Undergone previous evaluation
  • Used previously evaluated intervention model with a similar population
  • Used previously evaluated intervention model with a different population
  • Applied formal theory in program development
  • Applied informal theory in program development
  • Used another type of scientific evidence

Scientific Support From Prior Evaluations. In some cases, the intervention proposed has been evaluated during prior implementation (as may be the case with continuation funding of an intervention). This provides direct, empirical evidence of the intervention’s efficacy with the same population in the same setting in which it will be implemented during a particular round of funding.

Intervention developers can also generalize the findings from evaluations of similar programs to the situation for which they are planning. For instance, an intervention originally delivered to IDUs in an urban setting might also be adapted for IDUs in a more suburban or rural area. Similarly, a community-level intervention designed to decrease smoking might have some useful components that can be modified for decreasing unsafe sexual behaviors. The Behavioral Intervention Research Branch in CDC’s Division of HIV/AIDS Prevention has compiled a review of interventions with effectiveness determined through empirical research to help guide providers in selecting interventions (Figure 3.4 and Appendix A).

Figure 3.4

Interventions with Scientific Evidence of Effectiveness

In 1996, CDC began the Prevention Research Synthesis (PRS) project to create a database of science-based HIV/AIDS intervention studies. Among the several aims of this project is to identify interventions that have been evaluated using methodologically rigorous designs and have shown significant positive effects.

Studies were identified in two steps. First, studies were located from the PRS project using the project’s defined relevance and methodological criteria. Relevance criteria were applied to select behavioral, social, and policy studies—published or unpublished—reported from 1988 onward, conducted anywhere in the world, and aimed at reducing sex- or drug-related risk behaviors and HIV/STD incidence and prevalence rates, irrespective of positive, negative, or null findings. As of July 1999, the database has approximately 5,000 articles that report on HIV prevention; 200 of these interventions met the relevance criteria.

Further screening was done based on criteria for methodological rigor. These criteria were used to ensure selection of behavioral and social interventions that were evaluated using control or comparison groups and pretest/posttest designs, as well as policy studies with less rigorous designs. This process identified a subset of 108 primary studies that represent the best available intervention science meeting the conditions of the PRS project.

Next, additional criteria were applied to identify effective interventions from among the 108 well-designed studies. These criteria limited the selection to U.S.-based studies with at least one positive outcome related to HIV risk reduction that showed significant difference between the intervention and control or comparison groups, with no negative findings, and are “state-of-the-science.”

Twenty-five interventions that met all criteria have been identified. This collection of studies provides state-of-the-science information about interventions with evidence of reducing sex- or drug-related risks and the rates of HIV/STD infections. These interventions have shown effectiveness with a variety of populations, including clinic patients, heterosexual men and women, high risk youth, incarcerated populations, injection drug users, and men who have sex with men. They have been delivered to individuals, groups, and communities in settings such as storefronts, gay bars, health centers, housing communities, and schools. The source citations for the 25 interventions are found in Appendix A.

When the populations and settings of prior research are similar to those for which the intervention will be adopted, evaluators may have greater confidence in the findings. As the difference in populations, settings, and circumstances between the planned intervention and the previous setting increases, more subjective judgments are necessary to assess the validity of generalizing the scientific basis.

Scientific Support from Theory. Programs that are not objectively evaluated can still have scientific support. Behavioral, social science, political, and economic theories may provide frameworks for constructing specific HIV prevention interventions. A theory is a statement of the relationships that are believed to exist among a set of needs, activities, and outcomes. A theory should be stated in a way that enables it (given the time and resources) to be tested to determine if it holds up under operational conditions.

Theories can be both formal and informal. A formal theory is one that has been the topic of publications, forums, and research in the academic and scientific community. Bandura’s Social Cognitive Theory (Bandura, 1989) and Prochaska and DiClimente’s Stages of Change Model (Prochaska & DiClemente, 1992; Prochaska, Redding, Harlow, Rossi, & Velicer, 1993) are examples of formal theories that have been applied to HIV prevention.

“Theory” is sometimes criticized as being “out of touch with reality.” Informal theories are developed to meet the specific needs of a particular situation by articulating the relationships between the intervention components being implemented, the ways that they will affect outcomes, and the outcomes themselves. Not only is a good theory “in touch” with reality, it provides logical and realistic connections among a set of needs (e.g., risky behavior occurs because certain people are unable to negotiate safer sex with their partners), activities to address those needs (e.g., a communication group will help participants develop negotiation skills), and the specific outcomes that those activities will bring about (e.g., greater communication skills contribute to less risky behavior). Such a description allows the theory to be tested, making an informal theory a significant tool for intervention design.

The following example (Figure 3.5) shows a graphic formulation of these relationships for another type of intervention.

Figure 3.5 Statement of an informal theory to support a condom promotion campaign

Problem/Need: People at risk for HIV do not use condoms consistently.

Intervention
Create awareness of problem and ways to respond to problem

Mid-term Objective
Norms of community members change
Community members influence their peers with respect to sexual risk taking
Condoms are readily available
More people obtain condoms

Longer-term Objective
People who were influenced by their peers and have condoms accessible change their risk-taking behavior

Intervention Intensity. An assessment of the proposed intervention must also take into account whether the intervention will provide a large enough “dose” to each client to bring about the proposed outcomes. The size of the “dose” needed (sometimes called the dose effect) is a function of the strength of the intervention for bringing about a certain level of behavior change and the amount of exposure necessary to bring about that effect (e.g., the minimum number of outreach contacts needed to have an effect on a neighborhood or the number of individual counseling sessions required to bring about consistent condom use with 60% of counseling clients). It is difficult to expect an intervention to have an effect if the strength and duration specified in the intervention plan is too low. For example, a group counseling intervention proposing only one 30-minute session may be an insufficient dose to achieve a desired outcome of consistent condom use by all group participants.

Many providers implement interventions that have been previously evaluated by others. The original intervention that was evaluated had a specified level of strength and duration. If that same level of effect is to be expected when using that original intervention as a model, a similar level of strength and duration should be built into the intervention. Similarly, newly developed interventions must specify how much effect is expected and what duration is needed to achieve that effect (e.g., how many counseling sessions constitute a “dose” or how many viewings of a public service announcement are necessary to create awareness of an issue).

Table 3.6 summarizes the relevance and scientific soundness criteria that can facilitate evaluation of the choice of interventions.

Table 3.6

Evaluating the Choice of Interventions
Relevance Interventions that correspond with high priority strategies in the comprehensive HIV prevention plan reflect the central issue of HIV prevention community planning: “Does health department resource allocation mirror the strategies prioritized in the comprehensive HIV prevention plan?” In terms of relevance, an intervention that is consistent with a priority in the comprehensive HIV prevention plan (or a previous needs assessment at the local level) can be considered relevant to the jurisdiction.
Scientific Soundness The scientific merit of a proposed intervention can be evaluated in terms of:
  1. Whether the intervention has a basis in scientific evidence
  2. The anticipated strength and duration of the intervention

Scientific evidence can be in the form of prior evaluation or research that supports the intervention approach or a theory that provides testable assumptions about the relationship between the intervention and its intended outcomes. The more similar the populations and settings of the prior research, the greater the likelihood that the proposed intervention will be similar to prior research findings.

Establishing Intervention Goals and Outcome Objectives

To facilitate evaluation of the quality and effectiveness of an intervention, the intervention plan should include clear and measurable process and outcome objectives. Process objectives focus on the projected amount, frequency, and duration of the intervention activities and the number and characteristics of people to be served; process evaluation is discussed at length in Chapter 4.

Outcome objectives are statements of the intended effects of the intervention, such as increasing knowledge about HIV, changing risk-related behaviors, promoting community norms for safer sex, and reducing HIV transmission.

When developing outcome objectives, one must consider whether those objectives can, in fact, be achieved by the proposed intervention. Realistic assessment of the amount of change that the intervention is likely to bring about will help determine whether an intervention is the best use of staff time and financial resources. Assuming that the objectives can be obtained, the objectives must also be stated in such a way that the projected changes can be measured objectively—that is, different people measuring the change would get comparable results.

The goals and objectives for the intervention must meet two other standards. First, the provider must assess whether it has the resources, staff, and capability to implement the intervention (i.e. sufficient organizational capacity). Second, the intervention and its anticipated outcomes must be in keeping with the target population’s values and norms (i.e. culturally relevant). Table 3.7 summarizes the characteristics of well-written goals and objectives.

Table 3.7

SMART Characteristics of Goals and Objectives
Characteristic Questions to guide the development of goals and objectives
Specific
  • Are objectives stated as changes in particular behaviors?
  • Is the amount of change expected made explicit?
  • Can the changes be achieved through one intervention?
Measurable
  • Can the objective be measured in such a way that the success of the intervention
    can be determined?
  • Can these numbers or facts be presented in a report?
  • Are there data to compare these data with? (e.g., from a baseline or a control
    group)
Appropriate
  • Are these objectives culturally and educationally appropriate?
  • How will this program be accepted by the community?
  • Does the intervention fill a gap in current services?
Realistic
  • Are the goals and objectives attainable given the level of risk and the anticipated
    difficulty changing the risk behavior(s)?
  • Can the providing agency implement the proposed intervention?
  • Are the resources available to achieve the stated objectives?
Time-based
  • Can these objectives be accomplished within the available time frame?
  • Can we reasonably expect to detect changes within this time frame?

Developing An Implementation Strategy and Process Objectives

Following a clarification and justification of the general intervention type, the intervention plan should describe the specific characteristics of the intervention. These features represent the “nuts-and-bolts” of the intervention. In this section of the intervention plan, an agency should provide details about where the intervention will take place and how the provider will serve the target population.

Each of the three intervention types (HE/RR, HC/PI, and CTR/PCRS) has fairly distinct intervention elements. For example, an intervention plan for a social marketing campaign would need to discuss the distribution channels it would employ or the community mobilization procedures it would use.

In contrast, an intervention plan for an individual counseling intervention might describe the activities that would occur in each session with a client (e.g., risk assessment, determination of stage of readiness to change, or behavioral contracting).

It is important that the intervention plan be explicit about the type of written materials that will be distributed and the ways in which the appropriateness of the materials will be ensured. Likewise, the program design should discuss the types of HIV prevention items (e.g., condoms and/or safer sex kits) that will be distributed and why they are appropriate for the intervention and the audience.

Process Objectives. Health education and risk reduction; health communication and public information; and counseling, testing, and partner referral services each have fairly distinct intervention activities that can be conducted at varying levels of intensity. In this section of the plan, the provider should also specify the “dose” of intervention it expects to provide. Process objectives describe the specific intervention activities, the projected level of effort needed to carry them out, the people responsible for carrying them out, and when they will be completed.

Table 3.8 provides examples of process objectives for each type of intervention. See Table 3.9 for criteria for evaluating the implementation strategies.

Table 3.8

Examples of Process Objectives
Type of Intervention Sample Process Objectives
Individual- or Group-Level Interventions
  • 25 clients will be counseled per month
  • At least 60% of clients will complete all four sessions of individual counseling sessions
  • 250 condoms and 100 bleach kits will be distributed as part of the counseling in the next year
Outreach
  • Outreach contact will be made with 300 members of the target population at least twice during the year
  • 200 risk-reduction fliers will be handed out each month
  • 1,000 condoms will be distributed every 6 months
Prevention Case Management
  • PCM services will be provided to 50 injection drug users during the next fiscal year
  • Each IDU in PCM will attend at least three PCM sessions
  • New referral and coordination relationships will be developed with at least one new agency per month
Partner Counseling and Referral Services
  • Initial counseling sessions will be held with 20 HIV-infected individuals each month
  • Counselors will notify at least 90% of the partners they agreed to contact
  • HIV antibody tests will be performed for 80% of the notified partners
Health Communications/Public Information
  • Three radio spots on WXXX will be aired
  • Four new businesses per month will be recruited to distribute campaign materials
  • 15 community residents will conduct peer networking in their own neighborhoods at least twice per week
  • Hotlines will field at least 50 calls per week
  • Hotlines will provide referrals for at least 75% of all callers

Table 3.9

Evaluating Implementation Strategies and Process Objectives
Relevance In general, the implementation strategy needs to address the steps of the intervention, its cultural and language appropriateness, and issues of access and client involvement.

Process objectives need to be consistent with the activities outlined in the implementation strategy.

Scientific Soundness The intervention plan should provide a detailed enumeration of the activities that are needed to implement the intervention. It should also provide information on how the agency will ensure cultural and language appropriateness and optimize client access and involvement.

Process objectives should be clear, consistent, and measurable. Each intervention should have process objectives that specify the number, frequency, and duration of activities over the period covered in the proposal.

Assessing Characteristics of the Implementing Organization

The implementing organization is responsible for carrying out the activities in the intervention plan. In addition to describing the components of the intervention, the intervention plan should address the resources—human, financial, and institutional—available to the implementing organization to support the intervention.

An intervention plan is helpful if it provides a sense of an organization’s experience and its connections with the members of its intended clientele and with other providers in its service area. Implementing organizations should also describe their relationship with other collaborating organizations and how linkages and referrals will be made. Some interventions (such as prevention case management) are dependent upon strong linkages to other service providers. Acceptance of the provider by the community is essential to achieving desired results. These characteristics are partial indicators of provider familiarity with the needs of the community and the provider’s ability to “do business” there.

An agency must also have enough staff to implement the intervention at the level it proposes. Staff size should be related to the level of activities the provider has set forth in various process objectives as well as to the overall target population to be covered. One way of expressing this is as a provider-to-client ratio with a justification for how that ratio will serve the needs of the target population and the provider. Expected levels of staff retention and loss should also be addressed in this section of the intervention plan.

Similarly, a provider’s ability to effectively implement an intervention (and get results) is related to the quality of staff implementing the activity. Having an explicit set of procedures for quality assurance for staffing and implementation enhances the overall effectiveness of the intervention. A quality assurance plan is likely to include a thorough orientation, ongoing training, and clear lines of supervision and oversight.

Budget and Resources. The budget and its accompanying narrative should be considered part of the intervention plan. At the end of an agency’s budget periods, the budget can be compared with actual expenditures to determine whether the budget was a good estimation of expenses and to decide where refinements might be made.

Table 3.10 lists some criteria for reviewing the relevance and scientific soundness of the features of the implementing organization.

Table 3.10

Evaluating Implementation Strategies and Process Objectives
Relevance In general, the implementation strategy needs to address the steps of the intervention, its cultural and language appropriateness, and issues of access and client involvement.

Process objectives need to be consistent with the activities outlined in the implementation strategy.

Scientific Soundness The intervention plan should provide a detailed enumeration of the activities that are needed to implement the intervention. It should also provide information on how the agency will ensure cultural and language appropriateness and optimize client access and involvement.

Process objectives should be clear, consistent, and measurable. Each intervention should have process objectives that specify the number, frequency, and duration of activities over the period covered in the proposal.

Describing the Data System

The preceding steps in preparing an intervention plan have focused on the description of the intended intervention—its relevance in the jurisdiction, its scientific basis, and the protocol for implementing it. The intervention plan should also describe how the provider will manage and refine (if necessary) the intervention and how the provider will document progress to be accountable to stakeholders.

Concrete information about progress is essential to ensure that high quality prevention services are delivered as intended, intended clients receive those services, training and supervision are provided in response to identified needs, and resources are expended judiciously. A minimal data system to serve these purposes would document what has been done and would be used to assess intervention progress and help identify ways to improve it; such a system is at the heart of process evaluation, which will be discussed in more detail in the next chapter.

Management data are most useful when they are kept current. The characteristics of the intervention that have been proposed in the intervention plan are the first data that must be collected. Data about the implementation and outcomes of each intervention complete the data system. Therefore, it is important for the health department and its grantees to demonstrate the capacity and commitment to collect data relevant to program implementation.

Collecting process data is often viewed as a time-consuming process—time that could be better spent serving clients with direct services. Although everyone is concerned about providing the best possible prevention services to the most people, many people are willing to continue providing services without proven value. Stakeholders and funding providers—from federal policymakers to governors to community planning groups and members of the target populations— are demanding empirical evidence of what is being done for people at risk for HIV and how well those services work. The data to support these activities can be gathered fairly easily. However, as with anything of value, a commitment of effort must be made. The effort required can be minimized by integrating data collection into existing activities or creating administrative systems to handle data collection.

Such a data system could be a paper file system. However, one reason given by program managers in the past for not collecting or using data to manage programs is that shuffling a large number of paper records is cumbersome, time-consuming, and more effort than it is worth. Today, however, computers and database software are commonplace and easy to use, increasing productivity. The largest time demands of a computer system are the initial creation of the database and entering the data into the system. The time savings occurs in using the data for decision making, with the ability to examine the data in any number of permutations, create multiple reports, and, generally, have at-a-glance access to all information.

There are a few different scenarios for the development of a management information system database for intervention plan and process data:

  • Some health departments have existing, sophisticated computer systems in which these data are already collated; others have systems that could be expanded to include intervention data.
  • Some health departments may have staff who are computer-proficient and able to develop such a database using commercially available software (some of which may have been provided with the computer when it was purchased).
  • Another option is to have a programmer develop software for this purpose; this may be an opportunity to have financial systems, human resources, and other data systems situated in a single system.

Obtaining data from grantees and contractors may require the use of existing processes. For instance, health department subcontractors are usually required to report on a monthly, quarterly, or annual basis. These are excellent opportunities for receiving manageable amounts of data that can be entered into a central system with minimal effort at any one time. A data system for intervention information would only require standardizing the reporting expectations and requesting data that can be used systematically in a computerized system. This may entail using categorical or quantitative data when possible (as opposed to narrative or qualitative data, although this type of information is also helpful and appropriate in many instances). The example evaluation criteria are presented in Table 3.11.

Table 3.11

Evaluating the Data System
Relevance The data system should be linked to the process and outcome objectives identified in the intervention plan.
Scientific Soundness The data system should include variables to address each process objective (e.g., numbers of clients served, number of services provided, resources used, quality of services) and, where possible, each outcome objective. It should include a plan for collecting data that includes data sources, staff responsibilities for collecting and reporting the data, and a protocol for how the system will be implemented. While a computerized database for managing data is not a requirement, providers should consider the difficulties they might face when using a paper-based system.

In the Appendix

Appendix A contains the list of 25 citations that the Behavioral Intervention Research Branch has determined have scientific evidence supporting their effectiveness.

The example forms in Appendix B are provided as resources for grantees and their subcontractors to use if they wish to collect information about proposed interventions. There is one form for each of the major types of intervention (e.g., individual-level, outreach, health communication). These forms can be modified or added to as needed to meet the particular needs of the jurisdictions.

Back to top

References and Resources

Bandura, A. Perceived self-efficacy in the exercise of control over AIDS infection. In Mays, V.M., Albee, G. W., & Schneider, S. F., eds. Primary Prevention of AIDS. Newbury Park, CA: Sage, 1989. pp. 128-141.

Centers for Disease Control and Prevention. Planning and Evaluating HIV/AIDS Prevention Programs in State and Local Health Departments: A Companion to Program Announcement #300, Atlanta: Centers for Disease Control and Prevention, 1993.

Centers for Disease Control and Prevention. Planning and Conducting Street Outreach Process Evaluation, Atlanta: Centers for Disease Control and Prevention, 1994.

Centers for Disease Control and Prevention. Guidelines for Health Education and Risk Reduction Activities. Atlanta: Centers for Disease Control and Prevention, 1995.

Centers for Disease Control and Prevention. HIV Prevention Case Management: Guidance. Atlanta: Centers for Disease Control and Prevention, 1997.

Chen, H-T. Theory-driven evaluations. Newbury Park, CA: Sage, 1990.

Corby, N. H., & Wolitski, R. J., eds. Community HIV Prevention: The Long Beach AIDS Community Demonstration Project, Long Beach: University Press, California State University, 1997.

The Health Communication Unit at the Centre for Health Promotion, University of Toronto. Evaluating Health Promotion Programs, No date.

Mantell, J. E., DiVittis, A.T., & Auerbach, M. I. Evaluating HIV Prevention Interventions. Plenum Press: New York and London, 1997.

National Community AIDS Partnership. Evaluating Prevention Programs in Community-Based Organizations, 1993.

National Minority AIDS Council. The Program Development Puzzle: How to Make the Pieces Fit. 1997.

National Research Council. Evaluating AIDS Prevention Programs, Expanded Edition. Washington, D.C.: National Academy Press: 1991.

Prochaska, J. O. & DiClemente, C. C. Stages of change in the modification of problem behaviors. Progress in Behavior Modification, 1992;28:183-218.

Prochaska, J. O., Redding, C .A., Harlow, L. L., Rossi, J. S., & Velicer, W. F. The Transtheoretical Model of HIV prevention: A review. Health Education Quarterly, 1993, 21, 471-486.

U.S. Department of Health and Human Services. Making Health Communication Programs Work: A Planner’s Guide. Washington, DC: NIH Publication No. 92-1493, 1992.

Back to top

APPENDIX A: Interventions With Scientific Evidence of Effectiveness

For more information, contact:

Ellen Sogolow, Ph.D.
Health Scientist
Behavioral Intervention Research Branch
Division of HIV/AIDS Prevention
1600 Clifton Rd.
Mailstop E-37
Atlanta, GA 30333
(404) 639-1900
eds0@cdc.gov

CDC AIDS Community Demonstration Projects Research Group. Community-level HIV intervention in five cities: Final outcome data from the CDC AIDS Community Demonstration Projects. AJPH 1999;89(3):336-345.

Cohen D, Dent C, MacKinnon D. Condom skills education and sexually transmitted disease reinfection. J of Sex Research 1991;28(1):139-144.

Cohen DA, MacKinnon DP, Dent C, Mason H, Sullivan E. Group counseling at STD clinics to promote use of condoms. Pub Hlth Reports 1992;107(6):727-730.

Des Jarlais DC, Casriel C, Friedman SR, Rosenblum A. AIDS and the transition to illicit drug injection: Results of a randomized trial prevention program. British Journal of Addictions 1992;87(3);493-498.

DiClemente RJ & Wingood GM. Randomized controlled trial in an HIV sexual risk-reduction intervention for young African-American women. JAMA 1995;274(16):1271-1276.

El-Bassel N & Schilling RF. 15-Month follow-up of women methadone patients taught skills to reduce heterosexual HIV transmission. Pub Hlth Reports 1992;107(5):500-504.

Hobfoll SE, Jackson AP, Lavin J, Britton PJ, Shepherd JB. Reducing inner-city women's AIDS risk activities: A study of single pregnant women. Health Psychology 1994;13(5):397-403.

Jemmott JB, Jemmott LS, Fong GT. Reductions in HIV risk-associated sexual behaviors among Black male adolescents: Effects of an AIDS prevention intervention. AJPH 1992;82(3):372-377.

Kamb ML, Fishbein M, Douglas JM, et al. Efficacy of risk-reduction counseling to prevent human immunodeficiency virus and sexually transmitted disease: A randomized controlled trial. JAMA 1998;280(13):1161-1167.

Kegeles SM, Hays RB, Coates TJ. The Mpowerment Project: A community-level HIV prevention intervention for young gay men. AJPH 1996;86(8):1129-1136.

Kelly JA, Murphy DA, Washington CD, et al. The effects of HIV/AIDS intervention groups for high-risk women in urban clinics. AJPH 1994;84(12):1918-1922.

Kelly JA, St. Lawrence JS, Hood HV, Brasfield TL. Behavioral intervention to reduce AIDS-risk activities. J of Counseling & Clinical Psych 1989;57(1):60-67.

Kelly JA, St. Lawrence JS, Stevenson Y, et al. Community AIDS/HIV risk reduction: The effects of endorsements by popular people in three cities. AJPH 1992;82(11):1483-1489.

Kirby D, Barth RP, Leland N, Fetro JV. Reducing the risk: Impact of a new curriculum on sexual risk-taking. Fam Plan Perspectives 1991;23(6):253-263.

Lauby JL, Smith PJ, Stark M, Person B, Adams J. A community-level HIV intervention for inner-city women: Results of the Women and Infants Demonstration Trial. (unpublished report)

Magura S, Kang S, Shapiro JL. Outcomes of intensive AIDS education for male adolescent drug users in jail. J of Adolescent Health 1994;15(6):457-463.

Main DS, Iverson DC, McGloin J, et al. Prevention HIV infection among adolescents: Evaluation of a school-based education program. Prev Med 1994;23(4):409-417.

McCusker J, Stoddard AM, Zapka JG, Morrison CS, Zorn M, Lewis BF. AIDS education for drug abusers: Evaluation of short-term effectiveness. AJPH 1992;82(4):533-540.

O'Donnell CR, O'Donnell L, San Doval A, Duran R, Labes K. Reductions in STD infections subsequent to an STD clinic visit. Sex Transm Dis. 1998;25(3):161-168.

Rotheram-Borus M, Van Rossem R, Lee M, Gwadz M, Koopman C. Reduction in HIV risk among runaway youths. (anticipated publication, 1999).

Siegal HA, Falck RS, Carlson RG, Wang J. Reducing HIV needle risk behaviors among injection drug users in the Midwest: An evaluation of the efficacy of standard and enhanced interventions. AIDS Ed and Prevent 1995;7(4):308-319.

Stanton BF, Li X, Ricardo I, Galbraith J, Feigelman S, Kaljee L. A randomized, controlled effectiveness trial of an AIDS prevention program for low-income African-American youths. Archives of Peds and Adoles Med 1996;150(4):363-372.

St. Lawrence JS, Brasfield TL, Jefferson KW, Alleyne E, O'Bannon RE, Shirley A. Cognitive-behavioral intervention to reduce African-American adolescents' risk for HIV infection. J of Consult and Clinical Psych 1995;63(2):221-237.

Valdiserri RO, Lyter DW, Leviton LC, Callahan CM, Kingsley LA, Rinaldo CR. AIDS prevention in homosexual and bisexual men: Results of a randomized trial evaluating two risk-reduction interventions. AIDS 1989;3(1):21-26.

Wenger NS, Linn LS, Epstein M, Shapiro MF. Reduction of high-risk sexual behavior among heterosexuals undergoing HIV antibody testing: A randomized control trial. AJPH 1991;81(12):1580-1585.

Back to top

APPENDIX B: Example Forms to Summarize Intervention Plan Data

  • Individual-level interventions
  • Group-level interventions
  • Outreach
  • Prevention Case Management
  • Partner Counseling and Referral Services
  • Health Communications and Public Information Activities
  • Community-level interventions

Back to top

Go to Monitoring and Evaluating the Implementation of HIV Prevention Programs


1 Risk populations are characterized primarily by the risk behaviors associated with specific means of HIV transmission. This classification is not intended to minimize the importance of other ways of characterizing people at risk for HIV or in need of HIV prevention services. For example, there are good reasons in some situations to classify people according to demographic characteristics (e.g., age, race/ethnicity) or by occupation (“sex worker”). The use of behavioral risk populations has two primary advantages. First, it highlights the importance that CDC places on clarifying the behavioral risk that is the target of prevention efforts. Second, it provides a common denominator with which to describe the groups of people being served with CDC funds.
2 For many purposes, CDC will project numbers of people to be served in an ensuing year based on the CT data submitted by the jurisdiction the previous year. Therefore, the intervention plan example reporting form A for Counseling and Testing is limited to a narrative discussion of any differences that are anticipated between a previous year’s service level and the level expected in the next year.
3 I = Data requested for Intervention Plans
4 P = Data requested for Process Evaluations
Last Modified: October 15, 2007
Last Reviewed: October 15, 2007
Content Source:
Divisions of HIV/AIDS Prevention
National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention
spacer
spacer
spacer
Home | Policies and Regulations | Disclaimer | e-Government | FOIA | Contact Us
spacer
spacer
spacer Safer, Healthier People
spacer
Centers for Disease Control and Prevention, 1600 Clifton Rd, Atlanta, GA 30333, USA
800-CDC-INFO (800-232-4636) TTY: (888) 232-6348, 24 Hours/Every Day - cdcinfo@cdc.gov
spacer USA.gov: The U.S. Government's Official Web PortalDHHS Department of Health
and Human Services