Time Lines and Dues Dates
Membership Grid Data
Evaluating Linkages
Issues Related to Both Intervention Plans and Process Monitoring
Intervention Plans
Intervention Definitions
Population Definitions
Process Monitoring
Outcome Evaluation
Use of Evaluation Data
Relationship Between the Guidance, Other Evaluation Efforts,
and CDC Program Announcements
Technical Assistance

Listed below are CDC's answers to frequently asked questions (FAQs) about the Guidance. These FAQs were previously distributed at CDC's 2001 First Annual HIV Prevention Program Evaluation Meeting in Atlanta, June 19-20, 2001. They are reproduced here and some additional questions and answers have been added. Questions were identified from three sources: 1) issues that emerged during the initial Guidance trainings for health departments conducted in Atlanta, January - March 2000, 2) issues identified during interviews with health departments and other stakeholders conducted during the development of this manual, and 3) Guidance-related technical assistance requests to CDC. Answers to these questions were developed and approved by CDC with input from NASTAD, health department representatives, and other stakeholders.

Time Lines and Dues Dates

May jurisdictions phase-in process monitoring?

Data are due in April 2001. As is the case for all issues and concerns about the Evaluation Guidance, issues and concerns about the submission of process monitoring data should be discussed with project officers. CDC is aware of the challenges health departments may face in securing process monitoring data, especially for the first time, and will work with jurisdictions to help resolve any problems.

How should we coordinate the timing of process monitoring data and the progress reports?

Progress reports on activities that took place the previous year are due each April. Data on monitoring the implementation of prevention programs are due in April since the data cover activities that occurred the previous year. The first set of data for monitoring program implementation is due in April 2001 for the period, January - December 2000.

Since individual jurisdictions may have unique funding cycles, how should intervention plan data be reported?

Intervention plan data (chapter 3 of the Evaluation Guidance) should be submitted to CDC in September with health departments' applications for cooperative agreement funding. Intervention plan data cover the period January - December 2001. CDC is aware that some jurisdictions may not have their intervention plan data available in September because contracts with grantees for the year beginning January 1 may not be in place then. These situations should be discussed with project officers and a reasonable deadline for submitting the data should be agreed upon.

For outcome evaluation, what is actually due in September 2003?

Grantees receiving at least $1 million in cooperative agreement funding who choose to conduct outcome evaluation are to report the results of an outcome evaluation of at least one intervention in September 2003. The types of information to report are described in Volume 1 of the Evaluation Guidance. The Supplemental Handbook, Volume 2 of the Evaluation Guidance, contains more information on how to conduct outcome evaluation. Technical assistance requests should be channeled through project officers.

Membership Grid Data

Where do you count people on the membership grids who work with a population but aren't actually members of that population (e.g., people who counsel IDUs but aren't IDUs themselves)?

The "membership grids" ask for CPG (community planning group) representation by primary and secondary agency and primary and secondary expertise (among other types of representation). If persons work with at-risk populations but are not actually members of the population, they could be counted as an agency representative and/or a representative with expertise in behavioral or social science or interventions.

Evaluating Linkages

For Chapter 5 of the Evaluation Guidance on evaluating linkages between the prevention plan, funding application, and resource allocation, are jurisdictions to report service units or number of interventions?

Chapter 5 discusses the evaluation of two types of linkages: 1) linkages between the comprehensive HIV prevention plan and the CDC funding application and 2) linkages between the comprehensive HIV prevention plan and resource allocation.

To evaluate linkages between the comprehensive HIV prevention plan and resource allocation, jurisdictions should compare interventions funded in the previous year with interventions recommended in the prevention plan for that year. It is suggested that jurisdictions submit the worksheet found in the appendix to Chapter 5. That worksheet asks for interventions (recommended in the plan and funded) by name of intervention, not by service units or numbers of interventions.

To evaluate linkages between the comprehensive HIV prevention plan and the CDC funding application, jurisdictions are asked to report which recommended interventions in the plan are not included in the application. There is a worksheet in the appendix to Chapter 5 that can assist jurisdictions in listing the interventions recommended in the plan and funding application.

Jurisdictions should note that the interventions in the comprehensive HIV prevention plan that are compared to the CDC funding application and to resource allocation could be intervention types, such as individual-level counseling and street outreach, or interventions at specific locations such as individual-level counseling carried out at the St. James public housing development, or outreach conducted at the corner of 14th Street and Mulberry Place. Also, the target populations in the comprehensive prevention plan may not be the same as the target populations in the Evaluation Guidance. The Evaluation Guidance uses risk population categories, including MSM; MSM/IDU; heterosexual contact; and mother with/at risk for HIV while jurisdictions may have target populations in their plans that are not based on a risk behavior, such as the homeless, youth, and incarcerated persons.

Beyond these evaluations of linkages, jurisdictions are free to perform enhanced evaluations of linkages that will provide additional data useful for community planning. For example, an expanded worksheet could be used to indicate interventions that do not have CDC funding, such as interventions funded by the state. This enhanced information will minimize the appearance of "gaps" in service.

Can alternative means of demonstrating linkages between comprehensive plans, applications, and funded interventions be used instead of the forms in the Guidance?

The data on linkages need to be reported to CDC; the example forms in the Guidance are provided for reporting convenience. Other ways of reporting the same data are acceptable.

The Evaluation Guidance requests minimum data on the demonstration of linkages; jurisdictions may report additional data. CDC understands that looking at interventions funded solely by CDC funding may create the "appearance" of gaps, when - in fact - the gaps are filled by interventions receiving non-CDC funds.

Issues Related to Both Intervention Plans and Process Monitoring

On the forms for intervention plans and process monitoring, should we count all clients if the intervention is only partially funded by CDC, or should we use a "pro-rated" number?

For interventions where CDC cooperative agreement funding is only one funding source, health departments should "pro-rate" the number of clients who receive the intervention with CDC cooperative agreement funding. Departments should know what percentage of funding cooperative agreement funds represent for the intervention and use that percentage to figure out the "pro rated" number of clients. For example, if CDC cooperative agreement funding represents 75 percent of the funding for the intervention, then 75 percent of the clients should be considered CDC clients. The gender, race and ethnicity of these clients (and their ages, if possible) should also be identified. The distribution of gender, race and ethnicity for the 75 percent should represent the distribution for all clients receiving the intervention. For example, there are 100 clients; 50 are African American males; 25 are Latino males; and 25 are White males. The jurisdiction would report 75 clients: half (50 percent) are African American males = 38 African American males; 25 percent are Latino males = 19 Latino males; 25 percent are White males = 18 White males.

The forms in the Evaluation Guidance on process monitoring ask for statewide definitions or guidelines for the intervention being reported on, but the forms for intervention plans do not ask for this information. What does CDC want and when should the material be submitted?

CDC would like to receive one set of definitions or guidelines for each jurisdiction's interventions. This material should be submitted with intervention plan data since those data are due before the process monitoring data. For convenience, jurisdictions may submit one master list, rather than separate definitions or guidance for each risk population per intervention.

The forms in the Evaluation Guidance on intervention plans and process monitoring ask about interventions provided by various types of agencies. How are minority CBOs, faith communities, and individual agencies defined?

A minority board CBO has a board or governing body composed of greater than 50 percent of the racial/ethnic minority population to be served, and members of the racial/ethnic minority population to be served must serve in greater than 50 percent of key positions in the organization, including management, supervisory, administrative, and service provision positions.

The Evaluation Guidance refers to "Faith Community." For the Evaluation Guidance, a faith community can include faith-based CBOs as well as other faith-based entities funded to carry out HIV prevention, such as a coalition of clergy. Specifically in regard to faith-based CBOs, CDC defines them as organizations that have a faith, spiritual, or religious focus or constituency, and have access to local faith, spiritual and religious leaders and communities. Examples of faith-based CBOs include individual churches, mosques, temples, or other places of worship; a network or coalition of churches, mosques, temples, or other places of worship; or a CBO whose primary constituents are faith, spiritual, or religious community organizations or leaders.

"Individual" does not refer to an agency, but to an individual person not affiliated with a public or private agency or organization; e.g., an individual hired as a consultant.

How do you report the type of agency when it can fit more than one category for intervention plan and process monitoring data?

Health departments need to decide on just one code for an agency that can fit more than one code. Choose the description that BEST describes the grantee or the one code the grantee would use to describe itself.

Should the client designation on the Evaluation Guidance forms that reads "Asian/Pacific Islander" be reworded to separate Asian and Pacific Islander?

The race and ethnicity designations on the forms are being revised to conform to federal reporting requirements established by the Office of Management and Budget and CDC guidelines for consistency in data collection. The races will include "American Indian or Alaska Native;" "Asian;" "Black or African American;" "Native Hawaiian or other Pacific Islander;" and "White." The forms will also include "Hispanic or Latino," and "Not Hispanic or Latino." These revised forms will be available next year and should be used for the submission of intervention plan data in September 2001 (covering the period, January - December 2002) and process monitoring data in April 2002 (covering the period, January - December 2001).

What is the definition of Hispanic?

Hispanic or Latino is defined as "a person of Cuban, Mexican, Puerto Rican, South or Central American, or other Spanish culture or origin, regardless of race."

How should race and ethnicity be recorded when data are based on observation for outreach?

Best estimates should be used to record and report process monitoring data.

Why do the Evaluation Guidance forms include an "unknown" category for gender but not for race and ethnicity?

As noted above, the forms are being revised to meet federal directives and be more consistent internally.

Why are there different age categories on the Evaluation Guidance forms compared to the budget tables?

The budget tables refer to age in regard to budgets for one category - "young people" 13 to 25 years of age. The Guidance forms have three categories for age: 19 or younger; 20 - 29; and 30+ years old to capture three important age distinctions: youth, young adults, and older adults. The Division of HIV/AIDS Prevention is working to reconcile any differences in the ways age data are reported. Since different branches may report and/or collect age data in different ways (for example, one group may want more fine-tuned data than three categories will allow), CDC is working to assure that data can be "collapsed" so the categories can fit one another.

Will CDC understand that differences between intervention plan data on clients to be served and data on clients served in process monitoring may be due to difficulty documenting risk behaviors rather than interventions failing to reach clients?

Yes. CDC requests that health departments explain these challenges in a narrative format.

Intervention Plans

For intervention plans, should jurisdictions estimate clients or contacts?

Ideally, the best estimate for unduplicated clients to be served by the particular intervention should be reported. However, contacts are acceptable for outreach only. For all data collection by intervention, jurisdictions should do their best to collect unduplicated client counts.

If community planning considers scientific evidence and justification when prioritizing interventions, and the health department then funds these interventions, does this meet requirements for scientific evidence and justification for intervention plans? Or are grantees expected to submit additional information on scientific evidence and justification?

CDC's Guidance on HIV Prevention Community Planning, calls for CPGs to prioritize populations at high risk for HIV and to prioritize culturally and linguistically appropriate interventions for them. Criteria to be considered in prioritizing interventions include outcome effectiveness; relative costs and effectiveness; sound scientific theory when outcome effectiveness information is lacking; and values, norms, and preferences of the communities for whom services are intended. The Guidance states, "At a minimum, the community planning groups must provide a clear, concise, logical statement as to why each population and intervention given high priority was chosen."

With this in mind, intervention plans that include populations and interventions based on the priorities set in the comprehensive HIV prevention plan will meet the requirements for "evidence or theory basis for the intervention." This is the very minimum criterion for asserting the evidence or theory basis for the intervention. However, the community planning process will most likely not go into enough detail to provide evidence to justify application to the target population AND setting. In order to assert justification for the target population and setting, CDC prefers that health departments request logic models or depictions of program theory from applicants and/or grantees that show the proposed relationship between the intervention and expected outcomes for the particular target population in a particular setting.

Health departments that have Requests for Proposals (e.g., requests for applications, invitations to negotiate, etc.) that ask applicants to specifically discuss the evidence or theory basis of proposed interventions as well as justification for application to the target population and setting will meet requirements for scientific evidence and justification. In addition, if the RFPs also ask applicants to specifically discuss factors relating to the sufficiency of the service delivery plan (e.g., provider training and supervision, quality assurance and accountability mechanisms), this, too, will meet the requirements for sufficiency of the service delivery plan.

If the criteria above are met, grantees should not be expected to submit additional information.

What are the minimum bounds of acceptability for scientific evidence and justification for intervention plans? What would be an example?

Chapter 3 of the Evaluation Guidance contains discussion of how to assess the intervention's evidence basis and how to assess the intervention's justification to the target population and setting. There is also discussion on how to determine the sufficiency of the service plan. More extensive discussion is found in Chapter 3 of Volume 2: Supplemental Handbook. CDC's Guidance on community planning, referenced above, is another source of information on factors to consider in prioritizing interventions.

As noted above, the minimum bound of acceptability for scientific evidence is compliance with the CPG-approved priorities in the comprehensive prevention plan. However, the minimum bound of acceptability for justification is a logic model or program theory description that shows the relationship between the intervention and expected outcomes for the particular target population in a particular setting. If health department grantees were funded based on applications that provided a high quality discussion of the evidence or theory basis of interventions and justification to the target population in a particular setting, then those descriptions are acceptable.

What should one do if the intervention changes after it has been funded? Should health departments submit revised intervention plans? What are the implications for comparing intervention plan and process monitoring data?

The intervention plan data that health departments submit to CDC may be considered "benchmark" data for health departments and CBOs to use to set the stage for process evaluation; that is, understanding how and why process monitoring data differ from intervention plan data. If process monitoring data reveal that fewer (or even more) clients are being served than anticipated by intervention plan data or that different populations are being reached than those originally targeted, this is useful information to use to modify interventions to realistically meet client needs. This information should then be used to set more realistic plans for the next year.

If, for example, an intervention is dropped and another one added for a target population, this information should not be submitted to CDC. Health departments should not submit revised intervention plan data to CDC. Intervention plan data are to be submitted only once a year.

CDC recognizes that intervention plans change and a strict comparison of intervention plan and process monitoring data would often show major differences between the two sets of data.

What is to be written in the "Notes/Comments Field" on intervention plan forms?

As the Evaluation Guidance indicates, the "Notes/Comments Field" is an optional field health departments may use to provide explanation, clarification, or additional information about the data provided on the form. Health departments are not required to provide notes or comments.

Intervention Definitions

How do we distinguish between individual level interventions (ILIs) and counseling and testing in process monitoring?

An ILI may or may not lead to testing, and all ILI clients seen outside of the counseling and testing site per se -- whether they go on to get tested or not -- are counted in process monitoring for ILIs. Clients who are counseled as part of pre-test counseling should not be counted as ILI clients. Counseling and test site clients are reported on the HIV counseling and testing report form.

Is outreach for counseling and testing not considered part of outreach?

"Outreach" is generally defined as educational interventions conducted face-to- face in places where clients congregate. For the purpose of the Evaluation Guidance, outreach solely for the purpose of getting clients into counseling and testing, should not be included under "Outreach."

In regard to "Partner Counseling and Referral Services (PCRS), for intervention plans and process monitoring, are we counting HIV+ index cases or the partners of HIV+ persons who are notified and counseled?

The first page of the forms for intervention plan and process monitoring data for PCRS ("HIV-Infected Clients to Receive PCRS with CDC Funds" and "HIV- Infected Clients Who Received PCRS with CDC Funds," respectively) refers to HIV+ index cases. Page 2 of the process monitoring form for PCRS asks for data on the sex or needle sharing partners of HIV+ index cases.

Where do we report on CTRPN and coalition building as interventions?

The forms in the Evaluation Guidance for reporting intervention plan data as well as process monitoring data do not cover CTRPN and coalition building. It is suggested that you provide a narrative report that describes these efforts.

Can CDC funding be used for policy interventions?

CDC funding, like all funding from Congress, cannot be used to lobby federal or local legislative bodies. CDC funds may not be used for propaganda purposes or for the preparation, distribution or use of such items as publications or radio or television presentations designed to support or defeat pending legislation.

However, CDC funding may be used for community-level interventions that seek to lessen risky conditions and behaviors in a community through a focus on the community as a whole. As the Evaluation Guidance points out, this is often done by attempting to alter social norms or characteristics of the environment. Such efforts are also referred to as "structural interventions" and may be funded with CDC cooperative agreement funding.

Specific questions regarding structural interventions and whether they meet funding requirements should be referred to project officers.

What intervention would you use for a "chatroom" on the Internet; for example, a chatroom for MSM?

HIV/AIDS health education and risk reduction information provided to persons via a chatroom should be considered under "Other Interventions" on the forms for intervention plans and process monitoring. The intervention is not necessarily an individual-level intervention, according to the intervention types in the "Evaluation Guidance," since more than one individual is reached, and it's not necessarily a group-level intervention or health communications and public information. Use the form for other interventions or provide a narrative description.

The definition of Prevention Case Management (PCM) in the Evaluation Guidance seems more loosely defined than CDC's guidance on PCM. Which definition applies?

CDC's guidelines on PCM are not mandates for how PCM should be implemented. For evaluation, use the definition of PCM in the Evaluation Guidance. This broader definition will include the definition found in CDC's PCM guidance. As with all the intervention categories, national data about PCM will include some data from more rigorous implementation and some from less rigorous implementation. This is also true of ILI, GLI, and outreach interventions.

What constitutes "skills building" for GLI? Does every participant in a GLI need to demonstrate the skill or is it sufficient for one client to demonstrate the skill and the others to observe?

A variety of skills can be "built" during GLI (and ILI). If, for example, the skill is condom use and a phallic model is used to demonstrate how to fit a condom and at least one member of the group participates in the demonstration, the entire group can be considered as having participated in the skill building exercise. Critical thinking and decision-making skills are skills that can be enhanced during GLI. If these skills are discussed and demonstrated by members of the group through various exercises or activities, the entire group can be considered as having participated in the intervention.

What is really meant by CLI (community-level interventions) and social marketing? What is the distinction between CLI and a set of related but distinct interventions working toward a common goal (e.g., an agency implementing outreach, ILI and GLI targeting MSM in a particular community)? Should a CLI be deconstructed into its component interventions and then each intervention separated for intervention plans and process monitoring reporting?

As the Evaluation Guidance puts it, "CLI are interventions that seek to improve the risk conditions and behaviors in a community through a focus on the community as a whole, rather than by intervening with individuals or small groups. This is often done by attempting to alter social norms, policies, or characteristics of the environment. Examples include community mobilizations, social marketing campaigns, community-wide events, policy interventions, and structural interventions."

Social marketing is the application of commercial technologies to the planning and implementation of prevention programs. Social marketing is not social advertising, social education, attitude change, or socially responsible marketing of HIV prevention messages. Examples of social marketing programs at CDC include the "America Responds to AIDS" campaign and the "5-A-Day Nutrition" campaign.

The definition above of CLI indicates that it does not focus on individuals or small groups whereas outreach, ILI, and GLI do focus on individuals and small groups. If a grantee employs a set of related but distinct interventions working toward a common goal, it is appropriate to "deconstruct" that program into its component elements and report on each intervention separately for intervention plan and process monitoring data.

How should an intervention be categorized that counsels couples and includes skills building and/or service brokerage? What if it does not include skills building or service brokerage?

An intervention that counsels couples and includes skill building and service brokerage should probably be categorized as GLI (the intervention could be considered PCM if it meets the criteria for PCM established by the health department or grantee or if it is carried out in accordance with CDC's guidance on PCM). In this example, "counseling" refers to HIV/AIDS prevention counseling, not mental health counseling. Skills building (not service brokerage) must be a part of GLI. If there is no skills building, then the intervention cannot be categorized as GLI. Service brokerage is not considered a necessary component of GLI. It is, however, a necessary component of PCM.

What intervention type should be used to report condom drop-off activities (e.g., putting condoms in bowls in bars)?

Condom drop-off activities should be recorded under "Other Interventions" because they do not readily fit under any other intervention type. For example, "Outreach" is not appropriate because there is no face-to-face contact with clients. "Health Communications/Public Information" is not appropriate because no information is conveyed by the drop-off activities. When interventions are reported as "Other," the intervention should be explained.

What intervention type should be used to report brochures and other materials that health departments distribute to their grantees? What about materials they distribute to agencies they don't fund for HIV prevention?

The recipients of the printed materials distributed by health departments do not affect the intervention type that should be used for reporting. The intervention type is "Health Communications/Public Information" (print media distribution).

When does outreach become an individual-level intervention? For example, during outreach the outreach worker can spend a lot of time with one person on health education, risk reduction counseling, and skills building. If an ILI develops out of an outreach encounter, should health departments report on both interventions?

If outreach develops into an intervention that meets the criteria for ILI, then both intervention types should be reported.

Population Definitions

How should we categorize interventions focusing on women who have sex with women (WSW)?

WSW is not a risk population used in the Evaluation Guidance. The behavioral risk populations used in the Guidance are not intended to be exhaustive but to represent the majority of cases of transmission. For process monitoring (chapter 4), jurisdictions may report on risk populations that do not fit the categories in the Guidance in a narrative format using the variables indicated on the process monitoring forms in chapter 4 (e.g., gender, race, ethnicity, setting, etc.).

How should jurisdictions code a population whose risk includes both MSM and IDU but the intervention is focusing specifically on MSM routes of transmission?

Since the intervention is focusing on MSM, the primary risk population should be coded as MSM. MSM/IDU should be used to code the risk population when the intervention is designed specifically to meet the needs of men who have sex with other men and use injection drugs.

What if the target behavior is reducing crack use?

The question to ask for any intervention is, "What is the behavioral risk for HIV that is being addressed?" In the case of an intervention to reduce crack use, the assumption is that the behavioral risk for HIV would be sexual risk associated with crack use, either MSM or heterosexual. If this is the case, then one of these sexual risks would identify the risk population.

Whose HIV risk is being addressed when an intervention targets the population "mother with or at risk for HIV infection?" Is it the mother, the fetus, or both?

Regarding "Mother with/at risk for HIV," the Evaluation Guidance states, "Intervention will address the HIV prevention needs of women who have HIV or are at risk of becoming infected and who are pregnant and, thus, at risk of transmitting HIV to their infant." Therefore, if the pregnant woman is HIV- negative, the risk is for both mother and infant. If the pregnant woman is HIV- positive, the risk is for the infant. The risk population category remains "Mother with/at risk for HIV."

How do you code populations when you have an "open" counseling intervention and anyone can use the service?

For intervention plans, project numbers for each primary population (risk population such as MSM, IDU). For process monitoring, report the primary population as accurately as possible. Counseling implies that a risk assessment will be completed and this should help inform reporting.

What definition should be used for heterosexual contact - there's an AIDS surveillance definition and a broader definition suggested by the Guidance?

Use the Evaluation Guidance's broader definition. The risk population category, "heterosexual contact," does include heterosexual contact with multiple partners of unknown risk.

Also, heterosexual risk can include risk to the client as well as risk from the client (e.g., the primary population for an intervention is "heterosexual" because clients have sex with injection drug users; the primary population for an intervention is "heterosexual" because clients are HIV-infected heterosexuals).

For the risk population categories in the Evaluation Guidance, such as MSM, is the reference to high-risk sex or any sex? Where do transgender persons or crack users fit in?

The MSM and heterosexual behavioral risk populations defined in the Guidance reference risk; for example, MSM are at risk through unsafe sex; heterosexual men and woman are at risk through unsafe heterosexual sex. It is assumed that a jurisdiction which funds an intervention for MSM has decided that the intervention, in fact, is reaching men likely to be at risk for HIV.

Transgender persons should be counted as clients who receive a particular intervention but they are not a primary or secondary risk population according to the Evaluation Guidance. If their risk for HIV is sexual, the risk population is either heterosexual or MSM depending on their current gender identification. Similarly, crack users is not a primary or secondary population. Their risk for HIV is most likely sexual (either heterosexual or MSM).

The primary and secondary populations are the behavioral risk populations identified in the Guidance. Jurisdictions may collect data on risk populations as the jurisdiction defines those populations separate and apart from CDC's definitions.

How should we categorize a population when the intervention is directed to a group comprised of two or more subpopulations with distinct risk behaviors; for example, an incarcerated population includes some MSM, some IDUs, and a few MSM/IDU?

Every effort should be made to estimate a primary and secondary population in situations where an intervention targets both populations (note that data are reported only on primary populations). As a last resort, two populations that cannot be distinguished as "primary" and "secondary" should be reported separately as two primary populations. Because the members of the group cannot be distinguished by risk, the full population should be counted in each primary population report (i.e., they will be double-counted).

A jurisdiction may "split" the population for local reporting, but must be careful to match the specificity of the intervention plan reporting to that of process monitoring; i.e., if the population is split for intervention plan estimation, then it should be split for process monitoring reporting.

Why does the CDC strategic plan discuss "youth" as a priority population when this is not a risk population in the Guidance?

With the exception of "Mother with/at risk for HIV" and "General Population," the Guidance uses behavioral risk population categories (i.e., MSM, MSM/IDU, IDU, and heterosexual) because intervention types are used to influence particular risky behaviors that transmit HIV disease. CDC's strategic plan discusses youth because interventions should be targeted at the risky behaviors youth engage in. Data on youth served should be provided under the age range categories for intervention plans and process monitoring. In a similar vein, the prevention needs of HIV-infected persons are discussed in the strategic plan but HIV- infected persons are not a risk population category in the Guidance. Health departments are encouraged to fund programs that serve youth and HIV-infected persons, but the data to be submitted to CDC should reflect the risk population categories of the Guidance.

Is there a time-frame for specifying risk behaviors? For example, if someone has used needles in the past, does it have to be in the past year (or 6 months or 3 months) for them to be reported as an IDU? Does the time frame vary for different behaviors?

Agencies will likely have their own policies on conducting a risk assessment or otherwise determining risk behaviors. Current risk behaviors are most important because interventions will target behaviors clients are currently engaged in.

Process Monitoring

On the process monitoring forms in regard to staffing and expenditures, do you want to know the number of volunteers or the number of volunteer hours?

The number of volunteers providing interventions should be reported regardless of the amount of time they volunteer.

The process monitoring forms ask for the number of clients receiving interventions in various settings. The instructions indicate that a "Clinic/Health Care Facility" includes an STD clinic, but the form has "STD Clinic" as a separate setting. How will this discrepancy be resolved?

The instructions will be revised to match the forms. "Clinic/Health Care Facility" will not include an STD clinic. (The instructions also refer to "Social Services Agency" but there is no corresponding designation on the form under type of setting. For social services agency, the "other" designation should be used.)

If an intervention reaches clients other than those intended by the intervention, how are these clients reported for process monitoring? For example, if street outreach intends to target IDUs, but outreach workers also encounter a lot of high risk heterosexuals, how is the heterosexual population reported on the process monitoring forms?

The process monitoring forms should contain data on the primary risk populations being served by the intervention. Data are not reported on secondary risk populations. It is possible that new primary risk populations will be added to an intervention type over time, and health departments should provide data on them when process monitoring data are due. If you find that you are serving different populations than the ones you originally planned to serve in your intervention plans, you should report process monitoring data about that new population if you redesigned your intervention to accommodate the new population or the new clients you are serving total at least 25% of your caseload. In regard to the question's example, if the heterosexual population comprises roughly 25% or more of the population reached during outreach, then process monitoring data should be provided on that population.

Should clients who attend only one session of a GLI be reported under GLI or ILI?

Group-level interventions (GLIs) should consist of multiple sessions. There will undoubtedly be cases where clients do not attend all of the sessions. Clients who attend only one session of a GLI should be reported under GLI and not ILI since GLI was the intervention being delivered.

Can you report risk populations for process monitoring based on the intended audience for the intervention or do you need to assess participants' risk? For example, if 10 people participate in a GLI targeting MSM, can you report that you reached 10 MSM if you do not collect data on their risk behaviors?

For some intervention types, it is appropriate for the interventionist to conduct a risk assessment. For example, a risk assessment should always be completed for clients in PCM, and CDC strongly encourages risk assessments for other interventions as well. When there is no risk assessment, the intent of the intervention should guide reporting for process monitoring. If the intent of GLI, for example, is to serve MSM and there is no risk assessment to document the risk behavior, then clients should be reported as MSM since the intervention is targeted and tailored for MSM. Since risk assessments are not done during outreach, the venue for the outreach should be considered. For example, if outreach is taking place in gay bars, then the risk population should be reported as MSM. If no specific risk population is targeted by an intervention (this could be the case for health communications/public information), then "General Population" should be used as the risk population category.

How do you report the number of clients served if a contractor conducts teacher training with the intention that the teachers will then provide prevention education to their students? How do you report the risk population and demographics in this scenario?

In this scenario, health communications/public information seems to be the intended intervention. Students are the targeted population and there is probably no one risk behavior that is targeted. If this is the case, "General Population" would be the risk population. However, the numbers of clients served cannot be reported until those data are provided, in writing, by the teachers who received training. The teachers should report back to the Contractee after their prevention education session takes place. If the intervention is designed to address heterosexual contact as the risk, then that risk population category should be used for reporting when data are provided by the teachers.

How should health departments characterize the type of agency delivering the intervention (item #6 on process monitoring forms) when the intervention is conducted by an agency sub-contracted by the health department's grantee? Should the agency type be coded as the health department's grantee or the agency sub-contracted by the grantee?

The intent is to capture data on the types of agencies actually carrying out interventions. Therefore, the agency that has been sub-contracted by the health department's grantee should be used for agency type.

Outcome Evaluation

Can you use proxy measures for behavior change for outcome evaluation such as attitudes, beliefs, norms, or behavioral intentions or do you need to measure actual behavior change?

Since the ultimate objective of HIV prevention is to change risky behaviors, measures of behavior change are preferred for outcome evaluation. However, measures of change in knowledge, attitudes, beliefs, norms, or intentions are acceptable.

Use of Evaluation Data

How will data be used and how will CDC guard against misuse?

The Evaluation Guidance states that data provided by health departments will be used for three purposes: 1) To report to federal, state, and local stakeholders (including communities, health departments, local and national organizations, and federal policymakers) progress made through HIV prevention programs supported by CDC funds; 2) To improve national policies regarding HIV prevention; 3) To identify ways to improve HIV prevention programs nationwide.

CDC is interested in aggregate, national-level data. It is not CDC's intent to use local data in a punitive way. Data are collected and analyzed for the purpose of program improvement. State-level data will be shared with project officers. State-level data will not be shared with persons outside of CDC without consultation and discussion with state health department officials.

Interventions may vary within a jurisdiction; for example, prevention case management may be carried out with varying levels of intensity throughout a state. Will data on interventions at the jurisdiction-level be pooled together in a national data set?

Yes, data on interventions will be pooled together, with the acknowledgment of differences in how interventions are delivered. Health departments may provide narrative to explain variations in interventions.

Will CDC change its funding formula to reflect the effectiveness of interventions. In other words, will jurisdictions get more money if their interventions are effective?

CDC does not foresee linking funding to empirically demonstrated effectiveness.

Will CDC penalize jurisdictions who report reaching fewer people if that is the result of efforts to more specifically target their interventions to certain risk behaviors?

No. This would be seen as improving interventions, and large numbers are not necessarily a measure of success.

Relationship Between the Guidance, Other Evaluation Efforts, and CDC Program Announcements

Will the Evaluation Guidance being developed for CBOs be different from the Evaluation Guidance for health departments?

The CBO Evaluation Guidance -- a document on HIV/AIDS prevention program evaluation for CBOs directly funded by CDC -- is under development, and health department representatives are involved. The intent is that the CBO Guidance be as consistent as possible with the Evaluation Guidance for health departments, including consistency between the data to be collected from directly funded CBOs and the data collected from health department grantees.

How does the Evaluation Guidance relate to evaluation of the whole health department?

The Evaluation Guidance pertains to prevention programs currently funded under Program Announcement 99004. The ideas, principles, and methods outlined in the Guidance may also be useful for evaluating prevention and/or care activities undertaken with state or city revenues, with other federal funds, or with other resources. However, the Evaluation Guidance does not ask that efforts funded outside of CDC cooperative agreement funds be evaluated.

Health departments may be asked by funders other than CDC for HIV/AIDS program evaluation. The Program Evaluation Research Branch (PERB) is working with other branches in CDC and with HRSA to develop a common language for evaluation; for example, by standardizing definitions of populations and interventions.

Will CDC reconcile Program Announcement and Evaluation Guidance requests?

PERB and PBB are working together to reconcile any differences between program announcements and the Evaluation Guidance, including differences in the definitions of interventions and populations.

How will CDC reinforce the message that the Guidance intervention definitions will apply to future activities?

PERB is working to standardize definitions of interventions and populations. However, it is important to note that definitions in the Evaluation Guidance do not have to replace local taxonomies. Jurisdictions may use definitions of interventions and populations already in place locally. They just need to make sure local taxonomies are used consistently and that they fit categories in the Guidance.

What is the relationship between external reviews and progress reports?

Progress reports submitted in April will undergo a "technical review" by project officers. However, external reviewers may have the opportunity to refer to progress reports.

How do differences between Evaluation Guidance definitions for risk populations and surveillance definitions for exposure category relate to how budget tables are viewed? Are budget tables compared to surveillance data?

Chapter 5 of the Evaluation Guidance discusses the importance of linkages between the comprehensive HIV prevention plan and the allocation of resources. "Epi" or surveillance data should inform the prevention plan and there should be a strong and logical linkage between the plan and interventions and populations that get funded. PERB and PBB are discussing how Evaluation Guidance data, including budget tables and surveillance/ "Epi" data in the comprehensive plan, will be reviewed with the objective of improving community planning and prevention programming.

Can process monitoring data regarding expenditures replace the budget tables?

No. At this time, budget tables will continue to be submitted, but in April, rather than September. The form will be revised for health departments to reflect actual expenditures, to the extent possible. The revised table will be due in April 2001 to reflect the period, January - December 2000.

What is the implication/cost for doing evaluation in rural areas - is there a "ruralness" factor?

The Division of HIV/AIDS Prevention appreciates the challenges for program evaluation in rural areas, plans to discuss the issue, and will request feedback from rural states.

Technical Assistance

What additional tools are available to help with evaluation and community planning?

Technical assistance (TA) requests concerning community planning and the Evaluation Guidance should go through the health department's CDC project officer. CDC supports several organizations to provide community planning TA. This network is coordinated by CDC with assistance from the Academy for Educational Development.

What software can be used to manage data? Will CDC develop software for health departments?

Technical assistance channeled through project officers can put health departments in touch with other jurisdictions that have developed software to collect and/or aggregate data from their grantees (CBOs). CDC has plans to develop software that health departments can use to report aggregated data to CDC. In addition, CDC has developed a website that contains the Evaluation Guidance (Volumes 1 and 2) and other materials on evaluation. Health departments can download forms from the Evaluation Guidance to record the data asked for in the Guidance. The website address is http://www.cdc.gov/hiv/topics/evaluation/health_depts/guidance/.