 |
 |
National Center for Chronic Disease Prevention and Health Promotion
Healthy Youth
Healthy Youth Home | Contact Us
|
||||||||
|
|
|
|
|
|
|
||
National Center for Chronic Disease Prevention and Health Promotion Healthy Youth Healthy Youth Home | Contact Us |
|
|
||||||
|
 |
Resource Library The Handbook for Evaluating HIV Education: Booklet 1 Evaluating HIV Education Programs
Guideline 1: Focus on a manageable number of important program-related decisions. When many educators hear the expression "educational evaluation study," they almost instinctively think of a study designed to determine if an educational program is good or bad. That view of educational evaluation, fortunately, is way off the mark. An educational program is evaluated for one fundamental reason: to provide information to help people make better decisions about the program. Properly conceived educational evaluations help decision makers arrive at better decisions. The evaluator's responsibility, then, is to gather information, or evidence, for these decision makers. Sometimes, of course, the evaluator of an HIV education program will be the same person who manages the program. The kinds of decisions that must be made by those who staff an HIV education program might deal with (1) what content to include in the program, (2) how much instructional time to allot to different topics, (3) how to organize instructional components effectively, and (4) what to do when certain parts of the program appear to be unsuccessful. Two kinds of decisions Decisions that relate to educational programs can be classified into two major categories. The first category includes decisions that improve the program and allow it to function more effectively. These are program improvement decisions. The second category focuses on more fundamental go/no-go decisions, that is, whether to continue or discontinue the program. These decisions are program-continuation decisions. A decision might be made, for example, to terminate an existing HIV education program and replace it with a substantially different program. The type of decisions that need to be made directly determine the type of information you seek and the approach you will take in your evaluation. If you are carrying out an evaluation study designed to assist with program improvement decisions, you can be decidedly partisan. You are in every sense a "member of the team," and your chief responsibility is to boost program effectiveness. As we will see, a program-improvement evaluator can use data-gathering techniques that would be a poor choice for program-continuation evaluations. On the other hand, when carrying out a program continuation evaluation study, you must be completely objective and non-partisan. Your evaluation should be governed by your need to supply accurate and credible evidence to those who will decide whether to continue the program. Decision makers associated with an HIV education program typically fulfill one of two functions. First, they may be people who actually design and/or deliver the HIV education itself. These typically are teachers or health education curriculum specialists. Such people are mainly concerned with decisions to improve the program. Program improvement evaluation studies are particularly appropriate in the early years of a program's existence when the instructional staff is trying to eliminate the program's deficits and make it stronger. The second category of decision makers consists of those who authorize or fund an HIV education program. School board members at the district or state level usually set the policies that establish the programs. State or federal officials often supply funding for the programs. Both program authorizers and program funders are usually more concerned with program continuation decisions than with program improvement decisions. Program continuation evaluations usually are made after a program has been in place for a few years, when it is appropriate to determine if the program is worth the money it's costing. Most decision makers with whom you will work think of educational evaluation exclusively as a program continuation enterprise. When many school board members or district level administrators hear the phrase "evaluation study," they immediately imagine a study designed to determine whether or not they should continue the educational program. One of your tasks as an evaluator is to educate decision makers to realize that it often makes sense to evaluate programs in order to improve them, particularly in their early years of existence. Some people mistakenly assume that once a program has been evaluated, there is no need for further evaluation. In fact, however, program evaluation should be conceived of as an ongoing enterprise. As early versions of the program are offered, they can be improved via program improvement evaluation. Later, when the program staff believe the program is sufficiently mature, a program continuation evaluation might be undertaken. But even mature programs can be improved. Thus, evaluation of programs for program improvement should be a continuing activity. It is sometimes thought that evaluations of HIV education are successful only if they reveal that the program was effective. On the contrary, an evaluation that reveals a program's shortcomings can point the way to program improvements and ultimately program effectiveness. Evaluations that enhance the quality of decisions are successful evaluations. Guideline 1, to focus on a manageable number of important program-related decisions, stems from a basic human shortcoming: people can only make use of so much data. If they are given too much information, even at their own request, they are likely to become overwhelmed and, as a consequence, pay attention to none of it. The decision makers for whom you gather evidence in your evaluation will be no different. They will often want more answers than they can really use. Regardless of whether you pursue an evaluation study aimed at program improvement or program continuation, one of your early tasks is to focus on an intellectually manageable number of decisions related to the HIV education program. Skillful evaluators focus on significant decisions, not "nice to know" information. |
||||||||||||||||||||||||||||||||||||||||
|
|
An HIV education program staff usually aspires to bring about worthwhile changes in students. Those changes can focus on altering either students' HIV-risk behaviors or the factors thought to contribute to such behaviors. Put most simply, an instructional objective for an HIV education program should describe the post-program knowledge, skills, attitudes, or behaviors that the program seeks to promote. This is nothing more than a classic ends/means distinction. Identifying a program's objectives can lead to identifying the decisions on which you will focus your evaluation. A number of educators attempt to describe educational objectives as what the program itself will do rather than what it is intended to accomplish. Educational objectives have nothing to do with what the HIV education program is or how it was created. Instead, the objectives for HIV education must focus on program outcomes—that is, on what happens to students as a consequence of the program. Clearly stated, measurable objectives will provide a valuable yardstick for your evaluation of a program's effectiveness. If you can help a program's staff identify the objectives that they hope to accomplish, and if you can help the staff define those objectives as preprogram-to-postprogram changes in students, you will have gone a long way toward clarifying the focus of your evaluation. You can then recognize and isolate evidence bearing on key program decisions. Evaluators who wish to use an HIV education program's objectives to their advantage will need to be sure that the program is organized around only a handful of measurable objectives. Some researchers cite evidence suggesting that people have a difficult time concentrating on many more than six or seven issues at a time. Rarely permit your evaluation, therefore, to be organized around more than a half-dozen or so objectives—preferably fewer. A program staff may have a number of fine-grained instructional objectives to use in day-to-day instruction. As an evaluator, however, your responsibility is to isolate a few educational objectives that subsume such day-to-day objectives, then gear your data-gathering toward that smaller number of important objectives. Focusing on major decisions One of the best ways for you to focus an evaluation study on a manageable number of decisions is to encourage decision makers to identify their most important program-related decisions. Work with decision makers to identify the one most important decision at issue, then the next most important decision to be informed by the evaluation study, and so on. Another useful ploy to reduce the number of decisions is to group small-scope objectives into one broader yet still measurable objective. Suppose you were carrying out a program continuation evaluation of HIV education for a district's school board. You have learned that the teachers providing the district's HIV education program have listed four different objectives each dealing with a distinctive type of knowledge students need to acquire, such as "knowledge regarding HIV infection routes" or "knowledge regarding HIV-risk behavior patterns of teenagers." It would be relatively simple to create a single objective, "increased HIV-relevant knowledge," to effectively coalesce the four small-scope objectives. The teachers can still organize day-to-day instructional activities around small-scope objectives, and you can focus your evaluation on broad-scope objectives. In spite of your efforts to focus on a limited number of decisions, some decision makers may request information simply because it would be "interesting." Keep urging these persons to indicate how their requested information would actually make a difference in a decision dealing with the HIV education program. One good way to verify whether a requested set of evidence really bears on a program related decision is to present decision makers with hypothetical results and ask, "If the evidence turns out this way, what would your decision be?" Then present a divergent set of information, asking, "If the evidence turns out the opposite way, what would your decision be?" You may discover that it makes no difference to decision makers what the results are. In such instances, of course, encourage these persons to seek other, more relevant evidence. Must all decisions be linked to attaining objectives? Although the decisions addressed by evaluators are often linked to the achievement of a program's objectives, decision makers face many choices that do not depend on the attainment of objectives. For example, evaluators often gather evidence as to whether an instructional program is being delivered as intended. The decision at issue in this instance would be whether the program's staff must take steps to ensure that the program is being provided as its designers intended. Other such decisions include (1) whether community officials will permit controversial topics to be addressed in instructional activities, (2) whether students will regard HIV-related information as more believable if provided by peer counselors rather than teachers, and (3) whether the program's objectives are appropriate. There are also instances in which unanticipated effects of the program, that is, effects not foreseen in the program's objectives, might be significant in judging a program's effectiveness. In short, although the degree to which an HIV education program's objectives have been achieved can illuminate certain kinds of decisions, other kinds of decisions will demand that the evaluator adopt alternative approaches. Final thoughts about Guideline 1 The purpose of evaluating HIV education programs is to help those who must make decisions about the program do so in an appropriate manner. You will discover that evidence gathered in an evaluation study often plays only a minor role in the decisions ultimately made about a program. For many decision makers, a series of political and personal factors play a far more prominent role than any evidence of program effectiveness provided by an evaluator. As an evaluator, therefore, you will need to structure your evidence-gathering efforts to yield information that has at least a reasonable chance of affecting the decisions to be made. Back to Booklet 1 Table of Contents Back
to Handbook for Evaluating HIV Education - Introduction |
