Guideline 4: Use data-analysis procedures that yield understandable results.

Once you have gathered your data, that evidence must be summarized in such a way that is understandable to decision makers.

Practical versus statistical significance

Evaluators sometimes carry out data-analysis procedures that produce enough statistics to be "respectable." Such evaluators, however, must remember their audience. Unfortunately, statistical procedures that are among educational research's most useful tools are sometimes inappropriate for educational evaluation. In general, the audience for an educational researcher's efforts consists of other researchers or scholars to whom subtle, statistically significant differences may be quite important. The audience for evidence gathered by evaluators of HIV education, however, will most often be teachers, board members, or educational administrators. By and large, such decision makers are concerned with practical rather than statistical significance. A practically significant question might focus on whether a program's effect is large enough to warrant actions such as expanding the program's applications to other settings. In some cases, sophisticated statistical analyses can render an evaluation study's results virtually incomprehensible.

Return to top

As an HIV education evaluator, you will need to analyze data in the manner most appropriate to yield easily understandable results for decision makers. This usually leads to analyses involving easy-to-read indices such as percentages and arithmetic averages or easily understood data-representation schemes such as bar graphs. In recent years, most people have become familiar with news reports of surveys having an error margin of plus or minus a certain percent. If you can analyze your data so that the results can be cast in a form accompanied by a given error percentage, most decision makers will intuitively understand what you are reporting.

If more sophisticated analysis approaches are used, make sure that results can be easily communicated to decision makers. For example, analysis of covariance is a statistical procedure often used to account for initial differences between groups of students. Were you to employ this data-analysis technique, your report to decision makers could be something along these lines: "After statistical adjustments were made for the fact that the two groups were not initially equal, the HIV education group had 13 percent fewer reported incidents of unprotected sexual intercourse."

Suppose that, prior to an HIV education program, 35 of 100 students reported that they routinely had sexual intercourse without using a condom whereas several months after the program's conclusion only 28 of 100 reported such behavior. In other words, there was 20 percent reduction in sexual intercourse without a condom among students who engaged in such a behavior. These sorts of percentage-based results are easy for decision makers to interpret. People can make sense of percentage-based differences between students' preprogram and postprogram performances because people are used to dealing with percentages in other aspects of life. Most people are not used to dealing with statistically significant differences at the .05 versus .01 probability levels.

Percentage-correct may not be a suitable descriptive scheme for all assessment instruments you choose. For example, you might use a ten-item attitudinal inventory focusing on students' perceived ability to use refusal skills that yields scores from 10 points (low perceived ability) to 50 points (high perceived ability). For such an instrument, an arithmetic average of students' scores would be more sensible than results expressed as percentages.

Because you will typically be looking at preprogram and postprogram data for your evaluations, it will be a routine matter to compare the differences between such data to discern whether the HIV education program yielded its anticipated effects. Simple pretest-to-posttest percentage changes will usually fill the data-analysis bill satisfactorily.

This fourth guideline stresses the desirability of using data-analysis schemes that yield understandable results. You will discover in most instances that simple statistical procedures will take care of your data-analysis needs. In those few cases when you might need more sophisticated statistical analyses, you may wish to call on a statistical consultant to provide you with additional data-analysis guidance. Such situations might arise when it is unclear whether a difference in the performances of treated and untreated students is large enough to be meaningful.

One reason that Guideline 4 is included in this set of suggestions for HIV education evaluators is to dissuade you from believing you must carry out all sorts of complicated data analyses to make your evaluation study respectable. This is simply not the case. Your task as an evaluator of HIV education programs is to help the program's decision makers come up with better decisions. To be useful to busy decision makers, data-analysis procedures should lead to straightforward, readily interpretable information regarding program effectiveness.

Return to top

Back to Booklet 1 Table of Contents