Guideline 3: Solicit candidates.

Once the job description has been completed, it should be distributed to local colleges and universities and to professional organizations such as the American Educational Research Association and the National Council on Measurement in Education. It should also be advertised at least once in the local newspaper. The evaluation office and the personnel office of the state education agency can often provide assistance in identifying or soliciting viable applicants. Viable candidates with academic credentials in program evaluation can usually be found in college or university departments of educational research and statistics. Specialized academic centers, such as centers for health services research or educational research, are another potential source of candidates. You might also use your own existing network of contacts to identify persons who have conducted program evaluations for other organizations in your community.

Send copies of the job description to all identified candidates. When someone is identified as a good program evaluator, a telephone call or letter to recruit that person is appropriate. If an external evaluator is selected who resides outside your immediate geographical area, make sure the evaluator will be able to participate in regular evaluation committee meetings.

Return to top

Guideline 4: Interview and select the evaluator.

Does the candidate understand the difference between research and evaluation?

The primary purpose of research is to develop a new knowledge base or expand on an existing one. On the other hand, the primary purpose of program evaluation is to provide information related to specific program improvement or program continuation decisions. It is possible, of course, for program evaluation efforts to expand on an existing knowledge base while simultaneously providing program personnel with the information they need. However, external evaluators are sometimes more interested in conducting research peripheral to the evaluation needs of the program in part, because of their desire to publish in research journals. Such instances will usually lead to ineffective evaluation results. To prevent such a situation, have the candidate describe the difference between research and evaluation approaches. If a candidate does not understand or appreciate this difference, he or she will likely lean toward research during your evaluation because most graduate programs emphasize the acquisition of research rather than program evaluation skills.

Does the candidate understand your program?

Have the candidate describe his or her understanding of your program's intent (as reflected in the information you provide to applicants) and how its goals are to be attained. If the candidate misunderstands program goals or operational strategies, make corrections at this time to give the candidate a fair opportunity to respond to subsequent questions. It is important that you feel comfortable about the candidate's understanding of your program's goals and strategies.

What would the candidate's general approach be to your evaluation?

Have the candidate describe the general approach he or she intends to take for the evaluation. Pay attention to questions and issues the candidate believes should be the focus of the evaluation, the type of data to be collected to address those questions and issues, the method of data collection, and the presentation of the evaluation's results. If the discussion becomes very technical and a candidate is unable to present information that you can understand, it is unlikely that this candidate will meet your needs. A candidate unable to communicate effectively at this time will probably not overcome the problem during the evaluation. Effective communication is a key for success, and the interview gives committee members a good idea of how effectively a candidate can communicate.

Does the candidate believe your evaluation can be conducted for the available monies?

Candidates must indicate that their proposed evaluation approach can be carried out for the monies you indicated would be available. You may find a proposed evaluation plan to be excellent but unattainable under your anticipated budget.

All candidates who pass the initial screening process should be asked to produce a detailed budget for the evaluation. A detailed budget is useful for identifying the aspects of the evaluation that are being emphasized, as well as for providing the committee with a way of monitoring the overall evaluation effort.

What is the candidate's reaction to supervision by the evaluation committee?

The evaluator should report to the evaluation committee to assist the committee's supervision of the evaluation. It is reasonable to assume that a candidate unwilling to work under such conditions is not appropriate for the position.

Experienced evaluators might offer suggestions to help your committee's proposed project management operate more efficiently and effectively.

What is the candidate's prior evaluation experience?

Experience is an important factor to consider. A candidate probably will not have performed exactly the same evaluation that you require, but many similarities between previous programs and your own can be found. The candidate's prior experience will be your main opportunity to discover and weigh that person's strengths and weaknesses.

How useful are the candidate's previous evaluation reports?

Look for evaluation reports for which the candidate served as lead author. Assess the reports for their clarity, organization, readability, and potential usefulness for decision makers. Pay particular attention to how well they would help a program improve. Candidates providing technical, poorly written, disorganized, difficult-to-understand, or lengthy evaluation reports will likely compose similar reports for your evaluation. Evaluators with poor writing skills are all too common, and a poorly written report can ruin the best evaluation study.

Does the candidate have good references?

Candidates should be asked to provide the names of at least two persons or organizations for whom they have previously conducted evaluation projects. These references should be contacted for objective views of the candidates. Here are some questions you could ask the references.

• Did the evaluation approach used by the evaluator address the needs and desires of your organization?
• Was the evaluation conducted in a timely fashion?
• Was the evaluation conducted within your budget?
• Was the evaluation report useful to you?
• Would you hire the evaluator to conduct another evaluation for you?

Will the candidate's existing professional commitments interfere with the planned evaluation?

Good program evaluators are usually in demand. A candidate who is engaged in several projects, however, may be unable to devote sufficient time to your program evaluation. Ask the candidate to describe current and expected professional commitments. If the commitments seem excessive, ask how the candidate plans to conduct your program evaluation along with these other tasks. If the candidate indicates that other persons will be used to assist with the evaluation, determine which tasks will be performed by whom. Also determine if these other persons are capable of performing the tasks assigned to them. Using a team of trained and experienced persons to perform an evaluation is common, but the committee must satisfy itself that the team leader (i.e., the candidate) will be involved in all tasks that the committee believes require this person's direct involvement.

What is your general reaction to the candidate?

During interviews, be alert to the candidate's ability to communicate in a straightforward manner, and be alert to your own expectations of how effectively you and your colleagues can work with this person. A clash of working styles can certainly be a problem, and the chemistry between a candidate and committee members is not always right. Negative subjective reactions should be viewed as a serious problem that may not be easily resolved.

What is your overall rating of the candidate?

Following the interview process, committee members should individually rate the candidates on all of the issues previously described. Candidates might be rated on a five-point Likert-type scale, ranging from "Definitely hire as our evaluator" to "Definitely do not hire as our evaluator." "No opinion" should be the midpoint. (See Appendix B for a sample form for rating candidates.) After combining committee members' individual scores, the candidates should be ranked so that the position can be offered to the candidate most acceptable to the majority of committee members.

Return to top

Guideline 5: Write and negotiate the contract.

The desired relationship between the evaluation committee and the external evaluator is one of partnership and should be reflected as such in the contract. The contract should state, in a single paragraph if possible, the evaluator's general responsibilities. Also include in this paragraph a brief statement detailing your intended decision-making process and the authority of the evaluation committee. In another paragraph, list the contract deliverables and provide a timetable for them. Many evaluation contracts also specify who owns the data gathered during the evaluation as well as who has the right to publish the results of the evaluation study. Finally, indicate how the evaluator will bill for services rendered and a schedule of payment. Between 20 and 30 percent of the evaluator's fee should be withheld until the acceptance of the final report by the committee.

The contract should also detail the evaluation committee's responsibilities to provide the external evaluator with timely and appropriate guidance, to review and approve evaluation instruments and documents in a timely and constructive manner, and to assist the evaluator in solving problems that arise during the evaluation. (See Appendix C for a sample contract.)

Return to top

Guideline 6: Interact closely with the evaluator.

At the first meeting with the evaluator, the evaluation committee should again describe the overall project and express its expectations of how the evaluation should be conducted. Following this general discussion, the evaluator and the committee should schedule, and make agendas for, subsequent meetings to keep the evaluation moving in a timely and efficient manner. Later meetings should encompass a review of the sample selection process and a discussion of the data-collection plan. The more specific the agenda you make for subsequent meetings, the more likely that the evaluation will meet the needs of the program and be completed in a timely manner. Be sure to keep minutes of the committee meetings. The minutes need not be detailed but should record decisions made about the evaluation effort.

The committee's involvement in the evaluation process, however, should not be limited to periodic meetings with the evaluator. The committee, in part or in whole, should continually monitor the evaluator and the evaluation. A committee only intermittently involved in the evaluation process might not be aware that the evaluation is going in an inappropriate direction before considerable time, money, and effort have been wasted.

Return to top

Guideline 7: Prepare the final report and release of results.

As your HIV education evaluation nears completion, the committee and the evaluator should agree on a format for the evaluation report. The evaluation report should address evaluation questions directly and briefly and should be understandable to the target audience. Any report, of course, must provide useful and direct guidance for program decision makers. The committee and the evaluator should also agree at this time on the evaluator's role in the release of the evaluation's results. The committee may request that the evaluator be available to meet with decision makers, conduct interviews with news media, and make public presentations of the results.

The evaluator customarily submits a preliminary draft of the final report to committee members for review and comment. If the changes suggested by the committee are significant and would have the effect of changing the findings, recommendations, or overall focus of the report, they should be discussed at a meeting between the committee and the evaluator. If the evaluator does not believe that the suggested changes are consistent with the data, the evaluator has the right to be disassociated from the report. In such an instance, the evaluator may make the requested changes and assign authorship of the report to the committee. Such situations should be avoided, however, as they usually place the integrity and public acceptance of the report in jeopardy.

Return to top

Back to Booklet 7 Table of Contents