Evaluation Briefs No. 4, March 2006 Developing Process Evaluation Questions Evaluation questions help focus and provide structure to an evaluation, guide the evaluation planning process, facilitate decision making about evaluation methods to use, and inform discussions about how evaluation results can be used to improve school health programs. This brief defines evaluation questions and addresses steps to develop them. Evaluation Questions Defined Evaluation questions define the key issues to be explored by an evaluation. They are developed and prioritized by program staff, evaluation personnel, funders, and other stakeholders. Process evaluation questions address program operations, particularly the who, what, when, and how many of program activities and program outputs. Process evaluation questions provide information on whether program activities were accomplished (e.g., Were new HIV policies disseminated to all school districts during the past school year?); quality of program components (e.g., How much health education teachers improved their nutrition education knowledge due to our nutrition education curriculum workshops?); how well program activities were implemented (e.g., To what extent have teachers implemented, with fidelity, evidence-based tobacco-use prevention education?); whether the target audience was reached (e.g., Did all students with asthma receive Open Airways training?); and how external factors influenced program delivery (e.g., How are the new reading and math achievement test requirements impacting teachers’ ability to provide health education instruction?). The answers to these questions allow you to assess whether program activities are occurring as you had expected. They also can help determine areas in which a program needs improvement so that you can reach expected outcomes. Steps to Develop Process Evaluation Questions The most useful evaluation questions reflect a diversity of stakeholder perspectives, key components of your program, your most important information needs, and resources available to answer the questions. Steps include the following: 1. Gather your stakeholders and review supporting materials such as your program logic model, work plan, and data from the Indicators for School Health Programs. 2. Brainstorm evaluation questions about your overall program or a specific program activity. Use your logic model to facilitate this process, by focusing on inputs, activities, and outputs to generate process evaluation questions. Also, refer to your work plan and the Indicators for School Health Programs. You may identify many questions that can later be prioritized and reduced in number. 3. Sort evaluation questions from your brainstorming session into categories or groups that are relevant to you and your stakeholders. This process will help determine what resources you have for answering the evaluation questions you are most interested in. 4. Decide which evaluation questions to answer. You should prioritize evaluation questions that -- • Are important to program staff and stakeholders. • Address important program needs. • Reflect key goals and objectives of your program. • Reflect key elements of your program logic model. • Can be answered with available resources, including funds and personnel expertise. • Can be answered within the available timeframe. • Provide information to make program improvements. • Will be supported by your school health program administrators. 5. Once you have determined which evaluation questions you will answer, verify that they are linked to your program logic model, work plan, and the Indicators for School Health Programs. 6. Next, determine who, what, and how to collect the data that will be required for answering the evaluation questions. Determine who will be responsible for collecting the information and analyzing it to answer the evaluation questions. Possible data sources may include persons (e.g., teachers), documents, or records. Possible methods for collecting information include data from the Indicators for School Health Programs, interviews, etc. You also can refer to ETA Briefs on data collection and data management for further information. Example of How to Develop Evaluation Questions Preliminary Step: Suppose your program is focused on three activities: (1) establishing School Wellness Councils (SWC) in local schools; (2) providing peer led training to students on nutrition, physical activity, and asthma; and (3) professional development (PD) to school staff on nutrition, physical activity, and asthma. Stakeholders may include principals, teachers, other school staff, students, parents, SWC members, and community members such as physicians and nurses. Step 1. Gather your stakeholders and review supporting materials Meet with your stakeholders and review the logic model, work plan, and Indicators for your program. Step 2. Brainstorm evaluation questions Based on the program’s logic model, some of the process evaluation questions that you and your stakeholders might be interested in include the following: • How many local schools established SWCs? • Who sits on the SWCs? • How many schools have students for peer led programs on nutrition, physical activity, and asthma? • How many students took part in peer led trainings on nutrition, physical activity, and asthma? • How many staff PD sessions were provided? • Did PD sessions reach the intended school staff? Steps 3 and 4: Sort and decide on evaluation questions The next step is to sort your evaluation questions into groups for organizational purposes (i.e., SWC, peer led trainings, professional development trainings, etc.). Then the group needs to decide which of the evaluation questions are most important to the program and stakeholders; which can be accomplished given staffing, time, and other resource constraints; and which will provide you with information that will be used to improve the program. It could be that time spent on establishing a SWC and on staff PD is of primary importance to a majority of the stakeholders, although a few stakeholders are more interested in the peer led trainings that are taking place. The group as a whole then decides that the questions regarding PD and SWC will be the focus of the evaluation. Step 5. Verify linkage Once you have the evaluation question(s) determined, verify that they are linked to your program logic model, work plan, and Indicators for School Health. An example of a question from the Indicators for School Health Programs that you can use to document PD is “Did you provide PD to istrict or school staff on policies for nutrition?” Step 6. Who, what, and how to collect data Determine who, what, and how the data will be collected. Decide on a point person for the process and determine what records you will need to keep for data collection. For PD, some data sources might be registrations for trainings, on-site sign-in sheets, and evaluation forms completed at the end of training. Resources W.K. Kellogg Foundation. Evaluation Toolkit. Available at:www.wkkf.org/Programming/Extra.aspx?CID=281&ID=11. Accessed November 18, 2005. Patton M. Utilization-focused evaluation. 3rd ed. Thousand aks, CA: Sage Publications; 1997. Rossi P, Freeman H, Lipsey M. Evaluation: A systematic approach. 6th ed. Thousand Oaks, CA: Sage Publications; 1999. For further information or assistance, contact the Evaluation Research Team at ert@cdc.gov. You can also contact us via our website: http://www.cdc.gov/healthyyouth/evaluation/index.htm.