Evaluation Briefs No. 14 | November 2008 Data Collection Methods for Program Evaluation: Questionnaires This brief is about questionnaires as a data collection method for evaluation. The brief includes a basic overview of questionnaires; when to use them; how to plan and develop them; and their advantages and disadvantages. What is a questionnaire? A questionnaire is a set of questions for gathering information from individuals. You can administer questionnaires by mail, telephone, using face-to-face interviews, as handouts, or electronically (i.e., by e-mail or through Web-based questionnaires). When should you use questionnaires for evaluation? • When resources are limited and you need data from many people. You can disseminate questionnaires relatively inexpensively. Your costs will increase if you need to do a lot of follow-up to get a sufficient response rate. • To gather data about knowledge, beliefs, attitudes, and behaviors. Questionnaires are helpful in gathering information that is unique to individuals, such as attitudes or knowledge. • When it is important to protect the privacy of participants. Questionnaires are helpful in maintaining participants’ privacy because participants’ responses can be anonymous or confidential. This is especially important if you are gathering sensitive information. How do you plan and develop a questionnaire? Successful questionnaires start with thorough planning. • Define your objectives. The most critical part of developing your questionnaire is defining what you want from it and how you will use the information to answer your evaluation questions. By taking the time to define your purpose and objectives you will reduce the likelihood of gathering unusable information. • Select the number and type of participants for your questionnaire. Selecting the type of participants you want to include is part of determining your objectives. For example, if you need to know if health educators who participated in your training are using the information in their teaching, your participants will be teachers who participated in the training. You should also decide if you should include all possible participants or if a sample will suffice. This will depend on the number of possible participants and the resources you have available. • Develop questions that clearly communicate what you want to know. o Use clear and simple wording written at the reading level of your participants. o Avoid using abbreviations, jargon, or colloquial phrases. o The Question Appraisal System (QAS-99) is a method for identifying and fixing communication problems in questionnaires before formal pilot testing (i.e., a preliminary test of your questionnaire to make sure that intended participants understand and respond favorably to it). An overview of QAS-99 is in Brief 15 (see Resources below). • Decide when to use closed-ended versus open-ended questions. Closed-ended questions include a list of predetermined answers from which participants can choose. Open-ended questions allow the participant to answer the question in their own words. Closed-ended questions are easier to analyze. Open-ended questions can be useful if you do not know the possible answers to questions or for gathering insightful or unexpected information. However, open-ended questions are more difficult and time-consuming to analyze because you have to categorize and summarize the answers. • Include demographic questions. Questionnaires usually include demographic questions such as sex, race, age, education, and where the participant works or lives. The purpose of these questions is to describe subgroups of respondents. Limit the demographic questions to only those that are important for your analysis. For example, if you do not plan to compare the data by age, do not include age on the questionnaire. • Place questions in a logical order that flows well. Start with less sensitive questions and end with more sensitive questions. Order the questions in a way that makes sense to the participant, such as by topic area. • Pilot test the questionnaire. Testing your questionnaire before you administer it will help you find out if participants will understand the questions, if the questions mean the same thing to all participants, if it provides you with the data you need, and how long it takes to complete. Test your questionnaire with a small group who are similar to your intended participants. How do I get an adequate response rate? Response rate is the number of participants that responded to your questionnaire divided by the total number of participants you included in your evaluation. Higher response rates strengthen your evaluation results. There are several ways you can increase your response rate. • Communicate the value of your questionnaire. Participants will be more likely to complete your questionnaire if they understand its value. Communicate the purpose of the questionnaire, how you plan to use the data, and how the results will help participants. • Follow-up. If the questionnaire is administered by mail or electronically you will need to re-contact the participants, perhaps a few times. The more follow-up contacts, the higher the response rate. • Provide incentives. Giving modest financial or other incentives to participants increases the likelihood that they will complete your questionnaire. Resources Questionnaire Design. Georgia Institute of Technology College of Computing. Available at http://www.cc.gatech.edu/classes/cs6751_97_winter/Topics/quest-design/ Taylor-Powell E. Questionnaire Design: Asking questions with a purpose. University of Wisconsin Cooperative Extension. 1998. Available at http://learningstore.uwex.edu/Questionnaire-Design-Asking-Questions-with-a-Purpose-P1028C0.aspx Taylor-Powell E, Hermann C. Collecting Evaluation Data: Surveys. University of Wisconsin Cooperative Extension. 2000. Available at http://learningstore.uwex.edu/Collecting-Evaluation-Data-Surveys-P1027C0.aspx Checklist to Evaluate the Quality of Questions. Available at http://www.cdc.gov/HealthyYouth/evaluation/resources.htm For further information or assistance, contact the Evaluation Research Team at ert@cdc.gov. You can also contact us via our website: http://www.cdc.gov/healthyyouth/evaluation/index.htm.