Evaluation Briefs No. 15 | November 2008 Checklist to Evaluate the Quality of Questions To get usable and accurate data from questionnaires and interviews, it is critical that you develop questions that are easily understood by the respondent. The Question Appraisal System (QAS-99) is a method for identifying and fixing miscommunication and other types of problems with questions. Use QAS-99 before formal field testing of your questions. The QAS-99 includes a checklist composed of eight steps. Within each step, you can determine whether specific problems with a question exist, and, if so, check the “YES” box associated with the particular problem. Go to the QAS-99 user’s manual for more information on how to code the problems noted in each question and suggestions for correcting the problems. You can find the user’s manual at http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf. Below is the QAS-99 checklist. Although the QAS-99 was designed for reviewing telephone interviews, you can use the checklist for questionnaires, with the exception of Step 1. Questionnaire Appraisal System INSTRUCTIONS Use one form for EACH question to be reviewed. In reviewing each question: 1) WRITE OR TYPE IN THE QUESTION NUMBER. ATTACH QUESTION.Question number or question here: 2) Proceed through the form—Circle or highlight YES or NO for each Problem Type (1a…8). 3) Whenever a YES is circled, write detailed notes that describe the problem. STEP 1 – READING: Determine if it is difficult for the interviewers to read the question uniformly to all respondents. 1a. WHAT TO READ: Interviewer may have difficulty determining what parts of the question should be read. YES NO 1b. MISSING INFORMATION: Information the interviewer needs to administer the question is not contained in the question. YES NO 1c. HOW TO READ: Question is not fully scripted and therefore difficult to read. YES NO STEP 2 – INSTRUCTIONS: Look for problems with any introductions, instructions, or explanations from the respondent’s point of view. 2a. CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or explanations. YES NO 2b. COMPLICATED INSTRUCTIONS, introductions, or explanations. YES NO STEP 3 – CLARITY: Identify problems related to communicating the intent or meaning of the question to the respondent. 3a. WORDING: Question is lengthy, awkward, ungrammatical, or contains complicated syntax. YES NO 3b. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO 3c. VAGUE: There are multiple ways to interpret the question or to decide what is to be included or excluded. YES NO 3d. REFERENCE PERIODS (e.g., “during the past month”) are missing, not well specified, or in conflict. YES NO STEP 4 – ASSUMPTIONS: Determine whether there are problems with assumptions made or the underlying logic. 4a. INAPPROPRIATE ASSUMPTIONS are made about the respondent or about his/her living situation. YES NO 4b. ASSUMES CONSTANT BEHAVIOR or experience for situations that vary. YES NO 4c. DOUBLE-BARRELED: Contains more than one implicit question. YES NO STEP 5 – KNOWLEDGE/MEMORY: Check whether respondents are likely to not know or have trouble remembering information. 5a. KNOWLEDGE may not exist: Respondent is unlikely to know the answer to a factual question. YES NO 5b. ATTITUDE may not exist: Respondent is unlikely to have formed the attitude being asked about. YES NO 5c. RECALL failure: Respondent may not remember the information asked for. YES NO 5d. COMPUTATION PROBLEM: The question requires a difficult mental calculation. YES NO STEP 6 – SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for bias. 6a. SENSITIVE CONTENT (general): The question asks about a topic that is embarrassing, very private, or that involves illegal behavior. YES NO 6b. SENSITIVE WORDING (specific): Given that the general topic is sensitive, the wording should be improved to minimize sensitivity. YES NO 6c. SOCIALLY ACCEPTABLE response is implied by the question. YES NO STEP 7 – RESPONSE CATEGORIES: Assess the adequacy of the range of responses to be recorded. 7a. OPEN-ENDED QUESTION that is inappropriate or difficult. YES NO 7b. MISMATCH between question and response categories. YES NO 7c. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO 7d. VAGUE response categories are subject to multiple interpretations. YES NO 7e. OVERLAPPING response categories. YES NO 7f. MISSING eligible responses in response categories. YES NO 7g. ILLOGICAL ORDER of response categories. YES NO STEP 8 – OTHER PROBLEMS: Look for problems not identified in Steps 1-7. 8. OTHER PROBLEMS not previously identified. YES NO For further information or assistance, contact the Evaluation Research Team at ert@cdc.gov. You can also contact us via our website: http://www.cdc.gov/healthyyouth/evaluation/index.htm.