Question Evaluation

Question evaluation is a vital part of survey development. Evaluation reduces non-response error, which ensures comparable data and supports data users in their analyses of survey data. In addition, the Office of Management and Budget stipulates in Section 1.4 of The Standards and Guidelines for Statistical Surveys:

“Agencies must ensure that all components of a survey function as intended when implemented in the full-scale survey and that measurement error is controlled by conducting a pretest of the survey components or by having successfully fielded the survey components on a previous occasion.”

OMB Standards and Guidelines for Statistical Surveys Cdc-pdf[PDF – 318 KB]External

Question evaluation can determine if a question…

• Captures the intended construct: Ideally, questions should capture constructs as survey researchers and designers intend. Therefore, it is important to know if questions are not capturing the intended constructs so that they can be modified prior to fielding. A thorough examination of question performance also allows data users to understand the potential range of constructs that may have been captured in the data. Thus, question evaluation can enhance validity.
• Is interpreted consistently across respondent groups: Response patterns may differ across respondent sub-groups. Respondents’ interpretations of a question may differ depending on their social experiences and demographic factors. For example, respondents with different education levels might interpret or comprehend question terminology differently. Examining patterns of response across respondent groups can help researchers understand and reduce sub-group bias.
• Is difficult for respondents to answer: There are many possible reasons why respondents might have difficulty answering a question. A question may be confusing or social desirability bias may be a factor. Findings from question evaluation studies can reduce item non-response by leading to questions that respondents are willing and able to answer.

Question evaluation should be…

• Empirical – Empirical question evaluation is evidence based rather than relying on expert review.
• Transparent – Methods and analyses are clearly documented, so all conclusions can be traced back to original data. This transparency increases the credibility of the findings from question evaluation studies.
• Systematic – Consistency and rigor in data collection and analysis are key to good question evaluation. CCQDER uses Q-Notes help to ensure a systematic and transparent analysis across all studies.

Evaluation methods

• Cognitive interviewing
• Other research methods