Cognitive interviewing is a popular method for evaluating survey questions in the federal government. It offers a detailed depiction of meanings and processes used by respondents to answer questions—which ultimately impact the survey data. As such, the method offers insight that can enhance understanding of question validity and response error.
Characteristics of a cognitive interview study:
- Purposive sample selection: Because the goal is to identify the presence of problems, as opposed to making estimations or causal statements, a randomly drawn sample is not required.
- Qualitative methodologies are used to collect and analyze data
- Typically small samples: 20-50 respondents
Follow-up questions or cognitive probes are used to examine thought processes.
- Probes can target specific processes.
- Comprehension: What do you think “abdomen” refers to?
- Recall: How did you remember that?
- Judgment: How sure are you about that?
- Response: How did you pick an answer to that?
- Probes can also be more general and elicit narrative information about the respondent’s circumstances.
- Why did you answer that way?
- How did you come up with your answer?
- Tell me more about that.
Findings of a cognitive interview study:
- Are based on narrative information about how respondents answered questions.
- Are grounded in the experiences of the respondents.
- Provide insight into question interpretation and patterns of calculation.
Analysis of qualitative interview data
- Simultaneous pursuit of data collection and analysis
- Constructing analytic codes and themes from data (grounded theory)
- Make comparisons during each stage of analysis, for example:
- Explore question response in one interview (comprehension, recall, judgment, and response)
- Compare that finding to the next interview (Is it the same? If not, how is it different?)
Levels of analysis in cognitive interviewing
Level 1 – Conducting interviews
Level 2 – Summarizing interview notes
Level 3 – Comparing across respondents
Level 4 – Comparing across groups
Level 5 – Drawing conclusions about question performance