Cognitive interviewing is a popular method for evaluating survey questions in the Federal Government. Designed to understand how respondents, comprehend, retrieve, judge, and respond to questions.
Characteristics of a Cognitive Interview study:
- Purposive sample selection
- Typically Small samples: 20-50 respondents
- Provides rich detail in regards to respondents' thought processes in formulating an answer
- Data collected are qualitative
- Qualitative methodologies are used to analyze data
Follow-up questions or cognitive probes are used to examine thought processes.
- Probes can target specific processes.
- Comprehension: What do you think 'abdomen' refers to?
- Recall: How did you remember that?
- Judgment: How sure are you about that?
- Response: How did you pick an answer to that?
- Probes can also be more general and elicit narrative information about respondent's circumstances.
- Why did you answer that way?
- How did you come up with your answer?
- Tell me more about that.
- Are based on narrative information about how respondents answered questions. Validity is tied to the amount of rich detail that describes findings.
- Are grounded in the experiences of the respondents.
- Provide insight into question interpretation.
- Provide insight into patterns of calculation.
Findings of a Cognitive Interview study:
Analysis of Qualitative Interview Data
- Simultaneous pursuit of data collection & analysis
- Constructing analytic codes & themes from data (grounded theory)
- Make comparisons during each stage of analysis, for example…
- Explore question response in 1 interview (comprehension; recall; judgment; response)
- Compare that finding to the next interview (is it the same? If not, how is it different?)
Levels of Analysis in Cognitive Interviewing
Level 1 - Cognitive processing errors (individual response errors)
Level 2 - Patterns of interpretation and process (What the question captures)
Level 3 - Patterned differences in sub-groups (Potential For Bias)