Other Methods

Throughout the question and questionnaire design and evaluation process CCQDER researchers can use a number of methodologies besides, or in addition to, cognitive interviewing. Which of these other research methods are employed, and when, depends on the overall goals and timeline of the project. Some of these methods—such as focus groups and ethnographic interviews — are used primarily during the questionnaire design phase, while others complement cognitive interviewing during the evaluation phase. CCQDER commonly uses the following methods:

Focus groups:  Focus groups, or group discussions, are typically used during the question/questionnaire design phase of a project, and allow researchers to understand how potential respondents understand and talk about specific topics. CCQDER researchers rely on the group dynamics and conversation that focus groups provide to get information about the particular words or phrases respondents use to describe phenomena and which topics may be sensitive or burdensome. Focus groups are often combined with individual level activities—such as short surveys or free-listing tasks—that provide even more information about potential respondents’ perceptions and comprehensions of the topic areas under consideration.

Ethnographic interviews:   Ethnographic interviews are minimally-structured one-on-one interviews administered during the question/questionnaire design phase of a project. Like focus groups, they allow CCQDER researchers to understand the way that potential respondents think and speak about a particular topic, which then allows us to write questions about complex topics in ways they understand. The one-on-one nature of these interviews allow researchers to get a much deeper understanding about how a potential respondents’ social location and experiences affect the way they comprehend the subject of the project.

Usability testing:   During the evaluation of self-response survey instruments, CCQDER will often combine usability testing with cognitive interviewing. In these cases, CCQDER researchers are interested in understanding how respondents interact with a questionnaire—be it a pen and paper or web instrument—and how these interactions impact the respondents’ response processes. During usability testing, CCQDER researchers will typically observe these interactions and then retrospectively probe the respondent about them.

Web probing: In addition to its leadership on the development of cognitive interviewing methodology, CCQDER is one of the world’s leading research groups in developing and using web probes. Web probes are open- or close-ended items that are designed to capture how respondents understand, think about, and respond to questions under evaluation. These probes are embedded in a web questionnaire following the question or question the research team is evaluating. While CCQDER has experience with using and analyzing both open- and closed-ended probes, it focuses on the latter. The close-ended web probes that CCQDER uses in web probing are developed directly from the findings of cognitive interviews. The quantitative results that web probes provide are analyzed alongside the qualitative ones from cognitive interviews, allowing for an in-depth, mixed-method approach that allows CCQDER researchers to identify the distributions of interpretations and sources of measurement error not only across a survey’s target population, but also within subgroups in that target population. This gives researchers an opportunity to find out if certain groups—such as respondents with different levels of educational attainment or from different areas of the country—are more likely than others to use out-of-scope interpretations that may affect the final survey’s validity. CCQDER runs a semi-annual web survey, the NCHS Research and Development Survey (RANDS) as a vehicle to evaluate questions using this innovative method.

A more complete inventory of question and questionnaire design and evaluation methods has been developed by a subgroup of the Federal Committee on Statistical Methodology. CCQDER contributed to this inventory pdf iconexternal icon. For more information about CCQDER, please contact Meredith Massey mmassey2@cdc.gov