Cognitive Interview Evaluation of X Gender Marker Definitions for the U.S. Passport Application Form
Study year: 2022
The U.S. Department of State is updating passport forms DS-11, DS-82, and DS-5504 to better support all passport holders by adding an X gender marker. Several countries offer the option on passports, and 21 states and the District of Columbia currently offer the category on driver’s licenses and birth certificates. However, varying definitions of the X marker are used, and it is not known how the marker is interpreted and utilized by the general population. In partnership with The U.S. Department of State, the Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) at the National Center for Health Statistics conducted a cognitive interviewing study to assess how potential passport applicants would interpret different versions of the definition as well as how various options would inform response choices.
Citation: Willson, S., Miller, K. Cognitive Interview Evaluation of X Gender Marker Definitions for the U.S. Passport Application Form. 2022.
An Initial Cognitive Evaluation of a 2-Step Gender Identity Measure
Study year: 2021
This study investigates the performance of a gender identity measure for federal surveys. Specifically, the study examines a 2-step measure, whereby respondents are first asked sex assigned at birth followed by a question on current gender identity. The primary focus of the study was two-fold: 1) to determine whether respondents perceived the two questions as asking about distinct constructs, as opposed to seeing them as repetitive, and 2) to identify the ways in which respondents defined or conceptualized those constructs when formulating answers. As such, the study sought to investigate construct validity as well as to provide insight into potential response error.
Citation: Miller, K., Willson, S., Ryan, V. An Initial Cognitive Evaluation of a 2-Step Gender Identity Measure. 2021.
Mixed Method Evaluation of the RANDS During COVID-19 Telemedicine Availability Question: Results from the First Two Rounds of RANDS During COVID-19.
Study year: 2020
Due to the onset of the novel coronavirus pandemic in 2020, the National Center for Health Statistics (NCHS) had to adapt its data collections to the new environment. Surveys that relied on face-to-face data collection either had to adjust data collection modes or suspend data collection completely. To provide the public with timely data on the health-related outcomes of the pandemic, NCHS expanded the mission of its Research and Development Survey (RANDS) program to not only act as the center’s methodological research survey, but also to provide experimental estimates of a limited number of health-related outcomes. NCHS’ Division of Research and Methodology (DRM) consulted with both internal and external stakeholders when developing the RANDS during COVID-19 questionnaire. Through this process, telemedicine availability and use emerged as key concepts about which RANDS during COVID-19 could provide timely information. Specifically, NCHS wanted to collect and disseminate information about the prevalence of telemedicine availability prior to and before the pandemic, as well as use of telemedicine during the pandemic.
Citation: Scanlon, P. Mixed Method Evaluation Of The Rands During Covid-19 Telemedicine Availability Question: Results From The First Two Rounds Of Rands During Covid-19. 2022.
Cognitive Testing Evaluation of Survey Questions on COVID-19
Study year: 2020
This report documents findings from a cognitive interview evaluation of survey questions on the topic of coronavirus disease 2019 (COVID-19). The study was conducted in support of federal surveys that have incorporated (or intend to incorporate) COVID-19 measurements into their questionnaires. The questions evaluated include items on the RANDS during COVID-19 survey (a methodological survey housed at NCHS) and other federal surveys, such as the NHIS (National Health Interview Survey) and the ECHO (Environmental Influences on Child Health Outcomes) adult primary questionnaire, a project supported by the National Institutes of Health. The findings of this study serve two purposes. First, the results serve as a validity study for COVID-19 questions, so that survey data analysts can understand what constructs the questions capture. As a validity study, the cognitive interviews provide information about the patterns of interpretation associated with these survey questions. Second, this study explored the question-response process which identified problems respondents had in answering the questions and, by extension, possible sources of response error. Information from these findings may be used to improve question design for future surveys.
Citation: Willson S. Cognitive Testing Evaluation of Survey Questions on COVID-19. 2021.
2019 Evaluation of opioid-related questions for federal household surveys.
Study year: 2018
This report presents findings of a large nationwide cognitive interviewing study, including a companion pile sorting activity, to examine the performance of opioid use, impairment, misuse and disorder questions intended for population-based Federal surveys. The study consisted of 180 English and Spanish-speaking interviews in eight jurisdictions of the United States, of which 152 also included the pile sorting activity. Respondents varied in demographic backgrounds and ranged in their knowledge of and experiences with opioid pain medication. Interviews focused on the ways in which respondents interpreted the various questions as well as the specific experiences included in their answers. The research serves as a validation study in that the accuracy of survey question responses were assessed through in-depth follow-up discussions with respondents. That is, interviewers were able to determine false-negative and false-positive responses as well as reasons for that error. Analysis of those interviews, specifically, identifying patterns of commonality and difference, reveals the phenomena captured by each question as well as the ways in which, and circumstances by which, they may vary across groups of respondents.
Citation: Miller, K., Willson, S., Scanlon, P., Vickers, B., Massey, M., Creamer, L. 2019 Evaluation Of Opioid-Related Questions For Federal Household Surveys. 2022.