From Research to Response: Survey Shifts Gears During COVID-19
The scientists who designed the Research and Development Survey (RANDS) could never have foreseen the current global pandemic. But when COVID-19 struck, they recognized the tool was uniquely ready to meet the needs of the response.
The RANDS survey’s website allows users to view
experimental data on reduced access to care,
telemedicine, and loss of work due to illness,
shown here.
RANDS was developed at CDC’s National Center for Health Statistics (NCHS), which is home to our nation’s most comprehensive and longest-running health surveys. But unlike many of NCHS’ other surveys, RANDS wasn’t intended to collect data itself. Rather, it was designed primarily to conduct research about survey methods.
The team has now begun using RANDS to produce data about access to healthcare, use of telemedicine, and loss of work due to illness from COVID-19.
“For the first time,” says Jennifer Parker, who leads NCHS’ research and methodology group. “RANDS will be producing data that can be used to better understand a public health problem.”
Better answers lead to better outcomes
RANDS can be used to collect information about how people answering survey questions understand them, which lets researchers find ways to improve the quality of statistics they collect.
“The first step to getting reliable survey data is to make sure you’re asking good questions,” Jennifer says. RANDS can tell researchers “if the questions they’re asking mean the same thing from person to person and from survey to survey.”
Because the team had spent the past five years on this project, they were able to rapidly shift that knowledge toward the COVID-19 response. RANDS offered an advantage because it could easily be adapted to cover topics that are relevant to the pandemic, Behavioral Scientist Paul Scanlon says.
“When our usual surveys were suddenly interrupted by the pandemic, we realized that in RANDS we had created a platform that could quickly fill gaps in the data,” Paul says.
Asking questions about questions
“A useful feature of RANDS during the pandemic is that it can help us test the validity of new questions,” he adds. “Responding to COVID-19 requires that we ask questions we’ve never asked before. When we introduce new concepts, like quarantine, we need to understand how people think about that. RANDS is a tool, alongside cognitive interviewing, for examining how people understand and respond to questions so we can improve on them.”
As a behavioral scientist, Paul usually conducts one-on-one interviews to find out how people interpret specific survey questions. “When I interview people, there are no right or wrong answers. It’s just, ‘What do you think a question means?’
“For example, if a survey question says, ‘How is your health?’ we want to understand what factors people used to come
Question: Have you isolated or quarantined yourself because of the Coronavirus?
- Yes
- No
Probe question: When answering the previous question about isolating or quarantining because of the coronavirus, which of the following, if any, were you thinking about? (Select all that apply.)
- Staying inside your house and not leaving at all
- Staying in one room in your house as much as possible
- Limiting interactions with members of your household as much as possible
- Limiting interactions with people outside your household as much as possible
- Leaving your house for essential purposes only, such as grocery shopping, healthcare appointments, and exercise
- Staying six feet away from other people as much as possible
- Something else, please specify:
up with an answer,” he says. “Were they thinking about their diet and exercise habits? Or whether they’re sick or in pain right now? We don’t know unless we ask follow-up questions that dig deeper.”
Within RANDS are a series of unique questions, known as “web probes,” designed to get at this kind of information. The probes are closed-ended, multiple choice questions that ask what respondents were thinking about as they answered the other survey questions.
“If we can use RANDS to understand which questions work best, we can include them later in the year when we go back to conducting our longer-term surveys,” Paul says.
Getting it fast, getting it right
Another benefit RANDS brings to the response is the way it is delivered online. In addition to testing individual questions, RANDS has also been testing the uses of web panels recruited by research companies for gathering statistics.
“Web panels are not something we typically use,” says Katherine Irimata, the team’s mathematical statistician who evaluates new ways to collect data. “From a statistical perspective, online surveys present some very specific challenges we need to understand before we can incorporate them into our work.”
For example, the panels used for RANDS are recruited by outside vendors, and it’s important that they are selected using a reliable statistical framework. Also, response rates are lower for web surveys than NCHS’ usual surveys, and the format itself may have an impact on who responds or how people interpret the questions.
“We’re in a good position to evaluate how these issues affect the results,” says Katherine. “NCHS already has decades of data from the National Health Interview Survey, and we’ve been able to use that to help evaluate the data from RANDS. In turn, we’ve been able to use the RANDS data to inform the questions on our existing surveys.”
She notes, “Because we already had the web panels in place, and because RANDS has helped us understand so much more about how these data can be used, we were able to deploy quickly in the pandemic.”
The innovative science behind RANDS will open doors for better data in the future. Web surveys can reach a lot of people quickly and have the potential to be a flexible, cost-effective addition to NCHS’ arsenal, especially in an emergency response.
“When we built RANDS, we knew we could leverage the strength of our national survey data to evaluate, and possibly improve, health estimates from recruited panels,” Jennifer says. “Our idea is that this method can help not only NCHS, but eventually all federal statistical agencies.”
Paul adds, “We’re just beginning to explore what RANDS can do.”