Methodologic Changes in the Behavioral Risk Factor Surveillance System in 2011 and Potential Effects on Prevalence Estimates

In the past few years, all large population health surveys that depend on telephone interviews, including the Behavioral Risk Factor Surveillance System (BRFSS), have had to adjust to the rapid rise in the proportion of U.S. households that have a cellular telephone but no landline telephone. To maintain survey coverage and validity, surveys have had to add cellular telephone households to their samples. In addition, telephone surveys have had to make adjustments in weighting to account for declining response rates by adopting new methods of weighting to adjust survey data for differences between the demographic characteristics of respondents and the target population. Since 2004, BRFSS has been planning and testing the addition of cellular telephone households and improvements in its methods of statistical weighting. These new methods were implemented during the fielding of the 2011 BRFSS, which is to be released in 2012. This policy note describes the methodologic changes and their potential effects on BRFSS prevalence estimates. Preliminary assessments indicate that the inclusion of cellular telephone respondents and the move to a new method of weighting might increase prevalence estimates for health risk behaviors and chronic disease in many states. Carefully planned communication to public health officials and nonscientific audiences of the effect of changes in methods on estimates is needed to prevent misinterpretation.

BRFSS, begun by CDC in 1984, is a coordinated collection of population health surveys conducted by the 50 states, the District of Columbia, and five U.S. territories. Taken together, these surveys make up the largest ongoing public health survey in the world; in 2010, the number of completed interviews was 430,000 (1). With technical and methodologic assistance from CDC, state health departments contract with telephone call centers to conduct the BRFSS surveys continuously through the year using a standardized core questionnaire and optional modules, plus additional state-added questions. The federal government, state governments, and many universities, private organizations, and researchers use BRFSS data to identify the frequency of health behaviors and conditions, track progress toward health objectives, evaluate the effects of disease prevention activities, and rapidly assess emerging health problems (e.g., novel influenza and influenza vaccination patterns) (2).

Adjustment and improvement of methods is a part of all public health surveillance systems, including surveys such as BRFSS. All surveys must adjust their methods from time to time to account for changes in population, behaviors, technologies, and standards. In 2002, for example, the Substance Abuse and Mental Health Services Administration (SAMHSA) was obliged to change methods for the National Survey on Drug Use and Health to match current survey standards. Users had to account for discontinuities caused by these new methods that were not related to changes in real prevalence (3).

In 2004, an expert panel of survey methodologists met at CDC to consider the challenges facing telephone surveys and the implications for BRFSS. The panel made two major recommendations: 1) address the growing effects of cellular telephone–only households on coverage provided by the sample, and 2) develop improved weighting, adjustment, and estimation methods that could reduce the potential for bias and maintain validity as response rates declined and cellular telephone interviews were incorporated. CDC set a goal of implementing these changes with the release of the 2011 BRFSS dataset (4).

The proportion of U.S. households using only cellular telephones is rising steadily (Figure 1). Estimates for the first half of 2011 indicate that 36.4% of U.S. households rely exclusively on cellular telephones (5). In 2006, in response to the growing percentage of cellular telephone–only households and at the recommendation of the 2004 expert panel, CDC began testing changes in BRFSS survey methods to accommodate the addition of cellular telephones. In 2008, CDC funded a cellular telephone pilot study in 18 states, and by 2010, 48 states were conducting interviews of cellular telephone–only households as part of their regular data collection. These pilot studies allowed the states to test survey samples containing responses from landline telephone households and from cellular telephone–only households and helped them gain experience in administering and analyzing surveys containing cellular telephone interviews. CDC has provided each state with developmental datasets from 2008–2010 data, which include landline telephone responses with existing weighting methods, landline telephone responses with the new weighting methods, and combined landline and cellular telephone responses using the new weighting methods to allow the states to test the effects of the new methods on state-level estimates. The median proportion of all completed BRFSS interviews that are conducted by cellular telephone will be approximately 11% for the 2011 BRFSS dataset and approximately 20% for the 2012 dataset.

Since the 1980s, CDC has used a statistical method called "poststratification" to weight BRFSS survey data. Poststratification is a standard method for weighting survey data (6) and is a relatively straightforward process of simultaneously adjusting survey respondent data to known proportions of age (in categories), race/ethnicity, sex, geographic region, or other characteristics of a population taken from U.S. Census information. Poststratification is limited by access to information on each demographic characteristic for each of the regions or areas. For example, if researchers wish to weight information by county and proportions of weighting variables are unknown at the county level, poststratification is not an appropriate method of weighting.

In 2006, in accordance with the recommendations of the 2004 expert panel, CDC began testing "raking" (iterative proportional fitting), a more sophisticated weighting method. Raking, in contrast with the poststratification method, makes adjustments for each variable individually in a series of data processing–intensive iterations (7). As each variable in the weighting process is included, the weights are adjusted until the sample weights are representative of the population.

Raking presents several advantages over poststratification. Because raking does not require demographic information for small geographic areas, it allows for the introduction of more demographic variables suggested by the BRFSS expert panel (e.g., education level, marital status, and home ownership) into the statistical weighting process than would have been possible using poststratification, thereby reducing the potential for bias and increasing the representativeness of estimates. Moreover, because state level demographic characteristics of cellular telephone–only households are not available, weighting with poststratification is not feasible. Raking, which does not rely on information on smaller geographic areas, allows for the incorporation of a crucial variable, telephone ownership (households with landline or cellular telephones) in the weighting methodology of BRFSS. Beginning with the 2011 dataset, raking will succeed poststratification as the sole BRFSS statistical weighting method.

Evaluations conducted by CDC using 2010 and 2011 BRFSS data indicate that the addition of cellular telephone–only households will improve survey coverage for certain population groups. For example, the proportion of interviews conducted with respondents who have lower incomes, lower educational levels, or are in younger age groups will increase, because these groups more often exclusively rely on cellular telephones for personal communications. Inclusions of cellular telephone–only respondents thereby will increase coverage of portions of the population that are not included when only landline telephone interviews are conducted. Because these groups of respondents represent populations with higher numbers of risk factors, estimates of health risk behaviors likely will increase.

Adoption of the new methods also will result in BRFSS state-level prevalence estimates for 2011 and subsequent years that will vary from estimates that would have been achieved with previous weighting procedures (Figure 2). These discontinuities will vary by survey question and state, and they will be driven by state-to-state variations in demographic variables used for raking and the proportion of respondents who use cellular telephones. Assessments at CDC indicate that prevalence estimates for some of the most salient indicators of poor health or negative health behaviors measured by BRFSS will increase in the majority of states. Certain of these increases will be caused by the adoption of raking as the new statistical weighting method, and others will be caused by the addition of cellular telephone households. Differences resulting from weighting can be seen by comparison of poststratification and raking using landline-only data (Figure 2). The effect of cellular telephone inclusion is then seen by comparison of landline data with combined landline and cellular telephone data after raking. The use of raking also might change state-level estimates for chronic disease indicators (i.e., asthma, stroke, coronary heart disease, and diabetes), for the prevalence of self-reported "fair" or "poor" health, and for no physical activity, obesity, heavy drinking, and smoking.

Although, raking might cause state prevalence trends for certain risk factors to shift upward, in general, the shape of trend lines over time might not be affected. For example, in a particular state where the adoption of raking causes an absolute increase in the trend line for a particular prevalence estimate, the shape and slope of the line could remain stable. Data presented here cannot be used as national estimates because they do not include all states and do include data from U.S. territories.

One risk factor, current smoking, serves as an example of how estimates might shift in certain states. Preliminary analysis by CDC using developmental datasets for 2007–2010 reveals that adoption of raking shifts the aggregated trend line for current smoking upward by approximately 2.3–2.8 percentage points for the years 2007–2010, but the shape and slope of the trend line does not change materially (Figure 3). The addition of cellular telephone households to the aggregated state samples for 2009 and 2010 shifts the absolute estimates slightly further upward.

State and federal public health officials have expressed concern that trend line shifts in BRFSS prevalence estimates resulting from these changes in methods might be misinterpreted by the public, policy makers or legislators as real changes in the health behaviors of states' populations. This, in turn, could have adverse ramifications for public health funding and other support. The risk for misinterpretation can be reduced by a careful assessment of the changes in BRFSS health indicators in each state, and establishment of a proactive communication plan to explain the causes of discontinuities to public health officials, policy makers, legislators, and other nonscientific audiences. Each state has a BRFSS coordinator who can assist the state with analyses needed to guide responses to the changes and formulate an appropriate communications plan. CDC is working with the coordinators and other state public health personnel to provide additional materials that will help with these plans. Interpretation of changes in prevalence from one year to the next is a difficult task, especially in years where methods are adjusted. Communication plans should emphasize that 1) shifts in prevalence estimates for 2011 might not represent trends in risk factor prevalence in the population but instead merely reflect improved methods of measuring risk factors, 2) occasional improvements in methods, with accompanying effects on results, have been a necessary part of all public health surveillance systems, including population surveys, and 3) the changes in BRFSS methods are especially important to keep up with changes in telephone use in the U.S. population and to take advantage of improved statistical procedures.

Reported by

Carol Pierannunzi, PhD, Machell Town, MS, William Garvin, Frederick E. Shaw, MD, JD, Lina Balluz, ScD, Div of Behavioral Surveillance, Office of Surveillance, Epidemiology, and Laboratory Svcs, CDC. Corresponding contributor: Carol Pierannunzi, ivk7@cdc.gov, 404-498-0501.

Acknowledgments

Office on Smoking and Health, National Center for Chronic Disease Prevention and Health Promotion, CDC.

References

1. CDC. Behavioral Risk Factor Surveillance System: 2010 survey data. Atlanta, GA: US Department of Health and Human Services, CDC; 2011. Available at http://www.cdc.gov/brfss/technical_infodata/surveydata/2010.htm. Accessed January 6, 2012.
2. CDC. Interim results: state specific influenza vaccination coverage—United States, August 2010–February 2011. MMWR 2011;60:737–43.
3. Kennet J, Gfroerer J, eds. Evaluating and improving methods used in the National Survey on Drug Use and Health. Rockville, MD: US Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Office of Applied Studies; 2005. Available at http://www.samhsa.gov/data/nsduh/methods.pdf. Accessed May 31, 2012.
4. Hu SS, Pierannunzi C, Balluz L. Integrating a multimode design into a national random-digit–dialed telephone survey. Prev Chronic Dis 2011;8:A145.
5. Blumberg SJ, Luke JV. Wireless substitution: early release estimates from the National Health Interview Survey, January–June 2011. Atlanta, GA: US Department of Health and Human Services, CDC; 2011. Available at http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201112.pdf. Accessed December 20, 2011.
6. Groves R. Survey errors and survey costs. New York, NY: John Wiley & Sons; 1989.
7. Battaglia MP, Frankel MR, Link MW. Improving standard poststratification techniques for random-digit–dialing telephone surveys. Survey Research Methods 2008;2:11–9.