BRFSS Annual Survey Data

	Please note: This page is best printed in landscape view.

2003 Summary Data Quality Report

This report provides selected statistical indicators of data quality in the Behavioral Risk Factor Surveillance System (BRFSS). The report presents data on three general types of measures by state¹:

(1) Outcome measures, including response rates, which are based on disposition codes.

(2) Selection biases with respect to sex, age, and race/ethnicity.

(3) Missing values on income.

The measures in this report are designed to document the quality of BRFSS data. “Data quality” in this report refers to the accuracy of BRFSS data.

Outcome Measures

The factors affecting the distribution of disposition codes by state may be grouped into differences in telephone systems, sample designs, surveyed populations, and data collection processes. Different outcome measures are variously affected by differences in these factors.

Tables 1 and 2 present the frequency of the thirty-one individual final disposition codes by state. Table 1 shows the distribution of all telephone numbers of determined eligibility status in the sample. Table 2 presents this distribution for numbers of unknown eligibility. The number of completed interviews (110) plus the number of partial interviews (120) divided by the total number of records by state is the Efficiency Rate.

Table 3 presents brief descriptions of each final call disposition code. For more information on the BRFSS protocol and process for assigning disposition codes, as well as a list of interim disposition codes, see BRFSS Policy Memo 2001.1: BRFSS Disposition Codes.

Table 4 shows the frequency distribution and Table 5 shows the percent (of all numbers in the sample) distribution of disposition codes for each state grouped into several descriptive categories. The categories shown in Tables 4 and 5 and used in the calculations of the outcome rates in Table 6 are defined below. P in the table below is the proportion of records with a final disposition code of 210 for which more than half of the core questionnaire prior to the demographics section was completed. These interviews are included in the response rate numerator but are not used in calculating estimates of risk factors and prevalences.

2003 BRFSS Summary Data Quality Report

Table 6 provides six outcome rates for each state that are used to measure respondent cooperation, data quality, and data collection efficiency. The Resolution Rate is the proportion of all telephone numbers in the sample for which the status of the numbers as households with working numbers has been resolved. Records for which household status remains unknown are excluded from the numerator. The formula for the Resolution Rate is

The Screening Completion Rate is the proportion of all known households in which the presence or absence of an eligible respondent has been determined and in which, for eligible households, an interviewer actually spoke to the selected respondent. Households in which the presence or absence of an adult is unknown are excluded from the numerator. Its formula is

The Interview Completion Rate is the proportion of contacted selected respondents who successfully complete an interview. This rate is a type of cooperation rate. An alternate response rate definition is the product of these three rates. The formula for the Interview Completion Rate is

The Cooperation Rate is the proportion of all respondents interviewed of all eligible units that were actually contacted. Non-contacts are excluded from the denominator. This rate is based on contacts with households containing an eligible respondent. The denominator of the rate includes completed interviews plus the number of non-interviews that involve the identification of and contact with an eligible respondent. A Cooperation Rate below 65 percent may indicate some problem with interviewing techniques. The denominator of the Cooperation Rate consists of records with disposition codes of 110, 120, 210, 220, 250, and 260. Thus, the formula for the BRFSS Cooperation Rate is

A response rate is an outcome rate with the number of completed and partial interviews in the numerator and an estimate of the number of eligible units in the sample in the denominator. A proportion of the terminations (210) are included as partial interviews in the BRFSS CASRO Response Rate calculation because more than fifty percent of the core questionnaire was completed in these cases. The BRFSS CASRO Rate calculation assumes that the unresolved numbers contain the same percentage of eligible households as the records whose eligibility or ineligibility are determined. This estimated level of eligibility provides a conservative response rate due to the fact that the proportion of these unknown eligible telephone numbers that are eligible is probably quite low, given the fifteen or more call attempts required by BRFSS protocol. The formula for the BRFSS CASRO Rate is

The response rate is an indicator of the potential for bias in the results of a survey. It does not indicate the actual amount of bias. The actual amount of bias can be conceptualized as a function of two factors—the amount of non-response, which is measured by a response rate, and the differences between the respondents and the non-respondents. A response rate does not address the latter factor. If the non-respondents are highly similar to the respondents for the characteristics of interest, then even a low response rate will result in little non-response bias.

Table 6 also presents the BRFSS Overall Response Rate. The Overall Response Rate is a more conservative response rate that assumes that more unknown records are eligible and thus includes a higher proportion of all numbers in the denominator. The rate assumes that 90 percent of likely households are households and that 98 percent of households contain an adult who uses the phone number.

The Overall Response Rate formula is

The BRFSS Refusal Rate is the proportion of all eligible respondents that refused to complete an interview or terminated an interview prior to the threshold required to be considered a partial interview. Refusals and terminations (TERE) are in the numerator, and the denominator is the same as that of the Response Rate.

Selection Biases

Tables 7 through 14 present data on the differences between BRFSS and population data with respect to sex, age, and race/ethnicity by state. In these tables, BRFSS data are weighted for the characteristics of the sample design—disproportionate sampling by geographic and density strata (where they exist), number of telephones, and number of adults in the household.³ Since these factors are built into the sample design, they should be adjusted for before comparing survey distributions to population distributions. No standards exist to determine what constitutes a substantial difference between survey and population percentages. One approach would be to examine the distribution of discrepant values with the purpose of identifying extreme values, or outliers, which may indicate biased data.

Income Missing Values

Table 15 presents the percent missing (Don’t know/Not sure or Refused or either) income by state. Income is the variable in the survey data with the largest percentage of missing values. A larger percentage of missing values for income implies lower quality data for income and, by extension, for other variables.