Statistical reliability of estimates

Estimates for the total population generally have relatively small sampling errors and high precision, but estimates for certain population subgroups may be based on small numbers of respondents or events and have relatively large sampling errors or low precision. Different data systems have different standards for evaluating reliability. The criteria used to designate or suppress statistically unreliable estimates are indicated in the notes of the applicable tables or charts. Estimates that are unreliable because of large sampling errors, low precision, small denominators, or small numbers of events are noted with an asterisk (*).

Numbers of deaths obtained from the National Vital Statistics System (NVSS) represent complete counts and are not subject to sampling error. They are, however, subject to random variation, and standard errors are calculated to account for this variation in statistical testing. When the number of events and the probability of such an event are small, estimates may be unreliable. To protect confidentiality, rates and proportions based on a small number of events are suppressed.

For National Center for Health Statistics (NCHS) surveys, point estimates and their corresponding sampling variances were calculated using the SUDAAN software package version 11, which takes into consideration the complex survey design. Standard errors for other surveys or data sets were computed using the methodology recommended by the programs providing the data or were provided directly by those programs. Starting with Health, United States, 2017, updated reliability standards for proportions (usually multiplied by 100 and expressed as percentages) are used for selected NCHS data sources and years—specifically for estimates from the National Health and Nutrition Examination Survey beginning with the 2013–2014 cycle and the National Health Interview Survey beginning with 2016. Using these standards, the reliability of NCHS survey percentage estimates was assessed through a multistep approach based on minimum denominator sample sizes; the absolute and relative width of the Clopper–Pearson 95% confidence interval (adapted for complex surveys by Korn and Graubard); and degrees of freedom. Estimates identified as statistically unreliable are suppressed and replaced with an asterisk (*). Estimates whose complementary proportions are unreliable are presented but flagged with a double asterisk (**). This Korn–Graubard approach performs well for proportions near 0 or 1, incorporates information from the complex survey design including effective sample sizes, and is generally conservative (that is, a 95% Clopper–Pearson confidence interval includes the true proportion more than 95% of the time). The use of the Korn–Graubard modification of the Clopper–Pearson confidence interval for proportions is considered an improvement over the commonly used Wald confidence interval, which is known for its undercoverage (that is, a 95% Wald confidence interval includes the true proportion less than 95% of the time). The reliability of survey estimates for earlier data years was evaluated based on relative standard errors. (See Sources and Definitions, Relative standard error [RSE].)

For more information on the multistep approach, see: Parker JD, Talih M, Malec DJ, Beresovsky V, Carroll M, Gonzalez JF Jr, et al. National Center for Health Statistics data presentation standards for proportions. National Center for Health Statistics. Vital Health Stat 2(175). 2017. Available from: https://www.cdc.gov/nchs/data/series/sr_02/sr02_175.pdf. (Also see Sources and Definitions, Statistical testing.)