Report of the SLAITS and NIS Review Panel to the NCHS Board of Scientific Counselors (BSC)
As part of an on-going program review process, the Board of Scientific Counselors (BSC), National Center for Health Statistics (NCHS) commissioned a panel to review the State and Local Area Integrated Telephone Survey (SLAITS) and National Immunization Survey (NIS) and report its findings to the BSC. This preliminary report summarizes the review process, provides background information on SLAITS and NIS, describes the current status of the surveys, addresses factors affecting the surveys and the environment for health data collection and use, and presents a series of recommendations.
SLAITS is a data collection mechanism operated by the National Center for Health Statistics, Centers for Disease Control and Prevention (CDC), with the capacity to collect state and local health data on a range of topics. It is a random-digit-dial telephone interview survey platform which was established in 1997. SLAITS was initially conceived as a means of utilizing the sampling frame of the National Immunization Survey, a large scale RDD survey launched in 1994 to measure immunization rates among children in States and selected local areas. Working with a number of other agencies, various survey modules—many focusing on child health and welfare and children’s health care needs--have been developed and fielded through SLAITS over the past decade.
- SLAITS was developed as an offshoot of the NIS, to utilize the NIS sampling frame as a cost-effective method of collecting data at the local level. The NIS is conducted by the National Center for Immunization and Respiratory Diseases (NCIRD). The NIS was developed by NCHS--in collaboration with what is now the NCIRD (formerly the National Immunization Program)--and initially was conducted by NCHS for NCIRD. The NIS produces annual estimates of the immunization status of children 19-35 months of age in all states and a varying number of urban areas (14 in 2008). There were 28 fixed metropolitan areas oversampled by the NIS each year from 1994-2004. The current NIS oversamples in 6 fixed metropolitan areas that receive Section 317 immunization grant funds (Chicago, District of Columbia, Philadelphia County, New York City, Houston, and Bexar County, Texas), and an annually variable number of additional city/county areas chosen and funded by state immunization programs. For example, in 2008, ten additional areas are being oversampled, for a total of 16. To identify sufficient numbers of households with children of that age requires contacting about a million households annually, the vast majority of which do not have children matching the survey’s age range. Thus, a byproduct of the NIS is a large-scale, state and local sampling frame, only a small portion of which is used by the NIS. To make use of that resource, SLAITS was developed to provide the mechanism for appending additional surveys to the NIS sampling frame. SLAITS, with its ability to quickly collect State and local area data, was viewed as filling a serious gap in the capabilities of NCHS to monitor the nation’s health and the health data community in general.
In practice, SLAITS is a series of surveys operated in collaboration with sponsoring agencies. SLAITS receives no direct NCHS funding but operates on a cost recovery basis. SLAITS sponsors bear the entire cost of a survey fielded through SLAITS and also pay a 9 percent overhead to CDC and a 15 percent survey support charge to NCHS which covers SLAITS staff. To test the methodology of SLAITS, a health module modeled on the National Health Interview Survey was successfully pilot tested in 1997 in two States but no funding was available for widespread application. The following modules have been conducted by SLAITS and illustrate the variety of survey modules, in terms of topics, size, scope, sponsoring agency, design and operational features.
- Child Well-Being and Welfare Module investigated the association between public assistance program participation and factors that support child well-being. Sponsored by the Department of Health and Human Services (HHS), it was intended to test whether data to monitor the implementation of and eligibility for the State Children’s Health Insurance Program (SCHIP) could be produced through this mechanism. Fielded in 1998 and 1999. Cost:
National Survey of Early Childhood Health was the first national implementation of the SLAITS approach and produced national baseline data on satisfaction with pediatric care and its impact from the parent’s perspective. Primary funding for the survey came from the Gerber Foundation, American Academy of Pediatrics and the Maternal and Child Health Bureau (MCHB). Fielded nationally in 2000; seeking funding for fielding again in 2008. Cost $500,000 – 1 million.
National Survey of Children with Special Health Care Needs was designed to assess the prevalence and impact of special health care needs among children under age 18. The survey was sponsored by MCHB and HHS’s Office of the Assistant Secretary for Planning and Evaluation (ASPE) to provide the data needed by states for block grant applications and program evaluations. Fielded in 2001 and 2005-6 and planned for 2009. Cost: $10-12 million.
National Asthma Survey gathered data that would describe the health, socioeconomic, behavioral, and environmental predictors that relate to better control of asthma, as well as data on the content of care and limitations of those with asthma. The survey had two components: a national survey to produce prevalence estimates and a four-state in-depth survey of a sample of persons with asthma. The survey was sponsored by CDC’s National Center for Environmental Health. Fielded nationally and in 4 States in 2003. Cost: $5 million (includes multiple experimental phases).
National Survey of Children’s Health was designed to produce national and state-specific prevalence estimates for a variety of physical, emotional and behavioral health indicators and measures of children’s experiences with the health care system. Primary funding for this survey was provided by MCHB. The survey was conducted to assess how well each state and the nation are meeting MCHB’s strategic plan goals and national performance measures. Fielded in 2003 and 2007. Cost: $9-11 million.
National Survey of Adoptive Parents sponsored by the ASPE and the Administration on Children and Families, will provide data on the well-being of children adopted from foster care and other adoption outcomes. It is intended to provide the information needed to evaluate the effectiveness of legislation and program initiatives to support and promote adoption. Fielded in 2007. Cost: $800,000.
Survey of Adult Transition and Health is a follow up to the 2001 National Survey of Children with Special Health Care Needs. Children who were 13 to 17 years of age in the earlier survey are now being interviewed on their transition from health care providers who primarily treat children to providers of care to adults. Fielded in 2007. Cost:
Future plans call for SLAITS to repeat the National Survey of Early Childhood Health in 2009 (postponed from 2008 due to unavailability of funding) and the National Survey of Children’s Health in 2009. A SLAITS project budget for the National Survey of Children’s Health is attached (attachment 1) and shows annual funding and NCHS survey support.
SLAITS is in the Division of Health Interview Statistics, Survey Planning and Special Surveys Branch. SLAITS has five full time staff members who each have some responsibilities in addition to their SLAITS functions. For example, the Branch Chief serves as the SLAITS Director, overseeing budget and providing major program direction but also spends at least 50 percent of her time on Division or Center-wide management issues and projects. The SLAITS team is responsible for working with sponsors throughout the life of each survey; one statistician has the lead role for each survey. SLAITS is conducted under contract (currently with the National Opinion Research Center) and staff members work closely with the contractor as well as interface with the sponsor. Staff perform key functions through every phase of the survey from planning, methodological and analytical research, survey operations, data analysis and data dissemination. They interact frequently with data users providing technical assistance and represent the survey in various forums and conferences, both internal and external. The current staffing pattern for SLAITS is shown below. At this time, there is also an ASPH intern working with the survey.
GS-15: Branch Chief, SLAITS management, budget, content, procedures, operations, NIS liaison
GS-14: Lead statistician, methodologist and subject matter expert, survey oversight, weighting, QC
GS-13: Statistician, clearances, content, subject matter, analyst
GS-13: Statistician, analyst, subject matter, SAS expert, technical assistance
GS-13: Computer Scientist, data file prep, technical assistance, QC
The SLAITS and NIS Review Panel members (attachment 2) met on November 27 and 28 at the National Center for Health Statistics. In its deliberations, the Panel followed the “Procedures for Reviewing NCHS Programs” established by the BSC (attachment 3) which call for the reviewers to examine the current status, scientific quality and responsiveness of each program within the context of its mission. Further, the review procedures require that the review take into account future availability of financial and staffing resources, emphasize forward thinking and future plans rather than assess current operations, and conduct an interactive review that obtains information not only from written materials but also from presentations and discussion with program staff.
In advance, the Panel received a number of documents (attachment 4), providing background information on SLAITS and NIS. During the two-day meeting (agenda, attachment 5), they heard presentations (attachment 6) from James Singleton, Chief, Assessment Branch, Immunization Services Division, NCIRD, CDC; Marcie Cynamon, SLAITS Branch Chief; and Stephen Blumberg a senior scientist with SLAITS. Those individuals and other program staff were available throughout the meetings for in-depth discussion and/or provided additional information in response to specific questions or requests. The review focused on SLAITS and only dealt with NIS as necessary to understand issues and circumstances affecting SLAITS. Panel members reviewed and revised the preliminary report and provided specific additional information which has been incorporated. Panel Chair Judith Kasper presented the preliminary report to the BSC and received some additional comments which were considered and incorporated where appropriate.
SLAITS is currently facing a number of issues: short- and long-term funding, adequate staffing resources for survey management; on-going methodological research for optimum survey operations and data quality; marketing of the survey platform; expansion of potential client base; and role within NCHS and the Federal statistical community.
SLAITS has two modules tentatively scheduled for 2009 and no other modules in the planning stages. Many of the modules have been funded all or in part by the Maternal and Child Health Bureau, ASPE or a few other government or private entities. There is concern that this limited client list is insufficient to keep SLAITS in steady, continuous and permanent operation. There is little funding support possible through NCHS appropriations as NCHS itself is dealing with budget levels which may necessitate changes in other NCHS surveys and programs.
The SLAITS staff, while excellent, is small. They have managed to achieve a great deal and to compensate for their small numbers by working with exceptional creativity and dedication but it appears that they have reached the maximum extent to which that is possible. With multiple functions and the primary responsibility for a survey module, each staff member has limited time for wide-ranging research on either current aspects of the survey operations and data production and analysis or on future and alternative directions. SLAITS staff must keep the current operations in place and deal with current clients while at the same time exploring new relationships, new partners, and new opportunities.
In general, SLAITS like most population-based surveys is experiencing declining response rates and the fairly steep rise in the percentage of U.S. household that are without landline telephone phones. Both these factors pose a threat to the representativeness of the group that responds in telephone surveys, affect the quality of the data, and require the use of sophisticated adjustment and imputation techniques in order to provide accurate inferences about the characteristics of the population. SLAITS staff has been actively involved in this process and have contributed to the methodology in this field. In addition, NIS response rates have a direct impact on SLAITS rates. While lower response rates affect national survey results in certain ways, it is less certain and more dynamic how response rates and lower coverage rates affect the quality and reliability of state and local estimates. SLAITS has developed techniques for adjusting state and local estimates but lacks the research staff capacity to conduct the full-scale, ongoing research which not only will validate the results for local areas but also ensure that potential users of the state and local data will maintain confidence in the results and fully utilize the findings.
Function and Role
To determine the appropriate role for SLAITS and thus its key operating components requires time and attention, outreach, exploration and consultation. It also requires a keen and current knowledge of government programmatic and policy initiatives and goals, as well as the data needs generated in the process of managing and evaluating those programs, setting the policies, and establishing the goals. That information is critical to determine how SLAITS could and should function. To see the range of functions for SLAITS is to envision the range and function of its collaborators. SLAITS has functioned as a telephone survey platform allowing the collection of state and local data on a variety of health topics, frequently child health. Although each of these functions is important and potentially sustainable, it is critical to the long-term future of SLAITS to determine a priority for these functions and to use this in developing a consistent SLAITS “brand.” As described later, one of the recommended tasks for the SLAITS staff should be to diversify its collaborators and explore opportunities for cooperation with other agencies; identifying and promoting the SLAITS brand will be central to that effort.
NIS Status and Future
The future of the nation’s immunization program and the statistical systems supporting that public health program will affect the future of SLAITS. SLAITS operates directly off the NIS sampling frame and works closely with NIS to enable each SLAITS module to be designed and fielded. If the nation changes its childhood immunization policies or programs and/or the data collection efforts are modified or eliminated, there is an immediate and powerful impact on SLAITS. While NIS is at a critical juncture in its current telephone survey operations, there is much support from a strong and widespread coalition for the NIS to continue in some form. It is not likely that the NIS will be eliminated since there is a continuing need for an annual assessment of children’s immunization rates and there are many initiatives for immunization in other age groups which also need to be measured and evaluated. However, the survey could be modified in important ways with far-flung implications. As a result of internal and external reviews, NCIRD is considering a number of recommendations including the use of other sampling frames including Immunization Information Systems which are available in most states and are based on birth and other records and piggybacking on other survey sampling frames. While modifications of the NIS could require modifications of the SLAITS, it is hard to envision--in the current funding environment--that SLAITS could operate without the NIS. While a precise estimate for operating without the NIS framework cannot be determined, it is estimated that there would be an annual additional cost of about $2 million for SLAITS to operate without NIS. If SLAITS could find sufficient funding to counteract the loss of the NIS base, this would in a sense “free” SLAITS to conduct other types of surveys than in the past and to operate more widely within the health arena. SLAITS has a “do no harm to NIS” philosophy which governs the way SLAITS conceptualizes and develops each of its surveys. SLAITS must present each of its surveys as a follow up to the introduction of NIS to a potential respondent. Thus, the type and length of surveys and survey items, types of respondents (demographic and socioeconomic characteristics), and overall survey operations have to fit into the NIS framework.
Telephone surveys, landlines and cell phones
Perhaps the greatest challenge to telephone survey operations has been the drop in landline phone saturation rates and the dramatic and continuing increase in cell phone use. In 2007, one in eight U.S. households did not have a landline phone but had access to at least one wireless phone. That number has been steadily growing since 2003 when the question on landline use was first asked in the National Health Interview Survey and the results demonstrated that one in 20 were in wireless only households. (Blumberg, Stephen and Luke, Julian, “Wireless Substitution: Early Release of Data from the National Health Interview Survey,” National Center for Health Statistics, December 12, 2007) Not only are more people living in homes without a landline, this percentage is not consistent across the board and varies greatly by age and living arrangements. More than half of all adults living with unrelated roommates had only wireless telephones. Young adults, renters, men, and those in poverty were more likely to use cell phones and to be in households without a landline. In addition, the NIS and SLAITS like all telephone surveys, has seen declining response rates as a result of changes in the country’s telephone system, the increasing use of screening devices and less public willingness to participate in surveys. Telephone surveys will have to adapt to adjust for these factors in analyzing survey data and making projections about the characteristics of the population. Survey methodology may evolve and change to incorporate cell phones, alone or in some combination with the internet. Newly emerging internet-based survey methodologies may become more likely as the “usual” way of conducting telephone surveys becomes more difficult, more costly, and more problematic.
Rapid response survey capability
One key characteristic of SLAITS is it flexibility and its ability to field a survey in a relatively rapid timeframe. Once a survey has been designed and the questionnaire developed it can be in the field in about 6 weeks. Because SLAITS is a platform it can be more things to more people, easier and faster than large standard surveys. The ability to produce data on a timely basis is a quality long-admired and long-sought in the Federal statistical system, with greater or lesser success at various times. Many of the other major data collection mechanisms of the National Center for Health Statistics, specifically, or in the Federal statistical system, in general, lack the ability to rapidly field an entire survey or even add a limited number of new data items to an on-going survey. As the nation faces the possibility of emerging public health threats or unexpected developments in a dynamic health care system, it will be even more important to have a variety of mechanisms for determining the characteristic of those threats and changes, the population at risk, the speed and direction of the changes, and the most effective responses and reactions.
- Continue SLAITS. SLAITS meets a vital need for State and local area data. It is the only data collection mechanism within NCHS, with the exception of the National Vital Registration System, which is capable of producing a wide range of health data for all States and selected metropolitan areas. NCHS’s vital statistics system has an essential and established content set that can only be modified with the participation and agreement of the registration areas. SLAITS, however, offers the flexibility and quick turnaround that are the hallmarks of an effective telephone survey. Thus, SLAITS fills an important role in the health statistics structure of the nation. While CDC’s Behavior Risk Factor Surveillance System (BRFSS) is an on-going health survey system that tracks information on a number of health conditions, it differs considerably from SLAITS in the fact that the BRFSS is a collection of individual state surveys while SLAITS has the advantage of employing a standard methodology with sufficient detail in each state to produce state-based estimates. The SLAITS platform is ideal for surveys with in-depth data collection on a wide range of topics and the many factors affecting each. While SLAITS and BRFSS might benefit from more opportunities for collaboration especially on aspects of survey methodology, each fills a unique niche.
Determine and define the SLAITS brand. SLAITS can be described as a mechanism to collect State and local data and/or an innovative telephone survey platform. It also has a strong reputation for designing and fielding important and effective surveys of the health and welfare of children. Any and all of these descriptors can be solid selling points and form the basis for an effective marketing strategy, but defining and refining the brand will result in a sharpened image and more definitive strategy. A decision to emphasize data on children (not obtained in other state-level data collection efforts) would suggest broadening content and possible funding partners beyond a health focus (e.g. welfare, education, justice) and reinforces linkage to the NIS, for example. Greater emphasis on state and local data might suggest moving in the direction of collecting data across various age groups, staying within the health framework, and considering a future for SLAITS as complementary to other NCHS surveys.
Implement strategic planning process/explore independence from NIS. SLAITS is now inextricably linked to the NIS. This linkage has enabled the survey to operate over the past decade and to complete a number of successful health survey modules. However, the linkage also creates boundaries that restrict SLAITS in several key ways. For example, the modules conducted as part of SLAITS must be compatible with the overall NIS framework. Also SLAITS is subject to whatever challenges NIS faces, from survey response rates to operational features. SLAITS defers to NIS and accepts aspects of the survey that may not be advantageous to SLAITS. At some point, it may be decided that immunization rates have reached maximum probable levels and/or that the survey doesn’t produce new information and the NIS will be discontinued or modified in ways totally incompatible with SLAITS. SLAITS needs to have an independent survival plan in place. SLAITS should also consider and evaluate other modes of operation within NCHS. SLAITS could potentially operate in a complementary mode to other NCHS surveys and produce state and local estimates to supplement the national data produced by the basic data system. The complementary approach needs to be fully explored; cost estimates developed (including sampling costs without the NIS sampling frame); and scientific, statistical, administrative and management issues identified.
Improve and emphasize the policy relevance of SLAITS. SLAITS has shown that it can address the policy and programmatic data needs of government. Its earliest module, the Child Well-Being and Welfare module was designed to produce data to monitor the implementation of and eligibility for the SCHIP program, though it lacked the detail on household structure or income needed to accurately simulate eligibility for SCHIP. Later modules did produce data for state block grant applications and MCHB program evaluations. Another module explored the effectiveness of programs promoting adoption and the outcomes of those efforts. However, the policy related potential of the SLAITS program is not generally well known nor fully developed. There is a much greater potential than has been reached. With a better understanding of the policy and programmatic issues and the resulting data needs of a wider range of national, state and local agencies, SLAITS could develop proposals to meet those data needs and market its capabilities in a targeted, customized fashion.
For example, the National Survey of Children’s Health has the potential to provide policy relevant information in response to a new directive issued by the Centers for Medicare and Medicaid Services (CMS) in August of 2007. That directive requires states to achieve coverage rates of 95 percent or higher among low-income children in order to receive federal funding to cover children with family incomes above 250 percent of the Federal Poverty Level (FPL) under the State Children’s Health Insurance Program. Given the large state-level samples in the National Survey of Children’s Health--which are larger than the available state-level samples from the Current Population Survey, in many states, it could be used to assess coverage rates among low-income children. However, additional benchmarking and validation studies would need to be done to establish the credibility of the state-level coverage estimates from the SLAITS-based survey. This is a particularly important to the extent that the national insurance distribution from the National Survey of Children’s Health differs substantially from those found in more established surveys that do not rely on telephone-based sample frames. More generally, the SLAITS-based surveys have not been effectively tapped to help CMS or states monitor their Medicaid and SCHIP programs.
SLAITS should also explore the appropriateness/feasibility of producing precise estimates for the 28 metropolitan areas oversampled in the NIS. This again is a special capability of SLAITS to go beyond the national and even state level data to produce information which could be utilized at the local level for planning and policy purposes. The opportunity would add to the value of SLAITS without involving high costs to implement.
Seek stable funding and diversify collaborators. Currently SLAITS sponsors have been limited to just a handful of government agencies and private sector organizations. These sponsors provided essential financial support and the technical and scientific input to implement highly successful survey modules. However, the fact that there have been just a few sponsors in the history of SLAITS makes the funding stream uncertain and collaboration to support SLAITS vulnerable. MCHB, for example, has funded several modules and has a continuing interest in the data collection capacity of SLAITS. If changes occur within that organization in terms of mission, programs, budget, priorities, or leadership the interest in and ability to support SLAITS may diminish or disappear. SLAITS should seek to diversify and expand its client base. Federal agencies, in general, seem the most likely sources of support and collaboration. With the sizeable budgets for each SLATIS module, many private organizations or associations which might have an interest are probably priced out of the market. SLAITS long-term strategic planning should identify the Federal agencies whose programs and activities require data which could be obtained through SLAITS. Then outreach could follow in a targeted approach. The Substance Abuse and Mental Health Services Administration, the Center for Medicare and Medicaid Studies, even programs in the Department of Agriculture seem possible sources. Another possibility is to form a partnership with other agencies, which transcends an individual, ad hoc survey. For example, MCHB with its substantial on-going data needs might be amenable to a long-term agreement with some commitment to future survey modules but flexibility to develop over time. Opportunities to work with SLAITS could be presented to the HHS Data Council or to the Interagency Council on Statistical Policy. There are other possibilities for funding and partnerships to remain in the childhood data collection field. Consider welfare, education, agriculture programs with data needs. Specifically the Interagency Council on Families and Children may be a worthwhile consortium to pursue.
Expand methodological research. SLAITS should explore both short- and long-term methods research. As an immediate area of research interest, SLAITS should consider with NIS the pros and cons, factors involved, potential problems and benefits from using a more general introductory letter for the NIS. As has been stated, the SLAITS modules follow the initial NIS screening and must take into consideration the impact on respondents of the transition from the NIS to SLAITS. A more general NIS letter might broaden the scope of the possible SLAITS modules. On a long-term basis, SLAITS needs to invest in and give greater emphasis to telephone survey methods research. At this critical time for telephone surveys, research efforts should explore dual frame modalities, the use of cell phones, mixed mode methodologies and other methods of utilizing changing technology. As the population’s use of telephone and other communications devices changes rapidly, research to maintain or improve survey response rates, protect against sample bias, and take advantage of new opportunities is critical. Given that a unique strength of the SLAITS is the capability to provide more precise estimates for states and select metropolitan areas, it is imperative that more methodological resources be devoted to validating the state and local estimates that are produced from the survey. The survey’s low response rates and reliance on a land line sample frame introduces concerns about the reliability of the state-level estimates. With such an investment, it would improve the chances that the survey will be used to support policy-relevant analyses, which in turn, could increase support for it. SLAITS should also seek research partners, within NCHS for example through its Office of Research Methodology and expand the initial efforts with NIS staff. SLAITS should explore research opportunities across Federal agencies and perhaps pool resources for research among Federal agencies involved in telephone surveys. A presentation to the Interagency Council on Statistical Policy could introduce the program to potential Federal partners. Results from this research should be published in peer-reviewed literature to enhance the awareness and credibility of SLAITS and to advance the field in general.
Recognize resource limitations/increase staff. As noted earlier, SLAITS staff of five have accomplished a great deal. They’ve exceeded all reasonable expectations in the volume and quality of work they can produce. They have managed to plan and direct surveys, find sponsors, manage a contract, analyze and disseminate data, and deal with data users. However, existing staff is pushed to the limit and it is recommended that there be a modest expansion of staff in order to implement the recommendations developed by the Panel. Specifically the Panel recommends that some additional staff are needed to plan for the future, improve methodology, and increase the visibility and use of SLAITS. This small increase in staff would have major benefits. Staff time devoted to strategic planning could mean more effective outreach and an increase in interest in SLAITS and sponsors to develop and fund modules. Marketing the survey is labor-intensive because it requires both an understanding of the potential markets and the ability to develop and tailor the SLAITS marketing strategy. The best marketing strategy doesn’t just show how the customer’s needs could be met, but helps them identify how data can be employed to achieve their mission more effectively. Staff resources are also needed for validation studies, benchmarking analyses which enable states and local areas to fully utilize the SLAITS results. Finally, the full range of methodological research requires additional staff and staff resources would be well-placed in preparing and publishing analyses of survey results.
SLAITS is an effective program which has achieved solid accomplishments in the field of telephone surveys. It is now at a critical juncture, in terms of its future directions in a dynamic and changing environment. How it will meet those challenges will depend upon the full recognition of both the challenges and the opportunities presented. SLAITS can build upon past success, but to advance or even survive it must realistically assess and evaluate its future options. These recommendations present a framework for that assessment.
Procedures for Reviewing NCHS Programs
PDF Version (52 KB)
SLAITS and NIS presentations