Evaluation Design of New York City’s Regulations on Nutrition, Physical Activity, and Screen Time in Early Child Care Centers

This article describes the multi-method cross-sectional design used to evaluate New York City Department of Health and Mental Hygiene’s regulations of nutrition, physical activity, and screen time for children aged 3 years or older in licensed group child care centers. The Center Evaluation Component collected data from a stratified random sample of 176 licensed group child care centers in New York City. Compliance with the regulations was measured through a review of center records, a facility inventory, and interviews of center directors, lead teachers, and food service staff. The Classroom Evaluation Component included an observational and biometric study of a sample of approximately 1,400 children aged 3 or 4 years attending 110 child care centers and was designed to complement the center component at the classroom and child level. The study methodology detailed in this paper may aid researchers in designing policy evaluation studies that can inform other jurisdictions considering similar policies.


Introduction
In March 2006, the New York City Department of Health and Mental Hygiene (DOHMH) published a report on the prevalence of obesity among young children who attended Head Start programs in New York City (1). In response, the New York City Board of Health adopted revisions to Article 47 of the New York City Health Code governing licensed group child care centers in an effort to reduce early childhood obesity (2) (Box). Beginning January 1, 2007, centers were required to comply with the new regulations that restrict provided beverages, set minimum amounts for physical activity, and limit television viewing. We describe the multi-method approach developed and implemented to evaluate New York City's regulations of beverages served, physical activity, and screen viewing in group child care centers. This article provides a detailed description of the evaluation design and methods. The results of this evaluation are presented in the multiple manuscripts included in this special collection (3)(4)(5)(6).

Box. Child Care Regulations in Article 47 of the New York City
Health Code Children cannot be served beverages with added sweeteners. • Children can only be served juice that is 100% juice, and no more than 6 oz per day and only served to children 8 months old or older.
• Children 2 years old or older can only be served milk that is 1% milk fat or less. Children must have water available and easily accessible throughout the day.
• Children 1 to 3 years old must have at least 60 minutes of physical activity every day.
• Children 3 years old or older must have at least 60 minutes of physical activity every day, 30 minutes of it guided and structured.
• Children younger than 2 years old are not permitted to watch television.
• Children 2 years old or older are allowed a maximum of 60 minutes of television viewing per day (restricted to educational programs or those that engage child movement).

•
The multi-method approach to evaluate New York City's licensed group child care centers regulations involved 2 components. The first evaluation component (Center Evaluation Component) was designed to evaluate whether compliance was associated with center and staff characteristics. The second evaluation component (Classroom Evaluation Component) was designed to evaluate whether compliance was associated with staff and children's behavior. Institutional review boards of DOHMH and ICF International reviewed and approved all components of the evaluation. We present 1) the sampling methods and sample characteristics for the center and the classroom components of the study, 2) the instruments and data collection methods for the center component, and 3) the instruments and data collection methods for the classroom component.

Sample Selection
The sampling universe included all 1,656 DOHMH-licensed group child care facilities. Three District Public Health Offices (DPHOs) include catchment areas (ie, geographic areas within New York City that have high risk factor and disease burden and therefore increased need for public health services) that consist of low-income, high-risk neighborhoods, specifically East and Central Harlem, the South Bronx, and East and Central Brooklyn. It was thought that centers in these low-income neighborhoods might have the greatest challenge in complying with the new regulations, and they are eligible for training and technical assistance from the DPHO at no cost. To account for the differences in technical assistance received, the sample was stratified according to DPHO status (ie, whether centers were in a DPHO neighborhood or not).
Of the 1,656 licensed group child care facilities in New York City, 311 were in DPHO neighborhoods. Although 97% (301 out of 311) of the DPHO centers were in census tracts with 40% or more of families with incomes at 200% of the federal poverty threshold or below, only about 41% (549 out of 1,345) of the non-DPHO centers were in neighborhoods with such high poverty levels. To ensure comparability of DPHO and non-DPHO centers, only centers from high-poverty areas were randomly included in the sampling frame (300 in DPHO and 350 in non-DPHO catchment areas). The final center component sample consisted of a random sample of 130 centers in DPHO and 130 centers in non-DPHO catchment areas (Figure). Center eligibility was based on 4 criteria: 1) had a classroom of at least ten 3-to 4-year-old children, 2) had at least 2 teachers of 3to 4-year-old classrooms, 3) did not exclusively serve children with special needs, and 4) was not closing or had not already closed. Twenty-six centers were excluded because they did not meet these criteria, and an additional 58 centers elected not to participate. Although the focus of the study was 3-to 4-year-old children (94% of final sample), because the age of children within a classroom varied, our analysis included children 2 to 6 years of age (n = 1,427). Validated accelerometer cut points only exist for children older than 3 years old, therefore we excluded children less than 2 years 10 months of age (n = 29 aged 34-36 months). We also excluded children who were more than 6 years of age (n = 1). Most (92%) of the children were Hispanic or non-Hispanic black ( Table 2).

Center Evaluation Component -Staff Interviews and Site Inventory
The center component of the evaluation involved interviewing staff and conducting a site inventory. Center-level compliance with the regulations was assessed through a review of center records, the site inventory, and in-person interviews with center directors, teachers, and food service staff. DOHMH staff and sanitarians were interviewed on training, technical assistance, and enforcement of the regulations. Standardized, written data collection protocols were developed. In addition, DOHMH provided centerlevel data regarding dates of participation in DOHMH-provided physical activity and nutrition training programs, such as Sports, Play, and Recreation for Kids! (SPARK!), Eat Well Play Hard (EWPH), and EWPH Training of Teachers. These training programs are described in detail in Nonas et al (3) in this issue.

Instruments
The center component interview instruments were designed to assess center level compliance and the degree to which center staff members were familiar with the regulations. In addition, the instruments were designed to identify barriers to compliance with the regulations and the effect that DOHMH training had on staff awareness of the regulations. Teachers selected for participation were identified by center directors. Interview instruments were designed for each type of staff position and adapted from existing instruments (7,8). The site inventory recorded the types of beverages present in food storage and preparation areas, location and number of televisions, the availability of indoor and outdoor play space, and characteristics of the center's neighborhood that could facilitate physical activity, such as access to safe places to play (eg, a neighborhood park). Table 3 lists background resources and evaluation instruments developed for each evaluation component. The center component instruments were pilot tested by project team members at 2 child care centers and revised in advance of data collector training. The results of these pilot tests are not included in analyses. In January and February 2010, ten in-depth semistructured telephone interviews to provide context were completed with DOHMH staff members who enforce regulations and oversee various training programs but were not part of the evaluation team.

Data collection
Results of the center component are reported by Lessard et al (4) and Kakietek et al (5) in this issue.

Classroom Evaluation Component -Classroom and Child Direct Observation
The classroom component was an observational and biometric study of a subsample of the centers participating in the center component to determine whether compliance was associated with staff and child behavior. Classroom-level compliance with the regulations and child behavior was assessed through observation of beverages served, access to water throughout the day, and physical activity opportunities offered as well as documentation of center characteristics that support physical activity opportunities. Child-level outcomes were examined by using observed beverages consumed during meals and snacks and intensity and duration of physical activity measured by accelerometry. The classroom serving children aged 3 to 4 years was selected for the Classroom Evaluation Component. If a center had more than 1 classroom serving children aged 3 to 4 years, one of these classrooms was randomly selected for participation in the study.

Instruments and measurement
The classroom component included 2 days of classroom-level and child-level observation of foods and beverages provided and physical activity opportunities offered to children between 8 AM and 5 PM. Child-level measures included amounts of foods and bever-PREVENTING CHRONIC DISEASE ages served and consumed, child's height in centimeters and body weight in kilograms, and amount and intensity of physical activity each child achieved via accelerometry. Demographic information was collected by using a Child Information Form; data collected included date of birth, sex, race/ethnicity, and start date at the center.
A General Observation Form was created to record the episodes and context of physical activity, staff behavior related to foods provided and physical activity, children's access to water, and classroom staff participation in training. This instrument was adapted from forms used by DOHMH to verify compliance by their sanitarians and from other validated instruments (9)(10)(11)(12). An Accelerometry Form was used to record the time the accelerometer (Actigraph GT3X) was placed on and removed from the child, and the child's height and body weight were recorded on the General Observation Form (9,13,14). The Nutrition Observation Form was used to record food and beverage components of every meal and snack, including service style and preparation. This instrument was used to record staff behavior, such as providing second servings without the child asking, encouraging the child to try new foods, and drinking or eating less healthful foods in front of the children. The Mealtime Observation Form was used to collect information on all food and beverages served to and consumed by the children. The dietary observation instruments were adapted from existing literature on dietary assessment (12,(15)(16)(17).
The classroom component instruments and protocols were pilottested at 1 randomly selected, eligible group child care center and revised. The results of these pilot tests were not included in the evaluation results.

Data collection
Twenty data collectors were trained over a 5-day period that included classroom-based training and site visits. Two-day site visits were conducted between April and June 2010.
On arrival at the centers, data collectors measured height and weight for each child who had parental consent to participate. Height was measured in centimeters by using a portable stadiometer (Seca 213) and weight was measured in pounds by using a portable scale (Seca Clara 803). Each child was measured twice, and the data were recorded to ensure accuracy on the first day of data collection. Anthropometric measurements were averaged and a SAS program (SAS Institute, Inc) developed by the Centers for Disease Control and Prevention (CDC) was used to calculate BMI z scores. Children wore the GT3X Actigraph accelerometers for the duration of the 2-day observation period. Some children had 1 day of accelerometry data because they either refused to wear the accelerometer on the first or second day or did not attend the child care center on the second day of data collection.
Using the nutrition observation form, data collectors recorded all food served in the classroom over the 2 days. During a typical day, meals often included breakfast or morning snack, lunch, and afternoon snack. Three unique children were randomly selected per day. These children did not have to have parental consent to be observed. This resulted in a total of 6 unique child dietary records per center. Results of the classroom component are reported in Kakietek et al (5) and Stephens et al (6) in this issue.

Discussion
This evaluation is the first to measure compliance with beverage and physical activity regulations in a large sample of New York City child care centers in low-income neighborhoods. The data collected using this multi-method approach resulted in the creation of compliance scores for each center for the center and the classroom components, the calculation of a consistency of compliance score based on data from both the center and classroom components, and an analysis of the factors that are associated with compliance. By using mixed methods, we triangulated center-level and child-level data sources and conducted a multi-level assessment of the association between consistency of implementation of the New York City regulations and child behavior.

Limitations
A limitation of this evaluation is the absence of pre-intervention center-level and child-level data. The use of a post cross-sectional evaluation design limits our ability to assess whether the adoption of the new regulations spurred child care centers in New York City to improve their policies and practices regarding beverages, physical activity, and screen time. Also, even though the center component and the classroom component were conducted within a close time period, they were not conducted simultaneously, so it cannot be assumed that the regulation was being implemented the same way. Another potential limitation is that compliance measures were partially based on self-reported data (eg, the staff interviews), which are subject to recall bias and social desirability in responses. In addition, it is possible that the teachers surveyed in the center component were not the same teachers observed or interviewed in the classroom component, leading to a potentially large intra-center variation. It is also possible that the study sample was biased because of nonrandom refusal to participate. Although centers included in the center component sample were selected randomly from a sampling frame, about one-quarter of the eligible sampled centers (58 out of 234) refused to participate. Similarly, 66 centers that participated in the center component of the study opted to not participate in the classroom component. When compared with centers that participated in the center component only, centers that participated in the classroom component reported significantly more of the following: participation in CACFP, being a part of a larger parent agency, having a dedicated food service staff, being in DPHO areas, and participating in DOHMH training programs such as SPARK! and EWPH. It was possible that centers with poor compliance with the regulations were less likely to participate than centers with better compliance.

Strengths
Despite the limitations noted above, this study has numerous unique strengths. At the time of this evaluation, New York City was one of the only major municipalities to have strong regulations for beverages, physical activity, and screen time at licensed group child care centers. As a result, this study design was constructed specifically for the New York City regulatory and training environment. DOHMH staff members with intimate knowledge of the history and intensity of training and technical assistance provided to the centers before and after implementation of the regulations were involved in the study design. Additionally, the study sample focused on low-income neighborhoods that could have more difficulty than high-income neighborhoods in implementing the regulations.

Conclusions
The unique design of this evaluation contributes to the field both through findings and evaluation of practice. Although the results of this evaluation are limited to New York City's metropolitan, urban, low-income communities, these results have potential importance for communities across the nation. The original purpose of the 2 data collection methods for assessing regulation compliance was not to compare or contrast the results for assessing compliance, but the resulting data identified important differences in measuring levels of compliance when using different methods. Practitioners and researchers alike can benefit from understanding the differences (4). Furthermore, the use of accelerometry in the assessment of physical activity fills a gap in the field's knowledge of intensity of activity among children younger than 6 years old, for which there are no federal recommendations. Additional analyses of these data have the potential to add to the knowledge of the type and intensity of children's physical activities in structured and unstructured play.
Our use of multiple data collection methods to examine regulatory compliance in group child care environments contributes to the evaluation field because there are few studies systematically examining compliance and there is increasing demand for methods to assess policy implementation. This evaluation not only documented the extent of compliance by using a variety of methods but also identified factors that may affect a center's ability to comply. To further contribute to the field, future research might examine topics such as inter-rater reliability of observations of environment and child behavior and, in particular, the validity of using self-report data compared with direct observation data on compliance with nutrition and physical activity regulations. Finally, we hope the methods outlined here will provide guidance for future evaluations that build on this work.   DOHMH administrative data Primarily used for sampling stratification (center address and location relative to DPHO catchment areas), and indicators of center participation in DOHMH training programs. Low data collection cost and burden.
Center DOHMH staff interviews Provide context for the setting in which the beverage and physical activity regulations are implemented by interviewing staff responsible for providing child care center training and/or enforcing regulations. Low data collection cost and burden.

Context
Director, teacher and food service staff interviews Respondents' knowledge of and reported compliance with New York City's regulations for beverages, physical activity, and screen time at child care centers. Moderate data collection cost and burden on center staff and time required.

Center
Site inventory Beverages present in the child care center food storage and preparation areas and the centers' physical activity environment. Low data collection cost and burden. Center

Classroom Evaluation Component
Child information form Director reported child birth dates, sex, race/ethnicity, start date in the center, and number of days per week and number of hours per day attending the day care. Moderate data collection cost and burden on center staff and time required.