Wellness School Assessment Tool Version 3.0: An Updated Quantitative Measure of Written School Wellness Policies

Schools play an important role in promoting student wellness. As directed by the Healthy Hunger-Free Kids Act, the US Department of Agriculture updated the requirements for written school wellness policies in 2016. The WellSAT (Wellness School Assessment Tool) is an online tool that provides a quantitative score for wellness policy comprehensiveness and strength. The WellSAT has been updated 3 times over the past decade to remain current with federal law and best practices. In this article, we describe the process of updating to WellSAT 3.0. The steps included: reviewing the language of each item linked to a federal provision; adding and deleting items based on frequencies from the National Wellness Policy Study and the empirical support for specific policies; gathering feedback from a survey of experts (N = 77) about best practices and measure usability; and establishing intercoder reliability in a national sample (N = 50) of policies. We conclude with recommendations and guidance for the use of WellSAT 3.0.

mentation, and updates; complete triennial assessments on compliance, alignment with model policies, and progress toward goals; and designate one or more officials as responsible for school-level compliance with the policy.

Original assessment measure for SWPs
To support the empirical study of SWPs, a 96-item quantitative measure was published in 2009 (11). Policy items were identified by reviewing model policies, district self-evaluation tools, state checklists, and state policy classification systems by the National Cancer Institute (12)(13)(14). A rubric was developed to assess how completely each item was addressed, and each item was assigned a corresponding score: topic not addressed at all, score = 0; topic mentioned in a general manner or as part of a recommendation, score = 1; topic addressed in a specific manner or as part of a requirement, score = 2. The tool provided 2 summary scores: comprehensiveness, the percentage of items that received a 1 or 2; and strength, the percentage of items that received a 2. This scoring distinguished districts with strong policies that require or prohibit a practice (eg, "Teachers are prohibited from using food as a reward") while also acknowledging that districts may take an incremental approach to policy change by beginning with aspirational recommendations (eg, "We recommend that teachers use nonfood rewards").

Measure evolution
Over time, this measure was adapted to meet needs in the field. First, Chriqui and colleagues expanded it for the largest, ongoing nationwide assessment of SWPs. This endeavor originated in the Bridging the Gap research program and continued in the National Wellness Policy Study (NWPS) (2,15). To date, more than 5,000 districts nationally have been coded to provide reliable data on the content and strength of SWPs over time (2,10).
Second, in 2010 a national advisory group shortened the measure, named it the WellSAT (Wellness School Assessment Tool), and placed it on a free website, www.wellsat.org. WellSAT 1.0 had 50 items and 6 subscales: nutrition education (NE); standards for USDA school meals (SM); nutrition standards for competitive and other foods and beverages (NS); physical education and physical activity (PEPA); wellness promotion and marketing (WPM); and implementation, evaluation, and communication (IEC). After coding, the user receives 1) each item's score, 2) the comprehensiveness and strength scores for each subscale, and 3) the comprehensiveness and strength scores for the whole policy. Users can code multiple policies and download scores into an Excel file. Since its launch, the website has been accessed more than 100,000 times, and nearly 7,000 registered users have coded more than 9,000 policies from all 50 states and Washington, DC.
In 2014, WellSAT 2.0 was released to reflect the strengthened school meal standards of the HHFKA (5). After the final wellness policy rule requirements for school year 2017-18 were released (6,7,16), the measure and website were updated to version 3.0. This article describes the update and guidance for users of the tool. This study was determined to be exempt from institutional review board approval.
Updating to WellSAT Version 3.0 To revise the measure, the research team reviewed the language of each item inked to a federal provision; examined item frequencies from the NWPS; reviewed the empirical support for specific policies that were under consideration for inclusion or removal; gathered feedback from a survey of experts about best practices and measure usability; and established intercoder reliability. Table 1 lists the 19 WellSAT 3.0 variables that are based on a federal requirement, and Table 2 provides the final 67 WellSAT 3.0 items, the 12 WellSAT 2.0 items that were removed, and the reasons for the changes. The final measure is available at www.wellsat.org.

NWPS data and current evidence
In our second step, we examined frequency data from the 2014-2015 wave of the NWPS. This wave of 496 district policies was collected from a nationally representative sample of school food authorities that completed the director survey of the USDA's School Nutrition and Meal Cost Study (10). We used frequency data to inform decisions about adding and removing items. Items that rarely appeared in policies, were no longer relevant because of changes in the federal law, or were not considered best practices or evidence-based were considered for removal. Concurrently, we considered adding emerging topics and additional policies assessed in the NWPS.
All removed items are listed in Table 2. "School meals meet standards that are more stringent than those required by USDA" was originally created because meal standards were not in alignment with the Dietary Guidelines for Americans; however, this item is no longer necessary because of updated federal regulations (5). "Students leaving school during lunch periods" reflected concerns that students on open campuses may be more likely to eat fast food, but closed campus provisions appear infrequently in the NWPS. "Addresses school meal environment" was written to capture cafeteria features, including adequate space and a clean, pleasant environment; however, it is not in the NWPS, and the team determined that the lack of specificity made the item difficult to interpret. "Nutrition information for school meals (eg, calories, saturated fat, sodium, sugar) is available to students and parents" was important before the implementation of the updated nutrition standards (5) but is no longer a priority. "Recess (when offered) is scheduled before lunch in elementary schools" was originally included because it was considered a best practice; however, it was removed because of inconclusive evidence (22,23). "Foods and beverages containing nonnutritive sweeteners in high school" was removed because of low frequency in the NWPS and current scientific guidance (24). "Teacher-student ratio for physical education classes" was removed because of low frequency in the NWPS data set. "Physical education waiver requirements for K-12 students" was also removed owing to low frequency and because waiver provisions are typically governed by state law rather than district policies. "Staff involvement in physical activity opportunities at all schools" was removed owing to low frequency in the NWPS and a new, broader item was added to address staff wellness (WPM2). Finally, "District provides physical activity training for all teachers" and "Staff not modeling unhealthy eating/drinking behaviors" were removed to be consistent with NWPS. The item about positive staff role modeling (WPM1) was retained.
NWPS items were reviewed for inclusion in WellSAT 3.0 if they were present in coded policies and supported as best practices. "Nutrition education integrated into subjects beyond health education" (NE6) was added because it increases opportunities for nutrition education and appears frequently NWPS policies. "How to handle feeding children with unpaid meal balances without stigmatizing them" (SM4) was added because of increasing awareness of problematic practices and advocacy efforts to protect children (20). Because all foods sold during the school day must meet Smart Snacks standards, 3 new venue-specific items were added to be consistent with NWPS: à la carte (NS3), vending machines (NS4), and school stores (NS5). "Physical education promotes a physically active lifestyle" (PEPA3) (25) was added because it is present in 70% of NWPS policies, and "The establishment of an ongoing school building-level wellness committee" (IEC8) was added because it has been identified as a best practice (21).

Expert survey
The next step was to gather feedback about current best practices and measure usability from tool users and experts on school wellness policies. In April 2018, an online survey was emailed to recently active registered WellSAT 2.0 users (n = ~1,000); the Cen-ters for Disease Control and Prevention (CDC) Nutrition and Obesity Policy Research and Evaluation Network School Wellness Workgroup (n = 146); the Voices for Healthy Kids Nutrition and Physical Activity in Schools Coalition, led by Alliance for a Healthier Generation (n = 25); and experts from CDC (n = 2) and Action for Healthy Kids (n = 2). Some people received the survey more than once because of membership in multiple groups.
In the first part of the survey, we provided the revised list of items under consideration for each subscale and asked respondents to rank the importance of each item as low, medium, or high. There was an open response field for comments and suggested wording modifications next to each item. At the end of each subscale section, we asked, "Please comment on the items as a whole with regard to the policies associated with [name of subscale]. Collectively, do the items capture the range of policy considerations? Are there items you recommend adding or deleting?" The second set of questions included the scoring criteria for each item to achieve a 2 (ie, strong policy) and asked respondents to indicate if the criteria should be "weaker," "stronger," or if "it's just right." Each question also included a field for comments or wording modifications.
Across all items, an average of 8% of respondents rated items as low importance, 25% rated items as medium importance, and 67% rated items as high importance. The only 2 items rated "low importance" by more than 20% of respondents concerned school gardens and Farm to School activities. In the comments, respondents said that school gardens are not feasible for all schools and coders may not understand which activities are part of "Farm to School." To respond, the school garden item was removed and instead, gardens were included as an example in a new item on nutrition education addressing agriculture and the food system (NE8). A second new item addressed purchasing local foods for the school meal program (SM10). To help coders, there is a watering can icon by these items and links to the National Farm to School Network website to highlight these core Farm to School elements (19).
The experts recognized the importance of comprehensive physical activity but suggested that instead of 1 item referencing CDC's Comprehensive School Physical Activity Program (CSPAP) by name (26) www.cdc.gov/pcd/issues/2020/19_0373.htm • Centers for Disease Control and Prevention and community engagement (PEPA11); physical activity before and after school (PEPA12); and physical activity breaks during school (PEPA14) (26). The fifth CSPAP component concerns staff; however, the 2.0 item, "Staff involvement in physical activity opportunities at all schools," was found in only 18% of NWPS polices. Furthermore, experts recommended folding this into a broader staff wellness item. Therefore, the NWPS item, "Strategies to address employee wellness" (WPM2) replaced it.
The feedback on the "2" scoring criteria indicated that most experts felt the guidance should stay the same. On average across all items, 9% of the experts thought guidance should be "weaker," 12% felt it should be "stronger," and 79% said it was "just right." Additional comments from the expert survey informed wording changes and clarifications (Table 2). Experts suggested clearly distinguishing items required by federal law, so they are marked with a Capitol building icon, the label "federal requirement," and exact quotes of federal language with the scoring guidance.

Intercoder reliability
Our final step was to use the NWPS database of policies collected in the 2014-2015 school year (15) to draw a national sample of 72 school district policies that had been recently updated. From this sample, 50 school districts were chosen if their policy had one or more indicators of incorporating recent USDA rules: 1) Smart Snacks appeared within the text of the policy, 2) water availability at lunch appeared within the text, or 3) the policy achieved high comprehensiveness scores.
Per NWPS protocol, we used internet research with telephone or email follow-up to the superintendent's office to obtain the 2017-2018 school year policy documents for the 50 identified districts. Because 4 districts were nonresponsive, the next 4 districts identified using the selection criteria were substituted. The SWPs included the board-adopted wellness policy and the associated administrative guidelines, rules, and regulations. Written school wellness policies often list references to other related district policies, state, or federal laws within the text of the wellness policy to communicate that these additional regulations are also recognized by the district; therefore, we also included any additional policies that were incorporated by reference (eg, food service policies, state physical and health education standards). All documents were coded by 2 trained analysts using standard NWPS methods (2,4,10).
Two coders independently coded the policies and assessed reliability in groups of 5 SWPs. The percentage agreement on the first 2 batches was 73% and 76%, κ = 0.56 and κ = 0.60, respectively. The primary coder (E.P.) identified problematic items and wrote additional coding guidance. The percentage agreement on the next WellSAT 3.0 Data and Uses Table 3 contains descriptive statistics for each item in the full sample of policies (N = 50). All items were written into at least 1 policy in the sample. The mean comprehensiveness score across all categories of the 54 policies indicated that, on average, districts address more than half of the items on the measure, and the mean strength score of 33 means that only one-third of items include strong, definitive language.
WellSAT 3.0 has multiple uses, including self-evaluation by district wellness committees to comply with the requirement to compare their own policy with model policies and best practices (7). Similarly, state agencies can use it to complement their triennial reviews of SWPs, as required by the USDA (16). State agencies may also find WellSAT useful in providing technical assistance to school districts and tracking environmental policy changes. For example, New York State has used WellSAT to support a cohort of school districts (27), and evaluators in Arizona have used it as one of their SNAP-Ed policy strategies (28).
Although the summary scores are between 0 and 100, they should not be translated into letter grades. The coding reflects the strongest versions of policies, setting the bar to avoid a ceiling effect and help districts strive for excellence. To interpret scores, a stepped interpretation strategy is recommended. First, examine only the items that are noted as federal requirements, and ensure that they are present in the written policy. Next, evaluate each subscale comprehensiveness and strength score. If comprehensiveness is much higher than strength, consider using stronger, more specific language to clarify the expectation of the policy. If comprehensiveness scores are low, pick a subscale consistent with district priorities and consider adding new policies.

Comparing scores over time
The WellSAT has changed substantially since its inception in 2006. The first version emphasized going beyond federal requirements in multiple domains, including setting strong nutrition standards for school meals and foods and beverages sold outside of meals. The HHFKA now requires many of these practices, so WellSAT 3.0 items assess compliance with federal regulations. However, this measure also promotes further progress, such as extending strong nutrition standards to food sold or served after the school day.

Limitations
The primary benefit of updating this measure is to reflect changes in federal law and current evidence-based and expertrecommended best practices. However, districts that used Well-SAT 2.0 will not be able to track improvements over time. The WellSAT 2.0 tool is available on the website for those districts. Furthermore, the 50 test policies selected were chosen because they were comprehensive, so scores should not be interpreted as nationally representative. However, the results were consistent with the nationally representative NWPS finding that policies address many topics in an aspirational, nonbinding manner (10). Finally, some of the strong scores are due to state laws, not language specific to the SWP (10).
In conclusion, schools are a critical setting for health promotion, and strong SWPs are a key strategy to ensure consistent implementation of best practices. National surveillance is needed, particularly during school year 2020-2021 after the triennial assessments are required. Future research should evaluate the impact of the HHFKA wellness policies and regulations on local policies, practices, and ultimately student and staff health outcomes.     PUBLIC HEALTH RESEARCH, PRACTICE, AND POLICY JULY 2020 The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors' affiliated institutions.