Gathering Credible Evidence and Justifying Conclusions

Diverse creative team in modern office working on project together

In addition to the standard considerations around data selection and analyses, there are other considerations specific to the logistical, social, and political context of T21 policies that should be considered carefully when planning an evaluation. When looking at state, local, territorial, or tribal-level policies, the logistical and contextual considerations below can vary significantly between policies. While the national T21 law may dissipate much of the  variability among state, local, territorial, or tribal laws, it will still be important to understand how the law is structured, the environmental context in which it is taking place, as well as the historical context, which will vary between jurisdictions.

Compliance Checks: In addition to compliance activities conducted to enforce state, local, territorial, or tribal T21 laws, compliance checks for the federal T21 law occur at both state and federal levels. The Synar Amendment to the Alcohol, Drug Abuse, and Mental Health Administration Reorganization Act of 1992 requires states to enact and enforce laws prohibiting the sale or distribution of tobacco products to individuals under the age of 18 years to be eligible to receive substance abuse funding (Substance Abuse and Mental Health Administration, 2017). Under the Synar Amendment and other T21 laws enacted prior to the national T21 law, compliance was assessed by using underage decoys to determine if retailers were selling to underage persons. When the national T21 law was enacted, the Synar Amendment was also updated to reflect the new MLSA of 21. The federal agency that administers the Synar program has not yet provided guidance for states to conduct those compliance checks, and FDA has not yet finalized its regulations related to federal enforcement of the federal law.

Baseline Data Collection: States or jurisdictions should consider collecting baseline information when using pre-post designs to evaluate T21 policies. However, the rapid nature in which the national T21 law was passed presents challenges for having baseline data. This makes it difficult to measure changes in public opinion over time and, thus, policy impact. In such cases, existing data sources may offer opportunities for estimating baseline rates. Examples include, examining previous tobacco use among 16-17-year-old survey respondents, underage retail sales rates from Synar inspections, data collected related to more locally based T21 policies prior to the national T21 law, and retail scanner sales data (which is typically available retrospectively for several years). If information was not previously collected, a pre-post comparison will not be possible. However, other designs could be used to assess impact.

Comparative Assessments:  When baseline measures are impossible to obtain, quasi-experimental designs using comparison groups could be helpful, since this design allows for comparisons of samples similar in all aspects except for the characteristic of interest. For example, this design can be used to assess how contextual factors and variation in enforcement and compliance can have an impact on policy outcomes and impact (Coly & Parry, 2017).

Environmental Context: As with any policy evaluation, policies, campaigns, initiatives, or other environmental changes concurrent to the policy under review must be considered when designing and analyzing a T21 evaluation. This attention to the environmental context is especially important given that point-of-sale campaigns, increases in tobacco taxes, and other strategies may be enacted independent of T21 policies. Evaluators should consider both new and existing tobacco control policies and other interventions that may contribute to the outcomes being measured. In addition, evaluators can use evaluation designs that maximally control sources of invalidity (Campbell & Stanley, 1966).

Guidance Sections
Gathering Credible Evidence and Justifying Conclusions