Focusing the Evaluation

Focusing the evaluation/mission

Planning and implementing a T21 policy evaluation rely on having clearly articulated objectives and a sound plan for collecting and analyzing data. This section covers example evaluation questions across the domains of public awareness, compliance, health, and economic impact. Evaluation questions are organized by evaluation stage (Figure 4).

  • Content evaluation explores the process of identifying the problem and developing the policy.
  • Implementation evaluation explores the policy enactment and implementation, including enforcement.
  • Impact evaluation examines the policy’s impact on the intended short-, intermediate- and long-term outcomes, as laid out in the policy logic model.
Figure 4. Evaluation Planning and Implementation Flow
Evaluation Planning and Implementation Flow diagram described in section above

Click on the “View Larger” button above to open an image. Then right click the image and select “Save picture as…” to download the image.

Evaluating Policy Content

Evaluation of policy content is often included as a preliminary step of a comprehensive evaluation because it helps the evaluator properly interpret implementation and impact evaluation. Content evaluation of any policy, including T21 laws, takes an in-depth look at the policy’s strengths and weaknesses when compared with a model policy. For example, national tobacco control organizations created guidelines for a strong MLSA policy and developed model language (see below).

According to these organizations (Public Health Law Center, 2019.), “A strong tobacco minimum legal sales age (MLSA) 21 ordinance will:

  • Define tobacco products to include current and future tobacco products, including e-cigarettes;
  • Prohibit the sale of tobacco products to persons under the age of 21;
  • Require the tobacco retailer or their employer to verify the age of the purchaser prior to the sale;
  • Require tobacco retailers to post signs stating that sales to persons under the age of 21 are prohibited;
  • Designate an enforcement agency and establish a clear enforcement protocol;
  • Create a tobacco retail licensing program if the jurisdiction has the authority to do so under state law;
  • Dedicate funding to fully cover enforcements costs, either through licensing fees or as a provision in a state statute or local ordinance;
  • Provide authority for the state, county, or municipality to inspect tobacco retailers for compliance with MLSA 21 and a mandated minimum number of annual compliance checks for every tobacco retail establishment;
  • Provide penalties focused on the tobacco retailer, or licensee rather than the youth purchaser or the non-management employee. This would mean eliminating Purchase, Use, and Possession (PUP) penalties where they exist in current tobacco sales laws or polices.
  • Establish a civil penalty structure for violations rather than a criminal penalty structure to avoid unintended consequences that disproportionately impact marginalized communities and undermine the public health benefits of the policy, and
  • Where state legislation is pursued, ensure that local jurisdictions have the authority to enact more stringent regulations for tobacco products than state or federal law.”

Nineteen states and over 540 localities passed T21 laws prior to the passage of the federal law, and two additional states have enacted T21 laws since passage of the federal law (Campaign for Tobacco-Free Kids, 2020). Jurisdictions may continue to adopt their own MLSAs for tobacco products to help bolster compliance with the federal T21 requirements or to raise the MLSA higher than 21 years. If a state, local, territorial, or tribal law is not as strong as federal law, retailers still must comply with the federal law. Differences among these policies may be relevant for assessing and comparing the strengths of the various laws as well as any other relevant, existing state or local policy elements that are still being enforced. This can help inform future MLSA policy development, as well as other retail-oriented policies.

Examples of specific questions for a T21 policy content evaluation are listed below:

  • Is there support for the policy components? Is there opposition?
  • Is the policy consistent with gold standard policies?
  • Are enforcement activities, retailer education, and penalties delineated?
  • How are key components defined in the policy (e.g., tobacco products, restrictions, and signage)?
  • Did economic considerations or interests strengthen or weaken the policy?
  • Does the policy state the evidence about the expected economic impact?
  • If there are multiple local policies, how do they differ along these key components?
  • How did local education efforts affect policy adoption at the local level?
Evaluating Policy Implementation

A T21 implementation evaluation explores the activities involved in communicating about the policy, policy monitoring, and policy enforcement. For example, an implementation evaluation may examine media and other communications efforts (e.g., distributing signage) about the policy to the public and retailers. Implementation also covers the training, protocol development, and activities related to retailer inspections that make up compliance monitoring. For the federal T21 law, the FDA has not yet issued its final regulation, but has said that its enforcement will generally be carried out using the same process as when the federal MLSA was 18, and that it has begun using older underage persons in its compliance checks. (FDA, 2020).

Implementation evaluation questions could include:

  • Were education activities conducted, including addressing any concerns? Were they well-received?
  • Were education efforts effective at increasing awareness of the policy?
  • Were compliance checks performed according to the policy, or if not, how were they performed?
  • Are there geographic pockets of retailers who are non-compliant?
  • Were the right stakeholders involved in implementation?
  • Was the policy implemented uniformly across states?

As part of an evaluation of policy implementation, evaluators should examine all aspects of the policy. This includes the activities, actions, and the multiple perspectives for raising the MLSA for tobacco products to 21 years (for an expanded discussion, see Morian & Malek, 2017). A comprehensive evaluation will address all viewpoints by providing evidence of the policy’s impact and equipping policymakers and implementers with available data to mitigate any unintended consequences.

Guidance Sections
Focusing the Evaluation