Planning for Policy Evaluation

Group at a planning meeting

In identifying and engaging potential stakeholders for evaluating tobacco control policies, it is important to include participants from sectors in addition to those that are customarily involved in evaluation efforts. Promising groups include those involved in the policy implementation and enforcement. Federal, academic, or national health organization partners represent stakeholders that may have an interest in evaluation of the T21 law. Examples of these and other potential stakeholders are shown in Table 1.

Table 1. Examples of T21 Policy Stakeholders
Stakeholder categories and some examples of stakeholders
Stakeholder Categories Examples of Stakeholders
Policy Experts
  • National/state/local/territorial/tribal legislators and staff
  • National/state/local/territorial/tribal tobacco prevention and control program staff
  • Relevant enforcement agency staff (e.g., Department of Health, Attorney General’s Office, alcohol and tobacco boards, state enforcement agencies)
  • State/national/local nonprofit organizations (e.g., American Cancer Society, American Heart Association, American Lung Association, Campaign for Tobacco-Free Kids, Public Health Law Center)
Evaluation Experts
  • State, federal, academic, or contract evaluation research partners
  • Agency evaluation staff
Subject Matter Experts
  • State/local/territorial/tribal department of health and tobacco prevention and control program staff, SAMHSA-funded (Synar compliance) staff
  • Legal support partners (e.g., legal technical assistance centers)
  • State/national nonprofit organizations (e.g., Campaign for Tobacco-Free Kids, American Cancer Society, American Heart Association, American Lung Association, American Academy of Pediatrics, youth organizations)
  • Local substance abuse agencies, local arms of the state alcohol and tobacco agency
  • University research partners
  • National Network representatives (e.g., National LGBT Cancer Network, National African American Tobacco Prevention Network)
  • State and local partners of National Networks
  • Inspection or enforcement agency staff (e.g., Department of Health, SAMHSA-funded staff such as Synar compliance staff, Alcohol and Tobacco boards/agencies, local law enforcement)
  • Local enforcement agency staff (Department of Finance, Office of Consumer Affairs)
  • City/county boards/workgroups responsible for enforcing laws
  • Local advocates, coalition members, mobilized stakeholders
  • Mayor’s staff responsible for implementing new laws
  • Attorney General’s Office
  • Military stakeholders
  • Tribal stakeholders
  • Business associations, retailers selling tobacco (engaged only in the context of implementation-related outcomes and only as appropriate)
Describing the Policy Being Evaluated

Evaluation of a T21 law requires a clear understanding of the policy. Stakeholders and evaluators should understand the policy content, including provisions, implementation, and enforcement guidance. FDA’s guidance regarding the federal T21 law continues to emerge. Several key policy components are likely to affect implementation and enforcement and thus the evaluation. These components are highlighted below:

  • Policy definitions (e.g., tobacco products covered);
  • Enforcement authority;
  • Penalty schedule; and
  • Dates for when the policy is effective and active enforcement begins.

Understanding the policy and its intended effects enables evaluators to graphically display the theorized pathways of change in a logic model. The logic model should be tailored to the policy components and a specific jurisdiction’s evaluation priorities. A generic logic model that can be tailored to meet diverse evaluation needs is shown in Figure 1. In addition to standard logical model components, such as policy-related inputs, activities, outputs and outcomes (CDC, 2008), environmental context and unintended consequences are important considerations that also should be tailored.

Figure 1. Generic T21 Logic Model
Generic Logic Model

Click on the “View Larger” button above to open an image. Then right click the image and select “Save picture as…” to download the image.

Text description of Figure 1.

Evaluation of logic models specific to T21 policies passed in California (Figure 2) and Hawaii (Figure 3) illustrate state-specific tailoring of a generic logic model. California’s model was designed to support process and outcome evaluations. The California T21 policy specified that “any person, firm, or corporation” who sells or gives away tobacco products is accountable for violating the policy and included an exemption for active duty military personnel (Cal. Bus. & Prof. Code § 22958).

Figure 2. California State T21 Policy Logic Model
California Logic Model

Click on the “View Larger” button above to open an image. Then right click the image and select “Save picture as…” to download the image.

Text description of Figure 2.

The Hawaii T21 policy held “any person”—including the retailer making an illegal sale and the underage individual attempting to purchase a tobacco product—accountable for violating the policy (Figure 3; Haw. Rev. Stat. Ann. § 712-1258). It specified that persons under age 21 years in violation of the policy are subject to a $10 fine for the first offense and a $50 fine or 48–72 hours of community service for subsequent offenses. Hawaii did not explicitly exempt the military; the Army, Marines, and Navy indicated that they would voluntarily follow Hawaii’s T21 policy.2

Figure 3. Hawaii State T21 Policy Logic Model
Hawaii Logic Model

Click on the “View Larger” button above to open an image. Then right click the image and select “Save picture as…” to download the image.

Text description of Figure 3.

Below are several types of evaluation studies that state or local tobacco control programs can conduct to understand the process and outcome of a T21 policy.

  • Public Support. These studies use focus groups and/or surveys to assess public awareness of and support for a T21 policy. Public awareness/support studies can be used formatively to shape implementation or to document baseline public awareness and support for the policy and to measure changes over time.
  • Retailer Implementation. Telephone-based or in-person surveys of tobacco product retailers can be used to monitor implementation issues and unintended consequences (e.g., increases in purchases made on behalf of, or with the intent to sell to, underage consumers), as well as retailer support for or concerns about the policy.
  • Retailer Compliance. Assessments of T21 policy compliance by tobacco product retailers often rely on data gathered by the agency charged with enforcing the T21 policy, for example, tobacco purchase compliance checks. Such checks can be conducted by various agencies, depending on the jurisdiction, and may be part of routine retailer inspections for other purposes (e.g., FDA compliance, Synar compliance, state compliance). Data from compliance checks can be used to document policy compliance rates (i.e., proportion of retailers in a jurisdiction found to be in compliance with the policy), implementation of the enforcement regime, and/or types of venues and geographic regions that are more or less compliant with the T21 policy. This information can inform resource allocations for targeted retailer education, outreach, and enforcement efforts.
  • Behavior of Underage Youth and Young Adults. Questions on behavioral outcomes (e.g., reported ease of access to tobacco products; ever or current use of tobacco products; quit attempts) can be developed for existing state- or locality-based surveys of youth and young adults within the jurisdiction being evaluated. Studies of behavioral outcomes are most effective if they include data from before and after the T21 policy goes into effect.

Economic Impact. These studies use economic data to assess the cost-benefit of a T21 policy on retailers and other stakeholders. For example, a study on the cost-effectiveness of raising the MLSA to 21 in California used simulation models to generate estimates of benefits and costs (Ahmad, 2005). Substantial declines in youth initiation over 50 years with no net cost and a significant gain in quality adjusted life years were reported (Ahmad, 2005). Additionally, Ali et al. assessed the association between raising the MLSA to 21 with monthly sales of cigarette packs in California and Hawaii using difference-in-differences methods. Pre- and post- implementation estimates were obtained. The results showed that T21 policies were associated with a 13% reduction in cigarette pack sales in California and 18% reduction in cigarette pack sales in Hawaii (Ali, 2019). The costs, benefits, and methods included in these and similar studies may provide information that is helpful when planning studies of economic impact.

Guidance Sections
Planning for Policy Evaluation