Planning for Policy Evaluation
In identifying and engaging potential stakeholders for evaluating tobacco control policies, it is important to include participants from sectors in addition to those that are customarily involved in evaluation efforts. Promising groups include those involved in the policy implementation and enforcement. Federal, academic, or national health organization partners represent stakeholders that may have an interest in evaluation of the T21 law. Examples of these and other potential stakeholders are shown in Table 1.
| Stakeholder Categories | Examples of Stakeholders |
|---|---|
| Policy Experts |
|
| Evaluation Experts |
|
| Subject Matter Experts |
|
| Implementers |
|
Evaluation of a T21 law requires a clear understanding of the policy. Stakeholders and evaluators should understand the policy content, including provisions, implementation, and enforcement guidance. FDA’s guidance regarding the federal T21 law continues to emerge. Several key policy components are likely to affect implementation and enforcement and thus the evaluation. These components are highlighted below:
- Policy definitions (e.g., tobacco products covered);
- Enforcement authority;
- Penalty schedule; and
- Dates for when the policy is effective and active enforcement begins.
Understanding the policy and its intended effects enables evaluators to graphically display the theorized pathways of change in a logic model. The logic model should be tailored to the policy components and a specific jurisdiction’s evaluation priorities. A generic logic model that can be tailored to meet diverse evaluation needs is shown in Figure 1. In addition to standard logical model components, such as policy-related inputs, activities, outputs and outcomes (CDC, 2008), environmental context and unintended consequences are important considerations that also should be tailored.
Click on the “View Larger” button above to open an image. Then right click the image and select “Save picture as…” to download the image.
Evaluation of logic models specific to T21 policies passed in California (Figure 2) and Hawaii (Figure 3) illustrate state-specific tailoring of a generic logic model. California’s model was designed to support process and outcome evaluations. The California T21 policy specified that “any person, firm, or corporation” who sells or gives away tobacco products is accountable for violating the policy and included an exemption for active duty military personnel (Cal. Bus. & Prof. Code § 22958).
Click on the “View Larger” button above to open an image. Then right click the image and select “Save picture as…” to download the image.
The Hawaii T21 policy held “any person”—including the retailer making an illegal sale and the underage individual attempting to purchase a tobacco product—accountable for violating the policy (Figure 3; Haw. Rev. Stat. Ann. § 712-1258). It specified that persons under age 21 years in violation of the policy are subject to a $10 fine for the first offense and a $50 fine or 48–72 hours of community service for subsequent offenses. Hawaii did not explicitly exempt the military; the Army, Marines, and Navy indicated that they would voluntarily follow Hawaii’s T21 policy.2
Click on the “View Larger” button above to open an image. Then right click the image and select “Save picture as…” to download the image.
Below are several types of evaluation studies that state or local tobacco control programs can conduct to understand the process and outcome of a T21 policy.
- Public Support. These studies use focus groups and/or surveys to assess public awareness of and support for a T21 policy. Public awareness/support studies can be used formatively to shape implementation or to document baseline public awareness and support for the policy and to measure changes over time.
- Retailer Implementation. Telephone-based or in-person surveys of tobacco product retailers can be used to monitor implementation issues and unintended consequences (e.g., increases in purchases made on behalf of, or with the intent to sell to, underage consumers), as well as retailer support for or concerns about the policy.
- Retailer Compliance. Assessments of T21 policy compliance by tobacco product retailers often rely on data gathered by the agency charged with enforcing the T21 policy, for example, tobacco purchase compliance checks. Such checks can be conducted by various agencies, depending on the jurisdiction, and may be part of routine retailer inspections for other purposes (e.g., FDA compliance, Synar compliance, state compliance). Data from compliance checks can be used to document policy compliance rates (i.e., proportion of retailers in a jurisdiction found to be in compliance with the policy), implementation of the enforcement regime, and/or types of venues and geographic regions that are more or less compliant with the T21 policy. This information can inform resource allocations for targeted retailer education, outreach, and enforcement efforts.
- Behavior of Underage Youth and Young Adults. Questions on behavioral outcomes (e.g., reported ease of access to tobacco products; ever or current use of tobacco products; quit attempts) can be developed for existing state- or locality-based surveys of youth and young adults within the jurisdiction being evaluated. Studies of behavioral outcomes are most effective if they include data from before and after the T21 policy goes into effect.
Economic Impact. These studies use economic data to assess the cost-benefit of a T21 policy on retailers and other stakeholders. For example, a study on the cost-effectiveness of raising the MLSA to 21 in California used simulation models to generate estimates of benefits and costs (Ahmad, 2005). Substantial declines in youth initiation over 50 years with no net cost and a significant gain in quality adjusted life years were reported (Ahmad, 2005). Additionally, Ali et al. assessed the association between raising the MLSA to 21 with monthly sales of cigarette packs in California and Hawaii using difference-in-differences methods. Pre- and post- implementation estimates were obtained. The results showed that T21 policies were associated with a 13% reduction in cigarette pack sales in California and 18% reduction in cigarette pack sales in Hawaii (Ali, 2019). The costs, benefits, and methods included in these and similar studies may provide information that is helpful when planning studies of economic impact.
2 For more information: Armyexternal icon, Marinesexternal icon, Navyexternal icon.
