Evaluating Policy Impact
T21 policy impact evaluations can explore a variety of topics including changes in policy awareness and support, enforcement activities and compliance, retail sales, and underage tobacco access and use. Comprehensive impact evaluations intend to answer questions about whether the policy is having its intended effect, such as:
- How has public and retailer support for the policy changed since its enactment?
- Did T21 policy enactment affect public or decision-maker support for other evidence-based tobacco control policies?
- Did negative social norms about underage tobacco use change?
- Has retailer non-compliance changed over time as enforcement activities have continued?
- Did quitline call volume change?
- Did young adult tobacco use change?
- Did retail tobacco product sales change?
- Did cessation activity of underage youth or young adults change?
- Did retailer revenues and employment change after the policy went into effect?
Impact evaluations can employ quantitative and qualitative methods and a variety of data sources, including focus groups; public opinion polls; surveys of adults, youth, and retailers; enforcement agency reports; and retail sales data.
Table 2 presents example evaluation questions, study designs, indicators, and data sources for evaluation of T21 policy content, implementation, and impact.
| Domain | Example Evaluation Questions | Example Evaluation Designs | Example Meaningful Indicators | Example Data Source |
|---|---|---|---|---|
| Public and Retailer Support | Content Evaluation | |||
| Is there support for the elements of the policy under consideration (e.g., penalties)? | Qualitative (content analysis of news media)
Quantitative (analysis of opinion poll/survey data) |
Level of support for specific policy components | News media
Survey or opinion poll |
|
| Is there opposition to any of the elements? | Qualitative (content analysis of legislative history, testimony, news media) | Expressions of opposition in legislative history, news media | Legislative history
News media |
|
| Is the policy consistent with gold standard policies? | Qualitative (analysis of policy language, model policy language) | Presence of policy content in alignment with gold standard | Legal documents; published and grey literature | |
| Does the policy apply to more products than the federal law? | Qualitative (analysis of policy language) | Presence of definitions | Legal documents | |
| Implementation Evaluation | ||||
| Were efforts made to educate the public, retailers, decision makers, or other stakeholders about the policy? | Qualitative (analysis of administrative records, stakeholder interviews, focus groups) | Number and types of communication with stakeholders (e.g., in-person meetings, phone calls, emails, press releases, educational materials distributed) | Administrative records, press releases, stakeholder websites, stakeholder interviews, focus groups
Telephone or in-person retailer interviews |
|
| Were efforts made to address or alleviate concerns about the policy? Were efforts well received? | Qualitative (analysis of stakeholder interviews, focus groups)
Quantitative (analysis of opinion poll/survey data) |
Number and types of communication with stakeholders (e.g., in-person meetings, phone calls, emails, press releases, educational materials distributed)
Public awareness of outreach Percent of public who felt concerns were addressed |
Stakeholder interviews, focus groups, newspaper coverage
Opinion poll data
|
|
| Impact Evaluation | ||||
| How has public support changed since the policy was enacted? | Qualitative (focus groups, content analysis of news media)
Quantitative (time-series) |
Level of community awareness
Level of support for T21 policy
Level of support for policies, and for enforcement of policies, to decrease availability of tobacco to young people |
Focus groups (before and after policy enactment)
Adult tobacco survey, public opinion polls, news media |
|
| Did policy enactment change support for other policies to prevent initiation of tobacco use? | Quantitative (time-series) | Level of support for other tobacco use initiation policies | Public opinion polls | |
| Did negative social norms about underage tobacco use change? | Quantitative (time-series) | Level of support for norms about underage tobacco use | Adult tobacco survey, public opinion polls | |
| Domain | Example Evaluation Questions | Example Evaluation Designs | Example Meaningful Indicators | Example Data Source |
|---|---|---|---|---|
| Compliance | Content Evaluation | |||
| Are retailer, public education and enforcement activities explicitly referenced, delegated, and funded in the policy? | Qualitative (analysis of policy language) | Presence of policy components relevant to education and enforcement | Legal documents
|
|
| Were penalties for non-compliance specified in the policy for retailers and customers? Are the penalties evidence-based? | Qualitative (analysis of policy language, literature)
|
Presence of T21 penalties
Evidence from the literature supporting T21 penalties |
Legal documents, grey and published literature | |
| Implementation | ||||
| To what extent were the public and businesses aware of the policy? | Qualitative (analysis of stakeholder interviews, focus group data)
Quantitative summary analysis |
Knowledge of specific policy components | Stakeholder interviews (including retailers)
Focus groups
Public opinion polls |
|
| To what extent were compliance checks and enforcement actions performed according to the policy? | Qualitative (analysis of stakeholder interviews)
Quantitative summary analysis (post-intervention) |
Description of compliance checks in alignment with policy
Documentation of compliance checks |
Stakeholder interviews (including retailers), enforcement data | |
| Are there pockets of non-compliance that need additional enforcement? | Quantitative summary analysis (post-intervention) | Number of T21 violations by geographic area, store type | Enforcement data
|
|
| Impact | ||||
| Has non-compliance changed over time as enforcement activities have continued? | Quantitative (post-intervention time series) | Number of violations as proportion of inspections | Enforcement data | |
| How does observed retailer compliance compare to other states or localities without their own T21 policies? | Quantitative (post-intervention time series; strengthened with comparison) | Number of violations as proportion of inspections | Enforcement data | |
| Has reported retail access changed for youth under 18? | Quantitative (time-series, strengthened with comparison) | Proportion of young people who report having purchased a tobacco product from a retailer in the previous 30 days
Proportion of youth who report that they were refused sale of tobacco products because of their age during the previous 30 days
Locations where youth purchased cigarettes
|
Youth tobacco survey | |
| Has reported usual sources for tobacco changed for youth under 18? | ||||
| Domain | Example Evaluation Questions | Example Evaluation Designs | Example Meaningful Indicators | Example Data Source |
|---|---|---|---|---|
| Youth Behavior | Content | |||
| Is policy consistent with evidence base to eliminate harms to health and not increase disparities among population groups? | Qualitative (literature review) | Evidence supporting policy content regarding health and disparities | Grey and published literature | |
| Implementation | ||||
| To what extent was policy enacted uniformly in all communities affected by the policy? | Quantitative summary analysis (post-intervention)
Qualitative (stakeholder interviews) |
Proportion of stores inspected in affected jurisdiction(s)
Perception of uniform implementation Number of people affected |
Enforcement data
Stakeholder interviews |
|
| Impact | ||||
| How did quitline call volume change as a function of the policy? | Quantitative, time-series | Number of quitline calls (per week/month) | Quitline data | |
| How did rates of youth tobacco use change as a function of the policy? | Quantitative (time-series, strengthened with comparison) | Proportion of young people who have never tried smoking or using any other tobacco products
Proportion of young people who have smoked or used some type of tobacco product at least 1 day during the previous 30 days
|
Youth tobacco survey | |
| How did rates of young adult tobacco use change as a function of the policy? | Quantitative (time-series, strengthened with comparison) | Cigarettes: Proportion of adults aged 18-24 years old who have smoked at least 100 cigarettes in their lifetime and who now report smoking cigarettes every day or some days
Other tobacco products: Proportion of adults who have ever used a given tobacco product in their lifetime and who now report using the product every day or some days |
Adult tobacco survey | |
| How did overall tobacco consumption change as a function of the policy? | Quantitative (time-series, strengthened with comparison) | Retail unit sales of tobacco products | Retail scanner data | |
| How did rates of tobacco use cessation actions (quit attempts, quits) by young adults change as a function of the policy? | Quantitative (time-series, strengthened with comparison) | Proportion of tobacco users who stopped using a tobacco product for 24 hours in a quit attempt
Proportion of former tobacco users who last used tobacco 6 months to 1 year ago
Proportion of former tobacco users who have sustained abstinence from tobacco use for 6 months or longer |
Adult tobacco survey | |
| Domain | Example Evaluation Questions | Example Evaluation Designs | Example Meaningful Indicators | Example Data Source |
|---|---|---|---|---|
| Economic Impact | Content evaluation | |||
| To what extent did economic considerations or interests strengthen or weaken the policy? | Qualitative (content analysis of legislative history, testimony) | Evidence of economic considerations in legislative history | Legislative history | |
| Did the policy state the evidence about the expected economic impact? | Qualitative (content analysis of legislative history, testimony) | Presence of economic impact considerations in policy language | Legislative history | |
| Implementation | ||||
| How did events or activities in surrounding jurisdictions impact policy implementation (e.g., such that these may have encouraged or discouraged cross-border sales by young people)? | Qualitative (content analysis of stakeholder interviews, legal documents)
Quantitative (time-series, adjacent jurisdictions) |
Summary of policy and/or environmental context of surrounding jurisdiction(s)
Retail unit sales of tobacco products |
Stakeholder interviews, legal/news documents, scanner data | |
| Impact | ||||
| How did retailer revenues and employment change after the policy was enacted, all other factors being equal? | Qualitative (review of tax, sales, revenue information)
Quantitative (time series, post-implementation) |
Retailer revenue/tax filings for tobacco product sales
Perception of employment and economic impact after policy enacted Number of retail stores |
Tax receipts, sales revenue data, retailer survey | |
