Evaluating Policy Impact

Multiple graphs and a laptop

T21 policy impact evaluations can explore a variety of topics including changes in policy awareness and support, enforcement activities and compliance, retail sales, and underage tobacco access and use. Comprehensive impact evaluations intend to answer questions about whether the policy is having its intended effect, such as:

  • How has public and retailer support for the policy changed since its enactment?
  • Did T21 policy enactment affect public or decision-maker support for other evidence-based tobacco control policies?
  • Did negative social norms about underage tobacco use change?
  • Has retailer non-compliance changed over time as enforcement activities have continued?
  • Did quitline call volume change?
  • Did young adult tobacco use change?
  • Did retail tobacco product sales change?
  • Did cessation activity of underage youth or young adults change?
  • Did retailer revenues and employment change after the policy went into effect?

Impact evaluations can employ quantitative and qualitative methods and a variety of data sources, including focus groups; public opinion polls; surveys of adults, youth, and retailers; enforcement agency reports; and retail sales data.

Table 2 presents example evaluation questions, study designs, indicators, and data sources for evaluation of T21 policy content, implementation, and impact.

Table 2. Evaluation Planning Matrix – T21 Policy Evaluation
Table 2a. Public and Retailer Support
Domain Example Evaluation Questions Example Evaluation Designs Example Meaningful Indicators Example Data Source
Public and Retailer Support Content Evaluation
Is there support for the elements of the policy under consideration (e.g., penalties)? Qualitative (content analysis of news media)

 

Quantitative (analysis of opinion poll/survey data)

Level of support for specific policy components News media

 

 

Survey or opinion poll

Is there opposition to any of the elements? Qualitative (content analysis of legislative history, testimony, news media) Expressions of opposition in legislative history, news media Legislative history

News media

Is the policy consistent with gold standard policies? Qualitative (analysis of policy language, model policy language) Presence of policy content in alignment with gold standard Legal documents; published and grey literature
Does the policy apply to more products than the federal law? Qualitative (analysis of policy language) Presence of definitions Legal documents
Implementation Evaluation
Were efforts made to educate the public, retailers, decision makers, or other stakeholders about the policy? Qualitative (analysis of administrative records, stakeholder interviews, focus groups) Number and types of communication with stakeholders (e.g., in-person meetings, phone calls, emails, press releases, educational materials distributed) Administrative records, press releases, stakeholder websites, stakeholder interviews, focus groups

Telephone or in-person retailer interviews

Were efforts made to address or alleviate concerns about the policy? Were efforts well received? Qualitative (analysis of stakeholder interviews, focus groups)

 

 

 

 

 

Quantitative (analysis of opinion poll/survey data)

Number and types of communication with stakeholders (e.g., in-person meetings, phone calls, emails, press releases, educational materials distributed)

 

Public awareness of outreach

Percent of public who felt concerns were addressed

Stakeholder interviews, focus groups, newspaper coverage

 

 

 

Opinion poll data

 

Impact Evaluation
How has public support changed since the policy was enacted? Qualitative (focus groups, content analysis of news media)

 

 

Quantitative (time-series)

Level of community awareness

Level of support for T21 policy

 

Level of support for policies, and for enforcement of policies, to decrease availability of tobacco to young people

Focus groups (before and after policy enactment)

Adult tobacco survey, public opinion polls, news media

Did policy enactment change support for other policies to prevent initiation of tobacco use? Quantitative (time-series) Level of support for other tobacco use initiation policies Public opinion polls
Did negative social norms about underage tobacco use change? Quantitative (time-series) Level of support for norms about underage tobacco use Adult tobacco survey, public opinion polls
Table 2b. Compliance
Domain Example Evaluation Questions Example Evaluation Designs Example Meaningful Indicators Example Data Source
Compliance Content Evaluation
Are retailer, public education and enforcement activities explicitly referenced, delegated, and funded in the policy? Qualitative (analysis of policy language) Presence of policy components relevant to education and enforcement Legal documents

 

Were penalties for non-compliance specified in the policy for retailers and customers? Are the penalties evidence-based? Qualitative (analysis of policy language, literature)

 

Presence of T21 penalties

Evidence from the literature supporting T21 penalties

Legal documents, grey and published literature
Implementation
To what extent were the public and businesses aware of the policy? Qualitative (analysis of stakeholder interviews, focus group data)

 

 

 

Quantitative summary analysis

Knowledge of specific policy components Stakeholder interviews (including retailers)

Focus groups

 

Public opinion polls

To what extent were compliance checks and enforcement actions performed according to the policy? Qualitative (analysis of stakeholder interviews)

Quantitative summary analysis (post-intervention)

Description of compliance checks in alignment with policy

Documentation of compliance checks

Stakeholder interviews (including retailers), enforcement data
Are there pockets of non-compliance that need additional enforcement? Quantitative summary analysis (post-intervention) Number of T21 violations by geographic area, store type Enforcement data

 

Impact
Has non-compliance changed over time as enforcement activities have continued? Quantitative (post-intervention time series) Number of violations as proportion of inspections Enforcement data
How does observed retailer compliance compare to other states or localities without their own T21 policies? Quantitative (post-intervention time series; strengthened with comparison) Number of violations as proportion of inspections Enforcement data
Has reported retail access changed for youth under 18? Quantitative (time-series, strengthened with comparison) Proportion of young people who report having purchased a tobacco product from a retailer in the previous 30 days

 

Proportion of youth who report that they were refused sale of tobacco products because of their age during the previous 30 days

 

Locations where youth purchased cigarettes

 

Youth tobacco survey
Has reported usual sources for tobacco changed for youth under 18?
Table 2c. Youth Behavior
Domain Example Evaluation Questions Example Evaluation Designs Example Meaningful Indicators Example Data Source
Youth Behavior Content
Is policy consistent with evidence base to eliminate harms to health and not increase disparities among population groups? Qualitative (literature review) Evidence supporting policy content regarding health and disparities Grey and published literature
Implementation
To what extent was policy enacted uniformly in all communities affected by the policy? Quantitative summary analysis (post-intervention)

Qualitative (stakeholder interviews)

Proportion of stores inspected in affected jurisdiction(s)

Perception of uniform implementation

Number of people affected

Enforcement data

 

Stakeholder interviews

Impact
How did quitline call volume change as a function of the policy? Quantitative, time-series Number of quitline calls (per week/month) Quitline data
How did rates of youth tobacco use change as a function of the policy? Quantitative (time-series, strengthened with comparison) Proportion of young people who have never tried smoking or using any other tobacco products

 

Proportion of young people who have smoked or used some type of tobacco product at least 1 day during the previous 30 days

 

Youth tobacco survey
How did rates of young adult tobacco use change as a function of the policy? Quantitative (time-series, strengthened with comparison) Cigarettes: Proportion of adults aged 18-24 years old who have smoked at least 100 cigarettes in their lifetime and who now report smoking cigarettes every day or some days

 

Other tobacco products: Proportion of adults who have ever used a given tobacco product in their lifetime and who now report using the product every day or some days

Adult tobacco survey
How did overall tobacco consumption change as a function of the policy? Quantitative (time-series, strengthened with comparison) Retail unit sales of tobacco products Retail scanner data
How did rates of tobacco use cessation actions (quit attempts, quits) by young adults change as a function of the policy? Quantitative (time-series, strengthened with comparison) Proportion of tobacco users who stopped using a tobacco product for 24 hours in a quit attempt

 

Proportion of former tobacco users who last used tobacco 6 months to 1 year ago

 

Proportion of former tobacco users who have sustained abstinence from tobacco use for 6 months or longer

Adult tobacco survey
Table 2d. Economic Impact
Domain Example Evaluation Questions Example Evaluation Designs Example Meaningful Indicators Example Data Source
Economic Impact Content evaluation
To what extent did economic considerations or interests strengthen or weaken the policy? Qualitative (content analysis of legislative history, testimony) Evidence of economic considerations in legislative history Legislative history
Did the policy state the evidence about the expected economic impact? Qualitative (content analysis of legislative history, testimony) Presence of economic impact considerations in policy language Legislative history
Implementation
How did events or activities in surrounding jurisdictions impact policy implementation (e.g., such that these may have encouraged or discouraged cross-border sales by young people)? Qualitative (content analysis of stakeholder interviews, legal documents)

 

Quantitative (time-series, adjacent jurisdictions)

Summary of policy and/or environmental context of surrounding jurisdiction(s)

 

Retail unit sales of tobacco products

Stakeholder interviews, legal/news documents, scanner data
Impact
How did retailer revenues and employment change after the policy was enacted, all other factors being equal? Qualitative (review of tax, sales, revenue information)

 

 

 

Quantitative (time series, post-implementation)

Retailer revenue/tax filings for tobacco product sales

Perception of employment and economic impact after policy enacted

Number of retail stores

Tax receipts, sales revenue data, retailer survey
Guidance Sections
Evaluating Policy Impact