Effectiveness of Health Impact Assessments: A Synthesis of Data From Five Impact Evaluation Reports

Introduction Since the 1990s, the use of health impact assessments (HIAs) has grown for considering the potential health impacts of proposed policies, plans, programs, and projects in various sectors. Evaluation of HIA impacts is needed for understanding the value of HIAs, improving the methods involved in HIAs, and potentially expanding their application. Impact evaluations examine whether HIAs affect decisions and lead to other effects. Methods I reviewed HIA impact evaluations identified by literature review and professional networking. I abstracted and synthesized data on key findings, success factors, and challenges from 5 large evaluations conducted in the United States, Europe, Australia, and New Zealand and published from 2006 through 2015. These studies analyzed impacts of approximately 200 individual HIAs. Results Major impacts of HIAs were directly influencing some decisions, improving collaboration among stakeholders, increasing awareness of health issues among decision makers, and giving community members a stronger voice in local decisions. Factors that contributed to successful HIAs included engaging stakeholders, timeliness, policy and systems support for conducting HIAs, having people with appropriate skills on the HIA team, obtaining the support of decision makers, and providing clearly articulated, feasible recommendations. Challenges that may have reduced HIA success were poor timeliness, underestimation of time and resources needed, difficulty in accessing relevant data, use of jargon in HIA reports, difficulty in involving decision makers in the HIA process, and absence of a requirement to conduct HIAs. Conclusion HIAs can be useful to promote health and mitigate adverse impacts of decisions made outside of the health sector. Stakeholder interactions and community engagement may be as important as direct impacts of HIAs. Multiple factors are required for HIA success. Further work could strengthen the role of HIAs in promoting equity, examine HIA impacts in specific sectors, and document the role of HIAs in a “health in all policies” approach.


Introduction
Health impact assessments (HIAs) have been used for 2 decades as a tool to facilitate communication between public health professionals and decision makers in other sectors (1). More than 390 HIAs in the United States have been completed or were in progress as of early 2016 (2). An HIA is a "a systematic process that uses an array of data sources and analytic methods and considers input from stakeholders to determine the potential effects of a proposed policy, plan, program, or project on the health of a population and the distribution of those effects within the population. HIA provides recommendations on monitoring and managing those effects" (3).
Several reports have called for evaluation to determine the impacts and usefulness of HIAs (3,4). Reasons to evaluate HIAs include assessing whether they provide the expected impacts, improving methods, identifying positive and negative unintended consequences, and justifying requests for future resources.
Three types of evaluation can be conducted on HIAs: process, impact, and outcome (3). Process evaluations examine the process followed in conducting an HIA and compares it with the practitioner's intended plan or with established guidelines for HIA practice (5,6); several process evaluations have been published (7)(8)(9)(10). Impact evaluations examine the impacts of an HIA on selected decisions and other events; such information is valuable to those who decide whether or not to conduct HIAs. Outcome evaluations examine changes in health status or health determinants caused by the HIA, but these are rarely conducted. The objective of this study was to review and describe 5 large impact evaluations of HIAs, 3 of which were recently published. Methods I identified HIA impact evaluation reports by networking with HIA colleagues, searching literature in PubMed and Google Scholar, and reviewing references in published studies. I also searched major US and international HIA websites for relevant reports (2,(11)(12)(13)(14)(15) and reviewed recent HIA textbooks (3,(16)(17)(18)(19).
This study included HIA evaluation reports available in English that assessed and synthesized data on the impacts of multiple individual HIAs. Of 8 potentially eligible reports identified, 1 unpublished student report (20), 1 unpublished conference presentation (21), and 1 Canadian study now in progress (22) were excluded. I abstracted information about background, methods, and findings, including the impact of HIAs on decisions and on other relevant events, factors that contributed to the success of HIAs, and factors that reduced the likelihood of an HIA having an impact. I reviewed the similarities and differences in impacts, success factors, and challenges found across the various reports. To compare multistudy HIA impact evaluations with individual HIA impact evaluations, I reviewed a convenience sample of 8 impact evaluation reports that described 11 individual HIAs (23-30) selected for topic and geographic diversity.
I invited the investigators of the 5 major HIA evaluation studies to review a draft of this report to ensure their studies were accurately described and to provide additional comments. Investigators who provided comments are named in the Acknowledgments.  (35).

Characteristics of major HIA evaluation reports
None of the samples used in the 5 reports were representative of all HIAs. Davenport et al (32) reported on all HIAs identified in multiple databases as of 2004. The other reports purposefully selected HIAs from large databases by using criteria such as having "some potential for effectiveness" (35): reflecting diversity in sector, geography, timing, effectiveness, and funding (31,33): or relating to the agency's mission (34). HIAs were examined from such sectors as land use, transportation, housing, agriculture, energy, health services, and waste management.
All 5 reports included a review of the written findings and recommendations for each individual HIA. Three reports (31,33,35) included interviews of key stakeholders and decision makers to assess impacts. Davenport et al (32) reviewed commentaries and discussion papers and included an email survey. Rhodus et al (34) used only publicly accessible documentation from the Internet to identify HIA impacts.
Each report sought to document whether HIA recommendations influenced the decisions they were intended to inform. All reports identified impacts on decisions of some HIAs and generally provided examples.

Definition of HIA success
One reason to evaluate an HIA is to determine if it is a "success." Davenport et al (32) reports that "a successful HIA is one where its findings are considered by decision makers to inform the development and implementation of a [policy, program or project]." Bourcier et al (31) states that "success for HIAs should therefore be defined by both their impacts on decisions and on the environments in which decisions are made." Haigh et al (33) writes that effectiveness (success) can be defined as "the extent to which the HIA succeeds in bringing about the desired changes to decisionmaking and implementation . . . [but there are often disagreements] about what constitutes 'success' and what constitutes a 'desired change.'" In an unpublished survey, HIA practitioners defined success as educating decision makers about the health consequences of a policy, influencing the design of a project, creating new partnerships between health and other agencies, facilitating community involvement in a decision, and addressing community concerns (21). awareness but no specific changes are made in decision), opportunistic (favorable decision would have been made anyway), and ineffective (HIA ignored in decision). Haigh et al (36) found these categories difficult to use because different aspects of a single HIA may fall into multiple categories, and judging whether an HIA impact fit into a category often needed a graded characterization rather than a yes-or-no decision.
Wismar et al (35) also proposed measuring effectiveness in 3 dimensions: health effectiveness (avoiding negative health effects and strengthening positive health effects), equity effectiveness (assessing and preventing disproportionate impacts on vulnerable populations), and community effectiveness (incorporating a community's interests into the decision process). That review found varying levels of these dimensions of effectiveness across the 17 HIAs assessed. The other 4 reports did not use these dimensions in their analyses.

Direct HIA impacts
The 5 reports I studied indicated that many HIAs directly influenced decisions ( Table 2). Evidence fulfilling Wismar's direct effectiveness criteria was found by Rhodus et al (34) in 30 of 81 HIAs (37%) and by Bourcier et al (31) in 11 of 23 HIAs (48%), although the 2 groups overlapped. Examples of decision changes directly attributable to an HIA include those associated with a county bicycle plan (26) and a housing proposal (38). A linear model of decision makers obtaining information, making a decision, and acting accordingly often does not reflect actual circumstances (33), and the impacts of many HIAs may not fit neatly into Wismar's 4 categories. For some HIAs, other impacts such as community engagement may be more important than direct impacts.
The 5 reports indicated projects or policies are rarely cancelled because of unfavorable HIA recommendations. One HIA about rental housing voucher restrictions and another about a solid waste facility identified adverse health effects and contributed to those proposals not moving forward (34). Occasionally an HIA was favorable but the project was subsequently cancelled for other reasons (39).

Other HIA impacts
Three reports highlighted the raised awareness of health issues among stakeholders and decision makers caused by the HIA (Table 2). Although difficult to quantify, this raised awareness has the potential to influence decisions, thus providing an impact that extends beyond the initial reason for which the HIA was conducted.
Two reports found that an HIA may help create and strengthen interactions among stakeholders and other community members. Such interactions may continue after the HIA is completed and provide ongoing community benefits. Many HIAs helped facilitate the relationships between agencies across sectors (40).
Two reports emphasized the value of community engagement as part of conducting HIAs. The HIA process is designed to incorporate community input and may amplify community member voices in the decision-making process (31,41). Through participation in HIAs, community members may take action, increase contact with decision makers, and acquire skills that could help them influence decisions (42). Community engagement with the goal of empowerment is central to the HIA core value of promoting equity (43,44).
Other potential HIA impacts mentioned in various reports include developing new cross-disciplinary and interagency collaborations, identifying data gaps and questions for future research, establishing a foundation for appropriate monitoring, ensuring the public has accurate information on adverse and beneficial effects, and developing new forecasting methods (3).

Success factors
The 5 reports identified factors that contribute to success (also called "enablers" [32]) ( Table 3). The following factors were identified in at least 2 reports. The wording of success factors and of challenges varied widely among reports.
Engaging stakeholders. All 5 reports suggested that stakeholder input was valuable during the planning and conducting of the HIA, generally to provide a unique perspective and to increase knowledge and skills among the HIA team and the decision makers.
• Timeliness. Four reports suggested that an HIA must be conducted early enough in the decision process to have an impact.
• Policy and systems support for conducting HIA. Three reports indicated the value of having frameworks or legislation supporting HIA conduct and of following established guidelines for conducting HIAs.
• Engaging the "right" people on the HIA team. Three reports found that team members may be chosen based on their knowledge and experience with HIAs, teamwork skills, knowledge of the subject matter being assessed, and acquaintance with the community involved.
• Engaging and obtaining support of decision makers. Three reports highlighted the importance of ensuring the willingness of decision makers to receive and consider HIA recommendations. Clearly articulated, feasible recommendations. Three reports highlighted the value of clear, precise language to make it practical for decision makers to act on the HIA recommendations.
• Tailoring presentation of messages to decision makers and other audiences. Three reports indicated that HIAs with summaries of findings and recommendations written in language that is appropriate for the intended audiences are more likely to achieve the desired impact.
• Other success factors included taking advantage of HIA credibility to influence decision makers, quantifying health impacts, clarifying who pays the HIA costs, and using proactive positioning to recognize or create opportunities for HIAs and optimize HIA timing. These factors were each reported in one evaluation. One study reported development of an implementation and monitoring plan for recommendations as a success factor, but another study reported inadequate follow-up of recommendations as a challenge.

Challenges
The reports identified various challenges to an HIA being considered successful ( Table 4). The following challenges were identified in at least 2 reports.
Timing and timeliness. Three reports indicated that HIAs may have little impact if completed too late to affect a decision. For example, in legislative settings, the pace of decision making is sometimes faster than the pace of the HIA.
• Adequacy of resources. Three reports found that practitioners often underestimated the time and staff resources needed to conduct an HIA.
• Accessing relevant data. Two reports found that the absence of sufficiently local health data may reduce the value to decision makers of an HIA's findings and recommendations.
• Use of jargon. Two reports mentioned that use of technical terms or inconsistent terminology in HIA reports may reduce their value.
• Lack of support from, or involvement by, decision makers. Two reports found that HIAs are unlikely to affect decisions when decision makers are uninvolved in the process.
• Absence of requirement to consider HIA findings. Two reports mentioned that decision makers may give minimal consideration to recommendations by an HIA that was not required.
• Other reported challenges included being involved in controversial topics, lack of knowledge of HIA practitioners about policy and of policy makers about health, and disagreements on the breadth of the definition of health and on expert predictions in HIA reports.

HIA evaluation of individual HIAs
A review of impact evaluations of a convenience sample of 11 individual HIAs in 8 reports (Table 5) found impacts, success factors, and challenges similar to those described above. These impact evaluations were done by investigators associated with conducting the original HIAs, except for the 4 HIAs in Ireland (29). A New Zealand website has a list of 13 individual HIA impact evaluations (45), but it does not synthesize the common findings across those evaluations. No similar lists were identified for individual HIA evaluations conducted elsewhere.

Discussion
The evaluation reports reviewed consistently showed that some HIAs had direct impacts on decisions. Others may have had no direct impact on a decision because the decision maker was not receptive or the decision process was complex and nonlinear. Other impacts were found for most HIAs and may have been as important as direct impacts. Such other impacts included raising awareness of health issues among decision makers, building new partnerships among stakeholders, and engaging community members in issues that affect them. HIAs rarely led to project cancellations, which may be a concern especially for projects in the private sector.
Various success factors and challenges were identified across the studies, but few were consistently found in all reports. No single success factor was identified as necessary and sufficient to ensure the success of an HIA. Even timeliness was found to be not essential; Haigh et al (33) reported examples in which the HIA affected not the decision but the subsequent implementation of the policy. The findings on HIA success factors and barriers of this report are consistent with the findings of other studies. One survey of 47 HIAs in the United States documented the importance of community participation to increase the likelihood of HIA success (42). A British study based on 14 key informant interviews reported that leadership, integration of the HIA with existing organizational structures, and collaboration among key stakeholder organizations were central to encouraging HIA use in decision making (46). A master's thesis that reviewed 54 HIAs in the United States reported that time and funding constraints are often barriers to HIA effectiveness and that most HIA practitioners believe that the use of quantitative health data strengthens HIA credibility (20).
A survey of participants in the European Healthy Cities Network reported that the following factors facilitated the use of an HIA: political support, training in HIA, collaboration with a public health institution or agency, a culture of intersectoral collaboration, a supportive national policy, and access to HIA expertise PREVENTING CHRONIC DISEASE (47). The same survey reported the following barriers to HIA implementation: lack of skill, knowledge, and experience about HIAs; lack of a legal basis for implementation; and lack of political support (47). In a related report, factors enhancing the acceptability of recommended interventions included documented benefits, adaptability for stakeholder needs, an evidence base for the work, engagement of stakeholders, and the use of simple, clear language (48).
Promoting equity and reducing health disparities are often cited as major reasons for conducting an HIA (43,44,49). Equity may have improved as a result of some HIAs examined in this study, but no reports explicitly documented changes in equity or the factors contributing to reduced inequities. This lack of documentation may be due to the time required for documentation or difficulty in measuring such changes.
Process and outcome evaluations are complementary to impact evaluations. Results of several process evaluations of HIAs have been published, either alone (7-10) or as a combined process and impact evaluation (34, 35). Guidelines are available for process evaluations (17,50). Most process evaluations have found substantial variability in the extent to which HIA investigators followed guidelines for HIA practice (8,34).
HIA outcome evaluations would be valuable to document whether implementing a recommendation has actual effects on health outcomes (eg, rate of heart disease) or on health determinants (eg, physical activity level) (3,51). Outcome evaluations are difficult to conduct because 1) causal pathways are complex, making it difficult to attribute a specific change in a health outcome or determinant to a specific HIA recommendation, 2) there may be no comparison group to document what would have happened in the absence of the HIA, and 3) they require substantial time and resources (3,19,51,52). No outcome evaluations were found in my literature review. The most relevant example found was a class exercise that predicted dietary changes related to a new supermarket (53).
This analysis synthesized data on the characteristics and findings of multiple HIA impact evaluations. It found compatible results across 5 major reports that evaluated more than 200 individual HIAs. These reports examined HIAs that were conducted in diverse sectors, countries, and years, and by various methods. Particularly valuable were the interviews of stakeholders and decision makers in 3 reports, an approach that increases the likelihood of identifying which factors influence a decision. This analysis also compared various success factors and challenges that affect HIA impacts across multiple studies. This study has several limitations. The contexts and health systems vary across the countries in which the HIAs were conducted. HIA evaluations may have been missed if unpublished or if published in a language other than English. HIAs conducted in countries whose primary language is not English, such as Iran (54), Mongolia (55), and Thailand (56), may have different impacts on decisions, because of cultural, political, or economic differences. Each of the 5 evaluations used a different strategy for selecting HIAs; none were random samples of all HIAs. Three evaluations included potential effectiveness as part of the sampling strategy, which may have biased results toward finding an HIA to be effective. For some HIAs, decisions were being made as the HIA was being evaluated, so subsequent impacts of the HIA may have been missed. For other HIAs, the decision maker may not have remembered the impact of the HIA on a decision made long ago. Some decision makers may not have been able to state exactly how much weight was given to each of several factors (eg, health, economic, political, and social) in reaching a decision. Some HIAs involved politically sensitive topics about which informants may have been reluctant to provide full information. Some findings were difficult to quantify, such as the extent to which collaboration increased and awareness of health issues among stakeholders was raised. Not all of the evaluators interviewed decision makers; information obtained from HIA investigators or public sources may have not included data on influences of the HIA on the decision-making process. Finally, little information was available about the methods or impacts of HIAs conducted in the private sector (3).
The findings of this review lead to additional questions for which further research would be valuable. Several of these questions are now being researched by working groups of the Society of Practitioners of Health Impact Assessment (57).
First, because the impacts of HIAs are complex, no simple answer exists for policymakers and funders who ask how HIAs make a difference in decision making. HIAs may be more effective in some sectors than in others. Guidelines for best practices for HIAs in various sectors could be developed from reviews and case studies of HIAs in each sector (39,58,59;  Second, further work is needed to document the value of HIA in promoting equity and reducing health inequities. Such work requires identifying appropriate indicators of equity that are likely to be affected by decisions and for which changes can be measured (60). Third, outcome evaluations would be valuable to document the usefulness of HIAs. It may be possible to design outcome evaluations of some HIAs that focus on changes in selected health determinants, such as physical activity or diet.

PREVENTING CHRONIC DISEASE
Fourth, HIAs are a valuable tool in a health-in-all-policies approach to incorporating health into policies in many sectors (61)(62)(63). Further work could document examples in which HIAs were used to advance this approach. Impact and outcome evaluations in settings in which health-in-all-policies approaches were implemented could be conducted.
Fifth, HIAs range from rapid desktop analyses to comprehensive studies that require substantial time and resources. Further work could lead to guidelines on the depth and breadth of HIAs needed for projects and policies in various contexts. For some HIAs, such as those for living wage policies in multiple cities (64)(65)(66), the findings and recommendations may be generalizable, thereby making further HIAs on the topic unnecessary.
Sixth, the importance of an implementation and monitoring plan in the recommendations of an HIA is often underemphasized by HIA practitioners. Further work could establish best practices to routinely create such plans in HIAs and facilitate follow-up of these plans.
Finally, some HIAs are conducted independently of, or in conjunction with, environmental impact assessments (3,40). Further work could clarify the best means with which to incorporate health issues into environmental impact assessment processes.
HIAs can directly influence decisions outside the health sector and frequently lead to other benefits that may have long-term health promotion value for communities. An aggregation of factors is necessary for HIA success. A central objective of HIAs is to "have health at the table" when decisions are being made; the impact evaluation reports reviewed suggest that this purpose generally is being fulfilled.   The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors' affiliated institutions. The opinions expressed by authors contributing to this journal do not necessarily reflect the opinions of the U.S. Department of Health and Human Services, the Public Health Service, the Centers for Disease Control and Prevention, or the authors' affiliated institutions.