I. Potential Use of Foodborne Illness Surveillance Data to Evaluate FSMA

Background on FSMA Implementation

FSMA is the most sweeping change in FDA food safety authority in more than 70 years. The main themes of FSMA include prevention, enhanced partnerships, import safety, inspections, compliance, and response.

FSMA rules and implementation approach

FDA is implementing FSMA in three phases. Phase 1 consisted of developing regulations, guidance, and policy. Phase 2 includes designing strategies to promote and oversee industry compliance, and identifying performance metrics to measure success. Phase 3 involves transitioning industry strategies and metrics from design to operations and then evaluating success. In addition, as part of the planning process, FDA established multiple workgroups with operations and policy experts working together to develop a framework and multi-year implementation plan for ensuring compliance with FSMA regulations.

During FSMA Phase 1 implementation, FDA issued seven proposed rules, which are now all final and published (Table 1).

Table 1. FDA Final Rules

The framework and multi-year implementation plans that FDA developed for each of the rules encompass outreach, technical assistance, and training as well as data collection and analysis, performance goals and measures, inspections, compliance, and enforcement. FDA has placed an emphasis on (1) development of training for firms and regulators; (2) establishment of technical assistance networks (TAN) to provide central, consistent sources of outreach and technical assistance for industry and regulators; and (3) targeted education and outreach for firms.

Industry experience with FSMA preventive controls

The most significant change to industry resulting from the FSMA Preventive Control (PC) regulations has been improved documentation and improved or updated training, which have had a greater impact on medium and small companies than on large companies. Fewer changes were necessary for companies that prioritized hazard management prior to FSMA. Large companies have tended to have more robust food safety practices and environmental monitoring in place and thus had to make minimal changes to their food safety programs and testing in response to FSMA. However, many medium and small companies needed to make significant changes, and as such were given more time to implement the rules.

Since FSMA implementation, FDA inspections have become more collaborative; however, there are some emerging concerns on the part of industry. For example, taking photographs without permission has been a recent concern, but smart phones have made cameras ubiquitous. Additionally, some FDA inspectors have interpreted guidance as a requirement rather than a recommendation (e.g., Appendix 1 of Hazard Analysis and Risk-Based Preventive Controls for Human Foodexternal icon). Industry expects to learn more since it is still early in FSMA implementation.

Overview of FDA’s current thinking approach

As the FSMA rules were being finalized and published, FDA’s focus shifted to planning for implementation and measuring progress. FDA is using a results-oriented approach to develop measures to demonstrate achievement of outcomes. The approach first defines long-term outcomes (e.g., what do we want to achieve in 5 years) and identifies activities to achieve these outcomes. Then, risk-informed, direct, and practical performance measures are determined to measure and monitor if outcomes were achieved. As FSMA implementation continues, these outcomes and activities will be assessed for continued appropriateness and relevance. Outcomes for PC, Produce Safety, and Import Controls have been drafted, but are being further evaluated for feasibility and data limitations. In addition, activities and performance measures for each of these outcomes are still being drafted.

Measuring Public Health Impact of FSMA on Foodborne Illness

The Pew Charitable Trusts—metrics

Pew and the Robert Wood Johnson Foundation are co-sponsors of the Collaborative Food Safety Forumexternal icon. This workgroup has identified the following challenges to the FSMA implementation metrics:

1. Many factors impact public health outcomes.
2. Short- , medium- , and long-term measures are needed.
3. Not all metrics may be relevant across food types.
4. There are multiple relevant metrics: inputs, processes, outputs, and outcomes.
5. Data are imperfect, limited, and subject to reporting lags.

FSMA implementation should be evaluated using a web-metrics approach that includes public health metrics, industry performance metrics, FDA performance metrics, and industry segment–specific metrics. Public health metrics should center on reducing illnesses linked to FDA-regulated foods. Industry performance metrics should have a prevention-based focus, demonstrating a strong food safety culture. FDA performance metrics should directly track FSMA implementation, while industry metrics should be specific to a particular rule (e.g., produce, preventive controls, and the FSVP).

Review of food categorization schemes

Classifying foods implicated in outbreaks can allow public health and food industry professionals to target prevention efforts for pathogens and commodities. CDC has used the Painter commodity categorization scheme to categorize foods implicated in foodborne disease outbreaks since 2009. IFSAC, established in 2011, coordinates efforts by FDA, the U.S. Department of Agriculture (USDA)/Food Safety and Inspection Service (FSIS), and CDC to generate estimates of foodborne illness source attribution and inform food safety policy. One of IFSAC’s latest projects included expanding the food categorization scheme used by CDC to include more specific and useful food categories for regulatory agencies and stakeholders. To update the existing scheme, subject matter experts were consulted within CDC, FDA, and USDA/FSIS to identify new commodity categories, discuss proposed revisions, and provide expertise regarding the specific foods to be assigned to each category to ensure the new categorization scheme meets regulatory agency needs. A comprehensive food glossary was also developed to accompany the proposed commodity categorization scheme to illustrate the placement of specific foods into each commodity category. This effort resulted in the IFSAC Food Categorization Scheme, which consists of five hierarchical levels: (1) overarching food group; (2) major food type; (3) sub-specific food type; (4) food subtype varieties; and (5) specific processing, preparation, or consumption type. The IFSAC food categories reflect many of the food product definitions used by FDA and FSIS. The IFSAC Food Categorization Scheme may help to improve efforts for measuring the public health impact of FSMA; however, a high number of foods are unclassifiable (e.g., complex foods), and the unavailability of data at the lower levels may make classification of outbreaks into FSMA-regulated versus not FSMA-regulated groups challenging.

Considerations about data-driven approaches to measuring the public health impact of FSMA

FDA early approaches

Current surveillance systems such as FoodNet provide excellent data on the incidence of disease for several key foodborne pathogens; however, when evaluating FSMA, it is beneficial to look at illnesses that can be directly traced to FDA-regulated foods that are covered by the different FSMA rules. Outbreak data can provide this information on an ongoing basis for the entire country. However, outbreak data are limited in that they represent only a small fraction of overall foodborne illness, have surveillance bias such as analysis done on non-representative data, and have regional/year-to-year variability in how they are reported. In addition, in determining what outbreak data to use in analyses, considerations include

• Multistate vs. single-state outbreaks
• Selection of pathogens
• Aligning food categorization scheme and rules
• Point-of-contamination information
• Single vs. complex food vehicles

A number of approaches have been used to measure the public health impacts of food safety regulations, including (1) counting the number of outbreaks or number of outbreak illnesses, (2) estimating the overall disease burden in the population (based on outbreak data), and (3) statistical analysis to test a hypothesis of whether illnesses have been reduced after an intervention. Regardless of the approach, it is important to consider public health measures in the context of other FSMA measures and alternate hypotheses for apparent trends in outbreaks or outbreak illnesses. Continued CDC–FDA collaboration is needed to effectively evaluate the public health impacts of FSMA.

Trends over time (leveraging an IFSAC method)

Previous IFSAC work conducted to evaluate temporal trends in foodborne outbreaks and illnesses laid the foundation to focus on a simple Bayesian model for the current project. This project evaluates modeling techniques to produce estimates of foodborne outbreak and illness trends at the pathogen and food category level, and assess changes over time in foodborne illness outbreaks and illness for the four IFSAC priority pathogens. The new model developed in this current project (1) uses untransformed count data, (2) includes flexibility to capture real changes while remaining resistant to the influence of outliers (thin plate splines), (3) accounts for uncertainty and allows for evaluation of short- and long-term trends, and (4) can be used for similar scenarios or other pathogens of interest. One of the limitations is the granular ability to model specific foods and pathogens. Future directions may include incorporating epidemiological factors from the IFSAC P13 project, point of contamination, or other count data.

CDC early approaches

CDC’s Enteric Diseases Epidemiology Branch has performed preliminary analysis using the Foodborne Disease Outbreak Surveillance System database focusing on produce and to some extent processed foods. The analysis involved selecting pathogens and foods, using multistate outbreak data, counts of outbreaks, and number of illnesses per outbreak. Additionally, produce items were not included if they were not covered under the Produce Safety Rule, and outbreaks that could be attributed to an ill food worker were not included. Consistent with the ill food worker approach, outbreaks of norovirus were also not included. Determining which IFSAC food categories to include in the analysis for processed foods was more challenging than for produce. Ensuring data are specific to the regulation, having adequate data to monitor change over time, and the approach to incomplete and unknown data are among the many factors to contemplate. A multi-disciplinary effort and continued CDC–FDA collaboration are needed.

Discussion/Guidance

Discussion

The Working Group’s discussion included the following observations:

• Assessing FSMA Implementation
  • Outcome measures using human illness are a high bar to prove a relationship to the FSMA rules.
  • Ideally, data from some level of ongoing environmental and product testing would be available to compare with a baseline.
  • It is difficult to use illness data. Thus, measuring outbreak data and how it affects implementation of the rules may be the best option from a human disease aspect. Several members thought outbreak data were important even with their limitations to use as a metric.
  • WGS will be a “game changer.” Illness rates and outbreaks should be looked at regionally, and inspections and sampling for specific foods and pathogens should be targeted where indicated by the outbreak data.
• Consider industry’s potential contribution to measuring FSMA
  • From an industry perspective, since large companies did not need to make many changes to become FSMA compliant, they may not be the best place to look for impact. Small and very small businesses have not been impacted yet, since they have more time left to adopt the rules.
  • Further investigation is needed to understand whether the outbreaks and illnesses are associated with larger or smaller food companies. However, there is a large gap between large and small companies and the resources they have access to.
  • Industry has the capacity to reduce the level of pathogens; sharing what industry is finding and looking for concerning reductions in levels over time may be useful.
  • Industry could measure the number of events that happen before a product is distributed. These “close calls” are currently not recorded because they are not reported to FDA.
  • Mapping supply chains, supplier sources, and prevalence of pathogens may be helpful. Supply chain transparency can help with traceback and traceforward investigations that may lead to the source of an outbreak. Industry is working on this effort with technology like Block Chain.
  • Industry may have critical information that could be used to fill knowledge gaps during outbreak investigations. For instance, one company may not purchase product from a particular vendor because they have positive samples, but another company may buy that positive product.
  • Industry testing data alone may not be the best source to measure impact. There is a misconception that industry has a lot of testing data that they are choosing not to share with public health. Also, industry may be resistant to share the positive results they have because they do not always lead to a public health impact, but can lead to recalls and litigation. Litigation can tie the hands of the companies that want to do the right thing.
    • Government protections similar to those used for the vaccine industry may be worth considering.
  • Public health should work with industry to develop better measures for FSMA success. This may include identifying successful prevention of hazardous products from entering the marketplace and the number of employees per FDA-regulated facility trained in PC and having qualified individuals (and similar measures for Produce Safety and FSVP).
  • Using outbreak data is more important in determining how industry conducts risk assessment/analysis than how impactful the rule is. However, outbreaks are just a small portion of the overall public health implication.
• Examine the effect of other epidemiological factors on measuring the impact of FSMA
  • Further analysis of data from sporadic cases, such as determining the genotype of cases and attributable fraction related to an outbreak vehicle, could be useful. Additionally, analyses that use the population who seeks healthcare as a denominator may provide additional context and greater understanding of cases.
  • More emphasis is needed on the amount of a pathogen in a product rather than a dichotomous positive/negative. The impact of FSMA may be demonstrated by measuring lower levels of pathogen rather than lower proportion of positive samples.
  • Reports by industry under the Reportable Food Registry are a good data source for measuring FSMA impacts.
  • Attribution data could be used to measure the impact of FSMA, especially if it could be matched with food industries that implemented more standards.
  • It is difficult to measure changes that are the result of FSMA because there are other factors, such as the use of CIDTs and WGS, that are potentially increasing the number of detected illnesses and outbreaks. Additionally, not all foods and not all pathogens are relevant to FSMA, and changes in these areas cannot be attributed to the increased regulation.
  • Baseline testing of foods of high risk might be available from industry. Understanding the impact of changes in production processes on outbreaks is essential in quantifying the impact of FSMA. For example, documenting production practices that changed and resulted in less contamination may be useful.
  • Regulatory agencies should focus on eliminating risk factors. Additionally, looking for critical sources, targeting certain pathogens, and addressing them legislatively would have more long-term impact on public health. For example, using policy/legislation to remove sprouts and raw milk from the marketplace would help quantify the impact of FSMA.
  • Understanding why the outbreak happens and identifying contributing factors and whether or not these factors are reduced over time may be a way of identifying the impact of FSMA using outbreak data. For example, with FSMA there should be improved training for staff, which should lead to a reduction of outbreaks being caused by a lack of training. Another approach would be to assess the timeline for detection and response, or evaluate two or three food-pathogen combinations to see what factors may have changed over time.
  • Although the number of outbreaks (or the ones linked to a source) are small, a decline in those numbers would be meaningful and easily understood by the public. CDC should quantify the impact of better detection methods to help explain to the public why there are increases in cases.
  • More objective metrics are needed; outbreak data are only a small portion of foodborne illness data. USDA/FSIS has measured some objectives using baseline testing data. FDA should also develop prevalence data for pathogens in specific foods. Some FDA metrics may be flawed due to variability in practices among the inspectors themselves.
  • The number of recalls should be monitored to see if there is a reduction following regulation implementation. Additionally, the proportion of FDA recalls that are due to pathogens compared with all recalls (i.e., pathogen, foreign object, and allergen recalls) could be monitored to observe change.
  • The number of product positives could be compared with the number of human positives to see if there is a correlation.
• Increase capacity in states
  • It is not just important to identify more outbreaks. Public health agencies should also collect better data during outbreaks, including contributing factors and environmental antecedents. Part of the challenge is the large number of unsolved outbreaks and outbreaks not attributed to a source (or having contributing factors). CDC should continue state capacity-building efforts (e.g., the work of CoEs, FoodCORE [Foodborne Diseases Centers for Outbreak Response Enhancement], OutbreakNet Enhanced [OBNE]) to enhance outbreak detection and response activities at the state/local level, including providing training and funding to states to collect data that can be attributed to FSMA.
  • It is important to understand barriers at the state/local agencies that could be affecting their ability to solve outbreaks. Other issues that may be helpful to investigate include (1) what is the state and local capacity?; (2) how many epidemiologists per million people would be ideal?; and (3) is there a regional breakdown of where a source is not being identified, and is that related to a capacity issue?

Guidance

Based on these discussions, the Working Group highlighted the following recommendations:

• Improve the quality of sporadic illness as well as outbreak data
• Understand the effect of other factors (e.g., CIDT, WGS, investigative methods, better detection) on epidemiologic changes
• Engage industry and its wealth of information (e.g., supply chain and supplier source information, prevalence of pathogens at baseline and following interventions using best practices, “near misses” that avoided release of contaminated product) to measure the effects of FSMA rule implementation
• Increase capacity in states (not just funding, but informatics, training, surge capacity, etc.)
• Improve sharing of data between public health agencies, regulators, and industry
• Continue to improve collaborations and partnerships between public health agencies, regulators, industry, consumers, and academia