Expert decision making in exposure assessment.
Atlanta, GA: U.S. Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention, National Institute for Occupational Safety and Health, R01-OH-008513, 2009 Nov; :1-120
Accurate exposure assessments are critical for ensuring that potentially hazardous exposures are properly identified and controlled. The availability and accuracy of exposure assessments can determine whether resources are appropriately allocated to engineering and administrative controls, medical surveillance, personal protective equipment and other programs designed to protect workers. This project reports on a series of studies - desktop as well as in-plant - to evaluate the accuracy and potential bias of the accuracy of the exposure judgments made by practicing industrial hygienists. In the desktop studies, videos, task information and sampling data were provided to 77 industrial hygienists to evaluate the accuracy and potential bias of participants' exposure judgments. Desktop exposure judgments were obtained from occupational hygienists for material handling jobs with small air sampling data sets (0-8 samples) and without the aid of computers. In addition, data interpretation tests (DlTs) were administered to participants where they were asked to estimate the 95th percentile of an underlying log-normal exposure distribution from small data sets. Participants were presented with an exposure data interpretation or rule of thumb training which included a simple set of rules for estimating 95th percentiles for small data sets from a log-normal population. DIT was given to each participant before and after the rule of thumb training. Results of each DIT and qualitative and quantitative exposure judgments were compared with a reference judgment obtained through a Bayesian probabilistic analysis of the sampling data to investigate overall judgment accuracy and bias. There were a total of 4386 participant--task--chemical judgments for all data collections: 552 qualitative judgments made without sampling data and 3834 quantitative judgments with sampling data. The DlTs and quantitative judgments were significantly better than random chance and much improved by the rule of thumb training. In addition, the rule of thumb training reduced the amount of bias in the DlTs and quantitative judgments. The mean DlT % correct scores increased from 47 to 64% after the rule of thumb training (P < 0.001). The accuracy for quantitative desktop judgments increased from 43 to 63% correct after the rule of thumb training (P < 0.001). The rule of thumb training did not significantly impact accuracy for qualitative desktop judgments. The finding that even some simple statistical rules of thumb improve judgment accuracy significantly suggests that hygienists need to routinely use statistical tools while making exposure judgments using monitoring data. We characterized the impact of professional certifications, and characteristics relating to experience and education on exposure judgment accuracy. The judgments for each chemical on each different task were elicited from exposure assessment practitioners with varying levels of experience, education and related determinants. Logistic regression analysis indicated "years of exposure assessment experience" (p<0.05), "greater than 10 years of exposure assessment experience" (p<0.05), "highest EHS degree" (p<0.05) and a participant's "data interpretation test score" (p<0.05) directly impacted qualitative exposure judgment accuracy. Logistic regression models of quantitative judgment accuracy showed positive correlation with "greater than 10 years of exposure assessment experience" (p<0.05), "highest EHS degree" (p<0.05), a participant's "data interpretation test score" (p<0.001), rule-of-thumb data interpretation training (p<0.001), and the number of sample data points available for a judgment (p=0.004). Further analyzing participants with less than and greater than 10 years experience indicated additional correlations with Certified Industrial Hygienist and Certified Safety Professional certifications, total number of task exposure assessments, and career number of air surveys. The correlation of qualitative and quantitative exposure judgment accuracy with "greater than 10 years experience" supports similar research findings from other fields. The results of this study indicate that several determinants of experience, education and training in addition to the availability of sampling data significantly impacted the accuracy of exposure assessments. The findings also suggest methods for enhancing exposure judgment accuracy through statistical tools and specific training. We also conducted similar studies of exposure judgment accuracy of occupational hygienists across a wide range of industries. Participating companies provided monitoring information on specific tasks. A total of 49 hygienists from 6 companies participated in the study and 22 tasks were evaluated. The number of monitoring data points for individual tasks varied between 5 and 24. After reviewing all available basic characterization information on the job, task, and chemical, hygienists were asked to provide their judgment in probabilistic terms. 93 qualitative judgments (i.e., without reviewing monitoring data) and 2142 quantitative judgments (i.e., those made after reviewing monitoring data) were obtained. In addition to task judgments, hygienists' task experience and information on determinants such as educational background, statistical training and experience, years of industrial hygiene and exposure assessment experience, and measures of cognitive ability was also obtained using a questionnaire. Data interpretation training, with simple rules of thumb for estimating 95th percentiles was provided to all hygienists. A data interpretation test was administered before and after training. All exposure judgments were also collected before and after training. Hygienist data interpretation scores increased from 48% to 67% after training (p<0.001). Their quantitative judgment accuracy improved from 46% to 69% (p<0.001) post training. An improvement in agreement was also noticed. An agreement of more than 50%, between 2 hygienists, increased from 39% before training to 58% after training. The accuracy of qualitative judgments was not improved by the data interpretation training. The number of years hygienists had experience using the AIHA strategy and certifications held by hygienists were found to be significant determinants (p <0.0001) of judgment accuracy. Training (p<0.000 I), years of experience with a particular task (p<0.0001), certifications (p<.0001) and the number of job-tasks documented using the AIHA strategy (p<.0001) seemed to be significant determinants of task specific judgment accuracy. We also developed a framework for using exposure models in conjunction with two-dimensional Monte Carlo methods for making exposure judgments in the context of Bayesian decision analysis. A two-dimensional Monte Carlo scheme was developed by which the exposure model output can be represented in the form of a decision chart. Such a decision chart can be used as a "prior" in the Bayesian statistical framework, which can be updated using monitoring data to arrive at a final decision chart. Hypothetical examples using commonly used exposure models are presented, along with a discussion of how this framework can be used given a hierarchy of exposure models.
Exposure-assessment; Hazards; Engineering-controls; Medical-services; Surveillance-programs; Personal-protective-equipment; Personal-protection; Protective-measures; Industrial-hygienists; Humans; Men; Women; Materials-handling; Statistical-analysis
Dr. Gurumurthy Ramachandran, Professor, Division of Environmental Health Sciences, School of Public Health, University of Minnesota, MMC 807, 420 Deleware Street SE, Minneapolis, MN 55455
Final Grant Report
National Institute for Occupational Safety and Health
University of Minnesota, Twin Cities