Inter-rater reliability of assessed prenatal maternal occupational exposures to solvents, polycyclic aromatic hydrocarbons, and heavy metals.
In retrospective occupational exposure studies that use expert assessments, inter-rater reliability is often used as an indicator of validity when validity cannot be directly assessed. We evaluated inter-rater reliability of the exposure assessment of 7,729 jobs reported in the National Birth Defects Prevention Study. Jobs were classified as exposed, unexposed, or exposure unknown by two independent industrial hygienists; exposed jobs were further evaluated for intensity, frequency, and routes of exposure. Exposure prevalence ranged from less than 1 percent to 10 percent. Inter-rater reliability for exposure (yes/no), assessed by Kappa coefficients, was fair to good for cadmium (K = 0.46), chlorinated solvents (K = 0.59), cobalt (K = 0.54), glycol ethers (K = 0.50), nickel compounds (K = 0.65), oil mists (K = 0.63), and Stoddard Solvent (K = 0.55). Polycyclic aromatic hydrocarbons (K = 0.24) and elemental nickel (K = 0.37) had poor agreement. After a consensus conference resolved disagreements, and additional 4,962 jobs were evaluated. Inter-rater reliability improved or stayed the same for cadmium (K = 0.51), chlorinated solvents (K = 0.81), oil mists (x = 0.63), polycyclic aromatic hydrocarbons (K = 0.52), and Stoddard Solvent (x = 0.92) in the second job set. Inter-rater reliability varied by exposure agent and prevalence, demonstrating the importance of measuring reliability as a quality control step.