Appendix A: Validation
Findings from Validation Visits for 2010 ART Data
Site visits to assisted reproductive technology (ART) clinics for validation of 2010 ART data were conducted during April through June 2012. For validation of 2010 data, 35 of the 443 reporting clinics were randomly selected after taking into consideration the number of ART procedures performed, the number of canceled cycles, some cycle and clinic characteristics, and whether the clinic had been selected before. During each validation visit, ART data reported by the clinic to the Centers for Disease Control and Prevention were compared with information documented in medical records.
For each clinic, the fully validated sample included up to 40 ART cycles resulting in pregnancy and up to 20 ART cycles not resulting in pregnancy. In total, 2,070 ART cycles performed in 2010 across the 35 clinics were randomly selected for full validation, along with 135 embryo banking cycles. The full validation included review of 1,352 cycles for which a pregnancy was reported, of which 446 were multiple-fetus pregnancies. In addition, among patients whose cycles were validated, we verified the number of ART cycles performed during 2010. For each of these patients, we compared the total number of ART cycles reported with the total number of ART cycles included in the medical record. If unreported cycles were identified in selected medical records, up to 10 of these cycles were also selected for partial validation.
Discrepancy rates are listed below for validated items of interest. Overall, validation of 2010 ART cycle data indicated that discrepancy rates were low (<5.0%) except for “Diagnosis of infertility.” This field corresponds to “Patient Diagnosis” data in the National Summary table and the 2010 individual clinic tables in the 2010 Assisted Reproductive Technology Fertility Clinic Success Rates Report.
Discrepancy Rates by Data Fields Selected for Validation
|Data Field Name||Discrepancy Rate* (Confidence Interval†)||Comments|
|Patient date of birth||2.0%(1.0–3.0)||In about 55% of the discrepancies, the difference did not result in changing age category (Age of Woman).|
|Diagnosis of infertility||17.9%(13.3–22.5)||For approximately 20% of the discrepancies, a single wrong diagnosis was reported, mainly “Other” or “Unexplained,” instead of a specific cause. For another 50% of the discrepancies, multiple causes of infertility were found in the medical record, but only a single cause was reported.|
|Number of embryos/ oocytes transferred||<1%|
|Outcome of ART treatment (i.e., pregnant vs. not pregnant)||<1%|
|Number of fetal hearts on ultrasound||2.2%(1.2–3.2)||Of the discrepancies, 15% were misreported as single-fetus pregnancies instead of multiple-fetus pregnancies, whereas 20% of the discrepancies were misreported as having one or more fetal hearts when the medical records actually showed zero (0) fetal hearts.|
|Pregnancy outcome (i.e., miscarriage, stillbirth, and live birth)||1.4%(0.7–2.2)||For about 70% of the discrepancies, there was no information on pregnancy outcome in the medical records.|
|Date of pregnancy outcome||4.1%(2.8–5.3)||For about 35% of the discrepancies, there was no information on pregnancy outcome date in the medical records. For another one–third of the discrepancies, the date in the medical records was within 7 days of the reported date.|
|Number of infants born||<1%|
|Additional cycles in same reporting year||2.6%(1.1–4.1)||For approximately 75% of the discrepancies, fewer additional cycles were reported by clinics than were found in the medical records. The majority of the discrepancies were due to reporting one less cycle. A further analysis of the non-reported cycles revealed that 40% were canceled cycles and an overwhelming majority (> 95%) did not result in a live birth (success).|
Note: ART = assisted reproductive technology.
* Discrepancy rates estimate the proportion of all treatment cycles with differences for a particular data item. The discrepancy-rate calculations weight the data from validated cycles to reflect the overall number of cycles performed at each clinic. Thus, findings from larger clinical practices were weighted more heavily than those from smaller practices.
† This table shows a range, called the 95% confidence interval, that conveys the reliability of the discr epancy rate. For more information, see explanation of confidence intervals.
What is a confidence interval?
Simply speaking, confidence intervals are a useful way to consider margin of error, a statistic often used in voter polls to indicate the range within which a value is likely to be correct (e.g., 30% of the voters favor a particular candidate with a margin of error of plus or minus 3.5%). Similarly, in this report, confidence intervals are presented to provide a discrepancy rate range that we can be confident is an estimate of the proportion of all ART cycles, performed in a given reporting year, with differences for a particular data item.
Why do we need to consider confidence intervals if we already know the exact discrepancy rates for each clinic in 2010?
No discrepancy rate or statistic is absolute. Suppose that during validation, 100 cycles were reviewed, and a discrepancy rate of 15% was determined for a particular data item with a confidence interval of 10%–20%. The 15% discrepancy rate tells us that the average chance that a discrepancy occurred for the selected data field among all reported cycles was 15%. But because only a certain percentage of ART cycles were reviewed during the validation visits at a select number of clinics, how likely is it that this would be the discrepancy rate if we repeated validation? For example, if another 100 cycles were reviewed using similar validation parameters, would the discrepancy rate again be 15%? The confidence interval tells us that the discrepancy rate would likely fall between 10% and 20%.