Appendix A: Validation
Site visits to assisted reproductive technology (ART) clinics for validation of 2011 ART data were conducted during April through June 2013. For validation of 2011 data, 35 of the 451 reporting clinics were randomly selected after taking into consideration the number of ART procedures performed at each clinic, some cycle and clinic characteristics, and whether the clinic had been selected before. During each validation visit, ART data reported by the clinic to the Centers for Disease Control and Prevention were compared with information documented in medical records.
For each clinic, the fully validated sample included up to 40 ART cycles resulting in pregnancy and up to 20 ART cycles not resulting in pregnancy. In total, 2,059 ART cycles performed in 2011 across the 35 clinics were randomly selected for full validation, along with 191 embryo banking cycles. The full validation included review of 1,324 cycles for which a pregnancy was reported, of which 393 were multiple-fetus pregnancies. In addition, among patients whose cycles were validated, we verified the number of ART cycles performed during 2011. For each of these patients, we compared the total number of ART cycles reported with the total number of ART cycles included in the medical record. If unreported cycles were identified in selected medical records, up to 10 of these cycles were also selected for partial validation.
Discrepancy rates are listed on the next pages for validated items of interest. Overall, validation of 2011 ART cycle data indicated that most discrepancy rates were low (<5.0%).
|Data Field Name
|Patient date of birth||1.2% (0.6–1.8)||For approximately half the discrepancies, the difference did not result in changing the age category (Age of Woman).|
|Cycle intention||1.2% (0.4–1.9)||For almost all the discrepancies, an ART procedure cycle was reported as an embryo banking cycle.|
|Number of embryos/ oocytes transferred||1.2% (0.6–1.9)||For more than half of the discrepancies, the reported number of oocytes/embryos transferred was off by 1.|
|Outcome of ART treatment (i.e., pregnant vs. not pregnant)||1.4% (0.6–2.2)||The ART treatment outcome could not be found in the medical records for approximately 15% of the discrepancies.|
|Number of fetal hearts on ultrasound||2.2% (1.3–3.1)||Of the discrepancies, 25% were misreported as single-fetus pregnancies instead of multiple-fetus pregnancies, whereas 15% of the discrepancies were misreported as one or more fetal hearts when the medical records actually showed zero (0) fetal hearts.|
|Pregnancy outcome (e.g., miscarriage, live birth, and stillbirth)||1.2% (0.6–1.8)||About 40% of the discrepancies were misreported as live birth when there was no information on pregnancy outcome in the medical records to confirm the birth.|
|Date of pregnancy outcome||4.1% (2.8–5.4)||For about 40% of the discrepancies, there was no information on pregnancy outcome date in the medical records. For another 20% of the discrepancies, the date in the medical records was within 7 days of the reported date.|
|Number of infants born||<1%|
|Cycle count||2.9% (1.2–4.6)||For approximately two out of three discrepancies, fewer cycles were reported by clinics than were found in the medical records. The majority of the discrepancies were due to reporting one less cycle. A further analysis of the unreported cycles revealed that approximately 50% were canceled cycles and an overwhelming majority (>95%) did not result in a live birth (i.e., success).|
|Data Field Name
|Male factor||3.7% (2.1–5.2)||The following reasons for ART were under-reported: male factor.|
|Endometriosis||2.6% (1.3–3.9)||The following reasons for ART were under-reported: endometriosis.|
|Tubal factor||2.9% (1.6–4.2)||The following reasons for ART were under-reported: tubal factor.|
|Ovulatory dysfunction||2.9% (1.7–4.1)||The following reasons for ART were under-reported: ovulatory dysfunction.|
|Diminished ovarian reserve||8.4% (5.3–11.4)||The following reasons for ART were under-reported: diminished ovarian reserve.|
|Uterine factor||2.1% (1.3–2.9)||The following reasons for ART were under-reported: uterine factor.|
|Other factor||9.2% (5.4–13.1)||The following reasons for ART were over-reported: Other factor.|
|Unknown factor||6.5% (3.2–9.8)||The following reasons for ART were over-reported: unknown factor.|
Note: ART = assisted reproductive technology.
* Discrepancy rates estimate the proportion of all ART cycles with differences for a particular data item. The discrepancy-rate calculations weight the data from validated cycles to reflect the overall number of cycles performed at each clinic. Thus, findings from larger clinical practices were weighted more heavily than those from smaller practices.
† This table shows a range, called the 95% confidence interval, that conveys the reliability of the discrepancy rate.
What is a confidence interval?
Simply speaking, confidence intervals are a useful way to consider margin of error, a statistic often used in voter polls to indicate the range within which a value is likely to be correct (e.g., 30% of the voters favor a particular candidate with a margin of error of plus or minus 3.5%). Similarly, in this report, confidence intervals are presented to provide a discrepancy rate range that we can be confident is an estimate of the proportion of all ART cycles, performed in a given reporting year, with differences for a particular data item.
Why do we need to consider confidence intervals if we already know the exact discrepancy rates for each clinic?
No discrepancy rate or statistic is absolute. Suppose that during validation, 100 cycles were reviewed, and a discrepancy rate of 15% was determined for a particular data item with a confidence interval of 10%–20%. The 15% discrepancy rate tells us that the average chance that a discrepancy occurred for the selected data field among all reported cycles was 15%. But because only a certain percentage of ART cycles were reviewed during the validation visits at a select number of clinics, how likely is it that this would be the discrepancy rate if we repeated validation? For example, if another 100 cycles were reviewed using similar validation parameters, would the discrepancy rate again be 15%? The confidence interval tells us that the discrepancy rate would likely fall between 10% and 20%.