Breast Cancer
Analytic Validity

(95KB)

---

ANALYTIC VALIDITY

---

ANALYTIC VALIDITY

Question 9. How often is the test positive when a mutation is present (sensitivity)?
Question 10. How often is the test negative when a mutation is not present (specificity)?

Analytic sensitivity is the proportion of positive test results, when a detectable mutation is present (i.e., the test is designed to detect that specific mutation). The analytic sensitivity may also be called the analytic detection rate. Another way of expressing analytic sensitivity would be to divide the true positives by the sum of the true positives and false negatives. False negative results could be due to technical errors in the analytic phase (e.g., sample placement, contamination, expired reagents and cross-reactivity) or to administrative/clerical errors in the pre-analytic or post-analytic phases (e.g., incorrect interpretation of correct analytic result, sample mislabeling and incorrectly copying a correct result).

Analytic specificity is the proportion of negative test results when no detectable mutation is present. Another way of expressing analytic specificity would be to divide the true negatives by the sum of the true negatives and false positives. Analytic specificity can also be expressed in terms of the analytic false positive rate. The analytic false positive rate is the proportion of positive test results when no detectable mutations are present (1-analytic specificity). False positive results could be due to technical errors in the analytic phase (e.g., errors in placement, contamination, expired reagents, or non-specific reactions) or to administrative/clerical errors in the pre-analytic or post-analytic phases (e.g., mislabeling of samples, wrong interpretation of correct results, or copying results incorrectly).

Wrong mutations are a third type of error, along with false negative and false positive results. These occur when a mutation is present, but is incorrectly identified (e.g. base pairs are miscounted, resulting in an incorrect location of the mutation). For purposes of this review, wrong mutations will be considered false positive results, since there is an opportunity for correcting them by confirmatory testing. Wrong mutations occurring in any of the testing phases are included in the following analyses of analytic validity.

Gap in Knowledge: How should the finding of a wrong mutation influence computation of the analytic performance? The relationship between the third type of error (wrong mutation) and analytic performance has not yet been formally addressed. In this document, a wrong mutation will be considered an incorrect result, since this type of error could cause harm. For example, determination of breast and ovarian cancer risk could be affected by an incorrect mutation report. Also, family members would not receive correct information. Further, a wrong mutation finding will be treated as a false positive, since confirmatory testing of positive results will provide the opportunity to correct this type of error.

An Optimal Dataset for Analytic Validation

Few data sources exist for estimating analytic validity. Published reports of method comparisons and screening experiences provide limited information. The “true” genotypes of the tested samples are often undocumented (i.e., not confirmed by another methodology or laboratory consensus. Future analyses should be aimed at providing reliable, method-specific analytic performance estimates. One approach for collecting such data might include the following steps:

• An independent body (such as the College of American Pathologists, American College of Medical Genetics, Food and Drug Administration or the Coriell Institute of Medical Research (Camden, NJ)) would develop a standard set of samples, most of which would be randomly selected from the general population. Correct genotypes would be arrived at by consensus. Included in the standard set would be additional samples with known mutations or variants.

• The sample set would then be available for method validation. The current validation practice of having a laboratory (or manufacturer) run a series of samples with unknown genotype is inadequate, since there is no ‘gold standard' with which to compare. For example, how would a laboratory running an unknown sample determine whether a positive finding is a true, or a false, positive?

Appropriate sample size for determining analytic sensitivity and specificity has been discussed in detail in an earlier ACCE report (Prenatal Cystic Fibrosis Screening via Carrier Testing – Question 11 and 12). In brief, a target sensitivity (or specificity) can be chosen, along with an acceptable lower limit (assumed to be the lower limit of the 95% confidence interval). Given these targets, the number of necessary samples can be derived. For example, if a laboratory chose a target specificity of 98% and wanted to rule out a specificity of 90%, it would need to correctly identify at least 49 of 50 known negative samples (estimated using the binomial distribution). When the estimates approach 100% and relatively tight confidence intervals are sought, such as might be the case for BRCA1/2 mutation testing, a single laboratory would need to invest considerable effort. All of these suggested analyses could be done using a 2x2 table, and all rates could be accompanied by 95% confidence intervals (CI).

Analytic Performance of Sequencing Tests for BRCA1/2 Mutations

Myriad Genetic Laboratories Due to patent restrictions, full gene sequencing for clinical purposes can only be done in one laboratory in the United States. For this reason, data and methods used to calculate laboratory-specific analytic sensitivity and specificity reside only there. Publicly available data are limited to the “Technical Specifications” listed on Myriad's website (http://www.myriadtests.com/provider/doc/tech_specs_brac.pdf) A further public source of these data is a published study (Shattuck-Eidens et al., 1997) that states “sensitivity of the sequence analysis was at least 98 percent in validation studies using blinded analysis of known positive controls”. However, no data are provided to support this statement. According to Myriad's website:

• Analytic sensitivity of over 99 percent Myriad reports that “failure to detect a genetic variant or mutation in the analyzed DNA regions may result from errors in specimen handling and tracking, amplification and sequencing reactions, or computer-assisted analysis and data review. The rate of such errors is estimated from validation studies to be less than one percent (<1%)” According to Myriad, “In the first BRACAnalysis validation study, a total of 55 samples were used to determine analytic sensitivity. The genetic variations identified in these sample sets were previously characterized using one of the following three methods: allele specific oligonucleotide hybridization, linkage analysis or radioactive sequencing. Fifty-four of 55 samples with known mutations were analyzed with one false negative being reported. In conducting an investigation into the false negative, it was determined that there was an insufficient quantity of DNA for the sample. As a result of this finding, procedural changes were made in order to prevent recurrence of this failure mode. In particular, the volume of DNA in a microplate is now tracked via the LIMS and adjusted accordingly in order to account for evaporation and processing of the plate whenever a pre-defined DNA aliquot is removed. In addition, each specimen that tests positive for either a mutation or uncertain variant is reprocessed in order to confirm presence of the mutation or variant and during initial DNA extraction from whole blood, a second plate identical to the first is stored as a backup. In the event that the first DNA sample is consumed, the backup plate, that is identical to the first, is retrieved in order to resume processing. In practice, the current frequency of BRACAnalysis samples with insufficient quantities for DNA reprocessing was calculated to be 0.1 percent. Through the processes of well volume tracking and confirmation of all mutations and uncertain variants, we have minimized the potential impact of insufficient DNA quantities causing potential false negative results in BRACAnalysis.” From these data, we estimate the analytic sensitivity to be 98.2 percent (54/55) with a 95 percent CI from 90.3 to 99.9 percent.

• Analytic specificity of over 99 percent Myriad reports that “the incidence of a false report of a genetic variant or mutation resulting from technical error is considered negligible because of independent confirmation of all genetic variants. The incidence of a false report of a genetic variant or mutation resulting from errors in specimen handling and tracking is estimated from validation studies to be less than one percent (<1%)”. Myriad also reported that “the analytic specificity of BRACAnalysis was demonstrated to be 100% (46 of 46 samples with no known mutation were analyzed with no false positives reported).” From these data, we estimate the analytic specificity to be 100 percent (95 percent CI 92.3 to 100%).”

• BRACAnalysis Large Rearrangements Analytic Validity The BRACAnalysis Large Rearrangements was designed to detect five specific large rearrangements, either deletions or duplications, in the BRCA1 gene. Positive samples for this assay were obtained from a variety of researchers and clinicians. Internal samples obtained from Myriad Genetics, Inc. (the research section of Myriad) were also used.

• According to Myriad, “analytic validity of the assay was determined with a total of 27 samples, composed of 10 samples with known large rearrangements and 17 samples with no known large rearrangements. The analytic specificity for BRACAnalysis Large Rearrangements was determined to be 100.0% (17 of 17 samples with no known large rearrangements were analyzed with no false positives reported), while the analytic sensitivity was determined to be 100.0% too (10 of 10 samples with known large rearrangements were analyzed with no false negatives reported).” From these data we estimate the analytic sensitivity to be 100 percent (95% CI 69.2-100). The analytic specificity was calculated to be 100 percent (95% CI 80.5-100).

Gap in Knowledge: Are the data from which Myriad Genetic Laboratories estimates analytic sensitivity and analytic specificity sufficient? Myriad has provided limited data used to estimate the analytic performance of its sequencing technology. Estimates of analytic sensitivity and specificity for this laboratory cannot be considered robust. There is no appropriate external proficiency testing scheme available for blinded assessment of BRCA1/2 sequencing and it is unlikely that one will become available in the future. Testing is limited, in this instance, to one laboratory using a ‘home brew' technology. In this unusual situation, new creative approaches to establishing analytic performance estimates need to be developed. In the absence of these new approaches, it is unlikely that better estimates of analytic performance will be forthcoming.

Analytic Performance of Multiple Methodologies for BRCA1/2 Mutations

External Proficiency Testing in Europe: The European Molecular Genetics Quality Network (EMQN). The EMQN (http://www.emqn.org/emqn/) (last accessed 2/2008) was established in 1997 as an independent organization to provide External Quality Assessments (EQA) of molecular genetic tests. EMQN also promotes internal quality assurance by funding meetings to discuss “best practice” in disease and non-disease specific areas. The EQA schemes for the molecular diagnosis of familial breast/ovarian cancer gene mutations (BRCA1/2) were presented from 1999 to 2002 to assess the sensitivity of screening for unknown mutations in specified exons. Nineteen countries were represented through 2002. All laboratories used an automated DNA sequencing methodology to identify mutations for these challenges. However, a variety of screening methodologies was used to scan the specified exons. Table 2-1 shows the results of these studies, and Table 2-2 provides the calculation of the analytic sensitivity for the participating laboratories. The overall error rate for 1999 to 2002 is 2.7 percent (95 percent CI 1.6 to 4.2%). Incorrect responses included those that identified the sequence change, but either described the mutation with incorrect nomenclature or did not include a biological interpretation (i.e. the effect of the gene mutation on the protein function). No laboratory failed all challenges. Laboratories participating in these schemes include independent diagnostic facilities, parts of genetic/oncology centers, and research institutions.

Table 2-1. BRCA1/2 Mutation Testing: Results of the European Molecular Genetics Quality Network Survey

			Result		Type of Incorrect Result
Year	Number Of Labs	Alleles Tested	Correct N (%)	Incorrect N (%)	False Positive N (%)	False Negative N (%)	Wrong Mutation N (%)
1999*	14	80	78 (97.5)	2 (2.4)	1 (1.2)	1 (1.2)	0 (0.0)
2000	24	136	132 (97.1)	4 (2.9)	0 (0.0)	3 (2.2)	1 (0.7)
2001*	41	238	230 (96.6)	8 (3.4)	0 (0.0)	5 (2.1)	3 (1.3)
2002*	37	216	212 (98.1)	4 (1.9)	0 (0.0)	4 (1.9)	0 (0.0)
All	116	670	652 (97.3)	18 (2.7)	1 (0.15)	13 (1.9)	4 (0.6)

* Contained BRCA1 mutations only

Table 2-2. Analytic Sensitivity for Identifying BRCA1/2 Mutations According to Data from the European Molecular Genetics Quality Network Survey

Year	Analytic Sensitivity (%)	(95% CI)
1999	98.8	(93.2-99.9)
2000	97.1	(92.6-99.2)
2001	96.6	(93.5-98.5)
2002	98.2	(95.3-99.5)
All	97.5	(96.0-98.5)

In addition to genotyping, the EMQN scheme also attempted to score interpretation of results. It is EMQN's position that reports should contain all relevant data to make the report a ‘stand alone' source of information pertaining to the case in question. The criteria for scoring include:

• Are the patient's personal data (e.g., name, date of birth) clearly given?

• Has the clinical context been restated or has the clinical question been repeated?

• Are the results clearly presented?

• Has a clinical genetic interpretation of the results been given?

• With a negative result, have the limits of the applied test been mentioned?

• Have further options (for genetic testing and/or clinical management) been suggested?

The maximum interpretation score for each case is 2.0. The sum of the three cases is divided by three to compute the laboratory's score. The mean interpretation scores for 1999 through 2002 were 1.61, 1.38, 1.51, and 1.77. Frequent reasons for deducting points were:

• not mentioning further diagnostic options suitable to improve and/or complement the present test

• not mentioning the limits of the tests done

• incomplete interpretation of the consequences of the observed base pair change

• not mentioning that the result for a specific case increases the woman's cancer risk

Gap in Knowledge: EMQN: Analytic performance estimates are limited to sensitivity. While these data are not complete or robust, there appears to be no evidence of a problem in detecting a variety of BRCA1 and BRCA2 mutations, including variants of uncertain clinical significance, with any of the existing laboratory methodologies. Expansion of these challenges to include samples without mutations will help to provide estimates of analytic specificity.

Gap in Knowledge: EMQN: Analytic performance estimates are limited by the fact that laboratories are told which exons to examine. The DNA analysis of the BRCA1/2 genes is time-consuming and expensive. However, by instructing the participating laboratories to examine only one to three exons in these challenges, the EMQN falls short of assessing actual analytic performance in most clinical or research settings.

Analytic Performance of Single and Multi-site Tests in the United States

The ACMG/CAP Molecular Genetics Laboratory Survey Ten laboratories other than Myriad provide clinical DNA testing for the three common Ashkenazi Jewish BRCA1/2 mutations, as well as single-site testing for specific mutations known to exist in given families (www.genetests.org). The American College of Medical Genetics/College of American Pathologists (ACMG/CAP) Molecular Genetics Laboratory Survey External Proficiency Testing Program provides challenges for these laboratories. Few other data sources exist for estimating analytic validity in the United States. Published reports of method comparisons use direct sequencing as the “gold standard”, assuming that it has the highest accuracy. The ACMG/CAP Molecular Genetics Laboratory Survey provides a source of data that has several advantages, including: a large proportion of clinical testing laboratories that represent the range of methodologies presently being used and samples for distribution that have confirmed genotypes. However, basing analytic performance estimates on external proficiency testing also has drawbacks, including: