Draft Genetic Test Review
“This information is distributed solely for the purpose of pre dissemination peer review under applicable information quality guidelines. It has not been formally disseminated by the Centers for Disease Control and Prevention. It does not represent and should not be construed to represent any agency determination or policy.”
Breast Cancer
Analytic Validity
(95KB)
ANALYTIC VALIDITY
page 1 | page 2 | page 3 | page 4
ANALYTIC VALIDITY
Question 11. Is an internal quality control (QC) program defined and externally monitored?
Summary
- Internal quality control procedures are well described in governmental and professional published standards and
guidelines
- External monitoring is provided through inspections conducted by accrediting organizations such as the Clinical
Laboratory Improvement Amendments (CLIA), the College of American Pathologists (CAP) and New York State Myriad Genetic Laboratories is certified by both CLIA and New York State
|
Definition
Internal quality control is a set of laboratory procedures designed to ensure that the test method is working properly. An internal quality control program includes documentation that high standards are being practiced to ensure that:
-
reagents used in all aspects of genetic testing are of high quality to allow successful test completion,
-
all equipment is properly calibrated and maintained,
-
good laboratory practices are being applied at every level of genetic testing. To the extent possible, all steps of the testing process must be controlled.
Quality control procedures
Techniques that are used for analyzing DNA in testing for predisposition to breast/ovarian cancer are the same as those used for other molecular testing. These techniques are widely applied and well understood. As a result, it has been possible to design and publish generic internal quality control procedures, which many molecular laboratories already have in place. Table 2-9 lists published guidelines that, among other topics, describe reagent quality control, equipment calibration and maintenance, education of the technical staff, and other internal quality control procedures. The purpose of the quality control procedures is to rigorously control all steps of the DNA testing process to minimize the potential for test failure. Given that the internal procedures for establishing and maintaining good laboratory practice are readily available (Neumaier et al., 1998)
, the important next step will be to encourage, assist, and require laboratories to apply and document appropriate quality control procedures.
Table 2-9. Guidelines, Recommendations, and Checklists that Address Internal Quality Control Issues and Requirements.
| Clinical Laboratory Improvement Amendments of 1988 |
Federal Register 1992;57:7002-3 |
| Genetic Testing Under CLIA |
Federal Register 2000;65: 25928-24934 |
| New York State Laboratory Standards (9/00) |
www.wadsworth.org/labcert/download.htm |
Molecular Diagnostic Methods for Genetic Diseases:
Approved Guidelines |
National Committee for Clinical Laboratory
Standards MM1-A Vol 20 #7 |
| College of American Pathologists Checklist |
www.cap.org |
| Standards and Guidelines for Clinical Genetics Testing |
American College of Medical Genetics
www.faseb.org/genetics/acmg/stds |
External monitoring
All clinical laboratories performing genetic testing must comply with general regulations under the Clinical Laboratory Improvement Amendments (CLIA), and a CLIA certification should be considered the minimum acceptable level of external monitoring. One shortcoming of having only a CLIA certification is that CLIA inspectors often have less experience in evaluating genetic testing laboratories than other certifying organizations. CLIA is in the process of upgrading its regulations regarding genetic testing. The Task Force on Genetic Testing concluded that the current CLIA requirements are insufficient to ensure quality of molecular genetic testing. Laboratories certified by the College of American Pathologists (CAP) or by New York State Health Department will have undergone a more rigorous external monitoring that requires specific procedures and documentation. Myriad Genetic Laboratories is currently CLIA-certified and is licensed by New York State.
ANALYTIC VALIDITY
Question 12. Have repeated measurements been made on specimens?
Summary
- Having information about repeated measurements on the same specimen is important for determining the type and rate of errors in BRCA1/2 mutation testing
- External proficiency testing programs in the U.S. provide limited data for repeated measurements on the same specimen by multiple laboratories.
- To date, all participating laboratories (including Myriad Genetic Laboratories) have agreed on the mutation status of all challenges
- All clinical laboratories test control samples repeatedly, but results are not usually reported
Myriad Genetic Laboratories has performed internal method comparisons showing a high degree of agreement between gel-based and capillary-based sequencing |
Measurements made on the same specimen in different laboratories
Multiple laboratories have made repeated measurements on the same specimen, utilizing a variety of technologies. A collaborative external proficiency testing program, jointly administered by the American College of Medical Genetics and the College of American Pathologists (ACMG/CAP) provided three breast/ovarian cancer predisposition DNA challenges in 2001 and another three in 2002. A summary report of the results was also provided (Questions 9 and 10 give more detail). In the first two distributions of BRCA1/2 mutations, 28 of 28 participating laboratories (100%, 95 percent CI 93.0-100%) correctly reported the results for all three challenges. This survey was limited to the three predominant Ashkenazi Jewish BRCA1/2 mutations, and only one of the three samples in each year contained a mutation. The European Molecular Genetics Quality Network (EMQN) schemes for the molecular diagnosis of familial breast/ovarian cancer gene mutations (BRCA1/2) were presented from 1999 to 2002 (full data are not yet available from 2002) to assess the sensitivity of screening for unknown mutations in specified exons. Forty-one laboratories from eighteen countries were represented through 2001. All laboratories used an automated DNA sequencing methodology to identify mutations for these challenges. However, a variety of screening methodologies was used to scan the specified exons. The overall error rate for 1999 to 2002 is 2.7 percent (95 percent CI 1.6 to 4.2%). The overall sensitivity is 97.1 percent (95% CI 95.2-98.5 percent - Questions 9 and 10).
Measurements made repeatedly on the same sample within a laboratory
It is common practice for repeated measurements to be made on the same specimen (a control specimen) within a laboratory. For each assay, a positive control is usually included for testing. This internal documentation will remain within the laboratory but will be available for on-site inspections by certifying agencies as part of external monitoring. Thus, one avenue for collection of these data would be to use laboratory survey instruments. For DNA sequencing at Myriad Genetic Laboratories, typical controls (positive or negative) are not run. Clinical Laboratory Improvement Amendments (CLIA) has approved a quality control protocol that involves the correct sequencing from each batch of reagents, independent analysis of all deleterious and uncertain mutations, assurance of no contamination, and control of the majority of the sequence against 15 other specimens (16 specimens per run - Ward B, personal communication).
Myriad Genetic Laboratories has described an annual internal proficiency testing program, where 12 samples containing a wide range of known deleterious BRCA1/2 mutations are sent to clients who then blindly resubmit them (Ward B, personal communication). Since January 1997, a total of 144 such internal proficiency tests have been conducted for BRACAnalysis. The proficiency tests were conducted using a set of DNA samples with known mutations, and no sample was sent more than once. These samples represented more than 750 recurring genetic variations in the BRCA1/2 genes. To date, all 144 proficiency tests have identified the same genetic variants as the initial test. In addition, whenever there is a major change to its automated sequencing methodology, Myriad performs a revalidation using 80 samples with known deleterious BRCA1/2 mutations. This has occurred approximately once a year since 1998. Data from these exercises are not available for our analysis.
According to Myriad, “A number of internal validation studies have been conducted in order to ascertain the analytic validity of BRACAnalysis. Most recently, a large scale study comparing data between gel-based and capillary-based sequencers was conducted with internal samples. Samples were first characterized using gel-based sequencing in order to identify genetic variations in the BRCA1/2 genes. Following gel-based sequencing, the same sample set was analyzed using capillary-based sequencing.” Among the 128 samples with a mutation identified by the gel-based methodology, the capillary-based methodology agreed in all instances. Among the 910 samples without a known mutation, the two methods also agreed in all instances.
ANALYTIC VALIDITY
Question 13. What is the within- and between-laboratory precision?
This question is not applicable to testing for predisposition to breast/ovarian cancer, since such testing is qualitative. This question is only relevant to quantitative measurements such as repeat sizing.
ANALYTIC VALIDITY
Question 14. If appropriate, how is confirmatory testing performed to resolve false positive results in a timely manner?
Summary
- Confirmatory testing is additional testing to verify the finding of a mutation(s)
- It is likely to be useful because of occasional false positive test results
- There is little information about how often confirmatory testing corrects an error
|
Definitions
Confirmatory testing is performed to ensure that the initially positive test result is correct. Examples include:
-
a BRCA1/2 mutation, either known deleterious or variant of uncertain significance, is identified in an individual. The specimen is then re-run to ensure that the result is correct.
-
a methodology other than sequencing (e.g. protein truncation test) suggests a mutation. Sequencing is then used to identify and describe the mutation.
Four distinct types of confirmatory testing could be utilized, depending on the testing protocols in place and the circumstances in which the positive test result is obtained.
-
Repeating the same test protocol on another aliquot of the same specimen
-
Repeating the same test protocol on a different specimen
-
Performing a different test protocol on another aliquot of the same specimen
-
Performing a different test protocol on a different specimen
Reflexive testing is different from confirmatory testing, in that if a single or multi-site analysis does not identify a BRCA1/2 mutation, full sequencing can be performed.
Importance of confirmatory testing
It is important to determine how often ‘false positive' results will be identified upon confirmatory testing. Based on the European proficiency testing experience, false positive results may occasionally occur (Question 9, Table 2-1). For this reason, it may be useful to perform confirmatory testing, when a mutation is identified. This issue is dealt with in more detail under Clinical Validity (Questions 19 and 20). Myriad Genetic Laboratories routinely confirms all positive test results by repeating the same test protocol on another aliquot of the same specimen.
Gap in Knowledge: Impact of confirmatory testing on analytic specificity. Myriad Genetic Laboratories routinely performs confirmatory testing on all positive test results, but information is not currently publicly available to determine the impact of confirmatory testing on analytic sensitivity and, consequently, the overall screening process.
ANALYTIC VALIDITY
Question 15. What range of patient specimens has been tested?
Summary
- Whole blood, fresh or frozen tissue and buccal samples are acceptable for BRCA1/2 mutation testing for all types of DNA analysis
- Paraffin-embedded tissue is only suitable for single- and multi-site DNA analysis only
|
Molecular genetic BRCA1/2 mutation analysis has been successfully performed in a variety of specimens using available methodologies. DNA testing can be performed on:
- whole blood (DNA isolated from peripheral blood lymphocytes)
- fresh or frozen tissue
- paraffin embedded tissue samples from tumors (single- and multi-site analysis only)
- buccal samples
ANALYTIC VALIDITY
Question 16. How often does the test fail to give a useable result?
Summary
- Laboratory testing for BRCA1/2 mutations can be divided into pre-analytic, analytic and post-analytic phases
- In the pre-analytic phase, generally agreed upon criteria are in use to determine the appropriateness of testing. If these are not met, the test can be canceled
- In the analytic phase, samples fail for multiple reasons, and these failures are routinely documented in clinical laboratories but are not generally available for outside review
- When analytic failures do occur, repeating the analysis will often yield useable results
- Types of failures and their associated rates are rarely reported
|
Test ‘failures' in the pre-analytic phase of testing
In the pre-analytic phase, it may be determined that the sample is not suitable for testing because specific clinical criteria are not met, or because the sample is considered inadequate. While programs often monitor pre-analytic test cancellation rates as part of an overall quality assurance plan, these events are usually not considered a laboratory or methodologic ‘failure'. Table 2-10 lists criteria commonly used for deciding whether to reject a sample in the pre-analytic phase.
Table 2-10. Common Pre-analytic Criteria for Rejecting a Sample Submitted for BRCA1/2 Mutation Testing
Rejection Criteria Based on Clinical Information |
Inability to demonstrate informed consent |
Inappropriate referral
(e.g. a genetic counselor referral from a state where counselors are not
authorized to refer) |
Rejection Criteria Based on Submitted Sample |
Inadequate specimen quality
(e.g., hemolyzed blood or obvious contamination) |
Inappropriate sample
(e.g., whole blood with no anticoagulant or wrong anticoagulant) |
Inadequate specimen labeling |
Inappropriate handling prior to laboratory receipt
(e.g., sample too long in transit or exposed to extreme temperature) |
Test failures during the analytic phase of testing
Failures of individual samples or assays occur when preset quality control standards are not met and, therefore, test results are not considered reportable. Failures can arise for a number of reasons, such as improperly processed samples, problems with component reagents, or equipment malfunction. Many assay failures within the clinical molecular genetic laboratory are due to operator error. Automation and programs to properly train laboratory personnel can help avoid these problems. Only a few medical technology programs, however, currently provide adequate molecular components. Documentation of failures and subsequent corrective action is required by regulatory agencies such as CLIA and the College of American Pathologists. Unfortunately, failure rates and other information on assay robustness are often not published. Available data suggest, however, that repeating the analysis of an individual sample or assay run can often yield a satisfactory result. An estimated 10 percent of amplicons (BRCA1 is divided into 36 amplicons and BRCA2 into 48 amplicons) undergoing DNA sequencing at Myriad Genetic Laboratories are re-analyzed, due to unacceptable quality of data/amplifications or for mutation confirmation (Ward B, personal communication).
An irretrievable assay failure occurs when an apparently suitable specimen is submitted and approved for testing, but the assay yields a result that is clinically uninterpretable. Failures of this type are most often related to the quality of the original sample. Procedural problems during specimen processing and DNA extraction can also be responsible. Success rates for obtaining clinically interpretable results are close to 100% for blood samples.
Post-analytic failures, such as incorrectly or inadequately interpreted results, are considered separately from analytic test failures, as part of a review of overall quality assurance in the Clinical Utility section (Question 32).
Gap in Knowledge: Overall, and method-specific, failure rates. Clinical laboratories are required to document test failures. Test failure rates could be provided by laboratories participating in external proficiency testing administered by ACMG/CAP. Myriad Genetic Laboratories has provided the proportion of tests that fail initially, but has not yet provided information about how often the repeated analysis is successful.
ANALYTIC VALIDITY
Question 17. How similar are results obtained in multiple laboratories using the same, or different, technology?
Summary
- Data derived from external proficiency testing can be used to judge the consistency of results from laboratories testing for BRCA1/2 mutations
- External proficiency testing in the U.S. does not currently yield useful information for laboratories using sequencing methodology
- External proficiency testing in Europe does not currently yield useful information for laboratories using sequencing methodology
|
Comparing results from different laboratories using the same or similar methodologies
One potential source of data for evaluating differences in BRCA1/2 mutation test results from multiple laboratories using the same (or a similar) method would be external proficiency testing. However, the small number of participants in ACMG/CAP (17) and the relatively large number of methods (Table 2-8, Appendix B) preclude obtaining meaningful method-specific analyses. The number of participants in EMQN is greater (41), however, the methodologies used are not published. Even if available, such comparisons might be complicated because laboratories in the same methodological category could use different commercial or in-house reagent components and protocols. For example, although three laboratories might be grouped under the ARMS™ methodology, one might use a prepared kit, a second might use commercially prepared analyte specific reagents (ASR), and the third might use in-house reagents. These factors would make the comparison nearly equivalent to comparing different methodologies. To help in comparing methodologies, the ACMG/CAP MGL Survey Reports and EMQN EQA schemes might consider stratifying results into broad methodological categories.
Comparing results from different laboratories regardless of the methodology
As part of the 2001 ACMG/CAP Molecular Genetics Laboratory external proficiency testing survey, 17 laboratories were queried about their methodology for performing BRCA1/2 mutation analysis (Table 2-8, Appendix B). Limited data are currently available. To date, method-specific data on error rates are not available from these surveys. However, for this challenge there was a high level of agreement between laboratories for detecting mutations that were targeted by their specific method.
The EMQN scheme reported results from a number of laboratories using surveys from 1999 to 2002. These laboratories used a variety of methods to pre-screen for BRCA1/2 mutations (Appendix B). To date, method-specific data on error rates are not available from these surveys. However, for this challenge there was a high level of agreement between laboratories for detecting mutations that were targeted by their specific method.
Gap in Knowledge: Comparing results from different laboratories with the same methodology. There are no current data that compare results from different laboratories with the same methodology for BRCA1/2 mutation testing.
References
Ford D, Easton DF, Stratton M, Narod S, Goldgar D, Devilee P, Bishop DT, Weber B, Lenoir G, Chang-Claude J, Sobol H, Teare MD, Struewing J, Arason A, Scherneck S, Peto J, Rebbeck TR, Tonin P, Neuhausen S, Barkardottir R, Eyfjord J, Lynch H, Ponder BA, Gayther SA, Zelada-Hedman M, and et al. Genetic heterogeneity and penetrance analysis of the BRCA1 and BRCA2 genes in breast cancer families. The Breast Cancer Linkage Consortium. Am J Hum Genet 1998; 62(3):676-89.
Gayther SA, and Ponder BA. Mutations of the BRCA1 and BRCA2 genes and the possibilities for predictive testing. Mol Med Today 1997; 3(4):168-74.
Neumaier M, Braun A, and Wagener C. Fundamentals of quality assessment of molecular amplification methods in clinical diagnostics. International Federation of Clinical Chemistry Scientific Division Committee on Molecular Biology Techniques. Clin Chem 1998; 44(1):12-26.
Petrij-Bosch A, Peelen T, van Vliet M, van Eijk R, Olmer R, Drusedau M, Hogervorst FB, Hageman S, Arts PJ, Ligtenberg MJ, Meijers-Heijboer H, Klijn JG, Vasen HF, Cornelisse CJ, van't Veer LJ, Bakker E, van Ommen GJ, and Devilee P. BRCA1 genomic deletions are major founder mutations in Dutch breast cancer patients. Nat Genet 1997; 17(3):341-5.
Puget N, Stoppa-Lyonnet D, Sinilnikova OM, Pages S, Lynch HT, Lenoir GM, and Mazoyer S. Screening for germ-line rearrangements and regulatory mutations in BRCA1 led to the identification of four new deletions. Cancer Res 1999; 59(2):455-61.
Shattuck-Eidens D, Oliphant A, McClure M, McBride C, Gupte J, Rubano T, Pruss D, Tavtigian SV, Teng DH, Adey N, Staebell M, Gumpper K, Lundstrom R, Hulick M, Kelly M, Holmen J, Lingenfelter B, Manley S, Fujimura F, Luce M, Ward B, Cannon-Albright L, Steele L, Offit K, Thomas A, and et al. BRCA1 sequence analysis in women at high risk for susceptibility mutations. Risk factor analysis and implications for genetic testing. JAMA 1997; 278(15):1242-50.
Unger MA, Nathanson KL, Calzone K, Antin-Ozerkis D, Shih HA, Martin AM, Lenoir GM, Mazoyer S, and Weber BL. Screening for genomic rearrangements in families with breast and ovarian cancer identifies BRCA1 mutations previously missed by conformation-sensitive gel electrophoresis or sequencing. Am J Hum Genet 2000; 67(4):841-50.
|