Draft Genetic Test Review

---

Cystic Fibrosis
Analytic Validity

(1.14MB)

---

ANALYTIC VALIDITY

Question 8: Is the test qualitative or quantitative?
Question 9: How often is a test positive when a mutation is present?
Question 10: How often is the test negative when a mutation is not present?
Question 11: Is an internal QC program defined and externally monitored?
Question 12: Have repeated measurements been made on specimens?
Question 13. What is the within- and between-laboratory precision?
Question 14: If appropriate, how is confirmatory testing performed to resolve false positives in a timely manner?
Question 15: What range of patient specimens has been tested?
Question 16: How often does the test fail to give a useable result?
Question 17: How similar are results obtained in multiple laboratories using the same, or different, technology?

---

ANALYTIC VALIDITY

Question 17: How similar are results obtained in multiple laboratories using the same, or different technology?

Comparing results from different laboratories using the same or similar methodologies
The only potential source of data for evaluating differences in cystic fibrosis test results from multiple laboratories using the same (or a similar) method would be derived from external proficiency testing. However, the relatively small number of participants and the relatively large number of methods (Table 2-10, Appendix B) preclude obtaining meaningful method-specific analyses. Even if available, such comparisons might be complicated because laboratories in the same methodological category may be using different commercial or in-house reagent components and protocols. For example, although three laboratories might be grouped under the ARMS™ methodology, one might use a prepared kit, a second might use commercially prepared ASR's (analyte specific reagents), and the third might use in-house reagents. Each may also be targeting a different set of mutations. All of these factors would make the comparison nearly equivalent to comparing different methodologies. To help in comparing methodologies, the ACMG/CAP MGL Survey Reports might consider stratifying results into broad methodological categories.

Comparing results from different laboratories regardless of the methodology
As part of the 2000 ACMG/CAP Molecular Genetics Laboratory external proficiency testing survey, laboratories were queried about their methodology for performing cystic fibrosis mutation analysis (Table 2-10, Appendix B). Overall, the reported methodologies were used to detect between 1 and 70 mutations (median 12 to 14 mutations). To date, method-specific data on error rates are not available from these surveys. However, during the six years of operation (1996 through 2000) there was a high level of agreement between laboratories for detecting mutations that were targeted by their specific method.

The European Concerted Action on Cystic Fibrosis reported results from a much larger number of laboratories using surveys in 1996 through 1998 (136, 145 and 159 laboratories, respectively). Again, few method-specific data are available. However, all commercial kits were found to have problems identifying G551D and R553X.

The impact of a laboratory's mutation panel on proficiency testing results
There were instances when the responses varied greatly, because of variability of mutations being tested for by laboratories. For example, in 2000 MGL-17, DNA from a compound heterozygote (621+1G>E / G85E) was distributed.

• Twenty of the 41 participating laboratories (49 percent) obtained the correct genotype.
• Sixteen other laboratories (39 percent) did not test for G85E and thus identified the sample as coming from a heterozygote (621+1G>E/wild).
• Four other laboratories (10%) did not test for either mutation and reported a normal genotype (wild/wild).
• One laboratory (2 percent) did not test for 621+1G>E and thus reported a heterozygote (G85E/wild).

Although this genotype is rare, it demonstrates the wide range of laboratory responses that can occur when a mix of methodologies and mutation panel sizes occurs in practice. A more complete discussion of mutation panel size, composition and performance is contained in Clinical Validity.

Gap in Knowledge: Comparison of Methods for Cystic Fibrosis Mutation Detection
In order to compare analytic validity for various testing methodologies, proficiency testing data need to be stratified by methodological category. It would also be useful to identify subsets using the same commercially available reagents (e.g., in-house versus ASR reagents). Alternatively, a previously described method for validation (Question 9 and 10 – Optimal Sources of Data) could be employed that would provide not only analytic performance for a methodology, but also comparative data between methodologies.

Appendix B. Analytic methodologies used for cystic fibrosis mutation analysis

Introduction
Table 2-10 lists categories of methodologies that are used to detect cystic fibrosis mutations by laboratories participating in proficiency testing programs in the United States (ACMG/CAP MGL Survey) and Europe, along with the proportions using each method. Because many laboratories utilize “home brew” assays, these categories are not homogeneous. Some methodologies are relatively labor intensive and can only detect a few mutations (e.g., heteroduplex analysis), making them more suitable for research or diagnostic laboratories. When large numbers of specimens must be tested with short turn-around times (e.g., prenatal screening), other methodologies are needed. The European report documents a clear increase in use of commercially prepared materials/kits, 28 percent of laboratories in 1996 versus 50 percent in 1998 (Dequeker and Cassiman, 2000).

Table 2-10. Testing Methods Utilized by 36 US Laboratories and 151 European Laboratories According to External Surveys

Testing Method	USA (%)a	Europe (%)b
Allele Specific Oligonucleotide (ASO)	39	24
Electrophoresis for RFLP and size analysis	39	2
Allele-specific PCR/ARMS	19	15
Oligonucleotide ligation analysis (OLA)	14	18
All Mutation Scanning Methods	11	41
(Heteroduplex analysis)		(36) c
(SSCP)		(1)
(DGGE)		(2)
(Other)		(2)
Other methods	28	0

^a ACMG/CAP 2000 MGL Totals more than 100 percent (some laboratories use more than one methodology)

^b European Concerted Action for Cystic Fibrosis 1998

^c Mutation scanning methods available only as a total percent for USA. Numbers in parentheses delineate individual methods for Europe

Cystic fibrosis testing methodologies for prenatal screening ought to include the following characteristics:

• a reasonable number of mutations
• a low to moderate level of technical expertise
• a short turn-around time (one or two days)
• a high throughput (ideally, on an automated platform)
• a relatively low cost

These requirements might appear ambitious, but the evolution of other tests now used for screening in the clinical laboratory shows these goals are achievable. For example, immunoassays that are now routinely performed were originally developed in the 1960's by investigators with in-depth knowledge of immunochemistry and radiation detection methods. Over the ensuing years, these assays were revised and streamlined by manufacturers to meet the needs of clinical laboratories, including the development of automated immunoassay systems that minimize the chance for error. For FDA approved kits, the responsibility for ensuring reagent quality and instrument performance now rests primarily with the manufacturer. The laboratory's responsibility is to monitor the quality control measures set by the manufacturer to verify that assay performance meets specifications. A further development is a computer link to the instrument that automatically transfers test results to a patient record system for reporting. Automation is more expensive than manual assays in terms of reagents and instrument rental or purchase, but the overall cost per test can be the same or lower because of the reduced labor costs. This same development is beginning to occur for prenatal cystic fibrosis screening. Commercially prepared reagents have emerged using three major methodologies. The attributes of these reagents are summarized in Table 2-11.

Table 2-11. Characteristics of Commercial Reagents to Detect Cystic Fibrosis Mutations

	Commercial Cystic Fibrosis Mutation Detection Systems
	Elucigene	INNO-LiPA	Linear Array	PCR OLA
Characteristic
1. Method Type	ARMs	ASO	ASO	OLA
2. Company	Orchid	Innogenetics	Roche Molecular	Perkin Elmer
3. Mutations	29	33	31	31
4. Robustness	High	High	Not available	High
5. Special equipment	No	No	No	Yes
6. Total time (days)	1 to 1.5	0.5 to 1	2	0.5 to 1
7. Cost per patient	$30-40	$30-40	Not available	$55
8. Advantages	Low complexity Published data	Can be automated	Can be automated	Can be semi-automated
9. Disadvantages	Cannot now be automated		Automation essential for high throughput	Many steps
For more information	orchid.com (last accessed 3/2007)	innogenetics.com (last accessed 3/2007)	biochem.roche.com (last accessed 3/2007)	appliedbiosystems.com/   (last accessed 3/2007)

Notes pertaining to Table 2-11:

1. Method type: Methods displayed are those that are most commonly used and that are suitable for large-scale cystic fibrosis screening. These include the Amplification Refractory Mutation System (ARMS™), Oligoligation assay (OLA), and allele specific oligonucleotide assays (ASO). Both of the ASO assays use reverse dot blot strip technology.

2. Company: None of these commercial reagents have been approved by the FDA for clinical use. However, reagents may qualify under the FDA's Analyte Specific Reagent (ASR) rule which indicate that the assay building blocks are made under good manufacturing practices.

3. Mutations: This is the number of cystic fibrosis mutations that can be detected by the testing protocol. Manufacturers are modifying existing reagents to conform with the panel of 25 mutations. The EluciGene ARMS™ test cannot reliably identify individuals who carry two copies of a mutation other than delF508 (about 2-3 percent of affected individuals). This is not a critical requirement in population screening, where carriers are the initial target. Other methods can identify both carriers (heterozygotes) and homozygotes.

4. Robustness: Robustness describes how consistently and reliably a set of reagents performs when used by different laboratories, under varying conditions, and on different sample types (e.g., blood, buccal smears).

5. Special equipment: Some manufacturers require that specialized equipment be used to perform their assays. Although initially more costly, the equipment may allow more samples to be tested.

6. Total time: Estimated time to complete assay, including sample processing and reporting. Some methods only require one day, but laboratories may choose to extend the process to a second day for more convenient scheduling.

7. Cost per patient: Costs for the reagents and licenses to perform cystic fibrosis testing are extremely variable. Some laboratories perform ‘in-house' assays with relatively low reagent costs. In such cases, the cost of technical time for reagent preparation and QC/QA must also be considered. Costs of analyte specific reagents (ASR) can be relatively high, compared to traditional biochemical assays. However, the savings in technical staff time for preparation and QC/QA can offset reagent costs. For screening, the relevant figure is the cost per patient tested, rather than the cost per mutation tested.

8. Advantages: Reagents for prenatal cystic fibrosis screening should have high throughput with relatively low labor costs. Assays that can be efficiently automated can be cost effective. Peer-reviewed analytic validity data are helpful for validation.

Newer testing technology platforms with high potential for cystic fibrosis testing include various hybridization strategies (Roche and Luminex), arrayed primer extension (Orchid), mass spectrometry (Sequenom), sequence analysis (Pyrosequencing), and random coated array detection (Bioarray Solutions). However, there are no existing data that accurately compare these technologies with currently utilized methodologies or with each other.

References

Dequeker E, Cassiman JJ. 2000. Genetic testing and quality control in diagnostic laboratories. Nat Genet 25:259-260.

Provides link to non-governmental sites and does not necessarily represent the views of the Centers  for Disease Control and Prevention.