June 30, 2021 OneLab Network- Life of a Test Method: Validation, Verification, and Managing Quality

Download Slides

Requirements and Guidelines for Test Method Life Phases

Diagnostic Sensitivity and Specificity for Clinical Laboratory Testing

Transcript

Date of session: 06/30/2021

CHELSEA PARSONS: All right. Welcome, everyone. We’ll go ahead and get going. My name is Chelsea Parsons. And I’m a consultant with Guidehouse supporting the CDC OneLab Initiative. I want to make a couple of notes before we start the webinar. If you’re having any technical issues at all throughout the presentation, you can email the OneLab inbox for support. That’s onelab@cdc.gov. Again, that’s onelab@cdc.gov. Members of our team will be monitoring it the whole time. So they’ll help you if you have any issues.

If you have any questions throughout the session, you’ll see that in your bottom panel there’s a Q&A function. You can feel free to shoot a question there at any time throughout the presentation when a question arises. And we’ll have a quick 15 minute Q&A session at the very end of the presentation where we’ll try to get through as many of those questions as we possibly can. But you’ll have an opportunity to send us some later on if questions arise after the presentation, or if we don’t happen to get to your question.

You’ll notice that in the chat we posted a link to the live captions. If you’re going to use those, please be sure that you keep this Zoom meeting open as well as the live captions. You’ll have to have both open to get them to stay functioning.

So I’m just going to walk through an agenda really quickly before we get going. So we’re going to start today by introducing our presenters Dr. Triona Henderson-Samuel and Dr. Rex Astles. We’ll discuss some new OneLab resources. And then we’ll hear from our presenter, Rex Astles. And he’s going to present on the life of a test method. Validation, verification, and managing quality. We’ll have that 15 minute Q&A session that I mentioned earlier. And then we’ll end with a discussion of some upcoming network events.

So today’s presenters are Dr. Triona Henderson-Samuel and Dr. Rex Astles. Dr. Henderson-Samuel is a board certified pathologist in the DLS Training and Workforce Development Branch. Dr. Rex Astles, our presenter today, has been a health scientist within the Informatics and Data Science branch also within the DLS for over 25 years. He’s led many activities to assess and improve the quality of clinical and public health testing in the US since 2001. He’s also focused on building and reinforcing capacity for public health laboratory testing by fostering interactions between public and private laboratories.

He’s focused a lot on research to systematically improve the creation and implementation of laboratory practice guidelines. He’s a past director of both AACC board and the Board of the Academy of the AACC. He’s also been really active in the Clinical Laboratory Standards Institute. He was the chair of the EP19, which is a gateway document for using CLSI evaluation protocols for establishment and verification of performance throughout the life of a measurement procedure. And you will all actually be able to get access to this document after the event today. Within the next week, we’re going to post some really awesome resources that Dr. Astle has connected us with. He’ll mention this document a few times today, and you’ll get access to that later on. So I’m now going to toss it over to Dr. Triona Henderson Samuel from CDC’S Division of Laboratory Systems.

TRIONA HENDERSON-SAMUEL: Thank you so much, Chelsea. And before we get started today with our feature presentation, I’m just going to give you a few quick updates of some new CDC resources. So CDC has launched a new COVID-19 viral testing tool on the COVID-19 testing page that you guys have visited before. This interactive web based tool is designed to help both health care providers and individuals understand COVID-19 testing options using the most relevant actionable test related information.

For health care providers, the tool provides clinical decision support to help determine what type of SARS-COV-2 testing they should perform. Health care providers can also use this tool to quickly access the latest CDC test related content. And use this information to make decisions about a few steps such as whether or not to order a test, which tests they should order, the meaning of the test result that they’ve ordered, what to do with any conflicting test results, whether or not a confirmatory test is necessary, and how vaccination impacts decisions for testing. For individuals who do not yet have a test, the tool will help them determine what type of tests to get. For individuals who have a test result, the tool will help them determine the appropriate next steps, if any, based on the results.

Another resource that we want to share with you is the CDC’S Division of Laboratory Systems convenes regular calls with clinical laboratories to discuss the nation’s clinical laboratory response to COVID-19. These clinical laboratory COVID-19 calls, as we call them clicker calls, take place every other Monday at 3:00 PM Eastern time. Audio and transcript are posted online after each call.

On June 14th, Dr. Tim Stenzel from the Food and Drug Administration answered questions on this call regarding the effect of variants of testing validity and verification. As well as on recommended laboratory safety practices. The Q&A section was based largely on questions that emerged from the OneLab Network Training Needs Assessment. So we hope this session was helpful for those members who attended, and those who can now access the recording and the transcripts. Next slide, please.

Finally, CDC has created a quick reference guide for personnel trained to pack and ship suspected or confirmed SARS-COV-2 specimens. So [AUDIO OUT] some cool questions. And Chelsea is going to pull them up for us. It’s just going to take a few minutes to ask you some questions to better understand where you’re coming from, and who you are. So the first question that should pop up on your [AUDIO OUT] is, where are you joining us from? So you can select one. Mid-Atlantic, Midwest, Northeast, Northwest, Southeast, Southwest, West, or you’re coming from a non US or international territory.

And we’ll give you a few more seconds to answer that. And so you should now be able to see the results on the slide. So about 37% almost 40% of you are coming from the Midwest. Smaller from the Mid-Atlantic. And about equal portion from the Southeast, Southwest, and West. Thank you so much. So your second question. Do you have a role [AUDIO OUT]? Example, developing content. So your options here are three. It’s yes, no, or you are unsure. A couple more seconds. And perfect.

What we were hoping for is that the majority of you do have a role or active role in training and education. Thank you so much. And your final question is, do you participate– sorry. Did you participate in the OneLab Training Needs Assessment Survey previously? So that would be yes, no, or not sure again.

So about 50% of you did not participate in the needs assessment training. The training needs assessment survey, sorry. And 37% of you have. Thank you for taking the time to answer these questions. So finally, before we launch into Dr. Astle’s presentation, I want to give you all just a little context as to why we decided to prioritize hosting a session on this specific topic. Next slide, please.

So in this graphic, you’ll note that 49% of the network members who completed the training needs assessment survey indicated needing guidance in validating or performing tests. And issues cited by one of our network members is that they are, and I quote, “relying too heavily on manufacturer websites. And do not have the time or capacity to validate their instructions.”

I’d like to highlight that today’s topic is intended to provide context on the life of a test method, and provide material that you may find useful to share with your learners. We hope to ask some tough questions that laboratory professionals have grappled with such as, what if I modify a manufacturer’s procedure? What if I want to use an unauthorized reagent or material? What if I want to use a nonwaived test in a certificate of waiver setting?

We intend to have a follow up presentation in coming months addressing specifically how these concepts describe today apply within the context of an emergency use reauthorization. We know this is another area requiring education and training support. Here to present today is Dr. Rex Astles. As a reminder, today’s audio, transcript, and slide shared will be posted online afterwards. So you’ll be able to follow the hyperlink to the existing on the presented resources. Additionally, my team and I will be monitoring the Q&A section in case you have any questions throughout the presentation. I now hand the presentation over to Dr. Astles.

REX ASTLES: Thank you. That was a wonderful introduction. And I’m humbled to find myself working at the CDC with such fine scientists as yourself, Dr. Henderson-Samuel. It has been a great [AUDIO OUT]. I’m not planning to end it soon. But I wanted to provide a little context. I’ve been asked to provide a presentation that provides a baseline of a concept about how the test life can be thought of. And the two main stages in life, including establishment and implementation. And this will be helpful, I think, for understanding what gets modified, or left out, or abbreviated in emergency situations during public health emergencies.

So my short talk today is going to talk about the life of a test method, validation verification, and managing quality. Next slide, please. So let’s start with a pop quiz. And I’m going to title that, What is It? So you have successfully implemented an assay that you recently purchased. You continue to perform quality control as required by the manufacturer, and are performing all function checks. Is the ongoing process you’re doing validation of acceptable [AUDIO OUT] of adequate performance? Or demonstration of ongoing acceptable quality? And we’ll get back to this question at the end of the session and see if it’s a little clearer.

Dr. Henderson-Samuel has described the need and the purpose, really, of this talk as I’ve said, is to describe the functions of the laboratory system writ and large that is the entire laboratory system. If you work in a clinical or public health lab, of course, you have a critically important part in the laboratory system. But there are other players that I’d like to talk about.

So again, it will be how does the laboratory system function normally when not in a public health emergency? Learning objectives are to describe the test method life model, for you to be able to differentiate test method validation and verification, and to list some of the instructional resources that I’ll provide later. The outline really involves five different areas. [AUDIO OUT] system, as I mentioned. We’ll talk about the complexity model and its implications for laboratory testing. We’ll talk about the test method life paradigm. Sort of the overarching idea that ties all this together. And some of the important terminology used by the FDA and CLIA. And finally, we’ll talk about the importance of the instructions for use as a really critically important part of information that you need to know.

So there are many players in the laboratory system. Manufacturers, of course, are focused on manufacturing in quality, reagents that are stable, and are proven to be stable throughout the expected life of the test in the hands of the kind of operators who will use it. And supporting them are the federal agencies in HHS, CDC, CMS, and FDA. And we’ll talk a bit more about those two different stakeholders and players. There are accreditation organizations that you’re [AUDIO OUT]. The two biggest are the College of American Pathologists and the Joint Commission. And every year these are approved by CMS. They have deemed status as being more stringent than CLIA.

So the requirements that CAP, for example, has are inclusive of everything that’s in the CLIA regulations. But even more stringent. And also every year CMS approves the exempt states. There are two. Some of you may be in these. Washington state and New York state have requirements that are, again, more stringent than CLIA. And they do the inspections and take on that role of assuring that their labs are adhering to the essence of CLIA.

Professional organizations play a critically important role all the time, especially so during public health emergencies. And these, of course, are things like the College of American Pathologists, the American Association for Clinical Chemistry, American Society for [AUDIO OUT] that helped to collect questions. They advocate for their members of course. And they collect the questions their members have. And are able to put their members in touch with experts. And the huge social media increasingly. So it’s an important part of a response to a very dynamic and evolving situation like public health emergencies.

There are two standard organizations that I like to mention. CLSI, the Clinical and Laboratory Standards Institute, which was mentioned I’ve been very active in. And they only have a couple of standards that they have written. But they do write many, many guidelines each with many separate recommendations within them. And these guidelines, especially the evaluation protocols, are very helpful whether essential for the manufacturing community to know how to state and things like linearity, accuracy, and precision. And they’re also important for laboratories then to know how to verify that they are able to reproduce the kinds of performance claims that manufacturers have made.

And of course, clinical laboratories and public health laboratories are critically important. And I probably should have included public health agencies because increasingly, and especially during public health emergencies, labs need to know to whom they should report. And I know that can be very confusing.

Next slide I think it’s not new to you. They review and they allow marketing three main types of assays that we’ll talk about today. Premarket submissions for waiver for pre-market notifications or pre-market approval. And waiver requires, as you probably know, specific studies that show that the waived assay, if it’s granted waiver, is accurate, and relatively simple, and it can be used by untrained laboratorians. And pre-market notification basically hinges upon prior clearance of an assay. And it’s sort of showing comparability of this new assay to one that’s already been cleared or approved.

And the pre-market approval is the most rigorous process. And this is reserved for assays that have [AUDIO OUT] never been used clinically before. So as I said, FDA does grant waiver determinations. And the number of test methods that have been granted waiver has really increased every year. And so there are many, many different analytes that one can find now that have a waived assay. And within each category, there are increasing number of manufacturers that are individually branded waiver. So that’s been a boon for certificate of waiver labs. And has allowed some labs to switch from accreditation or certificate of compliance to a waiver category.

Finally, as you may have understood, the FDA does categorize tests into moderate or high complexity. And they use several criteria that essentially hinge upon the difficulty of interpreting the result, or maintaining calibration, and so forth. CMS is a important player because they do all the administrative work. They issue laboratory certificates like the one your lab has. They collect fees and conduct inspections. And approve accreditation organizations on a yearly basis. CDC helps them do that. And they monitor laboratory performance of proficiency testing or PT. And they approve PT programs on an annual basis. And they publish an update the Clinical Laboratory Improvement Amendments.

You may know that the CDC on the next slide, slide eight, has been involved in proposing changes to the proficiency testing regulations. And we’re in the throes of finalizing those regulations with CMS. So CDC Division of Laboratory Systems, where I work, does develop technical standards for CLIA. And [AUDIO OUT] to understand gaps in practices and where there are needs for changes. Either mandatory changes through CLIA or voluntary changes. We monitor PT program performance, and we manage the advisory committee, CLIAC, which meets regularly.

We develop and distribute technical information and educational materials, such as the Ready? Set? Test! booklet that helps laboratories understand how they can be sure that they are offering the best, highest quality test. And we try to strengthen partnerships with laboratory medicine stakeholders. The next slide speaks to the complexity model. And again, this has implications for laboratory testing. Excuse me.

So the complexity model works like this. There’s three levels. And the simplest level is a certificate of waiver testing. [AUDIO OUT] can only do that kind of testing. They need only have a laboratory director. There’s no other supervisory personnel that are named in CLIA that they need to have. And so they hire and maintain unskilled laboratory staff let’s say. And they’ll only be doing certificate of– they’ll only be doing waived test. And so they are not required to perform verification of waived test methods. They don’t need to do anything to show that the tests work. But they do have to follow everything in the procedure. The instructions for use or procedure manual that comes with the test must be followed exactly as it is. And that then is in effect their procedure manual.

Laboratories that don’t do waive testing, they can do non-waive testing. I said that wrong. Because labs that [AUDIO OUT] testing can do waive testing. So a lab that is able to do moderate complexity testing can also do waive testing. And they must verify manufacturer’s performance claims. And they have this additional technical consultant role that you don’t find in a waived setting. They cannot perform high complexity testing. Labs that are qualified to perform complexity testing can perform all kinds of testing– all three types. And they have to have these additional roles of general supervisor and technical supervisor.

Labs that perform high complexity testing is the only kind of lab that can perform laboratory developed tests– waived lab or moderate complexity lab should not be doing lab tests. And if a lab modifies a commercial test method it becomes de facto a high complexity test because it’s not been reviewed by FDA. And so labs that have [AUDIO OUT] you can’t do it really unless they are a high complexity test facility.

In the next slide, number 10, I’d like to introduce the test method life paradigm. So there’s really two stages. There’s method establishment, which is shown on this slide. And there’s method implementation. So establishment is a term that was in CLIA before EP19 grappled with and tried to structure this model. So we borrowed that term so to speak. CLIA does not talk about feasibility, and design, and development. In fact, they don’t talk about validation either.

The CLIA regulations say that if a lab establishes a new test, they have to do certain things. And we’ll get into that. But we know manufacturers are the ones who ordinarily do most of the establishment of new test methods. IBD manufacturers. And these are [AUDIO OUT] of this stage that are important for establishment. Let’s see if we need anything else on that slide. Yeah. Establishment is the determination of the clinical and analytical performance characteristics of a new or modified test method encompassing all phases in the life of that test method from feasibility through validation. That definition comes from EP19.

And the key part of it that is important that I want to emphasize is that it mentions clinical performance. For whatever reason, the CLIA regulations did not speak to clinical performance. And I’ll demonstrate that in a moment. But the CLIA regs do say that labs must establish performance for new assays. And so implementation, which is the next stage, includes all the phases of the life of a test method that happened after transfer of a validated test to an end user laboratory. So implementation is what happens after validation.

The next slide, number 11, I just wanted to share some definitions more or less paraphrased from EP19. So feasibility assessment is the consideration of various issues that are relevant to the advisability of developing the new test method. Issues may include the potential market for the attest, client [AUDIO OUT] and expectations, and strategic plans for the institution. The development– I’m sorry. Rather the design which ordinarily happens about the same time as feasibility includes considerations of whether the test method would be better than existing procedures. And whether it would meet a clinical need that’s not currently met.

The intended use for testing patients is agreed upon during the design phase. The relevant systems specifications may be established during the design phase. And then to [AUDIO OUT] improvement in hardware, software, reagents, and other system design elements to optimize performance and meet specifications. And all the time keeping in mind, of course, if it’s a manufacturer, any manufacturer, whether it’s a laboratory or an IED manufacturer will be trying to contain costs.

Clinical validation is the process the which one shows that test results are clinically meaningful, i.e. finding out whether the test method can detect or predict the disorder or condition of interest in targeted patient groups. So clinical validation requires quite a bit more work than just showing that a test can work analytically. It involves, of course, identifying and being sure you’ve correctly identified patients that have a disease versus those who don’t. Find finding patients that are relatively asymptomatic yet have the disease and so forth.

And so doing clinical [AUDIO OUT] I mean, clinical testing is onerous, expensive, and time consuming. And we all know that. The next slide is the first of four quick slides that I wanted to share that emphasize certain terms. And all of these terms are in EP19. And again, we’ll have a link to that at the end of this presentation. Intended use is that usage by the end user laboratory as specified by the test method manufacturer. That usage that has been specified by the manufacturer as originally designed and described in its instructions for use. It includes definitions of the measure, and that is the analyte and the specimen matrix. So for example, sodium in cold blood versus sodium in plasma.

The target condition is included in intended use and the clinical use including whether it’s for screening, diagnosis, or monitoring. All those concepts or part of intended use. The next term is target population, which is really very akin to the intended use. That’s the specific population for which the test method was validated. Possibly including patient age, sex, and the occurrence of other medical conditions.

The next term that’s important for validation is detection capability. And I wanted mention this because often the term sensitivity is used to mean detection capability. And sensitivity is probably more appropriately reserved for clinical testing, where you’re talking about sensitivity to detect a disease condition as opposed to analytical sensitivity, which is whether or not you can detect very low concentrations. So better terms p are used by CLSI EP documents or limit of detection [AUDIO OUT]. So that term sensitivity can be confusing.

The final concept I just wanted to introduce his performance claims. These are the analytical and clinical characteristics of a test method as validated and stated by the manufacturer in the instructions for use or the product insert. Excuse me again.

So let’s go on and talk now about what CLIA says regarding establishment of performance of a test method. As I mentioned, CLIA regulations omit any clinical proof, any proof, that the test method is clinically accurate. They do require for labs that are establishing their test, which is essentially a lab developed test, that the lab demonstrates accuracy, precision, analytical sensitivity– which is the same as detection capability, analytical specificity, [AUDIO OUT]. The ability not to be influenced by interfering substances as opposed to clinical specificity, which is identifying patients that don’t have the disease.

They also require that the lab establish the reportable range, and reference interval, and any other required performance characteristics. But again, there’s no specific requirement in CLIA for clinical validation. Perhaps if there had been, then this interest by FDA to somehow regulate laboratory developed tests might not have occurred. But there is a gap in what CLIA currently requires, and what FDA would like and we would all like for laboratories to be aware of and to do.

Slide 17 talks now about the second stage– implementation. Again, after the test validated by either the laboratory that’s establishing performance or the IVP manufacturer it’s transferred perhaps physically. At least functionally it’s transferred to the end user laboratory. And there are, as you see, five different phases [AUDIO OUT] implementation. So let’s talk about those in the next slide.

Preliminary evaluation is an optional initial determination of a new test methods suitability designed to quickly identify major flaws in the performance that would make it unsuited for meeting the end users’ needs. So sometimes laboratories might get a test kit either free or buy it from a potential vendor, and try it out. And see if it’s simple enough to use. And whether it perhaps gives a results that are consistent with the assay that the lab has been using. They might get a larger platform, and do some testing to try out precision. Maybe precision would be the most critically [AUDIO OUT]. It’s not able to be precise then the lab may say we’re going to pass on that test.

Increasingly, EP19, which is being revised by the way. Is thinking that this part, this preliminary evaluation, ought to include the idea that laboratories can get information without actually testing with the test kit. They can procure information from their colleagues, or published sources, or social media, or what have you about how a test works. And that’s really what happened during the SARS-COV-2 outbreak. Information flowed just as quickly as it could with whatever mechanisms it could flow. And so, again, the professional organizations were very helpful in sharing that kind of information. And that would be the kind of thing that we mean here by preliminary evaluation.

So verification is critically important. And again, it’s different than validation. Validation is what’s done to show its clinically [AUDIO OUT] end user lab does to prove that they can get the answers– rather the performance that the manufacturer stated. So the end user would be looking to make sure they can get linearity, precision, the accuracy, and can they probably be using EP19 documents to do that.

Launch is of course the first day the test goes live. And CLSI doesn’t have documents really about this. And CLIA doesn’t speak to it much either. But of course, on launch, you want to be sure that your informatics capability is able to transcribe the results into the electronic health record. And the clinicians know that this test is coming and so forth. So after launch, then it’s method maintenance. And by that we mean the method is being maintained at a high level of quality throughout the life of the method for however many months or years that method is used.

And finally, methods [AUDIO OUT] to retire. There are requirements for the laboratory to maintain records of the logs and so forth of the method itself. And I don’t have the number of years at hand here. But that’s all CLIA. The next slide, number 19, speaks to what CLIA requires as applicable to implementation. We’ve already talked about what CLIA requires for establishment. Now let’s see what they say about implementation. The onus is really on the manufacturer. Whether it’s the manufacturer as is usually the case, or a lab to consider the risk of analytical failure and calibration drift. And to determine the QC checks and their frequencies.

But CLIA says also that this is the lab’s responsibility. So determination of calibration procedures and control procedures [AUDIO OUT] specifications previously established or verified is [AUDIO OUT] 1253 b3. CLIA also requires appropriate control procedures in a separate section. And there, it speaks to monitoring the accuracy and precision of the complete analytic process. Establishing the number, type, and frequency of QC testing. So we ordinarily think about QC as testing liquid controls twice a day– once a day at two levels. But there are other kinds of QC that ought to be considered, especially for more complicated and highly manual assays for which the lab might determine there are many other things that can go wrong. And they should be testing for those.

Detecting immediate errors is of course critically important to do the system failure, adverse environmental conditions, or operator performance. [AUDIO OUT] the laboratory to monitor over time accuracy and precision that may be influenced by these factors. So identification and also monitoring over time. And then unless the laboratory has used the individual quality control procedure, the lab must adhere to the requirements for specific QC practices in that particular section. OK.

Let’s see. Now, we’re on slide 20. Additionally, I wanted to make this point that CLIA does not say that a lab must establish and then verify performance. If it’s a test method that the lab establishes, CLIA simply doesn’t say anything other than the lab must establish it. But the authors of EP19 and the speakers that I work with that present this session– we present a session based on EP19 [AUDIO OUT] that the end user to laboratory should also have to verify performance. Even if it’s basically the same research and development laboratory that is going to be doing the test on behalf of the clinical laboratory. Maybe even the same staff. We believe that there should be considerations of what should be verified if, for example, the reagents are going to be kept in a different refrigerator. Or there may be staff on offshore, weekends, or evenings that will be running a test. So that should be considered. And that’s the point I’m trying to make.

CLIA requires that the laboratory demonstrate that it can obtain performance specifications not for everything that was in establishment, but a much smaller list. So the lab should verify performance and accuracy. But they’re getting the accuracy they expect, the precision they expect, the reportable range of test results. That is that basically the assay is linear accurate over a broad dynamic range. And they can verify the manufacturer’s reference interval. And that is, by the way, a very difficult thing to do. It’s labor intensive.

And if the manufacturer use several hundred patients to establish the reference interval, how many should the lab use? And how do they define normal and so forth? So it can be quite a bit of work. So the 21st slide, and we’re getting near the end, talks about this point that the CLIA regulations don’t specify that the lab must verify previously established performance. And I’ve made that point. No specific requirements are in CLIA for verification of detection capability or analytical sensitivity. There’s no requirements that are clearly stated anyway for performance of qualitative tests. Although the [AUDIO OUT] listed under there that you can see at the bottom of the slide do apply. And maybe interpret it to be applicable to qualitative tests.

There are several things that can be found in the instructions for use on slide 22. And I just wanted to mention a few of those. Of course, the product name is likely to be included in the instructions for use. The intended use or uses for example in detecting pregnancy. The storage instructions both for agents and specimens stability should be there– must be there. And the limitations of the procedure. And this might include things like interfering substances. But it might include things like if further testing is needed, either a more specific or more sensitive testing, when certain test results are obtained for example, perhaps borderline test results that limitation that need should be stated in the [AUDIO OUT].

The Instructions for use should also described expected values. Including how the ranges were established and identification of the population, which the ranges were based. And any specific performance characteristics as appropriate including accuracy, precision, specificity, and sensitivity. So that’s quite a whirlwind. And I apologize for running through all that information so quickly. Perhaps I’ll reinforce it if we do a little exercise.

So in slide 23, I wanted to ask you some questions. What if your laboratory modifies a manufacturer’s procedure? The product insert. And by that I mean what if your laboratory wanted to reduce the sample size in half or reduce the incubation time by 10%? So the answer is that the de fecto– the default answer is that it probably has become a high complexity test. And your lab could do it if you’re a high complexity laboratory. If not, you shouldn’t.

And the test method probably needs to have clinical validation performed. So we might expect a change like that, like changing timing or specimen quantity, would likely change analytical performance. And it might change the limit of detection. But it could also have an important impact upon clinical care– clinical results. So it probably needs to be validated as if it were a brand new test.

And in the second question, what if you want to use unauthorized reagents or materials? The same answer applies. And the third question, what if you wanted to use [AUDIO OUT]. It’s like using plasma instead of serum. Or different target populations for different diagnostic purposes. Those really would probably require clinical validation studies as if it were a brand new lab developed test.

The fourth question I had is, what if you wanted to disregard the instructions in the product and including, for example, required quality control or training? If you did that, I’m not sure. It really is forbidden I guess is the answer. If you want to make modifications like that, I’m not sure how you– you would be using it off label. You would be using it against the manufacturer’s instructions. And I’m not sure that in that case, you could even treat it as a lab developed test. And sort of clinically validate that it didn’t harm anything if you didn’t require the training.

Finally, what if you wanted to use a non-waived test in a certificate of waiver setting? That’s really [AUDIO OUT]. You can’t do it. There’s no way. If you modified it, you’d have to be waived lab. I’m sorry. You have to be a high complexity lab, which you’re not. You’re a certificate of waiver lab.

So let’s go through this last question again. What is it? The answer is if you’re continuing to perform quality control and operating as you should be and doing everything correctly, then you’re demonstrating ongoing acceptable quality. Validation is what happens before the end user laboratory gets the assay. Verification is what they do to show that they can get the results that the manufacturer has the performance rather that the manufacturer has claimed. And those two terms are distinctly used, and that should be reserved really for those purposes.

So the 25th slide has a list of resources. These are web links that we hope will be used, and be very helpful to you. There’s some wonderful material in the DLS site. And the FDA site is also very useful. It lists a number of test methods that are available. And they’re CLIA categorization their test complexity. CMS [AUDIO OUT] of course is frequently valuable. And there’s a CLIA site, the Electronic Code of Federal Regulations, is really a link to the CLIA regulations. And the EP19 is the first bullet there under the Clinical Laboratory Standards Institute.

The CLSI makes that EP19 available freely to anyone who ask for it. And all of the material really that I’ve covered today for the most part is in there. Finally, I wanted to share with you a table, which we use in this ACC meeting that I mentioned. It’s probably very difficult to see on the PowerPoint slide. But we’ll provide you with the full table. This is just a piece of it. It will [AUDIO OUT] the different stages. Again, which are establishment and implementation. And the phases within each of those two stages. And the activity. And the reference to the quality requirements there for the FDA, QSR regulations, and CLIA. And the New York state requirements, which many of that if you do testing for patients that live in New York state, you have to get approved by them with your lab developed tests.

And finally, ISO requirements. And so the final far right column is simply a listing of the different CLSI documents that can be useful to meet those quality requirements. And so that concludes my talk. I’m hoping that Dr. Henderson-Samuel, you and miss Gatewood will be able to join you on camera for a Q&A session. Thank you.

JOHANZYNN: [AUDIO OUT], Dr. Astles, for that lovely presentation. Hey, everyone. My name is Jo Hanson. I’m going to be facilitating today’s Q&A. We’re going to spend the next five minutes or so answering some of your questions that you’ve been dropping in the Q&A. Thanks in advance for doing that. And if you haven’t already, please go ahead and put your questions in the Q&A. If we don’t answer your question by the end of today’s meeting, or if you have more questions after today’s meeting, please email onelab@cdc.gov.

And I’ll start with our first question for you Dr. Astles. Does this same model work when we’re talking about emergency use authorized laboratory tests?

REX ASTLES: Yes, it really should work because all the important functions that need to be covered, need to be considered, are laid out in this model. The great variances in the way the players– FDA, CMS, and CDC- has operated during the current public health emergency. So again, I think it’s helpful to know what the baseline is and what the requirements are. And so the instructions for use might not look the same as they ordinarily do. And we can talk about that in a later session. But anyway. I hope this helps, and will be helpful context and useful for thinking about emergency use authorizations. Thank you.

JOHANZYNN: Great. Thank you. Our next question might be a little specific. But I can see if you can answer it, Dr. Astles. Can I validate outdated swaps for clinical use?

REX ASTLES: Yeah. And so I assume that they’re talking about COVID-19 swabs that are outdated. So one of the things that happened during a public health emergency is that FDA in the need to transmit data, information, and guidance quickly has allowed laboratories to do [AUDIO OUT] that they might not ordinarily have permitted. And they have endorsed and allowed what they call bridging studies.

So I think it’s possible to verify performance for the swabs. Whether you could fully clinically validate them– you’re talking about outdated swabs– is probably not possible. But there should be ways. And FDA is eager, and very helpful with providing information around COVID-19 testing.

JOHANZYNN: Great. Thank you. Next question is, do you know of any good validation or verification references for evaluating LIMS, L-I-M-S, when custom testing workflows are developed?

REX ASTLES: Oh, that’s a great question. A very great question. Evaluation of laboratory informatic systems– I think you said LIMS, which I’ll take to [AUDIO OUT] is difficult because there’s so many things to make sure you are doing. You want to make sure the clinician is able to not only get the answer, but can see it. And can understand it, and not misinterpret it. So sometimes, particularly when free text is involved, lab information systems can be problematic in part because there’s not enough space sometimes in the window, so to speak, of information that can be allowed in the electronic health record. So I don’t have a good sense of that. But I can say the best most cautious thing to do would be to include clinicians advice in how to verify that they’re getting the results.

JOHANZYNN: Great. Thank you. Next question for you is, what should our laboratory do if we’re operating outside the recommended range listed for the test system?

REX ASTLES: Well, without knowing more, it’s hard to know. But let’s assume that they’re not so far outside that it presumably would impact results. I think the best advice I can give there is to probably talk to the manufacturer, and see what their advice is. So if it’s a test method that involves let’s say disks– what am I trying to say– just impregnated with some reagent. And if they get moist, and are moist for a long time, perhaps they deteriorate faster than ordinarily. And so the manufacturer might have some recommendations for that.

And moreover, we mentioned the individualized quality control procedure. So a lab in that situation might want to [AUDIO OUT] I mean, what is likely to go wrong. Figure out a way to quality control that that might involve running a control that is sort of borderline positive that might become negative if the reagents deteriorate. So I guess my advice is to talk to the manufacturer, and figure out if there’s something that can be done in the way of an IQCP.

JOHANZYNN: OK, great. Thank you. Next question is, why would my lab need to verify performance of a lab developed test that we created and already validated?

REX ASTLES: That’s a really good question. And it does seem to be onerous perhaps. I would say this. That if a lab has developed an established performance for an LDT, they at least should consider what things might go wrong or need to be verified with a benchtop analyzer. And then they move that benchtop analyzer into the production setting where the clinical lab is. They need to verify that it still worked after they moved it. Or as I’ve said, if they have different staff that are doing the testing, again, the people that have been using that assay 40 hours a week for the last six months are likely to get exactly the results that they would expect. The very best results. And I just think it’s prudent for the laboratory to consider at least what performance characteristics, not clinical validation, but verification of analytical performance might have changed after they have transferred it into production.

JOHANZYNN: Great, thank you. Do quality control materials have to be registered with the FDA for use with an FDA 510 K cleared test. And is this the same for QC for CLIA waived tests?

REX ASTLES: Do they have to be registered?

JOHANZYNN: With the FDA. Yep.

REX ASTLES: Yeah. So I’m not certain of the question. But if the lab has created their own QC materials, do they have to register? And I think the answer is, no. if we’re talking about a test– an IVD manufacturer test method that has been cleared or approved. And it either has quality control material or it doesn’t, if the lab has bought that test method and is using it, they to my knowledge don’t have to register anything with the FDA because the FDA doesn’t really control what labs are doing as of yet. It’s on the lab. CLIA– these things that I’ve talked about– can quality control be considered. And that it be used when there are things that need to be addressed.

JOHANZYNN: Is linearity testing possible for evaluating performance of waived testing methods or waived test method?

REX ASTLES: That’s a really good question. So no. First off, waived test methods you can’t verify anything. You’re not supposed to verify anything. You don’t need to. If you wanted to, you in theory could. But the thing about wave test methods is they’re often qualitative. They’re often yes, no, positive, negative, present, or not present. They’re not all that way. Many of them are quantitative. But the ones that are qualitative, you really can’t do anything as far as linearity of course.

For those that are quantitative, where you get a number and you could compare the number you get to some dilution of [AUDIO OUT] you could do it. But CLIA doesn’t require that you do anything to look at linearity for waived. You don’t have to verify anything.

JOHANZYNN: Great. Thank you. We have time for just one more question. So what would be the best approach to harmonizing reference ranges across a health system?

REX ASTLES: Oh, gosh. Well, the difficulty with harmonizing reference intervals is that by intent, each reference interval was supposed to apply to the local population. So if there is people in Istanbul, and people in Paris, and then let’s say Miami, they may have different expected values based upon the racial characteristics of the population for example. So it’s not really– I’m not sure how to respond. But I’m not sure how to get the reference intervals to accord.

Now if you’re talking about making sure that the results are similar, the best method is to use programs like the CDC’s host program and their CRMLN program with standardization of cholesterol testing in which commutable samples are sent to laboratories and manufacturers. And they test them and tune their calibration curves to get the results that CDC reference methods say are correct. So that is the best way to get harmonization internationally.

JOHANZYNN: Great. Thanks again, Dr. Astles for the presentation. And thanks to every one who put their questions in the chat. I’m sorry we weren’t able to get to all of them. But if you have more questions after today’s meeting, please email onelab@cdc.gov. We’re going to now quickly turn it over back to Dr. Henderson-Samuel for just closing remarks. Thanks again.

TRIONA HENDERSON-SAMUEL: Thank you, Jo Hanson. And thank you, Dr. Astles. And yes, we see a lot of really great questions coming up in the chat at the end. So please, if you would like them answered, please send them to onelab@cdc.gov. The next network meeting will take place on July 16th at 1:00 PM Eastern Standard Time. Additional information and Zoom link for that meeting will be coming to you tomorrow via email.

During this event, Dr. Dauphin, the acting CSELS director, will highlight lessons learned from her CDC deployment during times of public health crises. She will share with us what it takes to lead during a historic global pandemic when all eyes are focused on the critical and time sensitive work of our nation’s critical public health laboratories.

As a reminder, all of these slides will be posted at www.cdc.gov/onelab within the next two weeks. You’ll notice when this [AUDIO OUT] it will open up on your browser. We will use the results of this survey to further improve future OneLab Network events. The survey is voluntary and anonymous. We’re looking forward to continued collaboration, and being able to assist you in all your training needs.

Thank you, and have a great rest of your day.