Against a backdrop of dynamic federal regulations and a quickly expanding commercial landscape, labs are facing unusual obstacles in validating serology testing for COVID-19. Issues with test accuracy and supply, combined with a paucity of independent validations, mean most labs must basically start from scratch in vetting tests on their own.
Along with lab-based and high-throughput serology tests that can be validated as lab-developed tests, seven commercial serology assays have obtained Emergency Use Authorization from the Food and Drug Administration.
Additionally, there are currently 141 manufacturers that have notified FDA that they have validated and are offering serology tests in the agency's Policy D — which are necessarily deemed high-complexity tests until they are evaluated in the EUA process. These serology tests are listed on the FDA's website.
Serology tests, and particularly the lateral flow immunoassays, may or may not be available in large quantities as labs around the world scramble to acquire them, and few have been tested in large controlled trials. Sensitivity and specificity concerns—particularly in light of an unclear prevalence of SARS-CoV-2 infection in the U.S. — mean labs need to validate any test they bring before they use it on patients.
Indeed, a lengthy report issued this week from the Center for Health Security at Johns Hopkins University calls out the fact that validated, accurate serology tests are currently in short supply.
"These tests will be in high demand, and manufacturing should be scaled appropriately, but the first steps will be to ensure accuracy, validity, and comparability of available tests," the authors wrote.
A special article in the American Journal of Clinical Pathology noted, however, that COVID-19 serology assay validation is "a particularly daunting task" made more challenging because usefulness of the data hinges on knowing the prevalence of disease. For now, the authors conclude the situation requires a stance of "caveat emptor," or buyer beware.
Currently, there are 21 labs that are registered on the FDA website as offering serology testing, which means they have validated a test in the lab.
Labs at ARUP, the Medical College of Wisconsin, and the Mayo Clinic's Arizona location are also validating a variety of serology tests, and their lab directors have offered insights into the process.
As a large national reference lab, ARUP expects to soon be running a high volume of lab-based serology testing sent in from all over the U.S., according to Patricia Slev, medical director of ARUP's immunology core laboratory, where she and her team recently extensively validated a high-throughput serology test from Abbott Laboratories.
But the Abbott serology test has not yet been granted EUA, Slev noted in an interview.
So, not only does a CLIA-certified lab like ARUP need to validate it extensively, it is also somewhat unclear whether labs are then required to notify the FDA in any way if they are using a test from a manufacturer who has already notified FDA. "It is a real question in the lab community," Slev said.
Having EUA of high-throughput serology tests is urgently needed, she said, because it is likely a challenge for smaller hospitals and labs to do the types of validation studies required.
ARUP is also looking into a strategy of running confirmatory serology tests to accommodate for the low positive predictive value associated with low prevalence, and it is looking into tests from multiple manufacturers. As a national reference lab, it would not be feasible to determine the local prevalence for each sample sent in, Slev said.
Generally, the evolving FDA guidelines have also been challenging to stay on top of, particularly the Pathway D track that differs from the typical EUA process of the past. Access to positive serum or plasma samples needed for validation can also be a challenge for a lot of labs, Slev noted.
At the Medical College of Wisconsin, Nathan Ledeboer, concurred that sourcing specimens to conduct a thorough validation, particularly for cross-reactivity studies, has been a challenge.
Even more challenging, however, has been obtaining a sufficient supply of reagents within a time frame that was compatible with the increasing demand for testing, Ledeboer said in an email.
With his team at MCW, Ledeboer is now evaluating serology tests from several manufacturers. The majority are ELISA and automated immunoassays, he said.
"Earlier in the outbreak we evaluated one lateral flow test and found the specificity of the test to exceed 90%, but the sensitivity was 45%," Ledeboer said.
Thus, the lab is now limiting its evaluations to vendors that either have EUA or are in the process of obtaining EUA. It is also focusing on tests that have existing data to support performance, particularly independent data, and it is only looking at IgG tests.
"We are also expecting a significant level of demand, so our decisions are also being driven to methods that can support the volume of several thousand tests per day," Ledeboer said.
He has found the FDA's approach to be helpful in that it likely weeds out really bad tests. However, he added that the lab "recognized very early that broad and comprehensive validation would be required" for any serology test that it might begin using.
For molecular testing, the Mayo Clinic's Rochester location got EUA for a molecular test, while the Mayo Clinic's Arizona site had run the Centers for Disease Control and Prevention EUA test, then switched to an Abbott test. For serology, the Rochester site is validating plate-based assays, and Mayo-Arizona is doing the same, according to Thomas Grys, director of microbiology at Mayo Arizona.
But Grys said the lab also wants to bring on lateral flow tests since having a validated point-of-care test could potentially allow the health system to reach more patients, including healthcare workers and employees. The lab is currently in the validation process, beginning with the first step — trying to figure out which of the tests that are currently available might be any good, and also reliably available in large enough quantities.
That said, "the guidance for how to use these tests, and what the results mean, is definitely evolving," Grys noted. The fact that notification of validation is the only requirement for Pathway D is "a very low bar," he said.
"Everyone appreciates that there are efforts being made to make tests available, but that puts the onus on clinical labs — that are already busy—to figure out what's good and bad," he added.
While, Grys said he thinks the U.S. emergency regulatory strategy in the COVID-19 crisis will likely be debated for years to come, right now, "just trying to get our hands on some of the lateral flows has been difficult."
Companies seem to be trying to supply hard-hit areas first, and other countries as well, some of which have also endorsed certain tests.
To select tests that it could attempt to validate, Grys said his lab started with the FDA EUA list, and consulted preprints and colleagues, with the reputation of the company being a factor, as well.
Grys said he has been in touch with eight manufacturers and expects to actually get tests from half of those. He said that some companies wanted specimens from his lab to validate testing themselves, and making arrangements for that right now is difficult. Others have told him they are having shipping problems, or that their supply chains are disrupted. And some of the tests require a reader, which can lead to pricing differences that instigate another set of tradeoffs.
The lab is also considering the likelihood that it will need to validate a second serology test as a confirmatory test, but both tests would probably have to be validated together. "Operationally, it is another nightmare — do you need to confirm by a different antigen, or will any second test do?" Grys said. "Does it have to be plate based test, or another lateral flow? And you would have to validate the two as a pair — is [the second test] the same sample, or is it a new draw?"
Furthermore, in the current emergency, manufacturers are developing tests, but the designs may not be completely finalized, Grys said. He received some kits from a manufacturer who told him it was still kind of tweaking the test, but that he could have the first batch. The positive samples looked fine, but negative samples – including plain buffer alone – also tested positive, presumably due to some cross-reactivity in the reagents. Grys alerted the company and it is now reformulating.
"That's where some of these companies are at — they're like, 'We'll give you kits but we kind of want to know what you find, because if it's not working in your hands then we're going to change it.'" In this context, it is tough to publicly report results, or know whether if the lab chooses to go with a test, that the scenario will repeat itself, he suggested.
Word of mouth between lab managers is a source of leads on accurate tests, but Grys said he has not yet seen any specific kit being recommended enthusiastically. Admittedly, it is also hard to know if any companies have the capacity to scale testing, so there is a possibility that labs aren't sharing preliminary good results for fear that there will be a spike in demand and they won't be able to get the test after all the validation work.
Point-of-care lateral flow tests are still considered high-complexity tests if they have not been reviewed in an EUA application, and some test makers who notified FDA under Policy D are marketing them saying they are authorized. "There has been a lot of confusion for everybody," Grys said.
Some Policy D serology tests also have product inserts showing they have tested 60 to 100 samples, which Grys said is not that extensive. Coupled with the many FDA-mandated disclaimers, "it is hard to look at that and feel like you're impacting patient care in a positive way," he said.
"For me it's almost like a few years ago when direct-to-consumer tests were 'for entertainment purposes.'
That's almost where we're at, and it's like, if we're going to be there, then why are we doing this? What is the advantage of having allowed all of this?" he said.
