The U.S. Food and Drug Administration this week published the first results of a limit of detection comparison of molecular diagnostic tests for COVID-19.
Reported as nucleic acid amplification test-detectable units (NDU) per milliliter, the limits of detection for the first batch of 55 tests reported differed by 10,000-fold.
In a virtual town hall on Wednesday, Timothy Stenzel, director of the FDA's Office of In Vitro Diagnostics and Radiological Health, said the agency hopes the reference panel information will be helpful.
Stenzel said the FDA assessed relative limit of detections for tests that use viral transport media (VTM) and are extracted, as well as for tests that run from dry swabs and, separately, saliva tests.
FDA grew its own SARS-CoV-2 virus in culture to create the panel, then inactivated it, and tested that it was inactivated, Stenzel said. The agency began shipping the reference panel to diagnostics developers in May.
At the lowest end, or best-performing is the PerkinElmer New Coronavirus Nucleic Acid Detection Kit, with a reported limit of detection of 180 NDU/ml.
A test from ScienCell Research Laboratories reported 540 NDU/ml, while tests from Hologic, DiaCarta, BioCore, and Seasun Biomaterials had limits of detection of 600 NDU/ml.
The listing then jumps to 1,800 NDU/ml. Tests from CirrusDx Laboratories, Euroimmun, Helix OpCo, LabGenomics, Quest Diagnostics, Rheonix, and Roche fell into this category, as did one of the tests offered by Becton Dickinson.
The Centers for Disease Control and Prevention 2019-nCoV Real-Time RT-PCR Diagnostic Panel reported an LoD in the evaluation of 18,000 NDU/ml.
At the high end of the spectrum, three tests — from Diatherix Eurofins Laboratory, Luminex Corporation, and Qiagen — reported limits of detection of 180,000 NDU/ml.
In validation studies for the EUA, the test demonstrated 100% sensitivity, while a reference hospital in France using the test in an ER setting recently showed 100% sensitivity in an assessment of 69 clinical samples, the spokesperson said. A test of labs by the American Proficiency Institute also reported 100% sensitivity for five labs using the Qiagen that were challenged with a sample of containing 5,175 SARS-CoV-2 viral copies per milliliter.
The Qiagen spokesperson pointed to a disclaimer from an FDA statement that notes while a lower limit of detection represents a test's ability to detect a smaller amount of viral material in a given sample, "signaling a more sensitive test," the data "does not indicate how sensitive a particular test is."
Two direct-from-swab tests were reported separately. Abbott's had an result of 300,000 NDU/ml, while the Quidel's was 540,000 NDU/ml. There was a single saliva-based test in the dataset from Fluidigm, which had an LoD of 54,000 NDU/ml.
Alex Greninger, a virologist at University of Washington, said that at first glance the results don't completely square with his group's published evaluation of the Hologic, DiaSorin, Roche, and Cepheid assays, which found Cepheid's test to be the most sensitive.
The PerkinElmer test's secret sauce may have something to do with the extraction, Greninger speculated. "They're not extracting a ton of material but they use all of what's extracted, so the PCR sees a good amount of VTM," he said.
Masoud Toloue, vice president and general manager of diagnostics at PerkinElmer, commented in an email that the firm's achievement of the best performance is directly related to three key factors.
The test uses "highest-quality magnetic bead and PCR raw materials, a stringent manufacturing and quality control system for RT-PCR reagents, and, most importantly, years of our team's experience and expertise in developing NAT tests for blood screening," Toloue said.
In addition to listing LoD results, the agency is also listing tests as "Data not returned," "Data not interpretable," "Under interactive review," and "Did not provide shipping information."
In a statement on its website FDA said that as of Sept. 10 the agency had contacted developers of 176 EUA assays for shipping information and had sent the reference panel to developers of 152 assays.
"The FDA is reviewing results as they are returned, and continues to send the reference panel out to additional developers," the agency said.
Stenzel noted that the agency is working on finalizing LoD determinations for more tests and will post the next batch as soon as they are ready.
The agency said that developers who received the reference panel were asked to conduct testing and return results within two weeks of receiving the panel.
FDA noted that many developers returned data to the agency by Aug. 31, 2020, but in some cases, it did not receive the data, or the data was uninterpretable, or is still under interactive review.
One lab and one firm in the "under interactive review" category commented that this likely means the FDA is still reviewing the data that was already provided rather than that there is some back-and-forth process going on.
Ithaca, New York-based Rheonix, had an LoD comparable to tests from BD, Quest Diagnostics, and Roche. Richard Montagna, senior vice president for scientific and clinical affairs, commented that the firm is pleased that FDA has created a reference panel that all manufacturers can test against, so that laboratories can evaluate the relative analytical sensitivities of the currently available tests.
"Up until now, it has been difficult to compare tests on an 'apples-to-apples' basis," Montagna said.
Chris Mason, a genomics researcher at Weill Cornell Medicine and scientific advisor at Resilience Health, said his group will incorporate the FDA panel results into its analyses of EUA tests.
The team published results of its comparisons of FDA Instructions for Use documents in Nature Biotechnology, with the top assays lining up fairly well with those in the FDA's evaluation. Namely, the PerkinElmer test had the lowest limit of detection in this comparison, as well, and the top results included tests from BD, Roche, Quest, and DiaCarta, as well as ViraCor, Ezno, Abbott, and BGI.
"There is clearly a wide range of performance for various SARS-CoV-2 tests on the market, including multiple orders of magnitude difference in the LoD, plus varied methods for sample collection, extraction, priming sites, and overall processing," Mason said.
"The FDA work, our own work, and related benchmarking efforts at [the National Institute of Standards and Technology (NIST)], X-Prize, and the FDA will help adjudicate the limitations of any tests and help confirm that primers in use for diagnostics are valid," he said. Mason also commented that evolutionary selection pressure on the primer sites can lead to loss of primer reliability, and so, "Frequent validation is helpful to ensure that our target sites for primers are still efficacious."
In the town hall, Stenzel commented that as soon as an international standard for molecular testing is available, the agency will "anchor the FDA reference panel to that, and we will be able to translate the results we posted to international units."
Stenzel said, "We think that will be a huge advancement on top of what we've already done."
This story first appeared in our sister publication Genomeweb.
