Artificial intelligence has been touted for years as the next big thing in improving patient outcomes. But health systems haven't quite seen that promise materialize.
While there are thousands of research papers either introducing potential new AI models or validating products already on the market, there's a dearth of literature on how hospitals can actually put these tools into long-term use, and system leaders are mostly left to figure it out piecemeal and hope for the best.
"There's no centralized, coordinated fashion, in terms of vetting the products and putting them into practice. It's happening in silos," said Mark Sendak, population health and data science lead at the Duke Institute for Health Innovation. "Every health system is developing their own way of doing this, and it's our hypothesis is that there's probably a lot of really valuable learnings to be shared."
Software AI of all types, whether it's designed to flag sepsis early, predict if a patient will deteriorate within 24 hours or detect lung nodules, come with risks of unintended harm. The mechanisms a hospital puts in place to monitor AI can make all the difference when it comes to patient safety.
"The question is, 'Will the algorithm inadvertently cause harm and bias patient care?' because if it's missing patients, it's leading to a delay in diagnosis," said David Vidal, regulatory director at the Mayo Clinic's Center for Digital Health. "But then, they [health systems] don't know how to figure that out."
Vidal, Sendak and a group of researchers from the University of Michigan, Duke University and elsewhere have put out a call for health systems to publish reports on their experiences implementing AI in healthcare settings that others can use as guides for best practices. Frontiers, an open-access journal, plans to publish 10 such case studies next summer.
But while one hospital might have great success with AI software, another might fail. The nitty-gritty details of patient populations, how clinicians are instructed to use a tool and how it's built into processes ultimately shape how successful it can be.
UnityPoint Health, a multistate, not-for-profit system based in West Des Moines, Iowa, encountered the limitations of technology when it set up an AI early warning system for sepsis infections. The hospital chain empowered triage nurses to identify potential sepsis cases without using the software—and the humans caught more infections early than the AI did.
"It was unfortunate from an AI perspective, because I think the default thought is that these models will just walk into healthcare and revolutionize it, and that's just not an attainable goal right now," said Ben Cleveland, UnityPoint Health's principal data scientist. "We're talking about augmenting the intelligence of some of society's most educated and trained people. It's definitely going to be a process of figuring out where AI can meaningfully inform clinical decisions or operational decisions."
The landscape for healthcare software AI is massive, and most products do not currently go through the FDA clearance process as medical devices. Vendors have sought and earned FDA approval for only around 100 products as of late 2020.
"Practices and institutions need to be able to understand the software that we're picking and a little bit about how it was trained and how it was validated, to give them some understanding about whether or not they think that it'll work in their practice with their patient population," said Bibb Allen, chief medical officer of the American College of Radiology Data Science Institute and a diagnostic radiologist in community practice in Birmingham, Alabama.
At an October FDA workshop on AI transparency, Allen recalled the story of Cincinnati Children's Hospital and Houston's Texas Children's Hospital reporting software algorithms that missed diagnoses. The hospitals suspected that the model hadn't been configured to process pediatric data and imaging, he said. Stakeholders asked for greater clarity about how software is designed to avoid these kinds of missteps.
"It would have been nice to know or to have a warning that the model wasn't really trained for children," Allen said.
There are several routes that industry might use to make AI software more transparent and easier to use post-implementation. Hospitals could create algorithmic software stewardship programs, akin to initiatives to prevent overuse of antibiotics, for example.
In the future, hospitals may establish governance committees charged with choosing AI tools or creating formularies or vetted products along with insurance companies, said Dr. Nigam Shah, associate chief information officer at Stanford Health Care in California.
"They have a group of people who decide if it should be used, and what the guidelines are for when to use it," Shah said. "If the industry fails to self-regulate, then 10 years later, the government will eventually crack its whip.
Software companies themselves should be responsible for making sure their products are appropriately designed and used, said Suchi Saria, CEO and founder of software vendor Bayesian Health.
"It's the vendors job to bring the right kind of approaches to the table so that they're showing rigorous granular data on efficacy, adoption, engagement and safety," Saria said. "I don't expect a health system to suddenly become an expert in building the right monitoring infrastructure, but I think they need to understand the metrics for success."
The FDA may indeed broaden its oversight of healthcare AI software and establish guidelines for what vendors must do, Saria said. The agency currently doesn't exercise jurisdiction over software that's embedded in electronic health records systems.
Nebraska Medicine in Omaha has a team in place to evaluate AI tools, but the not-for-profit system still largely relies on word of mouth and other health systems' reported experiences when choosing software. For every new product, there are clinical experts who look at workflow and information management for individual units. And there are still barriers when trying to get clinicians to actually use the information the software generates.
"There's going to be a set of systems that really figure this stuff out, and they're going to accelerate rapidly in how they're using artificial intelligence, and then there's going to be a subset of systems that can't possibly keep up with that pace," said Justin Birge, Nebraska Medicine's director of provider informatics. "I'm hoping that's part of what this sort of initiative from Michigan and others are doing, is how we extend some of these developments and some of these success stories to smaller places."
