Health systems are moving from pilots to investments after determining which generative artificial intelligence products work best for their needs.
Generative AI has become a mainstay in healthcare more than two years after OpenAI launched the public version of its ChatGPT large language model. Health tech companies including Microsoft, Abridge, Ambience Healthcare and others are using generative AI to create ambient clinical documentation tools that turn a recording of a doctor-patient conversation into usable clinical notes for the electronic health record.
Related: Inside the competitive ambient AI vendor space
In a survey of health system executives, Deloitte found more than 40% said their organizations have achieved some return on their investments in generative AI, while 37% said it is too early to know. For the early adopters, there is a lot of optimism on the technology's potential.
Here is how three health systems went through the decision-making process before adopting generative AI.
Cleveland Clinic
Many health systems have tested multiple AI documentation vendors with physicians across primary care and multiple specialties. But few systems have been as committed to a bakeoff as Cleveland Clinic. The 23-hospital health system ran separate pilots with five vendors last year.
“We did five pilots and each of them was about three to four months long. Each one of them was with about 30 physicians,” said Chief Digital Officer Rohit Chandra. “We had more than 150 physicians participating in these pilots. We surveyed the physicians. We looked at the continued usage. We looked at time metrics and after-hour usage metrics, time and tool metrics, quality of notes metrics. We looked at a variety of different things just to assess how well the product was doing.”
Chandra said there was so much excitement over the technology's potential that the system had to turn away doctors who volunteered to participate in the pilots. Tech-centric physicians, who could understand the clinical and technical elements of AI, led the effort. All participating physicians checked in regularly and the leadership team compared notes.
At the end of the process, the leadership team chose Ambience because it envisioned working with the vendor over time, Chandra said. Cleveland Clinic, which declined to share how much it paid for the pilots and for Ambience, expects the product to improve physician morale and reduce the burden of documentation.
Chandra said doctors are waiting to get licenses to the technology and a systemwide rollout is expected to take place over the next three months.
Denver Health
Denver Health, a safety-net system, had the funds to do only a single pilot with one vendor, Nabla. The company went through an interview process with 10 vendors, evaluating cost and which of the companies were already integrated with Epic's electronic health records system. During a two-month pilot with Nabla, the most affordable option, the health system tested the product in 6,000 patient encounters.
The system has moved forward with Nabla for systemwide AI documentation. The projected return on investment was a major factor in the decision-making process, said Dr. Daniel Kortsch, associate chief medical information officer.
The system declined to share the cost of Nabla.
“Our organization is a safety-net and it’s really cost-conscious,” Kortsch said. “It is vital for us to recognize the value of this and to choose a vendor that can be appropriately priced. We had talked to partners around the country, other federally qualified health centers, some of which had attempted to use some of the higher-cost vendors and had to turn it off simply because they couldn't afford it in the long run.”
Kortsch said he initially asked the approximate 50 clinicians participating in the pilot to increase their productivity by a very small margin so the product's cost was offset by revenue gains. The requirement was later pared back and the tool’s success in making physicians happier justified the investment, he said. After the pilot, Denver Health surveyed the pilot's participants and 83% said Nabla would increase their desire to maintain clinical hours because it gave them more time to interact with patients, and patients liked it too, Kortsch said. He said he expects the technology will be a selling point as the system recruits physicians.
There is still room for improvement as the AI tool learns to produce an output that's more closely aligned with how physicians document during a physical exam, he said. The system will start using it in outpatient settings before moving it to the emergency room and inpatient care.
University of California San Diego Health
University of California San Diego Health has serious ambitions with AI. The health system hired a chief AI officer in December 2023 and tests numerous AI tools at its Jacobs Center for Health Innovation. It was an early adopter of using generative AI for physician-patient messaging when it worked with electronic health records vendor Epic and Microsoft on a five-month pilot in 2023.
The results of that pilot, published by the Journal of the American Medical Association’s Network Open in April 2024, were somewhat mixed. The generative AI tool didn’t save physicians any time, but it did reduce cognitive burden, said Dr. Christopher Longhurst, the system's chief medical and digital officer.
“It’s a lot easier to edit than it is to draft something de novo,” Longhurst said.
The lessons learned from the pilot will help UCSD Health as it decides on an AI documentation vendor. It is piloting tools from Microsoft and Abridge in a six-month process that will end this fall. Longhurst said 200 clinician will help test the tools.
While developing the generative AI messaging tool was relatively inexpensive, adopting a documentation vendor is more costly, Longhurst said. He is skeptical health systems will benefit financially after putting millions of dollars into these tools.
“I think AI and healthcare is probably going to change healthcare more than anything since antibiotics, but at the same time, it's not going to happen overnight,” Longhurst said. “We’re a risk-averse industry. We're dealing with people's lives. We need to be careful and thoughtful. Organizations like ours that have the resources to actually do this testing are obligated to be out front and sharing our lessons learned with others, so that people aren't making wasted investments.”
