Peer networks drive software decisions by hospital CIOs

University Health System—a 617-bed health system based in San Antonio—recently kicked off a systemwide electronic health record installation, scrapping its current Allscripts EHR in favor of one from Epic Systems Corp. It’s a high-priced project: between expenses such as purchasing the EHR, training personnel and retiring outdated systems, University Health is putting $170 million toward the transition. 

The breadth of Epic’s client base played a central role in its decision, particularly as the academic health system’s primary partner, the University of Texas Health Science Center at San Antonio, has used Epic for several years. The majority of other public health systems in Texas are also live on an Epic EHR, which University Health hopes will improve local interoperability. 

“Data-sharing was a driver,” said Bill Phillips, senior vice president and chief information officer at University Health.   

Despite vendors spending untold marketing dollars pitching their products’ standing on rankings produced by Black Book Research, KLAS Research and others, CIOs seem to be more driven by what’s happening in the market and good-old-fashioned peer networking.

“I’ve been in the industry for a long time, and I know a lot of the people,” said Chuck Christian, chief technology officer at Indiana-based Franciscan Health, describing one of the key steps he takes when considering a new software purchase. “So I’ll pick up the phone and call people who use the same software.”

Software purchases aren’t a decision to take lightly, as implementing a new system—such as an EHR—can involve millions of dollars. More than 90% of acute-care hospitals report being live on one or more EHRs, but hospitals often want to consolidate multiple products onto one system. That’s becoming a common case with rising rates of mergers and acquisitions.

Given the multimillion—if not multibillion—dollar stakes, it makes sense IT leaders would turn to the experiences of their peers. Nearly two-thirds of hospital executives indicate that either they or their peers have felt their job security was in jeopardy during an EHR transition, according to a seven-year survey of hospital leaders conducted by Black Book. At the manager level or above, 5% of respondents claimed they or a peer had been fired or asked to resign because of EHR replacement costs or productivity issues.

“With as much money as (new software) costs, those can be career-limiting decisions if you make the wrong one,” said Christian, who joined Franciscan Health April 8. He spoke with Modern Healthcare while at his previous role, vice president of technology at the Indiana Health Information Exchange.

Lisa Grisim, associate CIO at Palo Alto, Calif.-based Stanford Children’s Health, also highlighted the perspective she gains from the CIO community—with an electronic twist. IT leaders at Stanford Children’s have moved many of these conversations online, by participating in group email lists with their colleagues.

One of the electronic mailing lists Grisim is most active in is dedicated to CIOs at pediatric hospitals, which is coordinated through the Children’s Hospital Association.

“We’re very connected within the healthcare IT industry, particularly among specialty and high-acuity children’s hospitals,” Grisim said. “We’ll regularly send out questions about what people are doing in ERP (enterprise resource planning), PACS (picture archiving and communication system), or whatever it might be.”

This focus on peer-to-peer connections may, conceivably, give a leg-up to industry leader Epic.

Epic and Cerner are virtually neck and neck in deployment of EHRs, according to the Office of the National Coordinator for Health Information Technology. Epic’s website boasts that all 20 of U.S. News & World Report’s top-ranked hospitals in 2018-19 used its EHR, as well.

Add that to clinical staff’s familiarity. When Rochester, Minn.-based Mayo Clinic decided to implement a new EHR roughly five years ago, it searched for a system that could manage the academic medical center’s clinical, billing and revenue-cycle systems across all of its sites. Bringing in vendors to perform usability and functionality tests with hospital staff also played a major role in its final decision to choose Epic.

“Because Epic is so heavily represented in large academic medical centers, many of our newer staff, residents, fellows, have practiced in an Epic medical center,” said Dr. Steve Peters, co-chair of the Plummer Project, Mayo’s nickname for the EHR installation. Mayo brought its final sites live on Epic in 2018, more than a year after launching the systemwide project in July 2017.

But those rankings from research groups and input from consultants are not totally discounted. Grisim said Stanford Children’s uses the annual “Best in KLAS” rankings as part of its decisionmaking process, either before developing a request for proposals or after receiving responses, as a quick quality check.

KLAS Research’s rankings have been dogged over the years by questions about its methodology and a perception of pay to play. The 2019 “Best in KLAS” rankings were based on interviews that the market research firm’s analysts conducted with C-level executives, managers and physician leaders at 4,500 hospitals and 2,500 clinics, but many of the reports the firm releases are based on much smaller samples. In a report on patient engagement published in February, Epic was listed as the top patient portal solution based on interviews with representatives from just 37 customer organizations.

Epic scored 92 on the firm’s 100-point provider satisfaction scale. Athenahealth was the second-highest vendor listed in the report, with a score of 84.8, based on interviews with representatives from 30 customer organizations.

KLAS will flag when it has limited survey responses on a solution that’s included in a report, leaving a note that reads “insufficient data.” For most of the provider software tools that KLAS evaluates, insufficient data means that the firm surveyed 15 or fewer customer organizations. Taylor Davis, executive vice president of strategy and development at KLAS, said these thresholds are based on confidence intervals. 

“At 15 interviews, the confidence interval for somebody’s KLAS score tends to be plus or minus about 8 points,” Davis said. “If KLAS reports that your product is a 76 on the 100-point scale, you might be as low as the high 60s—it’s unlikely—or you might be as high as the low 80s, but you’re probably somewhere in the 70s. In about 15 interviews is statistically the first point, if we’ve done our job and randomly sampled, that we can start to say ‘Here’s where you are, generally.’ ”

The firm’s target for sample sizes is higher, in the high 20s or 30s. At 30 interviews, the confidence interval goes down to plus or minus 3 to 4 points, he added. “We include (sample sizes) in every report,” Davis said. “We’re really open about what we do on our reports, and we know providers look at those (sample sizes) and take the feedback based on what they see.”

KLAS has also been forced to tackle criticism that its rankings use a pay-to-play formula. KLAS addresses the concern on its website’s “frequently asked questions” section. The organization charges vendors fees to purchase its reports and to subscribe to KLAS membership—as a member, a vendor has access to the firm’s data and the opportunity to meet with analysts for feedback into its solutions. KLAS says that these payments do not affect a vendor’s performance score. 

Allscripts CEO Paul Black said the company paid KLAS $455,000 in 2018. That includes subscription fees, which KLAS determines based on a vendor’s annual healthcare revenue, as well as costs associated with attendance at KLAS’ summits. 

“I believe that KLAS should disclose how they get paid, just like industry analysts disclose if they have a position in a company,” Black said.

When asked to comment on how Epic—this year’s “Best in KLAS” overall software suite—works with KLAS, including whether it had made payments to the firm in 2018, an Epic spokesperson said: “As with all vendors, we are charged an annual subscription fee to get access to the data on their website, and the price is based on our revenue.” 

A Cerner spokesperson declined a request for comment, saying it’s the company’s corporate policy to not disclose contract amounts with partners due to a reasonable assumption of confidentiality between the parties.

Many hospital leaders, such as Grisim, said they review KLAS’ reports for access to the firm’s research—the free-form responses providers give to the firm’s survey questions, which KLAS includes at the end of its reports—in addition to the rankings. That’s par for the course, according to Davis.

“Providers report to us overwhelmingly that the most helpful research that we provide to them is the data and customer comments,” he said. It’s “the perspectives from their peers.”

For these hospital IT leaders, the reports offer a pulse-check on their colleagues, adding perspectives that they might not have been able to reach directly. “It’s things you want to hear about from others who have used the technology you’re not currently using,” said Dr. John Halamka, CIO at Beth Israel Deaconess Medical Center in Boston.

“I read these things to understand what others have experienced, but I’m a bit careful not to jump to concrete conclusions if sample sizes are small,” he said.

There are many resources a CIO might consider—peer perspectives, market research, third-party consultants. But there’s no single, magic bullet that hospital CIOs can rely on. It’s about weighing all of this input across the board.

“It’s looking at the KLAS research, picking up the phone and calling others you know in the industry, looking at (industry) blogs,” said Christian, who runs his own health IT opinion blog called the Irreverent CIO. “You have to take all of those as grains of salt as you piece the story together.”

RELATED CONTENT

Checklist: 6 steps to a new EHR purchase

Largest number of certified EHR products in use, 2019