EHR vendors fall short on usability testing

Despite federal rules meant to ease the use of electronic health-record systems, many developers of popular EHRs are falling down on reporting and meeting federal design requirements, according to research published in JAMA..

A team of researchers from the National Center for Human Factors in Healthcare at MedStar Health in Washington, D.C., found that not every vendor filed required

">reports

on their usability testing. The report adds to the mounting concern that EHRs are failing to raise the quality and safety of healthcare and lower its costs.

The MedStar researchers reviewed reports on usability test results from 50 of the most commonly used EHRs, which are required as part of the $31.3 billion federal EHR incentive payment program. The MedStar team found that vendors of nine of the 50 EHRs in their sample (18%) had no public report on usability on file with the Office of the National Coordinator, which oversees the testing and certification regime.

Of the 41 vendors whose usability reports were on file, roughly a third failed to state the type of user-centered design process as required for certification, according to the researchers.

Although not an ONC rule, the agency has endorsed a National Institute of Standards recommendation that at least 15 participants engage in user-centered design testing. But in the MedStar sample, 63% of the vendors with reports engaged fewer than 15 participants in end-user testing.

In addition, 17% of vendors with a report on file identified no physicians participating in usability tests of their computerized physician order entry (CPOE) systems. Two vendors in the sample used some of their own employees to do the testing.

The MedStar researchers said more than 90% of the provider attestations of meaningful use under the EHR incentive payment program from April 2013 through November 2014 were from users of the 50 EHRs in their sample.

The MedStar group focused on user testing of developers' CPOE functionality because that component of an EHR is most likely to be used by physicians or nurses and it is one that “presents significant safety hazards when not designed well,” the study authors said.

“The lack of adherence to usability testing may be a major factor contributing to the poor usability experienced by clinicians,” the study authors concluded.

The ONC became aware some vendors had failed to file usability reports earlier this year, according to Alicia Morton, who leads the IT certification program at the ONC. Morton issued a two-page guidance (PDF) in May that gave the organizations 90 days to get their test reports up to date.

The memo warned that future EHRs without the required design reports will be rejected for certification.

Last fall, the American Medical Association targeted what physicians perceive as generally poor usability of EHR systems as part of a broader campaign to improve physician satisfaction with the profession. A pair of recent surveys indicated that physicians continued to express dissatisfaction with EHRs on ease of use and interoperability.

To address provider complaints and patient safety concerns about EHRs, the ONC in 2014 said user-centered design processes must be applied to each of a dozen key EHR capabilities, including CPOE, electronic prescribing, medication lists and drug-to-drug and drug-to-allergy interactions.

But Raj Ratwani, corresponding author of the study and the scientific director at the National Center for Human Factors in Healthcare, criticized vendor processes that relegated user-centered design to the end of development, testing what are essentially finished products. The government should have been clearer in its guidance on what it wants vendors and independent testing and certification organizations to do, he said.

The study suggests industry efforts to incorporate user-friendly design into its development process have fallen short. The Electronic Health Records Association, a trade group of nearly 40 top vendors, over the past two years has been working with its members on improving usability, according to EHRA Chairman Leigh Burchell, vice president of policy and government affairs at Chicago-based EHR developer Allscripts. “We continue to hold education sessions for EHRA membership on UCD methods and techniques.”

Ross Koppel, an adjunct professor of sociology at the University of Pennsylvania who studies hospital workplace culture, said the MedStar research supports his view that the federal EHR incentive payment program is encouraging vendors to rush EHR software into use without proper testing.

“Vendors treated usability in the most haphazard way,” Koppel said. “The fact that usability is so primitive is a detriment to patient safety and clinician satisfaction.”