(Story updated with comment from Leapfrog.)
Just weeks after the Leapfrog Group released a controversial scorecard
assigning a letter grade for overall patient safety to more than 2,600 U.S. hospitals, the American Hospital Association is criticizing the group's data, calling it biased, inaccurate and unfair. NEW:
Leapfrog CEO Leah Binder responded in an interview that the AHA appears to be digging for ways to discredit the grades after hearing from members that didn't score well. "The old thing is kill the messenger, and this is the new version: kill the methodology," Binder said.
The Leapfrog Group, a Washington-based healthcare quality-improvement group formed by large employers, released its Hospital Safety Score
on June 6. The group said it based each hospital's letter grade—A, B, C, D or F—on performance on 26 quality measures, covering areas such as medication safety, infection prevention and patient falls.
But in a two-page letter, AHA President and CEO Richard Umbdenstock (PDF)
pointed to what he says are a number of “methodological shortcomings,” in the Leapfrog scorecard, including unreliable measures and the use of weighting across different hospitals. Most importantly, the AHA said, the Leapfrog Group favored its own survey when scoring hospitals.
“Specifically, for two of the scored questions (presence of a computerized provider order-entry system and intensivists in the intensive-care unit), those who provide information by responding directly to the Leapfrog Group's survey can earn up to 100 points for a fully compliant system, while those whose responses are derived from 'secondary data' can earn a maximum of only 15 points,” the AHA said in its comments.
Among its other concerns, the AHA said the scorecard uses a number of measures that have been shown in studies to be unreliable metrics of quality, including nine that are currently proposed for retirement from the CMS' Hospital Compare website. NEW:
Binder said the AHA seems to misunderstand the methodology, although she said the trade group was briefed on the scores and given an opportunity to ask questions in the weeks before the results were posted to the Web.
In cases where data from the Leapfrog survey was not available, for example, those measures were not factored into the score, Leapfrog responded in a letter to Umbdenstock
that takes on the criticisms point by point.
Most of the negative responses Leapfrog has received from hospitals have come from ones that weren't given a grade at all because the organization was unable to collect enough data for a reliable score, Binder said. "We have been really impressed with how constructive hospitals have been about this score," she said. "And that's what we wanted."