By the letter

A letter-grade system based on patient-safety measures has riled up hospitals that received low or average scores.

The Leapfrog Group, a not-for-profit that represents employer groups and purchasers, compiled hospital safety scores for 2,635 hospitals in the U.S. using publicly available data and information supplied by hospitals that participate in an annual Leapfrog Group survey.

The results—about 42% of hospitals earned a C grade or lower—triggered a wide range of reactions within the healthcare community. About a quarter of the hospitals received an A.

High-performing hospitals issued news releases promoting their A grades, while lower-performing hospitals, the American Hospital Association and other groups questioned the validity of the methodology and which measures were selected to compute the scores.

“We are, and have long been, in favor of sharing good reliable quality information with the public,” said Nancy Foster, vice president of quality and patient safety policy for the AHA. “That said, we find many of these measures to be unreliable and scientifically invalid, so we think this methodology needs a lot more work before it becomes something that the public should actually rely on to guide any of their decisionmaking.”

The Leapfrog Group brought in nine patient-safety experts to develop the hospital safety scoring system over a nine-month period.

The score is calculated using 26 measures and patient-safety indicators. Fifteen measures use data from the CMS or the Agency for Healthcare Research and Quality, and 11 measures were drawn from the Leapfrog Group's own annual hospital survey, which gathers self-reported information from participating hospitals at no cost.

“We recognize it's not perfect,” said Leapfrog Group President and CEO Leah Binder, “but I think we're all confident that we've given consumers a fair assessment of the relative safety of hospitals.”

Using data from both the CMS and the Leapfrog Group's own survey is one cause for concern. Rachel Sorokin, chief patient-safety and quality officer at Thomas Jefferson University Hospital in Philadelphia, said she is concerned that the scores for the process measures are weighted toward hospitals that participate in the Leapfrog Group's annual survey.

The 856-bed hospital, which received a C rating, has not participated in the survey. Sorokin said the safety scores are “primitive and not too accurate.”

Binder disputed concerns that there is a connection between hospitals that participate in the organization's annual survey and those that received better scores. “That is not the case, and I want to make that absolutely clear,” she said.

Many hospitals are confused about the methodology, which computes the safety score as a “z-score” that generates a number showing how far the hospital is from the mean, Binder said. It's not a point score.

However, the 904 hospitals that participated in the Leapfrog Group's annual survey generally earned better grades when compared with the total number of hospitals that received letter grades. About 50% of the survey participants received A grades and about 30% earned a C or lower, according to data supplied by the Leapfrog Group.

Hospitals can improve their scores in two ways: One, they can respond to the annual survey by Aug. 31 in an effort to improve their scores by November and, two, they can use the data to take a detailed look at their score and conduct an internal evaluation, a spokeswoman for the Leapfrog Group said.

“We used the best possible data,” Binder said. “What I would challenge hospitals to do is improve upon the best publicly available data. The better data we have, the better the score can be.”

Whether it was engaged consumers or concerned hospital CEOs, the site that is housing the safety scores brought in 50,000 unique visitors within two days of the launch and had about 500 visitors a minute at any given time on June 6. The site crashed several times due to high traffic.

Increasing consumer participation on hospital safety issues was another reason why the Leapfrog Group selected a letter-grade system; Binder noted the success that the New York City Department of Health and Mental Hygiene has had with providing letter grades to restaurants since the program has been in place.

“We want consumers to treat safety as one important factor before they choose a hospital,” Binder said. “Consumers don't pay attention to public reporting.”

The ECRI Institute, a not-for-profit organization that researches patient safety, expressed a different sort of concern with the grading system.

“It's entirely premature for any hospital to be getting an A grade on safety,” said Bill Marella, director of patient-safety reporting programs at the ECRI Institute.

Marella said the Leapfrog Group had made a “reasonable and fair attempt” in establishing the hospital safety scores but noted that the measurement of safety in a healthcare setting is an evolving field and will likely only improve when providers begin using electronic quality measures. The measures used to develop the hospital safety scores were not intended to compare hospitals, Marella added.

“The idea sounds great,” the AHA's Foster said, “but it's the details of making sure that the data are collected accurately and analyzed accurately and leads to a reliable comparison that can help inform someone's decisionmaking.”