CMS star ratings disproportionately benefit specialty hospitals, data show

The CMS' latest star-ratings formula disproportionately rates specialty hospitals better than major teaching hospitals, according to a Modern Healthcare analysis of quality performance data.

The CMS updated its star ratings in December with a new formula intended to improve accuracy and fairness. But a study of the most recent CMS data by consulting firm Sullivan, Cotter and Associates and Modern Healthcare show that the majority of specialty hospitals received five-star ratings—the highest rating a facility can receive—while a small number of major teaching hospitals received the same recognition. Of the 74 specialty hospitals that received a star rating in December, 45—or 61%—received a five-star rating and 16—or 22%—received a four-star rating, according to the Sullivan Cotter analysis. By comparison, of the 172 major teaching hospitals that received a star rating from the CMS, just 15—or 9%—got a five-star rating and 25—or 15%—received four stars. The analysis also found that specialty hospitals only reported about half of the measures the CMS considers to assign ratings and the measures reported were weighted higher under the methodology. Experts say the findings highlight consistent issues with the CMS' star-ratings methodology. "There is a clear bias here towards the critical-access hospitals and the specialty hospitals and much of it is driven by the methodology in terms of how the measures are aggregated into a single rating," said Andrew Wilson, research team leader at Altarum's Center for Value in Health Care, of the results. Hospitals care deeply about the star ratings even though it's unknown how often they are used by consumers. There haven't been any studies on how many Medicare beneficiaries use the Hospital Compare site, where the star ratings are featured, and the CMS said it's unable to determine accurately how many Hospital Compare visitors are beneficiaries. The American Hospital Association, the Federation of American Hospitals and America's Essential Hospitals all campaigned strongly against publishing the star ratings in 2016. The main concern for the organizations was the methodology, which it characterized as "flawed," saying that it "unfairly" penalizes teaching hospitals or those who serve large poor populations. Even though hospitals opposed publishing the star ratings, high performers tout their success in news releases and news stories and say the recognition is an indicator of quality care. The methodology Under the CMS' latest star-ratings formula, the agency removed winsorization, a tactic in which extreme hospital outliers in the data set were trimmed to minimize their effect on the overall ratings. Instead, the CMS runs k-means clustering "to complete convergence." This involves clustering hospitals repeatedly into five groups—or star categories—until the hospitals in each group are more similar to each other and different enough from the other four groups. The new formula addresses concerns that hospitals in different star categories performed similarly on the measures. The change slightly flattened the usual bell curve of hospitals that receive stars on a scale of 1 to 5, with 5 being the highest rating. In the previous methodology, few hospitals received one or five stars, and the majority have two, three or four stars. While that's still the case, the number of hospitals with one or five stars rose. Although the CMS changed that part of its methodology, the quality measures used and the weight applied to those measures weren't altered. All in all, hospitals can report up to 57 measures to obtain a star rating. These are divided and weighted into seven categories: mortality, safety of care, readmission, patient experience, effectiveness of care, timeliness of care and efficient use of medical imaging. The three outcome groups—readmissions, safety and mortality—are each weighted the most at 22% each. Patient experience is also weighted at 22%. Effectiveness of care, timeliness of care and efficient use of medical imaging are each weighted at 4%. To qualify for a star rating, hospital must report measures for at least three of the categories, one of which must be an outcome category like readmission or mortality. If a hospital doesn't report all of the measures, the CMS will weigh the submitted data more heavily. In December a total of 3,692 hospitals received a star rating. Critics of the star ratings, including the AHA, argue that how the measures are weighted skews the results so hospitals can appear to be better or worse performers than they actually are. "When you are looking at the star ratings, you really aren't getting as clear of a picture on the performance of the hospitals as you would want to," said Nancy Foster, vice president for quality and patient safety policy at the AHA. "You are getting this blend of measures that aren't adequately adjusted … you end up with this strange distortion." The CMS disagrees with critics, arguing that its star-ratings methodology "is not designed to advantage or disadvantage any type of hospital," a spokesman said in an email. He added, "Our analyses show that hospitals' star ratings are strongly correlated with the hospital performance on the individual quality measures." But the CMS data suggest that the specialty hospitals benefited from higher ratings by reporting fewer measures—particularly fewer outcome measures—than teaching hospitals. Differences in reporting Specialty hospitals that received a rating reported 27.2 of the 57 measures on average, according to Sullivan Cotter's analysis. Comparatively, major teaching hospitals that received a rating reported almost all of the measures, 51.37 of the 57 measures on average. The specialty hospitals were less likely to have the heavily weighted mortality measure considered in their star rating compared to the major teaching hospitals. Data showed the CMS didn't consider mortality for 71% of the five-star specialty hospitals. Jeff Softcheck, principal at Sullivan Cotter, emphasized that the specialty hospitals don't report all the measures because their patients don't present with those conditions. For instance, most specialty hospitals don't provide care to patients with congestive heart failure or stroke. By comparison, the 172 major teaching hospitals reported all of the measures needed to fulfill the patient experience, readmissions, mortality and safety categories. Because of how the methodology is set up, hospitals that perform below average on any of the four heavily weighted measures have to perform above average on the other three measures in order to get four stars or higher, Softcheck said. That can be hard for the major teaching hospitals because they are being compared to hospitals with much different patient populations, said Dr. Mark Rumans, chief medical officer at Sullivan Cotter. Major teaching hospitals, which typically treat complex conditions for patients with various socio-economic factors, have to compete against specialty hospitals that often work with insured patients on elective procedures. "If you end up at a major teaching hospital you are really sick … you are seeing multiple specialists, your length of stay is long," Rumans said. "Compare that to the experience at a specialty hospital, which has lengths of stay of one or two days. Those are such different levels of experience." The data show that 52.5% of the 40 four- and five-star major teaching hospitals performed below average on readmissions. Most made up for their score by performing above average in the mortality, patient experience and safety categories. Suggested changes Critics say a major problem with the star ratings is that it combines all hospitals together in the dataset and compares them all to each other. The AHA supports better risk adjustment so hospitals with similar patient mix or services are compared to each other. Others argue that the star ratings should display a hospital's performance on measures by specific condition so it's more helpful to patients. Right now, the star ratings can't tell a patient if the hospital is a good choice for a knee replacement because it combines all "these measures that aren't correlated with each other—that is the fundamental problem," said David Nerenz, the director of the Center for Health Policy and Health Services Research at the Henry Ford Health System in Detroit. Nerenz suggests that individual measures for individual procedures are displayed so consumers can decide what information is important to them. David Levine, senior vice president of advanced analytics and informatics at Vizient, also criticized the timeliness of the CMS data, which is mostly from 2016. "CMS is getting claims data all the time, they should be able to (update the data) on a more frequent basis," he said. The CMS said that it will continue to look for more opportunities to improve the star ratings and gather input from stakeholders. The AHA's Foster confirmed that the organization continues to talk to the CMS about ways to improve the star ratings. The CMS is expected to release updated star ratings in July. Editor's Note: Sullivan Cotter used its own methodology to categorize hospitals by five types: specialty, critical-access, small or medium community, large community or minor teaching and major teaching. The firm considers numerous sources to establish the groups including the American Hospital Association Guide. An edited version of this story can also be found in Modern Healthcare's March 19 print edition.