CMS' hospital star-ratings system changes will be modest

The modest changes, slated to be announced in December, are likely to disappoint some hospital leaders who hoped the CMS would nix its current model entirely.

"That is what we are going to be looking at, are there ways in which we can attenuate that significant impact for some hospitals so it's not quite as dramatic when a measure changes significantly," the official said.

The agency announced last month it would delay the July release of the star ratings after hospitals expressed concern over changes.

The CMS defended its current methodology, which relies on the latent variable model. The agency said the model works well because it's flexible enough to "accommodate changes in measures over time," the official said.

The latent variable model is a statistical approach that gives more emphasis to certain measures over others in the star ratings based on a number of aspects, including variation in performance among hospitals for that measure or how much measures correlate to each other.

The emphasis the model gives to each measure isn't decided by the CMS, but done strictly by the model.

"The advantage of this approach is it doesn't include some of the more arbitrary decisions that would happen if you were to apply an explicit weight" to each measure, said Dr. Arjun Venkatesh, a scientist at the Center for Outcomes Research and Evaluation at the Yale School of Medicine.

The CMS contracted with Yale to develop the methodology. The selection of the methodology involved feedback from consumers, hospitals, public-reporting experts and methodology experts.

The model is used to determine how hospitals perform in seven categories: mortality, safety of care, readmission, patient experience, effectiveness of care, timeliness of care and efficient use of medical imaging. The three outcome groups—readmissions, safety and mortality—are each weighted the most at 22% each, as is patient experience.

Each of the categories have a certain number of measures that the model will assign a loading coefficient, which determines how much emphasis each measure gets in each category. For example, in the mortality group, the model considers seven measures that make up 22% of the total group.

In the first four iterations of the CMS star ratings, the PSI-90 measure accounted for about 98% of a hospital's performance in the safety of care category because that is where the model likely saw the most variation in performance compared to the other measures in the domain, so the model assigned it the heaviest loading.

"I wouldn't necessarily see that as a good or a bad thing," the CMS official said.

The PSI-90 measure is a composite measure that looks at a variety of safety issues like pressure ulcers and sepsis rates, which can affect a lot of patients. While the CMS doesn't select which measures receive more emphasis, the agency believes PSI-90 rates indicate overall quality of care more accurately than other measures in the category.

However, in the proposed June release of the star ratings, the latent variable model, for the first time, assigned about 98% of a hospital's performance in the safety of care category to complication rates from hip and knee replacements. This change dramatically changed how a subset of hospitals performed on the star ratings, with some scores dropping or rising by two stars.

The CMS official said the agency was aware of the change "right away," even before it sent hospitals the preview report in May.

The agency determined that more emphasis was likely given to the measure for complication rates from hip and knee replacements because of the significant changes that have happened to the PSI-90 measure over the past two years.

The proposed June release accounted for the fact that coding for PSI-90 changed from ICD-9 to ICD-10. Additionally, the Agency for Healthcare Research and Quality recently removed some measures from PSI-90 in response to stakeholder concerns about the measure.

"It's not surprising that when you update a measure that hospital performance is going to change, and therefore the star rating is going to change," the CMS official said.

Although the CMS official said she wasn't surprised the change occurred, the agency is going to convene a technical expert panel to determine "statistical applications we can use within the existing methodology that can attenuate the impact" that measurement changes can have on hospitals' overall star ratings. The official added, "It wouldn't mean changing the methodology."

The CMS stands by the hospital star ratings, she said. "We believe it's still rigorous, accurate and reliable; and the kind of changes we saw that led to the delay, that could've been expected."