Consumers Union, perhaps best known for its ratings of household electronics and cars, became one of the latest to expand a hospital comparison Web site with the release earlier this month of how often hospitals fail to prevent costly, potentially deadly and avoidable infections.
How do they measure up?
Quality-reporting efforts advance but still uneven
Most hospitals, however, won’t be found on the newly expanded Web site. Patients won’t find information for roughly three out of four U.S. hospitals and some states are excluded entirely. That’s because the publisher of Consumer Reports cobbled together its data from 10 states that publish mandated infection reports or from hospitals that voluntarily disclose such data to a Washington-based not-for-profit organization focused on quality improvement: the Leapfrog Group.
Even as measures of hospital quality and safety expand, the Consumers Union’s recent effort underscores the challenges that continue to dog the nation’s halting push for useful public comparisons of hospital performance.
Health policy experts say quality reporting efforts have made gains in recent years, as public and private efforts have flourished, but have yet to broadly deliver the outcome and value measures considered most relevant to consumers and payers—thanks to uneven reporting, limited data and evolving research to develop measures themselves.
Further progress, they say, will depend on higher expectations for quality disclosure, investment in information technology to improve data collection and continued research to identify accurate and relevant measures of outcomes and efficiency.
“I think we’re all eager for the day when groups of doctors and hospitals realize they would be best off fully disclosing their performance information and using information as close to the bedside as possible,” says John Santa, a physician and director of Consumer Reports’ Health Ratings Center.
Despite the challenges, sources of quality ratings continue to grow. Consumers Union has twice expanded its Web site for hospital comparisons, first launched in May 2008, including using the most recent central-line bloodstream infection data from states and the Leapfrog Group.
The Commonwealth Fund went public roughly 14 months ago with online quality comparisons targeted toward hospitals and doctors. Anne-Marie Audet, vice president for the Commonwealth Fund’s program on quality improvement and efficiency, says the organization has also struggled to expand its quality Web site to include measures of preventable infections.
Audet says the health policy not-for-profit will report in March similar infection data to that reported by Consumers Union and is seeking to make public quality infection measures reported by states to the Agency for Healthcare Research and Quality. The AHRQ data are publicly reported at the state and national level, but not hospital-by-hospital, which is needed to effectively target quality improvement and identify the top-performing hospitals for the study, Audet says.
The Leapfrog Group, a decade-old quality improvement coalition of large employers, now operates in 46 states. The group recently expanded to include a coalition of employers in New Hampshire that will begin asking hospitals to voluntarily complete the Leapfrog survey in 2010. But the group was also forced to pull two measures on cesarean sections from its hospital quality survey after the data proved too cumbersome for hospitals to compile.
“It’s a very unfortunate statement” that measures so important to patients cannot be feasibly collected, says Leah Binder, Leapfrog’s CEO. She says consumers face widespread repackaging of existing measures and data and a limited number of players seeking to expand collection to measures not yet reported despite endorsement from the National Quality Forum.
HealthGrades, a publicly traded health ratings company, added organ transplant center outcomes measures to its Web site last November. HealthGrades used data already publicly available from the Scientific Registry of Transplant Recipients to launch the rating tool. Scott Shapiro, a spokesman for Golden, Colo.-based HealthGrades, says the value to consumers comes from aggregating multiple quality resources in an accessible format within a single Web site.
But as hospital comparison tools have grown, so has demand for standards to evaluate the validity of report cards’ quality measures, says Harlan Krumholz, the physician director of the 889-bed Yale-New Haven Hospital’s Center for Outcomes Research and Evaluation.
Comprehensive healthcare reform bills passed by the House and Senate, relegated to limbo after the Democrats’ Senate defeat in Massachusetts in January, call for the creation of national quality priorities, according to an analysis by the Henry J. Kaiser Family Foundation. The Senate bill would create a process to develop measures used for federal reporting and payment. Legislation in the House includes a center for quality improvement to develop and evaluate quality standards.
Krumholz was among the authors of the American Heart Association’s standards for assessing publicly reported outcome and health efficiency measures, issued in 2006 and 2008, respectively. Among the criteria endorsed by the association was detailed public disclosure of measures’ statistical methodology to address one critical issue for outcome rankings: adjusting for the severity of patients’ illness, or risk-adjustment.
Grading the graders
He has found such disclosure lacking from sources such as HealthGrades. “The graders need to be willing to be graded too,” he says.
Samantha Collier, executive vice president and chief medical officer at HealthGrades, says the company’s model relies on a widely accepted statistical method for adjusting data for risk—logistic regression—and cited an independent analysis published in the Journal of the American Medical Association in 2002 as a peer review of the company’s methodology.
Krumholz was one of five researchers behind the 2002 JAMA article, which was undertaken, the authors noted, because of the strong public interest in HealthGrades (with 1 million visitors to the Web site in 2001) but a lack of independent and public analysis of the company’s methods and performance.
The researchers compared HealthGrades rankings, which award stars on a scale of one (worst) to five (best), against hospital performance on seven measures of quality and survival for Medicare heart attack patients. Researchers also noted they did not have access to HealthGrades’ proprietary model for ranking hospitals and could not evaluate it directly.
The analysis found HealthGrades rankings broadly reflect hospitals with higher quality of care and lower risk of death. But the rankings were less successful when identifying performance of individual hospitals within each rating category. HealthGrades may best be used within the industry to spur quality improvement, but the rankings risk mislabeling individual hospitals with potentially significant consequences for consumers, the authors wrote in the article.
Heart attack patients’ survival 30 days after leaving the hospital was best among five-star hospitals and grew gradually worse with each drop in star rating. Overall, the difference in death rates within 30 days of leaving the hospital between HealthGrades’ top- and bottom-rated hospitals was 15.9% and 21.9%, respectively, according to the article.
Collier notes that the 6-percentage-point gap researchers found amounts to one life saved for every 17 patients transferred from a one-star hospital to a five-star facility. She says that no further peer-review articles on HealthGrades’ measures have been published since 2002, but that peer-review articles since then have supported the general statistical methods and data sources the company uses.
Results were more mixed on how well HealthGrades’ rankings reflected the quality of care heart attack patients received, the JAMA study found. Five-star hospitals were more likely to give heart-attack patients aspirin and beta blockers as they arrived and left the hospital than those treated elsewhere. But two- and three-star hospitals outperformed five-star hospitals on the use of acute reperfusion therapy.
And when researchers tested the findings with alternate statistical models to look at individual hospital performance, rather than performance of all hospitals in each ratings category, the findings suggested “ratings do convey some important information in aggregate, but provide little meaningful discrimination between individual hospitals’ performance in a manner sufficient to a public interested in making informed hospital choices,” the authors wrote.
Collier said that all rankings will misclassify some hospitals because of the limits of statistical models and described the company as an influential pioneer in the public reporting of hospital mortality rates. She says the company relies on experts across the country to review methodology.
The risk of publicly mislabeling hospitals derailed Medicare’s first effort to report hospital-specific death rates, first released in 1986 and ultimately halted in 1993 after sharp criticism over the methods used to calculate the figures from the American Hospital Association and American Medical Association. Bruce Vladeck, who was then administrator of the Health Care Financing Administration, the predecessor agency to the CMS, pulled its reporting, telling the New York Times that the agency’s methodology was “overly simplistic.”
“There was a real backlash from the hospital industry and very real concerns about the underlying data and risk adjustment,” says Janet Corrigan, president and CEO of the National Quality Forum, of what she describes as a pioneering effort at public hospital quality reporting.
Quality reporting has since improved and expanded, Corrigan says, but experts continue to struggle to identify valid data and measures that allow for accurate comparisons of performance. “There is no perfect data source, and there is no perfect measure,” she says.
Corrigan says the push to measure results—or the outcome of treatments and therapies—faces challenges other than risk-adjustment. Hospitals, doctors and nursing homes aren’t solely responsible for the outcome of patients’ care, she says. How patients care for themselves after treatment can determine whether they return to the hospital or regain their health. Not all patients understand doctors’ orders, and others do not closely follow them. And poverty can put medication or nutritious foods, needed to maintain health, out of reach for others.
One challenge facing architects of quality measures is to identify measures most closely linked to providers’ performance, Corrigan says. Another is to distinguish among the growing number of measures those most relevant to patients and quality improvement. The NQF has launched an ongoing review of its more than 600 endorsed measures and those in the pipeline for how adequately measures address the quality of care for the 20 most common medical conditions, she says.
Corrigan says she expects public reporting to grow increasingly useful for consumers who seek to make informed decisions. Composite measures, indicators of quality outcomes and cost, will give patients and payers information needed to compare the value of care, she says.
The Pennsylvania Health Care Quality Alliance, a voluntary quality organization in the state that includes hospital trade groups, major insurers and the Pennsylvania medical society, reports composite measures for how closely providers followed recommended treatment for heart attacks, heart failure, pneumonia and infection prevention. The NQF, which has endorsed measures for roughly a decade, approved its own criteria for endorsing composite measures last July.
Helen Burstin, a physician and the NQF’s senior vice president for performance measures, says several shifts are under way in the quality-reporting effort, including the development of composite measures.
Quality experts are increasingly seeking outcome measures and trying to identify process measures that most significantly influence outcomes (those that are “closer to where the actual action is,” she says). More than 160 of the NQF-endorsed measures track outcomes. “We have definitely seen growth,” Burstin says. Another 65 outcome measures are under review, along with new measures of resource use.
As such outcome and value measures increase, the challenge becomes to identify and package them in ways that are useful and meaningful to patients, Burstin says. Quality experts are now considering how to report measures for related services patients receive over time, just as policymakers are seeking to bundle payments to providers for groups of related services.
“We’re really just beginning to scratch the surface,” she says.
Send us a letter
Have an opinion about this story? Click here to submit a Letter to the Editor, and we may publish it in print.