Most of the hospitals that tested the core clinical indicators of the Joint Commission on Accreditation of Healthcare Organizations' clinical indicator monitoring system found them costly and ineffective aids for improving care.
Those findings are among data reported in JCAHO internal documents.
The JCAHO's board of commissioners approved many of the same indicators for use in the JCAHO's indicator system, known as the IMSystem, which accredited hospitals will be required to use as early as 1996. JCAHO officials said information in the documents supported, rather than undermined, the efficacy of the indicator project.
MODERN HEALTHCARE obtained copies of the internal documents last week.
The documents contain the preliminary results of an extensive survey of hospitals that tested three sets of clinical indicators developed by the JCAHO.
The board had a summary of the results prior to its deliberations over the indicators, said Deborah Nadzam, director of indicator measurement for the JCAHO.
"Raw research data should never be presented to a decisionmaking body," Ms. Nadzam said.
The indicators attempt to measure the quality of care in trauma, oncology and cardiovascular cases, and are considered to be the core elements of the IMSystem.
When the IMSystem becomes fully operational, it will cover up to 30 indicators in 10 clinical areas. Accredited hospitals will collect the raw indicator data, send them to the JCAHO for analysis and get back their results and comparative data from hospital peer groups.
The JCAHO hasn't publicly released the results of the "beta" test phase of the trauma, oncology and cardiovascular indicator sets, although they're slated to be added to the IMSystem next year during the second year of the voluntary phase of the project.
To date, fewer than 300 hospitals have signed up for voluntary IMSystem participation. The JCAHO had expected 2,000 hospitals to sign up by year-end.
The testing phase of the three indicator sets ended in 1993, and the JCAHO surveyed the test hospitals in October 1993. In questionnaires as long as 39 pages, the JCAHO asked hospitals about the indicators' usefulness and costs.
Some 109 test-site hospitals commented on the 12 proposed trauma indicators, 121 commented on the 11 proposed oncology indicators, and 121 commented on the nine proposed cardiovascular indicators.
The majority of hospitals said most of the indicators in each set didn't help them identify an opportunity to improve patient care or organizational processes, according to the documents obtained by MODERN HEALTHCARE. Even fewer took actions based on the indicators to improve care or procedures, the documents said.
For example, only 8.1% of the hospitals in the cardiovascular group said the clinical indicator on heart bypass surgery deaths identified an opportunity to improve care or procedures. Only 6.2% used the same indicator to take action to better care (See chart).
In the trauma area, only 14.1% of the test-site hospitals said the clinical indicator on the timeliness of computed tomography scans for head-trauma patients identified an opportunity to improve care or processes. Only 9.4% used the indicator to take corrective action.
In oncology, just 8.1% of the hospitals said the clinical indicator on the accuracy of diagnosing a specific type of lung cancer identified an opportunity to improve care or processes. Only 4.1% used the same indicator to take any action to improve care.
The average "opportunity identified" score of all 32 indicators tested was 16.9%, and the average "action taken" score was 10.9%.
In May, the JCAHO approved 15 of the indicators for use in the IMSystem.
The JCAHO, though, views the survey findings in a much different light.
Ms. Nadzam said the most important thing to look at is how many of the hospitals that identified an opportunity to improve care actually took action to do so.
For example, of the 28 hospitals that said one particular trauma indicator identified an opportunity to improve care, 23, or 82%, took action, she said.
"We are quite pleased with the results," she said. "This is good news."
Beta-site hospitals that tested the first two sets of clinical indicators-one in obstetrics and the other in anesthesia (now called perioperative)-had similar dim views of the usefulness of the indicators (March 14, p. 30).
Meanwhile, other internal JCAHO documents reveal that the cost of using the indicators may be much higher than the JCAHO has suggested in its marketing materials to hospitals.
For example, hospitals' median data collection and entry cost of using the trauma, oncology and cardiovascular indicators varied from 6 cents to 26 cents per case. However, the median cost per case jumps to $12 to $14 "if the costs of case finding, tracking accuracy and quality of data, interpretation, use, staff education, training, supervision, and telephone bills are included as well," the documents said.
In marketing material for the IMSystem, the JCAHO said the median costs per case for the obstetrics and perioperative indicators were $2.04 and $1.65, respectively, at the beta-site hospitals.