There's no handbook or shortcut for examining the embedded microchips in medical devices for signs of millennium flaws.
But if not done with care -- and a good sense of what can be tinkered with and what shouldn't be -- those analyses could make things worse instead of better.
"It's down-and-dirty, serious work," says Dan Forrester, whose testing teams at St. Joseph Health System have gotten the routine down as well as can be expected after checking medical devices at seven of the Orange, Calif.-based system's 16 facilities.
Just finding a possible offending chip is a job, because embedded processors often are "hidden under layers and layers of (circuit) boards," says Forrester, a 26-year veteran of the biomedical device industry with experience working for manufacturers, providers and the military.
Once a chip is isolated, the problem of testing may be only the beginning. Sometimes a microprocessor can't be directly interrogated by test equipment or the results of its electronic thinking judged by probing eyes.
"Most people can't test down to the embedded level," says Anthony Montagnolo, vice president of technology planning at ECRI, a not-for-profit technology assessment firm based in Plymouth Meeting, Pa. "If you just do a cursory test, you might kid yourself."
At St. Joseph, the testing involves consulting with a vendor's design engineer to size up the inner workings and come up with a proper way to see how it runs when warped ahead to 2000.
Unlike computer software, which allows the source code controlling an operation to be called up and picked apart, microchip programs are closed affairs, says Kenneth Kleinberg, research director for the healthcare division of the GartnerGroup, a Stamford, Conn.-based information technology research firm.
Though no one can actually see or get access to the microcode, Kleinberg says, diagnostic programs can be run to determine what's inside, much like a doctor does a workup on a patient.
But chip testers, like doctors, have to be careful about which tests they do, because some can be intrusive or even harmful.
A microprocessor that's built to do one basic thing may not be able to get back from the future after the testing is done. "You're taking it down a path that it hasn't taken before," Kleinberg says.
Some devices can't be tested, and others shouldn't be, Forrester says. St. Joseph's testing teams have been able to examine 54% of mission-critical inventory; for the rest, they're stuck with vendor reports.
"Our feeling is that we test whatever we can test," says Carol Barnett, coordinator of year-2000 efforts at PeaceHealth in Bellevue, Wash. "But where you can't see a date or don't know (a microprocessor) is there, . . . you have to rely on the vendor."
Some vendors come right out and tell inquirers about not only a device's year-2000 problems but also additional problems that would be caused by attempts to test it. "I respect a vendor for sharing that," Forrester says.
Of the 10 laboratory specimen analyzers St. Joseph flagged as millennium-buggy, four were identified from internal testing and the rest as a result of vendor notification that its lab products have flaws and that testing them would create a bigger problem, he says.
Looming as another obstacle in the remediation routine is a lack of answers to the problems, even the minor ones.
"A lot of this will be able to be fixed relatively easily," says Jack Beebe, who's directing the medical device dragnet at Catholic Healthcare West, a 37-hospital system based in San Francisco. But he's concerned about the dearth of solutions so far, because very few manufacturers have released fixes.
It may be tempting for a seasoned technician to go in and fix a device, but that could be risky, too. For example, St. Joseph pros could well have debugged one piece of equipment by replacing a $21 chip instead of spending thousands of dollars on repair or replacement. But the prospect of liability made them pull back, Forrester says.
"How much liability have you just assumed by doing that? Now you've just remanufactured a piece of equipment," he says.
St. Joseph's progress to date roughly parallels manufacturers' progress in assessing and testing equipment, but a firm sense of the remediation expense will be difficult until all the vendors have spoken on remedies. "I've got a lot more blank spots than figures right now, and I expect that to continue until the end of the year," Forrester says.
When testing started, it took about three hours to test a mission-critical device and about 41/2 weeks to complete testing at a hospital. Now that engineers have gotten the hang of it, the routine is down to an average of about an hour per device and about 11/2 weeks per hospital, he says.
An original dedicated staff of seven was expanded recently to 12 full-time computer engineers plus two managers. "This is critical," says Forrester about the hirings. "You've got to have the right people in front of your machines."
The dedicated staff is supplemented by 30 support staffers in the corporate office and more than 100 people working on teams at each facility. The system teams cross-train the facility crews so that they can continue monitoring and fixing devices after the principal team moves on.
No formal budget for the effort is expected until the end of August, when a better picture of remediation costs emerges, says Sharon Carlson, the system's year-2000 project manager. "We've decided it's silly to budget when you don't know what you're budgeting for," she says.
In addition to the repair and replacement expenses, there's a labor cost of implementing the remedies, training staff on new equipment, monitoring the performance of devices after they're installed or revised, and documenting all the remedies.
Like the bill for car repairs, the labor costs can be substantial, she says. For now, she added, what counts is a commitment from the St. Joseph leadership to spend what has to be spent to get that far.