Kenneth Dickie still shudders at the memory. One day in 1979, someone snuck into his secret office in the basement of the VA Washington Medical Center. The intruder stacked piles of patient records around Dickie’s DEC minicomputer, doused them with a flammable material, and set them on fire. Smoke filled the room, but fortunately for Dr. Dickie and for the future of American health care, an alarm went off in time, and the computer he was using to build the country’s first practical electronic medical record system was spared. Still, Dr. Dickie recounts today, he had to have the engine of his car rebuilt several times during this period because someone kept putting salt or sand into the gas tank.
Dickie was one of the Hardhats who developed what is today known as the VA’s VistA software program. VistA, which stands for Veterans Health Information Systems and Technology Architecture, is actually a bundle of nearly 20,000 software programs, most of which were originally written in the 1970s and 1980s by individual doctors and other professionals working secretly in VA facilities around the country. These pioneers had to do their best to hide their work from their superiors because it violated VA policy and was threatening enough to elements within the VA to provoke literal sabotage. But eventually, working without a plan and without a leader, these dissident doctors would wind up creating a wonder of “bottom-up” engineering that many experts say points the way to the future of twenty-first-century health care.
Today, after a long bureaucratic war that still leaves some of its developers congregating in online support groups, VistA has radically transformed the practice of medicine within the VA and made possible a new model of health care now being emulated around the world. This unique, integrated, publicly owned information system, written by doctors for doctors, has dramatically reduced medical errors at the VA while also vastly improving diagnoses, quality of care, scientific understanding of the human body, and the development of medical protocols based on hard data about what drugs and procedures work best.
The story of how VistA first came to be is inspirational on many levels. For one, it is a shining example of a time when the “Dilberts” of the world won, and their hidebound bosses were humiliated. Indeed, one of the ironies is that if the VA’s leadership hadn’t been so moribund for so long, the revolution that led to VistA would probably never have happened. A more savvy leadership at the VA probably would have contracted out with some private software developer to provide its information systems. The most likely result would have been computer programs imposed on, instead of created by, doctors and other medical professionals, costing billions of dollars, and written in a buggy proprietary code that ordinary users would have no ability to improve, modify, or integrate.
This story is familiar in the world of American health care, where what few electronic medical information systems are in place often inspire resistance and fail. That’s what happened, for example, at Cedars-Sinai Medical Center in Los Angeles, which in 2003 turned off its brand-new, computerized physician order entry system. Doctors complained that it took five minutes or more to log into the system and to enter the patient and medication data needed to fill a prescription. At least six other hospitals have shut down computerized drug-dispensing systems in recent years.1
But precisely because of its ossified traditions, the VA avoided this path. When its management failed to deliver workable information technology, the happy, if unintended, result was that various VA employees took it upon themselves to solve their own individual programming needs. Their individual efforts eventually created a highly effective hospital information system that remains unrivaled by any healthcare software developed by the private sector.
In 2003, the Bush administration’s top man at the Centers of Medicare and Medicaid Services, Thomas Scully, chastised private software developers for failing to come up with programs that could even begin to match the performance of VistA—let alone its price. VistA is open-source software, meaning that the code itself is free to anyone who cares to download it off the Internet and is accessible to individuals who care to modify it for their own purposes or to improve its performance. (Check out the demo at http://www.ehealth.va.gov/EHEALTH/CPRS_Demo.asp.) The only function VistA can’t do as well as its private-sector counterparts, at least without adding some code, is tracking patient billing. Instead, because of its origins, its focus is on patient care—something the private sector just can’t seem to imitate.
VistA’s origins lie in the late 1970s. Like most large institutions of the era, the VA had committed to large, centralized, mainframe computers, such as the IBM 650 Magnetic Drum Data Processing Machine, which it had been using since the 1950s for administrative purposes. These machines were jealously guarded by a tight circle of “high priests,” working out of the VA’s central offices and its main computer center in Hines, Illinois, who regarded anything involving bits and bytes as their exclusive preserve.
Predictably, as with many other institutions of the time, the software these high priests wrote, or more often procured from private vendors, wasn’t very good, in large part because the people who actually had to use it had little role in its development. Among the many scandals that dogged the VA in the 1970s was the poor performance of its information systems, which at one point in early 1976 completely broke down, causing 647,000 checks to veterans to go unwritten or to arrive late.2
Nor were the high priests, whose fiefdom was known as the Office of Data Management and Telecommunications (ODM&T), much better at developing software with medical applications. One project ODM&T embarked on, which was supposed to provide doctors with a computer system they could use in laboratories, began in 1968 and wasn’t ready for deployment until 1982. Just completing the VA’s seventeen-step bureaucratic process for approving new software typically took a minimum of three years of paper shuffling on top of whatever time the actual writing of the program required. In 1980, the high priests estimated it would take them at least ten years to develop even a rudimentary patient treatment file that could be stored in the VA’s mainframes.3
But as it happened, this was the dawn of the era of mini-and personal computers, and a handful of technically minded doctors sprinkled throughout the VA began experimenting with writing their own software to meet their various needs. One was Kenneth Dickie, an internist at the VA Medical Center in Washington DC, who, in an attempt to simplify and improve his own working conditions, began working on a DEC minicomputer in the hospital’s basement to develop a program that would combine lab results, patient history, and other data into a single electronic medical record. “It was unbelievably difficult to track down paper records,” he recalls today, “and unbelievably difficult to track down the data I wanted in those records.” In this era before laptops and wireless modems, Dr. Dickie’s idea was that doctors and nurses could use a single minicomputer on each ward to update, retrieve, and print out complete patient records.4
Meanwhile, Gordon Moreshead and Wally Fort in Salt Lake City began developing a clinical psychology data system to use in their own facility. Bob Lushene in St. Petersburg, Florida, developed online psychodiagnostic tests; Richard Davis in Lexington, Kentucky, was writing a nutrient analysis program for the treatment of diabetics; and Joe Tatarczuk in Albany, New York, was working to computerize nuclear medicine.5
Two other key players were the late Joseph (Ted) O’Neill and Martin E. Johnson. Both had been part of early government efforts to explore the potential of information technology in the practice of medicine. In late 1977, they found a new and precarious perch within the VA’s Department of Medicine and Surgery (forerunner of today’s VHA) and began working out of a small office, cryptically labeled “Computer Assisted System Staff,” from which they conspired to build a network of programmers within the VA who came to be known as the Hardhats. In December 1978 in Oklahoma City, O’Neill and Johnson managed to pull off a meeting of freelancing programmers within the VA and persuaded them to write in a common, user-friendly language and to share their code. But everyone had to be careful to work under the radar of those who controlled the VA’s centralized mainframes, even if it meant writing code under difficult conditions.
For example, many of the freelance programmers were forced to work on “word processors” that lacked tape drives. This was because buying a true personal computer, let alone one of the era’s minicomputers, would have, as one participant relates, “set off alarm bells in the Central Office back in Washington.” Programming on a Wang computer designed for secretaries made sharing information and updating software very difficult. The only ways to do it were with error-prone 300-baud modems or by physically carrying disk packs the size of cake trays from one site to another—a process some characterized as “committing portability.”6 Another key programmer, George Timson, worked out of San Francisco by remote access (“quite unauthorized and quite unpaid-for,” he states) with a Massachusetts firm to develop an elegant and highly effective file-sharing protocol that would become the heart of VistA.7
Yet, soon enough, the Hardhats ran into trouble from the high priests who manned the VA mainframes. Many Hardhats were fired or demoted; others had their computers confiscated. According to Timson:
In one case, in Columbia, Bob Wickizer went to lunch, and found, when he got back to his computer room, that his new PDP-11/70 [a minicomputer made by Digital] had been unplugged and was in the process of being crated. By all accounts, the machine never again processed another instruction, anywhere. The Enemy had won—or so it seemed.8
The turning point finally came in late 1981. By then, on orders from the central office, personal and minicomputers had been ripped out and locked up in closets where doctors couldn’t get to them. The VA’s central office had ordered a radiology system developed by Hardhats in Columbia, Missouri, to be shut down. It had also pulled the plug on a pharmacy system under development in Birmingham, Alabama, and another one in Albany, New York.9 When word leaked out to academic researchers of a promising patient discharge program developed by VA employees in Oklahoma City, the central office refused even to acknowledge its existence in public.
Stunned by these developments, many doctors and other medical professionals who used Hardhat software and saw its value at last broke out in open rebellion. The controversy, which burst into the medical trade press and caught the attention of Congress, finally caused the VA’s Chief Medical Director, Dr. Donald L. Custis, to take a field trip to the VA’s Washington medical center on North Capitol Street to see what all the fuss was about.
This facility is only six miles away from the VA’s central office on Vermont Avenue near the White House, but in those days it was also a world away. This is where Kenneth Dickie, joined by Marty Johnson, labored secretly in the basement, developing electronic medical record software. It was also where another key ally of the Hardhats, the late Paul Schafer, practiced surgery while also serving as executive director of the National Association of VA Physicians. When Custis arrived at the hospital, its director, A.A. Gavazzi, told him straight off that the Hardhats enjoyed “100 percent” support from the hospital’s doctors.
Custis observed all the homemade software systems in use and also all the programs clandestinely imported from other Hardhat strongholds around the country. These included programs that recorded drug prescriptions, printed pharmacy labels, analyzed psychological tests, maintained tumor registries, and much more. All were running on a DEC PDP 1134 minicomputer that Custis’s office had expressly forbidden use of for such purposes. But despite the obvious insubordination, Custis came away impressed. “It sounds like an ‘underground railway’ has been at work,” he was heard to say, “and doing good work.”10
Some Hardhats started calling themselves members of the “underground railroad” and even had business cards printed up with a drawing of a steam engine. But soon, this railroad was underground no longer. Swayed by Custis’s report of what he’d seen and fed up with the recurring problems with the VA’s formal computer division, the new Reagan administration’s top appointee to the VA, Robert P. Nimmo, and his deputy, Chuck Hagel (the future U.S. senator from Nebraska), signed off on the Hardhats’ initiatives and pulled the plug on the high priests.
Key allies in Congress, such as Rep. G.V. (Sonny) Montgomery, concurred. A conference report that would later take on historical irony noted that “any further delay in proceeding with the decentralized … system is not justified and will only result in VA’s medical computer system falling further behind the private health-care industry.”11 Hundreds of high priests got riffed—government parlance for “laid off.” And within a short while, recalls Hardhat Richard Davis, “many highly labor-intensive and error-prone systems of daily operations within the VA medical centers were dismantled.” The Hardhats had won.
Fortunately, all the different programs that became VistA were written in an easy-to-use, common language that lent itself to integration and file sharing. This meant that they could all be fit together, in literally less than a week, into a central module. Eventually that module grew to include more programs so that, for example, all the different forms of care a patient received, in all different parts of a hospital, as well as in clinics, could be combined into a single electronic health record.12 The benefits in coordination, patient safety, and adherence to evidence-based medicine have been so extensive that by 2007, the VA realized an estimated $3.7 billion in cumulative savings net of the costs the Hardhats incurred in developing VistA.13
Maybe this still does not seem like such a big deal. Even today, the potential of electronic health records to improve the practice of medicine is only beginning to become apparent to the public, or even to many private-sector health-care providers. But individual doctors practicing within the VA were already in many ways living in the world of the future. As early as the 1970s, the population they served, which was dominated by veterans of World War II, was aging rapidly, much as the U.S. population as a whole is now beginning to experience a rapid increase in the number of elders.
This meant that VA doctors in the 1970s, like doctors everywhere today, were seeing increasing numbers of patients beset with complicated, chronic conditions such as diabetes. These conditions, which were often accompanied by numerous comorbidities such as high blood pressure and cardiovascular disease, required constant monitoring and coordinated care involving dozens of people—specialists, nurses, radiologists, lab workers, physical therapists, counselors. The nature of these chronic diseases also demanded that patients become vitally involved in their own care, such as in measuring their own blood sugar levels, and that a system be in place for keeping track of such measurements.
The comparative frailty of the population served by the VA also made patients exceptionally vulnerable to medical errors, such as different doctors prescribing dangerous combinations of drugs. The advancing age of the veterans population also put a premium on record keeping that could quickly pinpoint who, for example, was due for a flu shot or a prescription refill. Since patients approaching the end of life often consume high volumes of expensive treatment, VA doctors and administrators also had an exceptional need for data about which of these treatments worked better than others and, indeed, about which didn’t work at all.
For all these reasons and more, the environment in which VA doctors were practicing medicine in the 1970s and ’80s made the value of electronic health records and other information technology easier to see than in many private health-care settings. Tellingly, Kenneth Dickie found his inspiration for developing electronic medical records while trying to contend with his caseload of VA nursing home patients in the 1970s.
A final and all-important consideration was that the VA as an institution maintained a near lifetime relationship with its patients. This meant there was a pressing institutional need to coordinate record keeping among the many different VA hospitals and clinics a veteran might use over his or her lifetime. And, crucially, it meant that any improvement to the quality of care the VA could achieve through its investment in information systems would rebound to its own long-term advantage. Managing diabetic care properly, for example, meant fewer expenditures for costly amputations down the road and would even help save on nursing home costs, for which the VA was potentially liable.
By contrast, in private-sector health-care settings, where patients typically move on to another plan every few years, investment in preventing long-term complications more often than not brings no return to the institution. Thus, from a very early date, both VA doctors and administrators were far more likely than their private-sector counterparts to see the value of investing in information technology that could improve the practice of medicine.
As Timson recalls,
By the mid-’80s everybody wanted everything. We finally kind of broke out of our illegitimate status as garage operations in different parts of the country and proved that we could put the pieces together. And then by the middle of the ’80s we were building complete hospital information systems, using, of course, hardware that was laughably limited compared to the PC that’s on your desk today.14
Private-sector vendors repeatedly pressured Congress to make VA doctors and technicians stop writing software. But VA doctors argued persuasively that there was no product available on the market that could compete with their own, user-made system.15
Since then, the growth in computer power and the emergence of the Internet, far from making VistA obsolete, has allowed it to grow still more capabilities. The original software is still in place in most facilities, but it is continually updated electronically with patches that fix bugs or add new features. Today these include electronic medical records containing X-rays, pathology slides, video views, scanned documents, cardiology exam results, wound photos, dental images, and endoscopies. The code that makes all this possible isn’t elegant by today’s standards, but it is stable and time-tested, and it works just the way someone trying to practice state-of-the-art medicine would want it to work.
To be sure, political appointees at the VA, often under the influence of proprietary software vendors and threatened by VistA’s decentralized Hardhat culture, have not always constructively supported the software’s continuing modernization. Especially during the late years of the Bush Administration, the VA’s political appointees began outsourcing health IT projects to private vendors, and shutting off innovation in the field. But the VA has more recently taken the important decision to keep VistA in open-source code, which means that the original vision of VistA as a program “written by doctors and for doctors,” still lives on. “The beauty of VistA,” says former Hardhat Greg Kreis, “is certain parts of it were not engineered in the early days in the classic top-down kind of design; it was more of a bottom-up design. What it may have lost in its engineering, it gained in its relevance.”