The most expensive software project failure ever: $3.7 billion

The modern, affluent standard of living depends on society’s infrastructure, both physical and intellectual: airports, highways, bridges, ports, telecommunication networks, databases, power plants, and the software that makes it all run.  150 years ago it was still common for massive bridge projects to meet with catastrophic failure. The engineers of the 1800s slowly mastered the art of working with iron and then steel. The projects grew in scale, and the skills of the project managers needed to keep apace.

Nowadays we rarely hear of catastrophic bridge failures. This is a particular type of infrastructure that has come to be well understood, both in its engineering and in the project management that oversees its construction. However, we, as a society, are still struggling to find the right way to develop large software projects. This is the newest type of infrastructure, and the correct way to engineer it and manage its development is still a source of controversy.

Robert L. Glass has compiled a book of entertaining and instructive stories regarding the massive software project failures of our time: Software Runaways: Monumental Software Disasters. Anyone with an interest in the management of software projects should read it.

The worst of these project failures, the most expensive and the most ambitious, was the attempt made by the Federal Aviation Authority to modernize the computer system it uses to keep track of what planes are  in the air. The effort began in 1981 and ended in complete failure in 1994. The government hired IBM to do the actual work, and over the course of 14 years, IBM burned through $3.7 billion dollars. Nothing was accomplished. The project was finally shut down by Congress. Nothing came out of the project, not a single piece of software, nor even a line of code, was ever used for anything.

Robert N. Britcher, who was involved with the project, offers a full write up of the project. I will try to entice you to buy the book with some excerpts:

The Advanced Automation System began, in concept, in 1981 and ended in 1994, “terminated for convenience” by the government. Billions of dollars were spent on it. It is hard to describe. You can’t learn anything from the name. You know it’s about air traffic control because I told you, or because you read about it in the papers. Maybe part of the problem was the name. It sounds like the system to end all systems.

One engineer I know described the AAS this way. You’re living in a modest house and you see the refrgerator going. The ice sometimes melts, and the door isn’t flush, and the repairman comes out, it seems, once a month. And now you notice it’s bulky and doesn’t save energy, and you’ve seen the new ones at Sears. So it’s time. The first thing you do is look into some land a couple of states over and think about a new house. Then you get I.M. Pei and some of the other great architects and hold a design run-off. This takes awhile, so you have to put up with the fridge, which is now making a buzzing noise that keeps you awake at night. You look at several plans and even build a prototype or two. Time goes on and you finally choose a design. There is a big bash before building starts. Then you build. And build. The celebrating continues; each brick thrills. Then you change your mind. You really wanted a Japanese house with redwood floors and a formal garden. So you start to re-engineer what you have. Move a few bricks and some sod. Finally, you have something that looks pretty good. Then, one night, you go to bed and notice the buzzing in the refrigerator is gone. Something’s wrong. The silence keeps you awake. You’ve spent too much money! You don’t really want to move! And now you find out the kids don’t like the new house. In fact, your daughter says “I hate it”. So you cut your losses. Fifteen years and few billion dollars later, the old refridgerator is still running. Somehow.

At $3.7 billion, the Advanced Automation System was one of the largest civilian computer contracts ever; maybe the largest. It was the largest single contract in IBM’s history. From the moment it was awarded, until near the project’s demise, IBM patted itself on the back. There was something for everyone, beginning with a great ball in Union Station, featuring Chubby Checker and “The Twist”.

At its peak the project employed over 2,000 people. About a million dollars a day. If you thought like the IBM project manager, this was a good deal. Many people were working and money was being made. It was going to last forever. No one considered that it wouldn’t. And everyone was getting ahead. (One of the ironies of conceptual work is that it is easy to believe you are farther along than you are.)

The AAS must have been the most supervised project in history: this atop its enormous size and complexity, and the extreme and constantly changing requirements. One programmer described it this way: “Working on the project was like working on a car inside the garage with the motor running. Eventually, even the crickets hopping around the tires suffocate.”

What I saw on the FAA’s Advanced Automation System would have made Sisyphus weep… Whatever commitment and discipline there was… was worn down by a battery of watchfulness that I can only ascribe to fear of failure. In spite of tens of millions of dollars spent on new computers for AAS, the most important piece of equipment on the project was the overhead projector. There were endless meetings attended by dozens of people – as if we were never quite sure about the whole thing. The people in charge simply lacked the confidence and the finesse of the space team: NASA, contractors, and astronaughts.

It has been noted by everyone from the New York Times to the Vice-President of the United States that the main problem on the Advanced Automation System was “changing requirements”. For those involved in large-scale computer systems, that is nothing new. No one can perfectly surmise the shape and feel of a system years in advance. Even replacing some aspect of a system you know by heart is not immune from thinking twice about it. … [But] the requirements churn (it was called) on the Advanced Automation System was not normal. It was the result of our enchantment with the computer-human interface, the CHI. The new controller workstation, fronted by a 20″ by 20″ color display, because it was capable of seemingly endles variety of presentations, mesmerized the population of AAS like the O.J. Simpson trial mesmerized the nation…

The project was handed over to human factor pundits, who then drove the design. Requirements became synonymous with preferences. Thousands of labor-months were spent designing, discussing, and demonstrating the possibilities: colors, fonts, overlays, reversals, serpentine lists, toggling, zooming, opaque windows, the list is huge. It was something to see. (Virtually all of the marketing brochures – produced prematurely and in large numbers – sparkled with some rendition or other of the new controller console.) It just wasn’t usable…

The cost of what turned out to be a 14-year human factors study did not pay off. Shortly before the project was terminated a controller on the CBS evening news said: “It takes me 12 commands to do what I used to do with one.” I believe he spoke for everyone with common sense.

Rummaging through one of the closets at the far end of the hall on the fifth floor one day, looking for some standards document, I found an envelope left by someone who left the company – as many did after so many years advancing against stone, while the wheels of commerce were accelerating on what everyone referred to as “the outside”. It contained “A Brief History Of The Advanced Automation System”. It was printed by hand and left, perhaps inadvertantly, or perhaps with the hope that some anthropologist might some day discover it and make a pronouncement. In every important way, it is the truth:

“A young man, recently hired, devotes years to a specification written to the bit level for programs that will never be coded. Another, to a specification that will be replaced. Programmers marry one another, then divorce and marry someone in another subsystem. Program designs are written to severe formats, then forgotten. The formats endure. A man decides to become a woman and succeeds before system testing starts. As testing approaches, she begins a second career on local television, hosting a show on witchcraft. An architect chases a new technology, then another, then changes his mind and goes into management. A veteran programmer writes the same program a dozen times, then transfers. The price of money increases eight times. Programmers sleep in the halls. Committees convene for years to discuss keystroking. An ambitious training manager builds an encyclopedia of manuals no one will ever use. Decisions are scheduled weeks in advance. Workers sit in the hallways. Notions of computing begin in the epoch of A, edge toward B, then come down hard on A + B. Human factors experts achieve Olympian status. The Berlin Wall collapses. The map of Europe is redrawn. Everything is counted. Quality becomes mixed with quantity. Morale is reduced to a quotient, then counted. Dozens of men and women argue for thousands of hours: What is a requirement? A generation of workers retire. The very mission changes and only a few notice. Programming theories come and go. Managers cling to expectations, like a child to a blanket. Presentations are polished to create an impression, then curbed to cut costs. Then they are studied. The work spikes and spikes again. Offices are changed a dozen times. Management retires and returns. The contractor is sold. Software is blamed. Executives are promoted. The years rip by with no end in sight. A company president gets an idea: make large small. Turn methods over to each programmer. Dress down. Count on the inscrutability of programming. Promote good news. Turn a leaf away from the sun. Maybe start over.”