The CHAOS Report

The Standish Group^[4] is a research organization that studies the software industry. Their flagship study, appropriately named the CHAOS Study, looks at a broad cross section of software development projects and evaluates their success based on various criteria. The study includes 34,000 software projects in a range from shrink-wrap software and operating systems to custom applications and embedded systems. The ten-year rolling study includes a wide range of different software projects and different sponsors. Every year they drop 3,400 projects that they were looking at from ten years ago and pick up 3,400 new projects.

The Standish Group sent questionnaires to 365 respondents, covering 8,380 applications in an effort to sort projects into one of three categories:

Successful

The definition the Standish Group uses for success of a software project is that it was “completed on time, on budget, and with all the features and functions as originally specified.”

Challenged

Challenged projects are a bit more difficult to classify. These are projects that were completed, but with some compromises either to budget—they went over cost—or to time—they shipped late—or they shipped with fewer features or less functionality than planned.

Failed (Impaired)

A failed project is any project that was canceled—it never saw the light of day. Typically, projects are canceled and fail for reasons outside the developers’ control. A project can fail for any number of reasons: insufficient funding, a shift in the market, a shift in company priorities…

In 1994, the CHAOS Study^[5] reported that 16% of the 34,000 projects they tracked were successful, 53% were challenged, and 31% failed. Ten years later, in 2004^[6] 29% were successful and only 18% failed.

In 2010, the CHAOS Study^[7] reported that 37% of projects were successful, 42% were challenged, and 21% failed. Just two years later, in 2012^[8] 39% were successful and only 18% failed.

While this is a vast improvement over the years, it still means the odds are against us making a software project “successful,” and the odds are worse than one in three that any one project will be successful, at least as defined by the Standish Group.

I believe that the increase in success in the 2004 study may be largely due to more maturity in the industry and greater adoption of Agile methodologies throughout the industry. But even with new ideas like Agile, the chances are only one in three that we’ll be successful.

I’ve worked with some very large organizations and for most, but not all of them, the success rate is much worse. Those that are not exclusively software development companies sometimes have success rates in the 5% range.

The software industry wanted this study so badly there was quite a bit of excitement leading up to it. But it wasn’t too long after its release that excitement turned into some significant questions, and not about the study’s findings, but about the study itself.