The data is stark: according to the 1994 Standish Group's Chaos report zero percent of software projects over $10m in size are deemed successful.
I guess the good news is that things can't get worse.
Zero percent doesn't mean zero, of course, but successes are in the noise.
There's no indication how many of these 35,000 projects are embedded versus traditional IT-like software, but since the embedded systems industry gets no respect, it's reasonable to assume few of the projects were embedded. Here are more data:
|Projects under $500k||38%||44%||19%|
|Projects $500k to $3m||27%||52%||21%|
|Projects $3m to $6m||16%||55%||29%|
|Projects $6m to $10m||4%||57%||39%|
|Projects over $10m||0%||66%||34%|
“Successful” means a project delivered mostly on-time, on-budget with most of the promised features. “Challenged” are those that were substantially over budget, quite late, and had many features scrubbed. “Failed” are those that were cancelled or unused.
The data mirrors my observations of firmware development. Few 4Kb 8051 programs fail to live up to expectations. Scale that to 32Mb of PowerPC code and the odds of getting anything working remotely like the marketing dweebs originally expected are small.
Especially horrifying is the number of abandoned projects. I don't know how many development efforts were in each dollar ranking, but for guestimation purposes let's assume the average size ran about $2m. The 21% of the 35,000 projects rated as failures equal some $15b in work tossed out the door, equivalent to about 100,000 squandered person-years of effort every year. And that's just for the surveyed programs, which are a bare fraction of all software development going on worldwide in any given year.
We're not doing well.
Something like half of the programs gave customers less than they'd been promised. An editorial in Windows magazine complained that they evaluated a number of shrink-wrapped programs and 17 in a row did not live up to the promises on the box.
Few industries thrive when they alienate the customer. Software is different than most other products, of course, as consumers simply don't know if the shiny widget seducing them from Wal-Mart's shelves will actually work as promised—or is a lemon. How do you know if that DVD player doesn't read older disks reliably? Or if the software shuts the engine down when the tank is under a third filled (as happened on a BMW)?
But the 'net is the great equalizer, and consumer sites which let readers rank products will eventually force vendors to provide products that meet and exceed the promises made in the glossy ads.
According to some sources firmware doubles in size every 10 months, which suggests that eventually even Brookstone's smart coffee spoon's code will eat all of the resources of a 500MHz ARM with half a gig of flash. But will it ever hit the shelves? Will it work correctly?
Marketing can take their share of the blame for exploiting the low-recurring cost of software by adding every conceivable feature to the simplest of products. Consumers are at fault, too, for their (uh, our) sheep-like pursuit of whatever Madison Avenue plasters across the billboards and TV screens.
But if we developers don't adopt better processes to bring in big projects on-time with reasonable functionality, our jobs will be at risk.
What do you think? How do you get big systems out in a reasonable amount of time without sacrificing the bulk of the features?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at . His website is .
Are you seriously writing this article based on data from 1994? When I went to the Standish Group link that was provided it points to a report from 1994.
– Dave Mosher
Jack replies – The Standish Group generally does not publish their Chaos report; this link is available only as a sample to encourage folks to buy the reports. But the data hasn't changed much; the 2000 report (not available on-line but referenced here) shows similar statistics.
The statistics presented in this article are very interesting.
I believe that these numbers largely represent the mismanagement of software development projects. Because there is still this lag in company investment for software management to be addressed separately from hardware engineering management, there is a lag in understanding how to quantify resources necessary to complete a project. In my my experiences there is also a larger problem of skirting proper design process, feeling software can be created and changed on the fly so the process of designing good specifications and defining the product in the early stages of development is compromised.
– Heather Magoon
We CAN do better. At my company we certainly do. It takes discipline (to establish a process and stick to it; to do code reviews for every change for example), good tools, and good management that understands management of software development. These are not independent variables, of course. Although I've seen several projects here that might be classified as “challenged” (depending on how that is determined), I can't remember one that had to be abandoned.
This is not nirvana. It's not out of reach. It's achievable.
– Joseph Schachner
How many of these projects are doomed from the start because the marketing dweebs as you called them and management have no concept of the required resources (man power, dev tools, TIME) for successful completion of a software project.
The running joke in my engineering department is “It's only 2 lines of code.” but unfortunately that's the real prevailing attitude of the marketing folks and it shows in our products. I have not been able to get a complete set of specs and then actually develop to them before they're changed and the project runs down a rabbit hole.
Part of this is a management problem as well. The engineers are not defended and allowed to do their job without interference from the marketing and sales people.
– Ken Wheeler
I think a lot of the 'failures' will get attributed to the story of Distributed Thinking…i.e..each department comes up with its own version of what is needed for the product and then as a compromise a product is specified which contains a little bit of everything and many things that dont work with the other.
Often Embedded Firmware design is done on a feature to feature Ad-Hoc basis with little or no understanding of the system engineering impact….And that in itself is one big reason why code size seems to follow Moore's law of some kind!
I have often seen( in just 3 years of experience!), that embedded code begins resembling a 'patchwork' of code that has been assembled as the different demands for features are accomodated. The problem is embedded software within constraints of timing and memory space is often not designed from a modular or a model based architecture..So the more changes you make, the less you can predict impact of those changes!! Failure is then, indeed, the option you have chosen right at the beginning!
– Deepak Alse
In my experience there have been two (2) major disconnects between the techies and the marketeers, that are often the reason for “failure”. Marketeers are sometimes given th impression by the Developers that anything is possible and so the marketeers put blue sky stuff onthe list, and the developes simply role over and play dead. Another source of disconnect in my experience, has been the developers reluctance to train the marketeers in reality. Marketeers sometimes see miraculous development, and they begin to expect that sort of development cycle on a regular basis, when in reality the marketeers don't see the sometimes months of preparation that goes into the miracle that in the marketeers mind only took a week.
It is our job as developers, technical staff to make sure that the marketeers, management understand that EVERYTHING takles time and money, and NOTHING is ever free.
– Steven LEach
Foundation, Foundation, Foundation. The design needs to be built so future features are added easily – a good solid foundation. Instead of “it doesn't matter” this is just one product. Next version and generations will be able to keep the foundation and only the top levels need modification.
– Steve Leggett
The report in question is over 10 years old. The fact the most major software development efforts are late/overbudget/underfeatured is not news.
– Rick Weyrauch
Those tiny, successful 4-kbyte pieces of firmware do at most a handful of things, and do them well. They're easy to bang on from all angles, and make bulletproof. Such a unit of firmware becomes a component in a device that may be greatly or only slightly larger in scope, but that component, if successful, is well-understood and reliable. It just works. For an embedded-systems guy, those pieces are fun and satisfying to develop, and rewarding to see in use, either alone, or in concert with other such units.
When scaling systems up to support the expectations of a feature-happy market, the temptation to move away from small, tight, and testable components to large, unwieldy blobs is great. Interfaces become convoluted, ill-conceived, painful, blurred, and poorly understood. Integration becomes a dirty business. Management of complexity is nearly impossible.
Forces (most often time pressure) push us away from laying a true foundation of solid blocks that fit together well and bear their share of the load reliably. When that happens, there isn't much doubt that the structure they support will collapse under the weight of complexity, inconsistency or poor integration.
Regardless of development tool, implementation language, or design methodology, the embedded mindset we share affords us a small-block perspective that can be applied nearly across the board in software development; a piece of software need not be “embedded” in the traditional sense for this to be true. More features in a product should call for an increased number of small, functional units of code before it it calls for components to grow in size, complexity and cost.
– Daniel Daly
So this just like any industry in its infancy. Really safe cars became standard nearly 80 years after the 1st cars. And meanwhile how many car projects tanked? Or for that matter take aircraft, appliances industry.
What is worrysome is the lack of scientific methodologies as foundation principles of design, testing etc. Currently they are more like catch phrases and afterthought.
The other above mentioned industries at least could bank upon metallurgy, chemistry etc. Why no synergies in software?
– kalpak dabir
This article seems to confirm my suspicion that we developers are just the start of that infinite monkeys experiment. Pound the keyboard, compile the output, if it does something, sell it.
– Mark Holdgrafer
I love how relentless you are in showing that the embedded software industry must get its act together. Keep up the good work.
Although many choose failure (maybe not on purpose) success is an option too!
Looking into the study a little further you will find success criteria identified. They site that “Hyatt had all the right ingredients for success: user involvement, executive management support, a clear statement of requirements, proper planning, and small project milestones.”
When I see “small project milestones” in the report's Success Criteria I think incremental delivery. There is also a strong correlation projects managed via a waterfall process and project failure. Iterative and incremental approaches have been around a long time and have a much better track record than waterfall projects. If you would like the numbers and background of this claim take a look at Craig Larman and Victor Basili's paper from IEEE Computer titled: Iterative and Incremental Development: A Brief History (http://csdl.computer.org/comp/mags/co/2003/06/r6047abs.htm)
The Agile Development has its roots in Incremental and Iterative techniques. Agile addresses many of the “Success Criteria” in the Chaos Report. There is certainly more than one way to success, but we are finding the principles and practices of Agile development are leading to more predictable and successful projects. Here is how an Agile development team would address the Success Criteria noted in the Chaos report
User Involvement – Users/stakeholders are part of the team, available for questions, and to try out the features as they are developed. Often user proxies are needed.
Executive Management Support
Top down buy in is needed as Agile development effects the whole organization. A spec can't just be tossed over the wall. Customers and developer have to work together to manage to a successful outcome.
Clear Statement of Requirements
Requirements are incrementally refined, and specified in automated test cases. The requirements start out vague, more like goals, and become very unambiguous as their acceptance criteria are defined.
An overall plan is created early, but it is constantly refined with the execution data from the team. Giving managers the ability to make trade-offs early and throughout the project regarding staffing, delivery dates and scope.
Expectation are kept realistic by using the teams own performance against estimate data to calibrate and monitor the plan.
Smaller Project Milestones
Big features are broken into smaller features that can me delivered incrementally. The whole big feature may be needed to ship, but progress can be tracked using incremental functionality. Ambiguities are resolved. Risks are confronted and mitigated. Internal deliveries are every couple weeks.
Team work is stressed, rather than individual heroics. Team member coach and mentor each other (more senior people on domain, architecture, real world. More junior people coach the senior on new technology like C++, Object Orientation, and tools)
Tasks are owned by individuals, but the project success is based on how well the development team works together to reach the common goal. Responsibilities are clear, developers own technical decisions such as code quality, design, architecture, and development estimated. Customers (marketing, systems engineering, management) own product business decisions such as acceptance criteria, feature priority, feature content, and scope versus date trade-offs.
Clear Vision & Objectives
Upper management has to define the business objectives. Each iteration, features selected must be moving toward those business goals. Uncertainty may exists at the feature level, but the features must align to target market and product objectives. Details are worked out over time.
Hard-Working, Focused Staff
Good people are a critical key to success. When you put good people in a situation where they can win, great things will happen. In a way this is not so much a cause as an effect.
– James Grenning
To Dave Mosher
I'd say the Standish report is every bit as valid today as eleven years ago. Too bad for us.
– James Grenning
Why is it when software developers read articles like this one we jump on the bandwagon and agree that something needs to be done, then when something IS done, like TL9000, we all jump off the bandwagon and say it is too difficult to implement? James Grenning hit the nail on the head in his response, and all of his concerns (and more) are addressed in the TL9000 standard. But, it takes work, discipline, and resources to carry it out, often more than we are able and/or willing to invest or sacrifice. Yes, it drives the cost of software up, but the old adage applies: you get what you pay for.
– Chuck B
I recently graduated from college focusing in Computer Engineering. For my senior design project (13 month embedded software project), my team of 3 ran smack dead into the project management issues of design and some serious trade-offs were made in the functionality in order to meet the deadline. I realize this was in part to us being green when assigning tasks in such a large project but the other part of it was that we received no formal training in organizing such a large project to make it manageable(realistic). I would like to ask those with experience if there are some nice books/guides for design methodology out there that I could get my hands on. Thank you!
– Joshua Frkuska
I submit that this effect may even “spill over” into the hardware arena. In 30+ years of working Ive only once worked on a project that was planned and funded at anything near what was required and which had a reasonably complete spec from the gitgo. Not surprizingly, this project came in on time, within budget with no scrubbed features. Funny thing though, now that I think back on it, There was no software in that project at all, it was all hardwired logic. It was a fairly sophisticated device too, a Yag laser for medical use. Also I've always wondered why it seems like you need about 10 software guys for every hardware guy on a project. Is that because when it is percieved to be simply a matter of software that we tend to put too much of the design in software? I think that many times the partitioning of tasks between hardware and software asks too little of hardware and thus overburdens the software. Just musings of a rusty old mind.
– Don McCallum
Hmm, this suggests that as an embedded systems designer, when I'm evaluating the cost/benefit of partitioning my design into hardware/software, I should take into consideration reducing the software component of the design to enhance chances of success. Interesting.
– Ingo Cyliax
What is really sad is that even if the data is 10+ years old, it is still valid. This has been an ongoing problem since I broke into the business (1979) and appears will still be here long after I'm gone.
Software engineering should be a discipline such as Mechanical and Electrical, but too often it appears more like the way we build houses in America – each one unique. As systems get more complex, we can't hope to keep this up. The business people of our companies will not be able to allow us to write off large chunks of time and effort just because eventually (in most cases) we ship something that resembles what Sales asked for in the first place…
– Tom Mazowiesky
Steve Leggett and Chuck B are correct: foundation and discipline. It took about two good hits for me to convince my young colleague that there is no such thing as one-off development. As soon as the feature “works” to any degree of the imagination, someone will either ship it or build something that depends on it. Either way that temporary “proto code” becomes part of the ongoing model and you are stuck with it forever.
– Kevin Kilzer
More often than not, we all start out with good intentions of putting together a 'well-crafted' product…with a well-defined architecture. However, temptation to 'hack-in' a solution usually overtakes us…and alas; we wind up having to keep working around the 'hack'. It is imperative to keep enforcing a discipline, and to keep away from the 'temporary' hack solution. These temporary solutions more often than not wind up becoming a permanent fixture for others to contend with.
– Ken Wada
Have I ever worked on a software project that was cancelled? Well, duh. “Has anyone NEVER worked on a SW project that was cancelled?” might be a better question.
– William Weber
The challenge seems to lie in scaling of development. The mindset, languages, techniques, processes and tools don't scale well when we move from small sized projects to large ones. The symptoms may not be confined to embedded systems only.
The way requirements come up from all directions should be improved but can we engineers handle it in a better way.
Ideally in a way that if deadlines are forced the feature being worked upon does not go in product but rest all the features are working fine.
Anyway if anybody has pointers to books/studies on this topic please pass the reference.
– Suresh Shukla