“Why are you pounding that nail with a brick?”
“Hammers cost too much.”
What about this: “We're looking for a cheap IDE; can you help?”
Equally silly, in my opinion, yet I get several such email requests each day.
Why is the only adjective describing IDE “cheap?” What about “reliable?” Or Windows vs Linux? I can think of a hundred critical technical qualifiers, yet all pale, it seems, compared to “cheap.”
The embedded tools business is microscopic. Various estimates peg it around $1b to $2b, but that number is deceptive. It includes sales and royalties from RTOSes, a segment that represents a huge but unknown percentage of all tools. Most embedded tool vendors are tiny. There's practically no demand for commercial tools.
When I was in the in-circuit emulator business, we found that bosses objected to spending thousands on debugging tools. “Just don't make any mistakes!” they'd rage when confronted with a $8k purchase request. “Think of the money we'll save!”
But the numbers are stark: C developers average a 5 to 10% error rate, after the compiler finds syntax errors. Even a small project can reasonably expect to deal with thousands of bugs. A tool that saves even a few minutes on each pays for itself many times over.
Debugging typically consumes about half the schedule. A tool that shaves a mere 10% from that is priceless. Yet vendors of static analyzers and other tools that automatically find bugs have a hard time getting management to sign checks.
The hardware world is different. I hardly ever hear: “We need a cheap logic analyzer. Any ideas?” Yet those tools, too, are primarily used to find an engineer's design errors.
What do you think? Why is it so hard to pry tool bucks from the accountants' stingy hands?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at . His website is .
Tools are important. Tools cost money. Generally, that money is well spent!
Now … that I have said this; let me tell you why I believe we are so stingy with the budgets.
Firmware/software engineers are not cheap. Firmware and software takes time to write and debug. This activity is extremely labor intensive. Thus, there is a general tendency to spend more on hiring engineers than on gear. Plus, a lot of times, the gear that we debug gets expensed on our budget for the full ASP! Upper management then says something to the effect: “Why do you need a debugger? Why, you have almost $2M in hardware debugging costs already spent on your budget!”
That “expensed” gear never shows up on the hardware manager's budget, because … when it “sits” on his bench, it is not a product yet!
– Ken Wada
It's difficult to get money for good tools because the “Budget Nazi” can't see the long term benefits. It's really that simple. Good tools last a long time, are reconfigurable, and upgradable, but cheap ones aren't!
– Steve King
I don't think it's just the cost. What I have found over the past 20 years is that most managers do not “get” software writing. No clue! “It's just typing” they say. They cannot fathom how a development system can cost thousands of dollars for just “typing”. And the hardware guys get all the fun toys!
– Bob S.
One problem that I haven't seen mentioned is that for many managers software engineering is a black art. In my current company, even though 75% of our engineers are software engineers, ALL of the engineering managers are former hardware engineers. I am sure they all did some programming at some time, but the difference between programming and software engineering is not clear to them.
– William Ames
Speaking as both a development grunt (with the scars to prove it – i worked on Motorola 6800 Exorcise debuggers), and as a manager of a development tem that encompasses both hardware and software, allow me to comment.
Software tools are just as important as the hardware tools. Compilers, debuggers, simulators, ICE, whatever, every tool I've ever fought ofr or signed a PO for has always returned its value several times over. What needs to be done is to tell management what the benefits are in terms they understand – time and money. After spending money on tools, whenever we hit a problem that was particularly tough, once it was solved using a tool, I would always report that the tool was a major factor in finding the problem, and shortening the development cycle. This makes the manager feel good, because he knows he made a good decision to sign off on the tool.
You can predict to some degree what enhancement that particular tools will bring – C compiler vs. assembly code, debugging tools, etc. This doesn't have to be down to the last percent, but management needs to get a flavor or feel for what the benefit is – standing in the guys office and screaming that 'I can't work without this!' is not justification – it's just a tantrum. In our personal lives we make the same decisions, just with different items – new car vs. used, home purchase vs. rent, etc. You wouldn't make any of these decisions without information to justify them, would you? When you have the smarts to use the tool effectively, and know how it will benefit you in your project, you have the knowledge to make a coherent argument for tool purchases. Managment doesn't need a 20 page report (in most cases), but they need enough information to justify it to the financial side of the business. Every business has a budget that it works to. Sometimes these can be stretched easily – in other company's that may be harder, so you need to give the financial people the information so they understand not what the tool does, but why it will make you more effective in your job.
– Thomas Mazowiesky
I think the problem is our own doing.
How frequently do we actually hit the schedules we set for ourselves? How often do we have quantitative results to show progress before/after spending some money on other tools?
Do the strictly hardware guys do that? NO! It's impossible to see what is happening without a scope or logic analyzer, and the boss can understand that concept.
In the software and firmware areas, we need to show a similar inability to do ANYTHING w/o the proper tools.
I can't get $500 of software tools because mgt doesn't see where to hook the scope probes to the box. If I could get them to understand that this debugging stuff is the software equivalent to a scope, it might be easier.
Perhaps the tool vendors can help us make that sort of justification.
– Andy Kunz
Regarding tool costs and why it seems to be easier to get managmentto buy a hardware tool than a software/firmware tool. Here are someof my thoughts…
1…I think generally speaking, hardware design is considered to be morechallenging than firmware (the nail is bigger and the wood is harder,so the boss lets you buy a bigger hammer). This is obviously not thecase in reality; however, I'm sure it is a factor when it comes to toolpurchases.
2…I think the mentality is that you need 1 hardware designer with good toolsand then just throw an army of firmware developers on it and you're good togo.
3…In mature organizations, it's likely that more of management comes froma hardware background, since there was hardware before there was firmware.As a result, more managers do have a hardware background; hence they aremore sympathetic to the hardware developer's story.
4…Hardware is “hard” (can't be remotely upgraded); software is “soft”(we'll just fix it in the next release). This is just a mentality thatMicrosoft has put on us all and we're stuck with it.
5…I think all the embedded linux hype hasn't helped either. Un-savy managersthink that once the hardware is done, we “just put linux on it” and allthe “free” open source will just work. This is obviously not the case;however, it is presented this way quite a bit.
Some of the fault, IMHO, lies with the tool vendors. I know you're historyis in emulators, so you're a bit “tools biased”; however, let's face it…The tools have bugs too!
In many cases, I find it more efficient, more satisfying etc.. to avoid usinga whole lot of tools because then I force myself to know what I'm doing a bitmore. There's nothing better when tackling a problem, than to thoroughlyunderstand what the problem is. Many of these tools throw layers of abstractionon us that make many developers totally unaware of the fact that they are workingon an embedded system that has a very different personality than a PC.
There are obviously pros to that; however, there are definitely cons as well.
Bottom line, as is always the case with embedded stuff…
You just gotta pick and choose your toolset (or the lack thereof) carefully.
– Ed Sutter
I think the size of the penalty box is directly proportional to the amount that will be spent on the tools.
The rev of a circuit board or the respin of an ASIC device is measured in weeks or even months. A software revision is measured in hours or maybe a day or two.
So the hardware managers are willing to spend more money to insure that the design is right the first time. The software managers seem to accept the fact that the software/firmware will have software defects, and that they can be repaired quickly. So, they don't seem to have the need to 'get it right' the first time.
My bigger question is why we are doing software the same way that we have for the last 30 some years? We should be an intelligent bunch, and would want to find a better way. This is why the hardware/silicon guys now design in VHDL, Verilog, etc, rather than drawing individual transistors on sheets of velum like they did 30 some years ago.
– Howard Smith
I'm self employed as an independent consulting engineer and work out of my home in a world class engineering office. I've had to make substantial investments over the years for software for both mechanical engineering and software engineering tools.
One embedded software company in Laurel, MD where I worked as a senior C/C++ software engineer for eight months literally spent more money on chocolate than they did on software engineering tools. They had an entire cabinet full of bags of chocolate candy and did all of their work with “free open source” garbage software. “Free open source software” is the biggest lie in the history of software engineering. I lost two teeth before I finally quit in disgust. I'm probably the only person you will ever meet who left a six figure job partially because they were giving away free chocolate.
Another embedded software/hardware company in Annapolis, MD hired me as a senior mechanical engineer to do Finite Element Analysis (FEA) and 3D solid modeling. They refused to hire me as an independent consultant and insisted that I become a fulltime employee. I should have known better. On the first day they tried (and failed) to steal all of my intellectual property. When it came to tools, they had nothing – for six months I did all of the FEA and solid modeling on my computers with my software. At six months I gave up and quit.
Clearly American managers have excelled at Stupid School. And don't forget – we Americans are too expensive to hire for all those 5-200 Million dollars a year American CEO's. We are becoming a society that is literally strangling to death on its own collective stupidity.
When it comes to software tools, especially “expensive” ones, it's doubly hard for developers to build a case because doing so risks appearing like an admission of incompetence. Developers may imagine a manager reacting with, “What? You don't know how to write software or find and fix your own mistakes?”
What so many managers fail to realize is that while writing a piece of software that works is easy, writing a piece of software that works with thousands of other pieces of software, all at the same time, on custom hardware, under varying conditions, is a near miracle. Virtually any tool that:
a) helps you create better code in the first place, and
b) debug it faster
Is a wise investment that will help keep development schedules and development costs in check while improving product quality and reliability. As one of the other readers pointed out, perhaps tools vendors need to do a better job of helping developers sell the value of tools to their own management.
– Jan Liband
even though I agree that you shouldn't be pennywise and pound foolish when it comes to software development, I see in many replies that “open source” software is partly blamed for the problem. Someone anonymous talked about “free open source garbage software”.
I do most of my development using the Gnu Compiler, a most excellent piece of open source software. My favourite operating system for development is completely open source (and it is not Linux). This open source software saved me a lot of money and I am satisfied with it.
Use open source software when it helps you out. Skip it and go commercial when that provides a better alternative. But don't blame something free for your troubles.
– Ewout Boks
This is an interesting topic, because where I work, it's not so much an issue of money, unlike folks in traditional industry where it appears to be very much the case. My manager more than once told me not to worry about the cost of the tools I am evaluating. We already have the licenses, it's a matter of convincing the IV&V folks which ones they should use.
In our case, resistance is met when it comes to getting people to actually use the tools. That is why we tend to place a big emphasis on usability. If I can use a tool right out of the box without having to read much of the manual, I know it sounds more promising than one which I have to constantly delve through a thick manual. Even though both tools might produce comparable results.
I was told we came up with a slick in-house tool to interface with Flexelint but it has been gathering dust because it was too cumbersome to use.
In the business of code analysis, I am finding that one tool just doesn't do the full job, but rather two or three will each produce interesting findings. Part of the challenge is reconciling the data.
– Joe Sallmen