I used to be in the in-circuit emulator business, back in the glory days when developers routinely forked over $5k, $10k, or more for an ICE. Cross compilers ran $3k to $5k; licenses for RTOSes might start at $20k/seat plus royalties. Few engineers complained, their companies seemingly happy or at least resigned to spend large sums to get a product to market. For that was the way we always had done things, and there were few alternatives.
In the ICE business, at least, this has changed. Emulators for higher end processors mostly don't exist, killed off by their rapidly increasing costs and competition from other tools like BDMs and JTAG debuggers. Today most developers use these inexpensive tools that offer far less performance than an ICE. This blows my mind, since the problems we're solving today are far more complex, with much more code and a bigger real time component than projects of old. More code, more complex code, and less capable tools do you sense a disconnect here?
I'm big on automated source code analysis software. An example is Lint in my opinion programming without Lint is like riding motocross without a helmet. Also important are metrics tools, software that evaluates source code to find excessive complexity. We know that convoluted code is error-prone and maintenance-resistant; a complexity analyzer quantitatively identifies sections of source that must be simplified. (Here's a cool site with freebies).
A commercial version that gets great reviews is Programming Research's QA-C. Yet it's a $5k tool. Most developers recoil in horror when they discover the price.
$5k is a lot of $k. Add on top of that maybe another $3k to $5k for a cross compiler, a few thousand for a BDM with source level debugger, hundreds to thousands for version control, editor, IDE, and before long a basic development environment might cost $10k to $15k. Put ten developers on the team and the total tool costs exceed $100k, a lot of money by any standard.
And yet, the latest programmer salary survey shows developers earn around $80k to $90k in the USA (you have to register to get to this information, but it's not too painful). Loaded with benefits that approaches $150k per person per year. That ten person team sucks a cool million five from the organization. $100k to $200k for tools starts to sound cheap, especially when you amortize the cost over a multi-year project, or over multiple projects.
For some reason it's different on the hardware side. EEs routinely get logic analyzers ($10k to $100k), scopes ($5k to $30k), various signal generators, analyzers, monitors, schematic capture, PCB layout, and simulation. Back when I had a boss getting approval for anything from Tektronix or HP (now Agilent) was easy; software tools forget it.
What's different? Is it because EEs produce something visible, something management can see and fondle, whereas we're just cranking invisible bits? Or has the GNU movement convinced management that free is the norm?
Every day new software development tools appear. Some look pretty awesome; others actually are. The cost to develop code continues to skyrocket. It seems to me that anything that shaves just a bit off the development costs is a must-have.
Does your company make you get by with a minimal toolchain? Why?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He is conducting a seminar about building better embedded systems on December 5. Contact him at . His website is .
The poll asks 'What will your company pay for an optional tool, like a source code analyzer?'. What's optional about that? People make mistakes. Tools might not be perfect, but they *are* consistent, and generally find the mistakes that people make. I work on the DiscWorld principle that if there's a one in a million chance of something happening, nine times out of ten it will happen. And (in my experience) tools reduce those odds significantly, as does using a repeatable process including things like peer reviews and unit testing (using tools!).
– Paul Tiplady
Part of the problem is engineers not knowing how to present a business case. Instead of saying “it would help me out a lot” or even “it will save one man-month of mytime”, they should be saying “Our time to market will be one month earlier. Thisprojects to be worth $1.5M by marketing estimates”. Makes $50K seem small.
Unfortunately, development tool prices hit small innovative companies the hardest. Theyare on the cutting edge and can't afford to toss lots of money around unless they havealready sold their souls to the VCs.
– Steve Nordhauser
For some companies, especially in developing countries, the tool's prices are too high,and the availability is very limited.
I have developed successful embedded systems, probably not too complex, but using limitedtool chains. My experience has shown me that it is more important to focus on thesoftware design stage, rather than rely on the capability of a tool to find the bug forus. It is better not to introduce that bug on the first place.
In my opinion, more emphasis should be put in using better and safer programminglanguages, and in using software development practices more strictly.
– Alexander R.
Your article of is really right on the problem. Hardware companies promote BDM and JTAGand make them so convincing that management as well as most engineers believe ICE isunnecessary. I found if timing is not important, BDM and JTAG are ok. However, iftiming requirement is very tight and there are lot of tasks interacting, ICE isnecessary. Nowaday, there are less and less people realize the importance of good tools. Cost is number one, time to market is number two factor in most development. Majority ofdecision makers in companies have little experience (I mean deep experience) in trulycomplicated development so good tools are less appreciated.
– Luke Lu
I think Mr. Ganssle overstates the importance of ICEs in the developmentprocess. Although they offer extraordinary insight into some problems, they can only doso in very limited situations.
The ICEs I use typically end up being fragile hangar queens: attached to speciallymodified PCBs in benchtop mock-ups of real systems. Of course, many of the mostinteresting problems occur in full-size systems that are not conveniently located in thelaboratory.
BDM and JTAG debuggers offer the possibility to hook up to a malfunctioning systemwherever it is located, without tearing it completely apart. Perhaps this explains whythe huge cost premiums ICEs demand are becoming harder and harder to justify.
– Scott Barnes
I DO use a low end ICE for my Atmel uC projects, but the only reason I got itwas because it was cheap. It is VERY difficult to sell software tools to managementbecause they do not use them & have no idea whether or not they're really needed. Inaddition, when you're constantly on the ragged edge of profitability, it is hard tojustify spending money for something that isn't “real”.headline : Do tools cost too much?
– Dave Telling
Part of the problem is that people (accountants, managers, etc.) have becomeaccustomed to the idea that any arbitrary piece of software should cost between $50 and$350 per seat. All they have to do is go down to CompUSA and check the shelves. Even thecompilers (for PC development) fit into this range (mostly). This can make it a hard sellwhen you show up a request for an embedded toolchain that costs $8K/seat.
– James Thayer
I read your column “Do tools cost too much?” with great interest. The problem is notthat software tools cost too much but they deliver too little value at any price. I havebeen in the software industry since 1980. I have developed software to automate testingof diesel engines, analyze flight test data and now monitor traffic control system(automobile traffic control).
In every company large sums of money was spent on software tools. With very fewexceptions the tools became shelf ware in fairly short order. The reasons were legion:the tool would only work if we allocate certain system resources (timers) that wereessential to our product, incompatibility between tools, outrageous upgrade costs eachyear, and plain they did not work. Some of the tools I have used are: CardTools fromReadySystems (remember them), CCC from Softool (cvs does a better job), System Architect(I liked System Architect but they missed the OO world), Rational Rose, RequistePro (itis faster and more effective with Word and Excel), DOORS, TeamWork, Interleaf, Softwarethru Pictures, Developer Studio, ClearCase, etc.
Except with the simplest tool set (make, cvs, TrackRecord, and your choice of compiler)the projects that used the top of the line tools delivered code slower and with the sameproblems as code without the top of the line tools.
So right now what I see with tools is “Less for More”. I have a very hard time sellingthat to upper management.
– Keith Patton