Why don't vendors understand us and give us the tools we want? The tools are out there but the vendors can't seem to reach us. Is anyone to blame?
Last March I wrote an article on Embedded.com stating that vendors of software tools have done a lousy job of selling us on their value proposition.1 You know the drill. Take UML, for instance, or static analyzers that examine source code and identify errors before debugging starts. The products look pretty good, and those alluring magazine ads create a bit of tool lust. The salesman isn't at his desk, of course, but a disembodied voice vows that “this call is very important to us.” Two weeks pass with more calls diverted to voicemail before the company's representative finally rings back. His strong voice is assuring but strangely lacking in specifics. Sure, there's a heart-warming story or two about some apocryphal project saved by the tool. You're encouraged to read the product literature on the vendor's site, but these cite unreadable academic papers, which gleaned insight on a 500-line program written in Smalltalk, developed by three undergrads whose real objective was a passing grade.
For some reason too many vendors don't understand that their customers are engineers, people who make all choices in life, to our spouses' regret, based on quantitative data. Don't tell us your product will improve productivityprove it. Give us the numbers. Back up your claims with data and a wealth of documented success stories from people willing to discuss their projects with us.
The article created quite a furor. Many readers wrote to agree. Others suggested management is at fault, as too many managers won't spend money on software tools or are afraid of changing technology. One developer said he's not even allowed to use a compiler! Another complained that his boss won't part with a lousy $239 for Gimpel's must-use lint.
In the poll conducted next to the article only 36% of 260 respondents felt vendors do a pretty good job.2 That's a flunking grade by any measure. Another poll showed that only 14% of readers can get their company to fork out more than $1,000 for any tool that's not absolutely required, like lint or a source code analyzer.3
So the tool hucksters aren't impressing their engineer customers, and haven't convinced management that spending money will save bigger bucks in the long term.
No vendor responded to the article via the public forum, though several wrote to me privately. All agreed that for reasons they understand and probably many more, they've done a poor job of getting their message across. One in particular expressed frustration with customers for not “getting it,” and dissatisfaction with their own marginally effective marketing efforts. They asked for a meeting to discuss these issues at the Embedded Systems Conference San Francisco. I showed up with a couple of pages of thoughts on the subject stuffed into a pocket. Perhaps it was the beer; somehow interest in the subject had waned.
I'm passionate about tools and feel strongly that most of us can benefit from broadening our arsenal of code development tricks and techniques. Yet there's no market for tools. Pundits peg the entire embedded tool industry (for everything from compilers to emulators to RTOSes) at a paltry $764 million, a third of which all goes to one company. The EE world is very different; Agilent alone sells $2.5 billion of test equipment into engineering labs every year. Ask your boss for a $1,000 software quality tool and watch the door slam in your face. EEs routinely get $50,000 logic analyzers. Yet software consumes something like 70% of the development cost for electronic products.
The problem of limited use for modern software tools and techniques is real and boils down to three major issues: the nature of developers, the boss, and the vendor itself.
It's us, darn it
Few of us would go to the boss and say “my software sucks. I need lots of dough to get it under control.” We have a primal fear that asking for software quality tools is akin to admitting incompetence. Version-control systems, compilers, editors, and the like are safe requests as they are proficiency-neutral. Debuggers, emulators, and BDM debuggers stand on shakier ground but are expected purchases. Code-quality tools fall into a whole different battleground. “What's the big deal?” we tell ourselves. “Just put the quality in from the get-go. Then you won't need these things!”
Process terror also afflicts too many of us. We're afraid that new tools and procedures will stifle creativity. Developers view their next project as a blank canvas just waiting for the artist to express anything in any manner. Software would be better and delivered faster if only we had a paint-by-numbers approach.
Many developers simply don't want smart tools. Want proof? Consider how many engineers ignore compiler-generated warning messages. Compilers, which know the language far better than most of us, emit warnings when there's something possibly fishy in the code. Why would anyone pooh-pooh such portents of possible peril?
We're skeptical of the untried. A new tool or new process means taking a leap of faith in the middle of an already compressed schedule. Worse, we don't really trust anyone else's code, because we know how bad ours is. So developers view big software products as neatly wrapped bug packages.
Finally, in the world of embedded systems most engineers have only a passing acquaintance with any sort of disciplined software process. Few
participate in a software process improvement network. We're like alcoholics. That one nasty code patch leads to another, and then another. Pretty soon you're furtively hacking, cheating the VCS so no one sees your bad habits. Process improvement needs a social component, rather like AA, to keep us on the straight and narrow.
For all of these reasons, when we're promoted to the exalted level of boss the entire cycle repeats.
Nah, it's the boss
The boss, of course, answers to his superiors or to the stockholders. All demand fiscal discipline. It's hard to justify software quality tools when the costs might be high and the promised results nebulous at best. Anyone can see economic justification for not spending, but the wisdom of investing for future benefits eludes most people.
It's easier to buy a logic analyzer or scope than a pile of bits. The test equipment has physical manifestation. There's a place to hang a property tag. A $5,000 piece of software looks like a nickel CD with a book. And generally not a very well-written book, at that. Accountants know how to depreciate the scope, but what are the rules for software?
Seasoned bosses have learned through bitter experience that software is a problem. It feels inherently intractable. Schedules slip while engineers mumble incomprehensible explanations. Bugs surface with alarming frequency, and again the developers offer little solace. When the boss expects software to be the road to perdition he naturally views vendors as snake oil salesman.
The top dog, who's never part of any sort of process improvement network, who probably reads nothing about software engineering, hasn't the faintest inkling that change is possible, that there are ways to soup up the firmware efforts. He figures thrashing about in the quicksand, just trying hard to keep your nose above the sand, is the natural way of things.
Shipping always comes first, it seems. Schedule über alles, über even quality. I'm reminded of that old cartoon of the raging medieval battle, arrows flying and swords being wielded with great gusto. The general complains to his aide that he's in the middle of a battle; he simply doesn't have time to see the salesman, who is standing there, anxious to sell a cartload of machine guns.
Finally, virtually no one buys anything that incurs upfront costs for long-term benefits. That's why the rate of personal savings is so low in the U.S. Progress comes, it's felt, only from generating code. Planning? That's an obstacle to progress. Ditto for design, and code inspections, or running the source through lint or other analysis tool. Just crank some more code, fast!
Maybe it's the slimy vendors?
The vendors are mired in the muck of their own filth as well.
Every salesman over-promises and under-delivers. Supporters hail every tool, every methodology as offering the bright hope of emancipation from the morass of missed schedules and buggy products. Though many of these tools and methods do help a lot, none live up to the hype. So users are jaded.
Vendor honesty is too often an oxymoron. Sales people must learn that practicing engineers despise hokum. We'll respect you more when you tell us where your products don't apply. As it is, all developers fear the painful and expensive product evaluations. Does it work in my environment? How can we expose the limitations early?
There are too many competing tools and methods. All promise miracles. Should we get started with Extreme Programming, Test Driven Development, Feature Driven Development, or Shlaer-Mellor? Is UML the right tool? Where does RUP fit in? And what about those zany SEI people pushing CMM, CMMi, PSP, and the TSP?
As I mentioned earlier, the dearth of hard numbers and believable user stories makes most engineers roll their eyes when facing an earnest salesman. Marketing stories that read like marketing stories don't sell.
Vendors have no effective channels of communications with customers. Ads are swell but rarely get much attention. Web sites work fine after a potential customer identifies a need. None of us want to see a salesman or a dentist; they're both viewed as evils to be avoided whenever possible. Magazine articles touting a specific product are rarely read puff pieces.
Why not a scope?
Hardware designers have little trouble securing funding for budget-busting items like oscilloscopes and logic analyzers for a number of reasons. First, a scope is a low-risk purchase. There are no illusions, little hype, and a general sense by everyone of what these things do.
The datasheets are as specific as the formula for Viagra. A few pages spell out bandwidth, vertical resolution, sensitivity, and number of channels in clinical detail. The devices sell themselves based on bald facts, hard numbers understood by vendors and customers alike.
Finally, the entire electronics community finds these devices indispensable. Heck, I had a scope (a surplus vacuum tube version without triggering) in 8th grade. There's no debate about either their efficacy or their essential role in the electronics lab.
Can vendors of software tools morph their marketing into the oscilloscope model? I'd sure like to see a lint datasheet that said “300 studies confirm the product finds 68% of C errors for 22% of the cost of debugging.”
To reduce risks vendors must find better ways to help prospective customers evaluate the products. Frankly, I'm sick of downloading eval copies of compilers and other tools. My hard disk is a tangled mess of registry hacks and incompletely uninstalled software. Either give us an utterly clean and trustworthy way to uninstall the products, or devise a web-hosted approach.
Risk reduction also requires lots of user stories. Surfing over to iLogix's web site I find only eight descriptions of how customers used the well-respected Rhapsody; most read like the promo material they are. Few numbers appear. At Programming Research's site I could find no user stories for their quite cool QA-C.
Ultimately vendors need to stealthily infiltrate the heads of prospective customers. That happens best when friends and colleagues tout the virtues of a tool or method. Today's buzzword for this is viral marketing, a difficult and often slow process.
And customersusmust participate aggressively in the world of software engineering. Software engineering is an evolving field; the state of the art is much different than when we left college. Stagnate, do things the same way you did 10 years ago, and watch your career wither.
Staying involved gives the vendors a better chance to get their message across. Horrors! Who wants to talk to vendors? Well, it better be all of us. Many of these products and techniques are excellent adjuncts that will improve your code and reduce development time.
Ultimately, for us to compete effectively in this new world order, we'll have to exploit every trick, tool and technique available.
Jack Ganssle is a lecturer and consultant on embedded development issues. He's conducting seminars on this in Boston and Las Vegas this fall. Contact him at .
- Ganssle, Jack, “A value proposition,” Embedded Pulse column, Embedded.com, March 3, 2004. http://embedded.com/showArticle.jhtml?articleID=18201946
- “What's your general level of satisfaction with your embedded tool vendors?” Embedded.com poll, http://embedded.com/pollArchive/?surveyno=26500001
- “What will your company pay for an 'optional' tool, like a source code analyzer?” Embedded.com poll, http://embedded.com/pollArchive/?surveyno=12900001
I just finished your article on the Value Proposition — Unfulfilled. I couldn'tagree with you more. Something needs to change and quick. I have hadthe experience of asking for software tools (specifically a VCS) only to beturned down and later find the money used to go to some useless tradeshow. It seems to be in our nature to dismiss any value placed on somethingthat you can't physically touch no matter how much it may improve productivity.I know from past experience that getting a “C” compiler and an IDE mightbe the most you could hope for.
In too many instances, I have had embedded projects cancelled or redefinedbecause the design and debug of the software would delay the productrelease significantly. A new embedded project that is still in discussion willrequire approximately 40-60 man hours of hardware design time but theestimate for software design and debug is 6 months! That estimate is basedon writing the majority of code in “C” without the benefit of a top-down designapproach. I am sure the programmers will begin by writing code in an attemptto have something working in less than 6 months. They are aware that they willpay for this haphazard design approach in extended debug time but managementis willing to let the development continue if it appears to work. Unfortunately, theydon't know it will take another 6 months to completely debug it.
We have looked at the availability of embedded productivity tools only to find there isn'tmuch available that helps when doing an embedded software project. If the embeddedmarket is ten times that of the PC market why hasn't anything been done about it? Itwould seem to me there is money to be made in embedded tools. Am I missingsomething here? In one instance a salesman from Rational Software tried to sell us atool that clearly was intended for use in the PC application world. They build a goodtool for the PC application arena but they don't seem to care about the embedded world.Meanwhile, Microsoft is offering this bloated, buggy, convoluted OS as the answer toour embedded software design problems. Management sees this as a clear reduction inthe software design schedule and they wholeheartedly support it. Inevitably, we end upwith a major hardware redesign to support the Microsoft OS, a schedule that extends wellpast 6 months and customer support staff ten times normal size. If this mindset continueswe will all be running an OS that is not well suited for use on embedded products. We needtools intended for the 8, 16 and possibly even 32 bit embedded markets. OSes, debuggers,IDE's and structured design tools to shorten these long schedules. I firmly believe that oncewe believe that software design is no more of a constraint than hardware design in newdesigns, we will see more products with better quality.
– Dirk Cooley
Thanks for a thoughtful article on software tools. Many points were well taken. I'd like to extendthe 'scope analogy further, though. What I say is undoubtedly skewed by the fact that EmbeddedSystem Resources is a two person company that cuts a fairly wide swath through the hardware andsoftware community. Out of our limited revenue, we need to buy hardware test equipment. Compilersand software tools, FPGA development tools, Schematic capture and board layout tools, not to mentionrecent enough computers and OSs to run them on. Obviously, purchasing decisions are made carefullyand as infrequently as justifiable.
We own two oscilloscopes, one analog, one digital. They meet our routine needs quite well, whetherits debugging a switching power supply, an RF amplifier or converter, an audio processing system, oran embedded CPU board. Why? They are general purpose in design, and standardized in interface. Ican plug any probe from any manufacturer into it, and the worst thing I might encounter is a need fora probe power supply or lack of automatic 10x probe identification.
I have four C compilers! And that is for just four CPUs! Not only that, but I'm embarking on avariant x86 CPU shortly, and will need a fifth compiler for it. Why? It has a slightly differentsegmentation scheme and I don't have access to the internals of my existing x86 compiler to make theminor changes required. Also, it has a JTAG port, and I do not have access to the debugginginterface on my existing compiler to bolt a JTAG interface to it. By contrast, my oscilloscope has aservice manual available for little more than the cost of the paper it is printed on which containscomplete schematics. I can do any repairs, or special interfacing I wish. The compiler doesn't eveninclude library source, unless I part with another large wad of cash! Program source–forget it!
If I go down the recommended path, the necessary hardware and software will be $8000! I won't besolving the problem in that manner. The compiler will be used on less than 1/4 of the work on aproject that has a total revenue of $36,000. It is true, it will be used on future projects later,but do the math. Basically my labor using the compiler on this project would be free! And by thetime I use it on enough other project to make it start paying, there will be mainanence releases tofix bugs that never should have been there that will cost several hundred dollars more!
My oscilloscope probably has almost as many lines of code in it as the above mentioned compiler, plusit contains several hundred dollars worth of tangible electronic parts, which cost the manufacturerseparately for each unit sold. I paid $1500 for it on E-Bay. I would have paid maybe twice thatmuch if it was new, but that is still less than the price of the software tool! Furthermore, I amnot likely to find the software tool on E-Bay–the life expectancy is too short, and the manufacturerprobably won't support someone other than the original purchaser. Manufacturer support is important. I just found out that one of my current C compilers breaks on structures of certain size ranges (noI'm not talking > 64K, more like ~ 512 bytes!) It also has the unique ability to rememberinformation about previous compiles such that when compiling:
if ( a > B)
x = 1;
x = 0;if you swap the two assignment statements and re-compile, the source code will be different, butsometimes the object code will remain the same! They have not admitted the existence of the problem,let alone come up with a solution! I must remove the code fragment, re-compile and then put it backin order to get correct object code.
In summary, there are several problems contributing to the lack of use of software tools, some moresolvable than others:
1) Standardization – all software tools need to conform to an industry standard specification so thatthey can be freely mixed, matched and added to. Even processor or language specific features,ideally should be table driven in a known format. The only product that I know of that even comesclose on this one is GNU/GCC.
2) Openness – This is scary, because of intellectual property rights, and the fact that software ismuch more easily replicated than hardware, but the fact that I can't even supply a debug driver to myIDE because of the secrecy with which it is guarded is appalling.
3) Pricing – software tools are simply priced too high for many users in many cases. The real issuehere is the wide range of incompatible hardware that needs to be supported, and the wide range ofcompanies competing for the resulting fragmented market. There are only three companies I wouldconsider buying an oscilloscope from, and one of them makes specialty products I haven't needed. Theother two each have a fairly large market share and can afford to make affordable tools. There arehundreds, maybe a few thousand companies making embedded software tools, each targeted at a smallniche. Many may only ever sell a few hundred copies. That is a problem!
4) It didn't get much attention above, but the learning curve is a major factor, also. It is evenharder to justify a new tool that may save time and money in the long run, if the learning curve andchanges to other parts of the process are so long and disruptive that it will destroy the marketwindow for the product being worked on. I can walk up to almost any oscilloscope from anymanufacturer, without a manual and in a few minutes be taking measurements and using most of itscapabilities effectively. I would be foolish to even try using most of my software tools withoutspending a week reading the manuals and working through the meaningless sample exercises. Someonewould say, “but there are too many options, it isn't a fair comparison”. I disagree. Back in theDOS days I was asked to update someone else's xbase information system. I decided a compiler wasnecessary to get the desired speed. There were two good candidates: Clipper and FoxBASE. Clipperhad enough command line arguments to fill a chapter in the manual. FoxBASE had none, and generatedfaster code! Which do you think I used? Even when options are necessary (and I do like control ofmy process) the interface can still be clear and intuitive. Let's take optimization for example. Some compilers have 20 or 30 options. Most of us really need three or four: NONE – for stepping andtracing, SPEED where time matters and SIZE for where memory is more important. Add a couple featuresabout a few things that pointers are allowed to do and that about takes care of it. As detailoriented as I am, I really don't see why I should care whether optimization is local, global,peephole, involves loop unrolling, code re-ordering, register remapping, etc., etc. If it is safeand produces faster or more compact code, I want it, if it doesn't I don't dare–end of discussion.
– Wilton Helm
Sorry, gotta vent a bit after reading your excellent article “The Value Proposition-Unfulfilled.”
We sell a UML based code generation tool called visualSTATE. Getting customers to look at it is a bit harderthan finding a politician with a sensible position in an election year.
Re: your comment: “Two weeks pass with more calls diverted to voicemail before the company's representativecalls you back” Not true! If I'm gabbing with a customer when you call, I'll be back to you before the dayis, next day at worst. If you are interested in buying IAR tools, I'm on you like a duck on a June-bug.
Re: “For some reason too many vendors don't understand that their customers are engineers, people who makeall choices in life, to our spouses' regret, based on quantitative data. Don't tell us your product willimprove productivity-prove it. Give us the numbers. Back up your claims with data and a wealth of documentedsuccess stories from people willing to discuss their projects with us.”
I would love to be able to do this, but to do so, I would have to have data from a customer himself on howhis/her software build process changes as a function of tools usage. I have been selling tools for 5+ yearsnow, and I have yet to run into one company that has anything close to real software productivity metrics. Noone seems to be collecting realistic data on this topic. Even if they did, what incentive do they have totell the outside world about it, maybe giving their competitors info they don't want them to have?
Re: “Every salesman over-promises and under-delivers.” No, nyet, non, nein. First, I am a developer. I KNOWhow hard it is to write code. Secondly, I realize that many engineers are working from a “back of theenvelope” spec from a manager/marketing person who doesn't really know what they want, but tells the engineer”make it good.” I have actually been there done that, have the tee-shirt, and gotten paid for it. I still getchecks “paid to the order of Mark Moran” for real embedded code that people spent real money on. When I'mspeaking to a customer, I speak to them as developer-to-developer, not salesman-to-customer. Most of thempick up on this and are glad that I'm not here to bullshit them.
More specifically, when I pitch our visualSTATE tools, I SHOW them what's going on. In my Internetpresentations, I do the following:
– show them the specs of a simple application
– describe exactly how to use visualSTATE to implement the specification
– show the customer the test/debug/verification capabilities of the tool
– show them the generated code
– show them the generated documentation
– discuss exactly what part of the application the generated code covers and what part they have to write byhand.
All this takes about 45 min. but many customers have questions and comments and the discussion frequentlystretches to 1 hr.
I don't claim this is perfect, but if the customer does not understand the strengths and weakness of thistool after this presentation, then he should turn in his geek credentials and find work outside ofengineering.
Re: “So the tool hucksters aren't impressing their engineer customers and haven't convinced management thatspending money will save bigger bucks in the long term.” In presenting a tool like visualSTATE to engineers,I have yet to fail once in getting the engineer salivating about this tool. Then he/she runs into a brickwall known as “management.” This you have ably described in your article and I have only a resounded”s'accord!”
I have a little true story on my web site at http://mysite.verizon.net/vzeeezij/id5.html about typicaldealings with management when I worked as an engineering manager. Incidents like this are the reason I'mselling software tools these days.
So what's the answer? I don't know, but I love the fact that you are addressing these issues in your column.Keep 'em coming!
– Mark Moran