They say there's no greater love than that of one dead drunk for another. The same, apparently, applies to battle-scarred embedded folks.
This June, on my annual summer sailing vacation, I stopped in Bermuda for a crew change and some R&R before heading south. Having put a number of sailing stories on the Net it's not too unusual for someone to recognize “Voyager,” my 32-foot ketch. So, tied up to the wharf in St. George, I wasn't surprised to be hailed from a gangly fellow on the dock. He was headed to the White Horse Tavern and wanted to buy me a drink.
Wizened by the sun, with a week-old growth of beard and soiled T-shirt, he looked more like one of us sailors than an engineer. But the high-tech California lifestyle had been financially good to him, and he had retired to a life of beaches, salt air, and rum.
We chatted over Dark and Stormies far into the night, greatly enriching the pub's proprietor. A regular ESP reader, even here in this near-tropical idyll, he debated with me some of the fanatically held opinions I so often express in these pages. I enjoy these sorts of sessions, always learning much even when we hold diametrically opposing opinions. Though some folks grow out of such passionate banter after their sophomore year, I find that a vigorous exchange, whether on politics or engineering, is educational, interesting, and great fun.
Jamie told me he had spent much of his career working for a small contract design outfit. Their 25 engineers created some 50 new systems per year, for clients as varied as the Defense Department (where paperwork mattered more than product) to dot-com startups requiring some embedded intelligence.
I spent 15 years building an in-circuit emulator company, which gave me the wonderful chance to look over the shoulders of literally thousands of developers creating an astonishing variety of products. It was like vicariously participating in these projects, since anyone who sells debugging tools gets called in only when the project is at its absolute nadir.
I've observed that most of us work in small teams. We tend to learn mostly from our experiences, but those are necessarily limited by working on only a couple of products per year. If our company produces the same sort of product based on similar technology year after year, then our experience base has little chance to grow.
It's quite interesting to compare engineers from companies that produce products to those working for smaller consulting outfits. At the risk of making an overly patronizing generalization, many of the consultants have a wider range of “tools,” abilities, tricks, and techniques to get a product to market. They might work on a dozen products in a year, each in very different fields. This broad scope of experience helps them make better systematic decisions (such as 8-bit vs. 32-bit). They'll often use better tools and more high level design processes, for example, because over the course of so many projects they find that these techniques work.
So Jamie and I shared many of the same experiences, both with technology and the people that build the systems.
Perhaps the most striking impression we've both brought from our careers is the astonishing level of heroics used to complete most projects. Every new product starts with optimistic folks making very quick progress in the design and coding stage. Early debugging, in unit test, is quick. Integration of the software with the hardware or of tested software units into a more complete whole, is where schedules collapse and design flaws become glaringly apparent.
Developers almost universally panic, and then, through late nights and other heroics, beat the system into submission. This is where features are discarded willy-nilly in an attempt to minimize schedule slippages. A recent study by Electronic Market Forecasters showed that 57% of us reduce schedule slippages by last minute feature shedding.
I was once fiddling with business statistics and was shocked to discover that some 70% of our emulators were shipped FedEx overnight. Turns out that most of the customers had planned to use much more limited tools, but were stymied by problems far more difficult than imagined. Jamie, too, had found that front-end planning to deal with inevitable back-end surprises was critical to project success.
Though I'm a pretty happy guy, I consider unbridled optimism at the outset of a project to be a sign of future disaster. Before every ocean passage, I tear the boat apart from stem to stern inspecting, testing, and ensuring that the vessel is safe for sea. So too for an embedded project: I've learned to quench the enthusiasm and probe the deepest recesses of the plans, designs, and schedules for risks and flaws. Things always turn out more difficult than expected.
Too many choices
I asked Jamie if the consulting company had a standard set of tools and technology. In my own business I was loathe to let engineering switch compilers and processors, having found the cost of such change to be astronomical. A small company, with end users who cared little what was in the box, we were reasonably successful at keeping a consistent technology for many years.
He harrumphed, and said that each customer mandated a certain processor and often even a toolchain. Costs were thus much higher than they could have been. Switch even a very smart developer from the ColdFire to a PIC, and watch productivity collapse for the duration of the learning curve. Add in the need for perhaps tens of thousands of dollars for new tools-plus the time to master the infernal things-and it's easy to see the cost of capricious processor selection.
He admitted the difficulty of creating standard library-like components that work across processor families. Clearly, device drivers share little common structure; even something as inherently simple as a timer has totally different I/O requirements between parts from Motorola and Intel.
Consultants often cannot share technology developed for different customers. Non-disclosure agreements, written by legal departments that don't have a clue about the costs of product development, frequently explicitly prohibit the sharing of any part of a product's source code.
Other consulting companies I've worked with have managed to educate some of their customers about the savings inherent in sharing some of the source. When clients sell to different industries, there's usually little threat to a common code base. But this means the consultants must teach the rarely receptive lawyers the benefits, a difficult task given the legal profession's aversion to risk of any sort, real or perceived.
Jamie told me of their storage room, filled with costly development systems for projects shipped long ago. What an unproductive waste.
Motorola, Intel, and others dealt very effectively with customers' needs for more horsepower by making CPU families that share a common architecture yet offer a wide mix of performance and on-board peripherals. As an ICE vendor, I always admired the customers who took full advantage of this breadth. If the compiler could stay the same, the developers needed no retraining. All emulator vendors offered simple and cheap pod changes that optimized the customers' investment.
Despite the costs inherent in changing processors, Jamie did say they managed to keep the costliest of all engineering resources constant: process. Their firmware standards and their ways of going about building a product stayed mostly unchanged regardless of client or technology. He admitted, as I've discovered with my own customers, that creating a development process, training the developers, and then helping them turn what is first perceived as a pain into wholly-embraced religion, is difficult and time consuming. But the benefits mean more predictable scheduling, more reliable products, and happier customers.
Perhaps our most spirited debate concerned tools, especially debuggers. My ex-ICE-vendor status puts me squarely in the “price is no object” camp. If spending $10,000 or more saves a significant amount of debugging time, the cost is quickly recouped.
But a continuing frustration of all debugger people is convincing customers of this “truth.” All of us sincerely believe it, yet it's a hard sell. I remember a cost/benefit analysis prepared by the biggest ICE vendor many years ago that attempted to prove just how much money a big-buck emulator saved. Later, an executive at the company admitted to me that this report always fell on deaf ears.
It's pretty hard, after all, to ask management for $10,000 for a tool whose sole purpose is to find your mistakes. They're inclined to tell the developers to do it right the first time. A noble goal, perhaps, but certainly elusive.
My non-scientific observation of hundreds of customers showed that half of the average project is spent in debugging. We're making a lot of mistakes, for sure.
Jamie's company was one of the few that tackled bugs from the outset, minimizing them with code inspections, careful design, and by using all of those great process ideas the pundits preach but few really employ. With so many projects happening per year, they had a chance to collect enough statistics that clearly showed the benefits of that approach. They just didn't need a lot of debugging horsepower. And with so many different products based on an equally diverse set of architectures, the cost of tools was staggering.
So I challenged him about features. Modern debugging tools, like BDMs, offer a bare set of resources, a set much reduced from those of an emulator. There's no doubt in my mind that, for very good reasons, BDMs are the present and the future of debuggers. CPU speeds are too high for ICEs to keep up with anymore; SMT and BGA connections make too many un-probable. Yet it's odd that despite the increasing size and complexity of our products, our debugging capabilities are reduced from the days of yore.
Jamie was adamant that features are of little value. That exotic cool thing used once a year to find a really tough bug, in his opinion, doesn't really exist. If you don't know how to use the feature, or maybe even forgot about it, then it's of no help.
Well, yeah. Sure. But shouldn't we be experts at our tools? Shouldn't we continually sharpen our skills?
But then I thought about my old customers. Few ever learned more than single stepping and setting breakpoints. The vast depth of features were almost always untapped.
Then he turned the tables on me: “Just how many of Microsoft Word's feature do you really know? Or Excel?”
I use these programs for hours a day, yet haven't a clue about most of their capabilities. Maybe there's a lot of truth there. Me, I hate to load new software on my PC because the thought of reading yet another manual is nauseating.
And there's the dark secret of most products: count on bugs and problems in those features that are rarely used.
Business is hard
The last impression we shared of this funny industry is our distorted notion of the ease of turning an idea into a product, and then into a profitable business. Jamie told tale after tale of customers backed by venture money, generally from unsophisticated investors more gullible than the average professional VC, who failed despite creating a working product.
There's a euphoria associated with starting a business, especially during the frenetic '90s. Some think that the unbridled optimism of those years was a unique feature of the short-lived dot-com zaniness. Not true. In 20 years of starting and running various companies I've seen a constant pattern of people-especially engineers-whose very cool idea distorts their vision of reality.
Other business owners confirm my observation: people come to us with ideas, constantly, in an effort to start some sort of partnership. Almost without exception (in my experience, at least), major flaws unrelated to the idea itself clearly doom the invention.
Engineers always seem to think that if the product costs $100 to manufacture, it can sell for $110. None anticipate the overwhelming costs of marketing and sales, of paying for overhead, and the cost of capital. In my ICE days our manufacturing costs were about $1,000 per unit, yet we had to charge around $8,000 to squeak out a minimal profit.
Others feel that placing an ad will make the world beat a path to their doorstep, which has been disproven a thousand times, even before the Web showed us, yet again, this bald fact: advertising is crucial but inadequate. A successful business learns to build marketing and sales through a complex and expensive set of interconnected approaches. Advertising is crucial but, by itself, totally ineffective.
And so our discussion and sometimes loud but always friendly debate roared into the night. I was delighted to find another embedded person in this remote island, and even more pleased by Jamie's spirited and thoughtful comments.
So if you see Voyager in some port, give us a shout and let's trade ideas. Or don't hesitate to let me know your feelings about this fascinating business at the Embedded Systems Conferences. There's nothing like a great debate.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at .