Internet Appliance Design

Have you chosen the best programming language for your current embedded project? Your answer may very well depend on how the phrase “best programming language” is defined. Perhaps you've chosen the language that produces the most efficient code for your particular processor; or maybe you've selected the one that will allow you to finish the coding more quickly or with fewer bugs. Then again, maybe you—like so many of us—are just using the same language you always have.

This month we have a pair of feature articles concerning the reality of connected-device design. The first, by Larry Mittag, looks at the state of the hardware, software, and connectivity suites we need to build true Internet appliances. It seems there is a bit of a lag between what big business thinks they can do in the short term and what embedded designers are actually able to do at this point.

In a separate article, Jack Ganssle looks at the state of Universal Serial Bus as a connectivity option for embedded systems, especially those that have to work with or alongside PCs. He begins with a brief introduction to USB and its networking paradigm. He follows that with a ton of helpful information for developers just getting started with the technology.

Before reading those, please join me for some language lessons.

Best fit
Even considered within the narrow scope of embedded systems, the decision of what language to use to implement the solution to a given programming problem is a difficult one. Many factors must be considered and different weights given to each of them. The factors relevant to a language decision probably include at least:

• Efficiency of compiled code
• Source code portability
• Program maintainability
• Typical bug rates (per thousand lines of code)
• The amount of time it will take to develop the solution
• The availability and cost of the compilers and other development tools
• Your personal experience (or that of the developers on your team) with specific languages or tools

All of these factors are important. Yet, in all of my years as a software developer, I have never once worked with any engineer or manager who gave either discussion or serious thought to the matter of language selection. Nor have I ever seen a formal “language selection process” documented at any company. The majority of embedded programmers just seem to assume that (almost invariably assembly or C when you're talking to embedded folks) is always the right choice. The fact that some of them always pick assembly and others C is one clue that the language decision is more complicated than they would have the rest of us believe.

During my years as an embedded developer, I've broken many of the conventional language assumptions in this field — with great success. I've used assembly language on occasion, but only once to improve the efficiency of a routine beyond what could have been done by an off-the-shelf C compiler. I've used C on memory-constrained eight-bit processors. Even more surprisingly, perhaps, I've made use of many parts of the C++ language without ever once paying a performance penalty. Heck, I've even written a piece of embedded software in Java (complete with one of those “too big and too slow for use in any embedded system” Java virtual machines). All of these things have been done in the process of creating real products that worked and were sold, and were mostly completed on schedule.

I'm absolutely convinced that matching your choice of programming language to the computing task at hand is essential. If you choose correctly, you will be rewarded with a straightforward design and an even easier implementation. If you choose wrong, you may run into problems at one of those points or, worse, during the subsequent integration and/or testing phase. A bad language choice may not keep you from finishing, but it could cause a lot of headaches along the way.

I think a big part of my success on past projects has been a result of good language choices. Unfortunately, I must admit to feeling that this may be a combination of experience, talent, and luck that can't be translated into a formal decision-making process. Too many variables and too many project-specific issues must be considered.

So, when asked to moderate a birds-of-a-feather session at last year's Embedded Systems Conference in San Jose, I chose language selection as the topic for discussion. I wanted to see if there was knowledge in my brain and in the brains of others that could be used to instruct people in the ways of language selection. This open forum provided an excellent opportunity to bring such knowledge out into the open. The discussion was lively and interesting, but did not lead to any major revelations. Since then, I've continued to give this issue thought from time to time.

I haven't yet been able to formalize my thinking on language selection. But I'm hoping that someday soon I will be able to. In the meantime, I'd like to tell you about a project that gave me reason to think as carefully as ever about language selection and the results of a related experiment.

Mission impossible
Your mission, should you choose to accept it…

While working as a consultant, you receive a call from an old colleague. It seems that a small local company, XYZ, Inc., has gotten itself into serious trouble with a new product development. They are in desperate need of someone to straighten out their embedded software. You express interest in the work, thank your friend, and make a quick call to the engineering manager at XYZ.

It turns out that XYZ has not traditionally been in the product development business. They've been manufacturing, selling, and supporting the same product designs for more than a decade. However, about two years back they realized that they would not be able to get all of the parts to build their flagship product for much longer (including one arcane eight-bit processor). If they were to stay in business, XYZ's owners realized, they would have to redesign this product and start manufacturing the new design by a certain date.

Shortly thereafter, an electrical engineer (fresh out of a local university) was hired to design the embedded hardware and software for the new design. He spent about a year learning his way around their old system, researching their options, and designing the new hardware. Then he spent another year writing 2,000 lines of assembly code to control the system. (An abominable one instruction per hour.) A few weeks before the first shipment of this new product was supposed to take place, the engineer (now with two years of “experience”) accepted a new job and left the company.

In the wake of their loss, XYZ's management realized that the new product was far from shippable. Despite the incredible amount of time spent on the software development, a fair number of bugs remained to be worked out of the firmware. The list of bugs even included problems like occasional system lock-ups that were indicative of major design flaws or implementation errors. In addition to that, several issues related to product performance (not bugs, per se) still needed to be addressed. No one else at the company knew anything about the assembly code and few notes or comments could be found in the engineer's working areas.

Despite a backlog of half a million dollars, XYZ may be on the verge of going out of business. The new design is not working and no parts are available to build the old one. Your new client wants you to attempt either a fix of the existing assembly code (the engineering version of a Hail Mary play, given their list of bugs) or a complete software redesign and implementation. Which would ultimately be faster? And, if you chose the latter route, what language should you use? In short, your client's existence as a company and the fate of its 40 employees may well be riding on your decision. So you'd better make a good one.

Decisions, decisions
Not so long ago, I was put in this very situation. The pressure to make the correct decision was, obviously, quite palpable. Because of the risks, I sought to be as logical and formal as I ever had been and to include the client in the entire process. The options at our disposal seemed to be:

• Fix the existing firmware. The biggest risk here was that this might not even be possible. The fact that the original author had gotten 80% of the functionality mostly working did not imply that the bug fixes and remaining 20% could be built around his infrastructure. I had encountered situations in the past in which software was so poorly designed that it was essentially beyond repair. I thought this might be another of them
• Redesign and implement the firmware. This was more of a sure thing (I could definitely make the system work this way), but it might take longer than fixing the existing code. And I'd probably have some bugs of my own. Who knew what would take longer? And, if I were going to rewrite the software, what language should I choose?
• Assembly. I've found that assembly language is rarely the correct choice. It suffers from a lack of portability and maintainability. Also, I wasn't familiar with this particular assembly language myself, so the learning curve might slow down my work
• C. The problem here was that the target processor was an eight-bit Microchip PIC16Cxx with just 8Kwords of ROM and 368 bytes of RAM. Before I could make a language decision, I'd have to figure out how much of an impact using C might have on the image size and program efficiency. Were there enough memory and instruction cycles to spare? And, if I chose C, whose compiler should I use?
• Others. I didn't think any other languages were worth considering for this project

To get a feel for the existing code, I decided to fix one of their minor bugs before making any decisions. This took 22 hours in all. A large chunk of time for one minor bug, but I'd learned a lot in the process. I'd learned what their existing development environment was like (and installed the tools on my computer), and I'd learned that the original author of the software didn't understand PIC assembly much better than I did. I also inferred — from the structure of his code — that he had no master plan at all.

By this point, I was leaning toward a complete redesign of the firmware. Judging from their list of problems and the state of the code, I wasn't sure if I'd ever be able to make it do what they needed. However, I thought I understood what the system had to do now and was pretty certain that the man-year the original programmer had spent on this was way too long. My gut told me this was a two-month project — max. I didn't want to waste time trying to fix the existing code if I was just going to end up rewriting it after all. A couple of wasted weeks might cause serious financial difficulty at XYZ. So I was going to play it safe and start over from scratch. Now I just had to figure out what language to use.

Assembly or C?
It turned out that one discussion I'd had with the client was the ultimate decision-maker. XYZ's management didn't like that the original programmer had spent a year developing a program that no one else at the company could understand. From what they'd heard, having the program written in C would make it both more portable (should the processor ever need to be changed again) and maintainable (far more C programmers exist than PIC assembly programmers, by any measure). They were right on both accounts.¹

I was also quite sure that writing the program in C would be much faster than writing it in assembly. However, I needed to give several issues more thought before fully committing to C. I needed to be sure that I wouldn't encounter any actual size or efficiency surprises. My experience has been that these are not limiting factors in 99% of all cases, but an 8-bit processor with 368 bytes of RAM and an odd-ball call stack was outside the scope of my experience base.

A little Web-based research showed that at least three C compilers were available for the PIC16 family. I downloaded demo versions for two of them and ran some experiments. I wouldn't need to make any library calls, so the extent of the impact of using C would be:

• Any extra RAM usage associated with more frequent function calls (my program would be more structured than the assembly version)
• Any extra ROM usage associated with the C startup code, compiler-supplied routines (for example, the PIC16 family doesn't have a multiply instruction), and overhead introduced by the compiler
• Any performance impact associated with extra instructions that can't be avoided when generalizing at the compiler level

It turned out that the extent of these three factors was relatively minor. The compiler would create overlays to minimize RAM usage. The C startup code was very small. Multiply routines would indeed add a few hundred instructions — but, then, the current program had to implement a primitive multiply function too. The existing program contained about 2kwords of code and 4K of look-up tables, in an 8K ROM. Assuming I'd need those look-up tables (it turned out that I didn't), that still left me room to make my program twice as big as the assembly version. I deemed that the worst case, given the requirements and my experiments, and selected C as my language.

Tortoise and hare
This is where things got really interesting. After better understanding the trade-offs between fixing their existing program and developing a new one (and having been burned once), management at XYZ didn't want to take any chances with this. We agreed that a second consultant would be sought to fix the existing code in parallel with my efforts to create a new program. For the first time in my career, I would have a chance to test an engineering decision in a real-world competition. Had I made the right choice in deciding not to fix the existing code? Would I be praised as a hero or burned in effigy?

Sure enough, another consultant (a PIC assembly expert, no less) was located within a few weeks. He didn't have as much time to devote to the project as I did, but the results were still instructive. It turned out that my new C program was ready before the assembly program had even been fully repaired. Hours worked were tracked on both sides and, at the point that we shipped the C code, here's how the numbers broke down:

• Fixing the existing code (including my minor bug fix): 86 hours
• Rewriting the code in C (including design and performance improvements): 185 hours

After 86 hours of repair, the assembly program was growing short on bugs. The other consultant had done an amazing job of eliminating the system lock-ups and almost all of the other flaws. I was impressed. However, none of that software's performance problems had yet been addressed. From my experience with the C code, these performance “tweaks” required a major rethinking of the design. I'm still not sure if the existing program could have been restructured to handle this paradigm shift or not.

After just 185 hours of designing, coding, testing, and debugging, the new firmware was ready to be shipped. It included all of the needed performance enhancements and pieces of it were later easily ported to another, similar piece of hardware. The C implementation was also modular, commented to be easily understood by others, and had even passed a thorough code inspection. All of that in about 10% of the time it took to write the original (bug-riddled) assembly program. Clearly, this program should have been written in C in the first place.²

More surprises
A major chunk of time I didn't include in the above figure was 26 hours I spent working with an XYZ engineer to debug a race condition in his software. This race condition was exposed when my C code was unable to communicate with a remote system with which the assembly code had no trouble exchanging information. After two long days in the lab, we finally determined that the true source of the problem was that the C code was twice as fast to respond to a sync command than the assembly code had ever been. With a shorter latency in the communications channel, a race condition in the remote software was exposed. Surprise, surprise, surprise… C had beaten assembly in terms of program efficiency.³

But that wasn't the only surprise result on this project. Another was that the new firmware image was actually smaller than the old one. Recall that the original assembly program consisted of about 2kwords of code and 4K of look-up table data. The C program contained about twice as much code (4kwords), but the look-up table data was eliminated in its entirety. Compiler-supplied mathematical routines made it much more natural to use equations to fit the necessary curves. These calculations did not need to be performed especially quickly, and the use of equations made calibration of the system significantly more flexible. The system's performance was thus improved at the same time that its memory utilization was reduced and the code made more readable and portable!

Choosing language more tuned to the problem at hand made the system design easier, the implementation faster (by a factor of 10), and the whole project run more smoothly. The end result was a program that could be easily maintained and/or ported to new hardware.

I would love to hear your thoughts, good and bad experiences, and other comments in the area of language selection. So, if you have two cents to offer on the subject, please drop me an e-mail. Perhaps together, we can nail down some objective criteria for selecting various languages.

Next month I plan to begin a protracted discussion of TCP/IP implementation details. In the meantime, stay connected….

Michael Barr is the technical editor of Embedded Systems Programming. He holds BS and MS degrees in electrical engineering from the University of Maryland. Prior to joining the magazine, Michael spent more than half a decade developing embedded software and device drivers. He is also the author of the book Programming Embedded Systems in C and C++ (O'Reilly & Associates). Michael can be reached via e-mail at mbarr@netrino.com.

References
1. In fact, they'd wanted it written in C in the first place. But the engineer they'd hired decided on assembly, apparently because C scared him. He convinced them to go along by saying vague things like “C is too big” and “C is too slow.”

2. To be fair, I did benefit from the work of the original programmer. In certain instances, I was able to borrow the algorithm for an essential peripheral interaction from his code, rather than reconstruct the sequence from a databook. If I hadn't had access to these working parts of the code, I probably would have spent about 100 more hours on the project (50 of them staring at a logic analyzer).

3. The speedup was actually the result of a better design. Assembly will always be at least as fast as C when executing the same algorithm.