College taught me many things besides forging fake IDs. By senior year I could run circular integrals around any function and solve (some) differential equations. Work and energy were my friends. A year of studying Statics and Dynamics gave me tools I still use to understand the forces that keep the boat's mast up. Circuit Theory covered phasors (rotating vectors). Electromagnetics baffled with brain-crushing equations but I sorely wish I'd paid closer attention as fast digital logic is now all about impedance and transmission lines. Transistor theory, logic, programming, and a slew of other classes were meant to prepare us budding EEs for The Real World.
A new grad, though bursting with theoretical knowledge, is generally incapable of designing circuits, and worse at troubleshooting them. Technicians sniff with disgust as the newly-minted EE struggles with the scope, or burns up parts because he can't relate those work and energy equations to the real world of a quarter-watt resistor.
Over time, with experience, the recent graduate turns into a productive engineer, mostly by acquiring experience via the school of hard knocks. His fingers become calibrated temperature sensors, able to discern a hot chip from one that's operating on the verge of failure. That two-nanosecond glitch isn't an issue; the data bus is always a little uncertain when the drivers are all tristated. He may not remember the 2N2222's beta, but knows that with a 1k base resistor it'll drive that relay just fine.
There's a “feel” that derives from experience. Perhaps a definition of an engineer is one who combines analysis with feel to produce products. Where a first-year civil engineering student might thoughtlessly design a structure operating just at the edge of the materials' strengths (because, after all, the equations show that's a valid approach), the seasoned professional tosses in a bit more margin than required. He's seen things fail and his arm still aches from that incident in Reno years ago.
Embedded.com reader John Johnson sent an interesting link to an article that shows how humans subconsciously use a Bayesian approach to acquire a feel for the nature of the world. I slow down for a traffic light a certain distance away without thinking about the decision at all, the result of many years behind the wheel. My teenager is not yet quite so gifted, though his reaction times and physical capabilities far outstrip mine. It's amazing how, with practice, a batter can hit that little baseball flying towards him at 80 MPH or more. Yet the novice swings and misses every time.
The Bayesian algorithm that's somehow hardwired into our brains takes the result of prior experience, merges that with the current situation (say, crossing a busy street), and updates our experience file with the outcome of our action (we crossed uneventfully, or made it but are white with terror, or EMS scraped us off the pavement after being clipped by a Hummer).
I worry that many firmware folk avoid acquiring a similar feel for their creations. Developers guesstimate execution times and code sizes all the time. They're usually way off.
But mostly they don't know their guess was bad. Because if the damn thing works, we ship it. That ISR we figured would need 50 microseconds actually consumes nearly one millisecond at times, but no one knows, because no one measured.
It works. Ship it.
The schedule slips by three months but no one calculates actual hours spent on the project and compares those to the original estimate. No feedback loop means an important learning experience is cast away.
Too many developers simply don't allow their inner Bayesian processor to do its job, partly because of the stealth nature of software. There's nothing to fondle, see, nothing overheats or explodes (in the software, that is). Everything is under the hood. The system announces success or failure via bugs, but the “how is it working” lies deeply buried. Don't instrument working, apparently perfect code and you're effectively cheating your Bayes subsystem.
And that's a tragedy.
What do you think? What tricks do you use to improve your “feel” for firmware?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at . His website is .
Too many developers simply don't allow their inner Bayesian processor to do its job, partly because of thestealth nature of software.
I have seen it as a battle between Management and Engineering. Somehow, to the Management world, it looksbetter to cut the schedule in half and see engineers working nights and weekends to try and get the job done. It somehow makes the engineers who do so look “dedicated” and “committed”. And every manager wants a team ofdedicated engineers. So in a strange way, there actually is a reward system in place for a “tight schedulework your butt off” arrangement.
Also, I believe many managers do not trust an accurate estimate from Engineering. They somehow think theschedule is padded with tons of fun time and needless hours spent over-engineering the product. That is a bitodd, and even sad when most managers were once engineers – you'd think they would understand.
So in either of those cases, the engineer is not rewarded for developing an accurate schedule. I think that isone reason the engineer is dissuaded from “following their inner Bayesian processor”.
Lately, it seems I have been getting a better feel when I do unit testing on the code I write. Years ago, Iused to just do full-integration testing. But when working on some medical products, I learned the benefits ofunit testing. Somehow, it gives me a warm fuzzy that the code will run the first time it is loaded (almost)and that it will be consistently reliable for a long time. Unit testing almost doubles the code writingeffort, but some of the problems caught at that level can save countless hours at a customers site on a sweatyfactory floor with an oscilloscope.
– Lou Calkins
There are a lot of ways that a software engineer gets a feel for the craft. You give the example of aninterrupt routines timing. There are instruction set simulators for many, many different processors out thereranging from lowly 8051's to 1GHZ TI C64xx DSP's. This will let you set up and measure the ISR under a varietyof conditions before you have hardware designed, so you can do nice things like go back to the hardware guysand tell them to split the design into two processors with a dual port or shared memory BEFORE you build thefirst board and have to scrap it. TI's code composer studio for it's DSP's and ARM's lets you get the codesimulated in an instruction set simulator before you have hardware for example, so you have a lot better ideaof throughput, codesize, etc.
Other tools like this exist for other processors. Can't afford a simulator, then ASMFLOW or other assembly language cycle counters, plus a spreadsheet can do ina pinch on a simple processor like the 8051, or even an 8086.
It only takes one project that is low of memory to quickly give one a FEEL for doing codesize and memoryestimates before boards are done. Soon one knows about what will fit in an 8051, an 8086, and what takes thebig silicon.
Processor speed and throughput is a related area where one starts to get a feel with experience. One may learnafter a number of years in the industry that an 8051 will generate one DTMF tone, and not have much left overfor much else, while a top end TI C6X DSP can generate scores of DTMF tones and put each one on its own T1 timeslot, and have left over CPU bandwidth.
Good tools can help where you have not done something before, as can quality documentation from silicon andtool vendors. (Of course reading documentation and doing performance calculations and memory and code sizeestimates takes time which in these days of ever compressed schedules gets harder to justify)
– Bill Murray
I'm fortunate in that I develop both hardware and firmware. Because, as you say, I have developed afeel for circuits, it extends into a feel for firmware. Knowing how the hardware works is paramount to writinggood code. Today, to many programmers are separated from the hardware so never develop the synergy that comesfrom doing both (in many cases, simultaneously). Even with good optimizing compilers, knowing a lot about themicroprocessor is necessary. I don't mean only memorizing the data book becasue we know that data books don'talways tell the truth, or leave out some very important data. I feel you have to play with the actuallprocessor to see what it will do. Then, I think you develop a feel for what your code can and can not do. Atleast, it works for me.
– Bill Larkin
First, I'm not a firmware developer. I'm a general programmer. Started off as trades person who gota computer for Christmas and taught myself how to program it (machine language).
I guess that is why I'm a little different from the people I work with now. I could tune a car, rebuild anengine, etc. I liked the hands on stuff. When I started teaching myself about computers I used analogies tounderstand what was happening in the computer. I imagined electrons on the data bus to be postal deliverypersonnel driving on the highway. The memory addresses were homes that parcels needed to be delivered to.
As I learned more I realized that some of my analogies didn't work universally. People still use Newtonianphysics. I still use my analogies with the understanding that they aren't quite right.
I also noticed that as an instructor at a large university, many of my students would ask me “What happens if Itry X?” where X is some programming attempt. I ask them, “Have you tried?”
It seemed almost as if the students were afraid of trying things. Or at least unsure how to see if it worked.
I understand when you talk about using your finger to tell if a chip is running too hot. I can write a programthat blinks some LEDs or generates a tone. If the timing of the code is off I will see it in the LEDs or hearit in the tone. In other words, I cannot use my sense of touch with software but I have other senses.
One thing I have noticed that concerns me about software/firmware development is that seeing nothing wrong ispresumed to be something good.
Too many times I see someone assume the code works fine because they didn't see anything wrong when it ran.They are not really looking though. They are just looking for the obvious. They are not measure how right thecode is behaving.
– Darrell Grainger
I have the liberty of working with fairly large systems (several hundred k or even a few M offlash). I liberally sprinkle log messages through my code. Whenever something goes wrong, however small, Ilog a message. When doing something that changes the nature of the system (a new device attached, etc), I loga message. Those log messages get sent out a serial port or through a UDP syslog. As the system grows, Iwatch the log messages scroll and I get a feel for what a “normal” run looks like. There have been severaltimes when I'm not focused on each line of the log but I know something went wrong just because something “feltwrong” as the log scrolled by. I think that's the software analogy to the temperature checking finger.
– David Poole
I design high-performance ignition systems & related accessories, many of which use embedded 8-bitmicrocontrollers. I have found over the years that certain functions (interrupt latencies, for example) arevery predictable, and I can count on them to work within a certain time frame. I toggle port bits at thebeginning & end of certain functions to get an idea of how much time they take, especially if I am doingsomething new. What is interesting to me is that I use C for virtually all fo my projects, and I rarely run outof time to do the things I need to do. This leaves me with a lot of margin in case I need to tweak thingslater.
– Dave Telling
I wish I was as much a professional todo the things you are talking about. To that end, we'retrying a new tool for us: Rational Rose RT. There is a considerable learning curve, but we're hoping it willlead to less “stealthiness” in our implementation. Also, the forced component and publish-subscribe naturewill leave us better prepared to make the kinds of tests you are talking about, rather than just doingsystem-level tests because there is no time to test components. We don't have anything to do with Rational, soI sincerely recommend looking into it.
– Jim Beck
Good firmware engineers always instrument their code. I always (almost always) have an extra I/Obit or two that I can pulse to indicate software execution of an ISR or some other critical code. Attach ascope and you have real world feedback, not just a calculated execution time.
In addition to I/O pin, most modern embedded designs generally have some uncommitted RAM or EEROM that can beused to document firmware function, errors, critical paths, etc. In larger systems, an event log or file maybe invaluable to provide feedback not only in the case of product failure but also for routine analysis ofexecution margins. These logs can be evaluated after a beta is completed or at the time of a repair.
CLOSE THAT LOOP!
– Michael Weisner