Everything you learned in kindergarten was wrongThe complexity of designing embedded systems requires a special kind of teamwork. Here are some practical tips for managing embedded systems design projects.
Kindergarten is the time when little humans start learning about life with other people. They learn that one truck/doll/book/puzzle is enough for 10 kids if they know how to share. They learn how to sit quietly while someone else is talking, raise a hand when they have something to say, and wait for their turn to come. They learn that a job worth starting is a job worth finishing. Above all else, they learn that anyone can become president of the United States if they only put their mind to it. In short, they learned everything they need to know to sabotage an embedded systems design project.
When it comes to managing an embedded design project, everything you learned in kindergarten was wrong. The rules to follow for project success are:
- Sharing is NOT caring; selfishly protect your resources
- DON'T wait your turn; interrupts in a design are desirable
- DON'T finish what you started; evaluate every approach often and abandon one quickly if it's not working
- Everyone CAN NOT be president (or king or boss); choose a leader and follow him/her
Sharing is NOT caring
Almost every child entering kindergarten struggles when it comes to sharing his or her toys. Every kindergarten teacher feels the nagging doubt of failure if at the end of the year a student still refuses to share. And yet, shared resources are the number one danger to an embedded software project. If you truly desire a successful project completion, BE SELFISH!
No embedded project worth noting has a cakewalk of a schedule. To get the most out of a team in the shortest amount of time requires minimizing context switches. Superfluous corporate meetings, irrelevant training, offsite team bonding, these all jeopardize an embedded project's success by diverting the team's attention. There isn't an engineer (worth having on your team) who looks forward to a company all-hands meeting any more than a shopping excursion with his or her spouse. This kind of diversion eats into a project by taking an engineer away from what he or she loves: solving hard problems.
Many engineers leave work at the end of the day with a nagging problem on their mind only to get new inspiration between 6pm and 8am the next day. These same engineers are anxious to jump back into the design, problems or not, and any dilution of their attention erodes the end result in some way. Let your team dedicate itself to the project at hand, especially when you get into the meaty problems and looming deadlines. There will come a time when the project winds down and diversion is a good thing; until that time, do everything you can to let the team focus. Not only will this improve results, it will increase the engineers' morale.
A more insidious form of diversion occurs when your stars get pulled away to "consult" on other projects (generally the suffering ones). These interruptions are euphemistically called "multiplexing" (also privately referred to as "fire-fighting"), and the interrupter frequently appeals to ego to get otherwise sane people to accept ludicrous assignments, all while maintaining faith (publicly) that the main project will not suffer. It's good engineering practice to design your hardware to handle multiplexing, but not good human-resource practice for managing teams. Let your engineers concentrate on one task (yours). Just say no (and throw a temper tantrum in the conference room if necessary).
DON'T wait your turn
At the start of kindergarten, all parents worry about their child when the time comes to let the little lamb join the flock. Each of these children has likely been the center of several adults' attention and in the past five years had every worldly need satisfied in a timely manner. What a shock when 12 or 20 other humans like themselves are now vying for the attention of a single teacher, and the old methods that gained them attention in the past are exactly what that ambitious teacher is attempting to suppress. Get in line, raise your hand, wait your turn. Nice philosophy, but an embedded project disaster.
When it comes to an embedded systems project, decisions can't sit quietly in a circle with a hand raised; design tasks can't stand in queue quietly waiting to be ushered up to the drinking fountain. Projects must flow full-speed ahead in parallel, and the project team must be prepared to make simultaneous decisions. This can be disconcerting when one decision may have a ripple effect on other portions of a design, but this is all the more reason to attack problems as a team in parallel, applying the cross-functional strengths of the team to help flush out the what-ifs and tangential cause-and-effects. Projects can't afford to make every decision serially, because if you try to, you'll find the queue never ends.
This theory plays out even in the operation of the embedded processor, where in completing its assigned tasks waiting in line would be deadly. Embedded systems projects of any size (not just large ones) must be broken down into multiple threads and tasks, perhaps split between a foreground and background, perhaps with the assistance of a commercial real-time operating system (RTOS). Every portion can be designed to operate more-or-less independently if all abide by certain rules and budgets. Think of the U.S. Navy's and Air Force's crack flying teams: they seem to be flying around chaotically, coming together, and passing at high speeds in lucky near-miss situations. A project with multiple threads and tasks working independently with careful planning can take a seemingly chaotic set of operations and "fly" together just as smoothly, not stopping and waiting for everything else to finish, but each accomplishing exactly what it needs to do, letting another task pass by at the right time without collision.
In kindergarten we learn not to interrupt others as they are speaking; we wait patiently until they finish before we start to speak. When it comes to the internal workings of embedded systems, however, interrupts are the foundation of an efficient design. Constant interrupts are bad if nothing else gets accomplished, but this is just a failure to use interrupts wisely. Interrupts are the basis for a more efficient multitasking design. One example is a communications handler that quietly collects bytes passed in and stores them in a buffer until a foreground task is ready to process a new command. In this case, use an interrupt to indicate that the byte buffer is full, so that the main processor can briefly stop what it is doing, transfer the byte to the storage buffer, and then go back to whatever it was doing.
DON'T finish what you start
The hardest lesson I had to learn in kindergarten (and I'm not sure if I did learn it) is that once you start something, you need to finish it. Or often put another way, a job worth starting is a job worth completing. Perhaps you heard the teacher say, "you need to put your mind to one thing and complete it before starting something new." Please don't take this mindset into your embedded systems project.
The reality in embedded systems design (and in many of life's endeavors) is that the first approach is not the best, and at times entire projects that one day appear to be exactly what needs to be done as fast as possible soon reveal themselves to be symptomatic distractions, not the far-reaching exhaustive solutions you'd hoped for. For this reason you need to remain open to the possibility that a chosen approach will not work and stay critical of your decisions until they prove themselves out.
It isn't good practice to willy-nilly start down one path one day and just switch to the next for no reason. Every problem has multiple solutions, so it's necessary to collect the data that drove a particular approach and to revisit the data and conclusions often, so you can realize when new data indicates that a better approach is possible or required. Make each solution prove its merit with real data, not just theories and hypotheses. Be careful not to become emotionally attached to an approach and lose sight of mounting data against the original assumptions.
This flies in the face of "let's finish something first before we start something new" but when you stay objective, data-driven and critical, prepared to make quick shifts in your approach intelligently, with your eyes open, you can get to the final destination faster. We don't like to believe it, but experienced embedded systems designers know that the last 20% of a project really does takes 80% of the time, and the last 20% of a poor design takes 80% of your weekends. Making intelligent approach changes early on might look like a lot of effort is thrown away, but in reality earlier is the best time to make a change (and remember the 20/80 rule when someone says it is too late to make a change).
Everyone CAN't be president
It's a grand and glorious notion that in the United States, any boy or girl who sets his or her mind to it can become anything they want to be and perhaps even rise to the highest office in the land. These days there might not be many parents who would wish the presidency on their child, but the widely accepted notion is still that anyone can achieve the top spot given sufficient motivation. If an actor and a peanut farmer could do it, theoretically any American could.
In your embedded systems design project, this too is a disastrous notion, and it needs to be stamped out. Although some democracy is useful for gathering opinions from multiple perspectives to obtain a complete picture of a problem and its solutions, majority rule is not the way to run an embedded systems project. Benevolent dictatorship is a better analogy. One person needs to be the decision maker, and not everyone is equipped to take that spot. And it isn't always the smartest guy in the room who gets to choose, either. Nor will the solution he chooses always be the most elegant one.
One leader, one "the buck stops here" decision maker is required, and only one. That person must be able to make the hard choices ("we take path A; I'll go tell management we have to slip the schedule") and must be able to collect the data and present it in a way that supports the decisions to both the team and the rest of the world ("we need to release this version as our beta and tell the customers about the XYZ problem, so we can get good feedback on the ABC feature"). These kinds of decisions can result in casualties, but someone has to stand up and make the decisions consciously, confidently.
As in any successful restaurant, you can't have too many good cooks but more than one chef is too many. The cooks need to accept it and not hold the secret belief that "if I really wanted to be the chef, I could do it." In an emergency, lack of respect for leadership can cost lives. This is also true of an embedded systems project, which may look like an endless series of emergencies; the price on such a project is paid more slowly over a longer period of time. Chose a leader, work with your leader, give him or her good information, and respect the value of all the roles on a team. Since the leaders of the next project come from the ranks of past projects, everyone benefits by following these guidelines.
Kindergarten was useful, just NOT in the embedded lab
Let's face it, Mrs. Miller (or Mr. Dalrymple, whoever your kindergarten teacher was) might say, "Shame on you," but embedded systems design is not the same as circle time. Share your stapler, but guard your team resources. Wait patiently at the drinking fountain, but fill your design with preemptive interrupts and tasks. Complete the training presentation describing evacuation assembly points in the parking lot, but rapidly kill a feeble design approach (or doomed project). Encourage a workmate to lead the company softball team, but make sure everyone on the project knows who casts the deciding vote. Then watch successful embedded systems design projects flow from your team.
Jon Pearson is the development tools product manager for Cypress Semiconductor's PSoC programmable devices group. Jon has led and participated in many embedded systems designs over his 20-year career and has experienced the good and bad aspects of all concepts he described. Jon attended kindergarten at Plymouth Junior High School in Minnesota, which may explain his confusion of the lessons they tried to teach. You can reach him at email@example.com.
I agree 100% with the point of not deluding your project momentum with worthless 'diversions'. Embedded engineering is not a 'touchy-feely' undertaking. It requires pragmatic, dedicated thinking and commitment. Have the beer & pizza party when the beta unit ships!
- Ron Owens
Sr. Design Engineer
Very good article
- Ganesh Naik
Sr. Software Developer