Hardware engineers employ all kinds of design reviews and processes, including design for manufacturability and design for testability. It's time software engineers stood up and asked for what they need most: design for debugability. Here's a list of ideas to make systems easier to debug.
Embedded software's close connection to hardware means programmers spend almost as much time looking at schematics and probing circuits as our hardware-oriented brethren. And yet we programmers are often not invited to hardware design reviews. Well I say, “No More!” We're always hearing about DFM (design for manufacturability) and DFT (design for testability). I say we need another acronym: DFD-design for debugability.
This article outlines some hardware changes that will make debugging easier. Most of the suggestions do not provide any direct benefit to the hardware designer, and some of these tips may even make their lives more difficult. As a software designer, you may have to fight for some of these changes. If so, just point out that debugging software is usually on the project's critical path. Anything the hardware designer can do to shorten the time spent debugging pays off directly in the schedule.
Of course, hardware designers do think about debugging when they draw up the schematics and lay out the board. It's just that their debugging needs and our debugging needs can be quite different. They have typically proven that large parts of the hardware function correctly either through simulation or breadboarding, so they don't worry about probing each and every signal. And often the easiest parts to get right, like address and data buses, are the parts programmers most want to probe.
Most projects include some level of hardware documentation, such as requirements specifications, design documents, and interface documents. A lot of information in those documents can be helpful during debugging, but having a pile of documents with you in the lab can be more hassle than help. When I'm debugging, the only two things I want to look at (besides my code) are a schematic and the printed circuit board (PCB). That means that I need to transfer some of the documented information onto one of those two media.
One of the most common ties between software and hardware is the addressing scheme. Most designs have ROM, RAM, memory-mapped I/O, and other addressable devices. The hardware designers often select these addresses either with discrete address decoding logic or with a programmable logic device (PLD). Sometimes the software sets up those addresses using programmable chip selects. Either way, the addresses of each device should be labeled on the schematic right next to the reference designator.
Hardware designers might argue that addresses could change during debugging, so this information shouldn't be on the schematic. I believe that you should put the information where it can be most helpful, and if that means changing a document or at least marking it up, then that's what you do.
The hardware designers spend weeks working on the schematic and looking at data sheets for each chip, so when they see “74AC244” printed next to a chip they think “4+4 Tristate Buffer”. Software designers aren't as familiar with chip numbers, so it would be easier for us if a descriptive label were printed on the schematic.
This goes for pin names, too. The pin names on some chips are too generic to infer any functionality. These can't be changed because they're tied directly to the part in CAD's tools library. But the hardware designer can add labels to the schematic identifying individual pins by function; for example, “Backlight” instead of “P1.0”.
Oftentimes the schematic is littered with node names. Typically these are cryptic shorthand descriptions like “POS_OFF_CH” for positive offset channel. If you can't convince the hardware designers to spell out their node names, at least try to convince them to add
full-length labels near the chips. It can also be helpful to label a signal as an input or output, as in “POSITIVE_OFFSET_CHANNEL_IN”. Of course, doing this for every pin around a chip would clutter up the schematic quite a bit, so it might make sense to only label key pins or groups of pins.
One place where I definitely want to see a label for every pin is on a connector. This includes signal pins, no connects, voltages, and grounds. Also, connectors should be laid out on paper just like they are in real life. If the connector is a two-row header with pin 1 in the upper left corner, then its symbol should be two rows with pin 1 in the upper left corner. Connectors should also have functional labels like “Serial” or “Display”. If there's room, ask the hardware designer to put the part number and manufacturer next to the connector. If there's still room, ask them to add the part number of the mating connector. If there are a lot of connectors, it may make sense to create a separate schematic sheet that gives the mating part numbers for every connector in the system.
If the board includes application-specific LEDs, suggest adding labels that match their functionality like “Alive” or “Rx”. If there are generic LEDs just for debugging, label them with I/O identifiers like “P1.0” or address:bit identifiers like “0xFF00:2”. Another helpful piece of information is the polarity of the device. If the LED is active low, precede the label with a slash as in “/Rx”. With these labels in place, a quick glance at the schematic tells you everything you need to know to use the LED as a debugging tool.
It's common for boards to have some kind of hardware selectable functions using DIP switches, jumpers, or solder shorts. When these are present on the schematic, ask the hardware designer to add some text next to the device explaining the different options. For example, a jumper might be labeled:
J3 = RS-232
/J3 = RS-485
Some chips have mode pins that are typically pulled high or low for a given application (as opposed to being controlled by the software). In these cases, add some text around the part stating what mode is being used.
In general, think of pin names as variables and schematic labels as comments. Would you want to read someone else's code if they used variable names like “XCVR_CK” or “POS_OFF_CH”? What about function names like “U12” or “SAA1064”? Once you understand what a given signal or chip is doing, coming up with a descriptive name for it is easy.
Figure 1: A schematic drawn with ease-of-use in mind
Figure 1 shows a schematic that has been supplemented as suggested in many of these examples.
Programmers typically consult a schematic when they need to figure out how something works. That's why most of the previous schematic comments dealt with describing the circuit's functionality. But when programmers look at a circuit board, we're usually trying to find a signal we need to probe.
The labels on the printed circuit board (PCB) should help us find the signal we're looking for, and the layout should make it easy to get at those signals. Of course, it would be nice if the PCB incorporated all the labels and text placed on the schematic, but that's just not going to happen. Imagine telling the hardware designer to increase the size of the enclosure by 10% to make room for labeling every pin on every connector!
The most common thing programmers do in the lab is hunt for a specific pin on a specific chip. Unfortunately, chips seem to be simultaneously getting smaller and sprouting more pins. Have you ever tried to find pin 135 on a 208 pin QFP? The following simple labeling rules will help you find the pin you need quickly:
- Clearly mark pin 1
- Label the pin number of each corner
- Mark every fifth or tenth pin with a dot or line
An example of this is shown in Figure 2.
Figure 2: Pins labeled on PCB for ease-of-use
The first signal you need to find when debugging is a ground for the scope probe. There are ground pads and vias all over the board, but they're hard to find without labels. Ask the PCB designer to label a ground pad or via near each group of signals you may want to probe. If board space is too tight to write “GND” all over the place, try using a symbol like the standard three-line ground on schematics. Grounds on connectors are especially important. Not only are they great places to clip a probe, but they also provide some visual feedback when plugging in a cable (if the red wire in the cable is attached to the connector's ground pin, you'd better think twice before turning on the power).
Ground is most commonly used for scope probes, but for some reason most scope probes come with a relatively huge alligator clip. Trying to attach that thing to a ground connection without shorting something out can be pretty tricky. In The Art of Designing Embedded Systems, Jack Ganssle advocates cutting off the alligator clip and replacing it with a micro grabber. I applaud the idea, but I'm not sure what my manager would say if he saw me take a pair of wire cutters to the brand new $150 scope probe he just bought. Ganssle also suggests soldering through-hole resistors into some of the ground vias, then cutting the lead just below the resistor body. This results in a handful of stiff, 1 to 2 in. stubs that you can easily attach that ugly alligator clip to. My manager doesn't care how many resistors I cut up, so I usually choose this option.
Other signals that programmers commonly look for when probing include triggering signals like chip selects, read and write lines, and clocks. Ask the PCB designer to clearly label these signals. Once in the lab, you may want to attach the resistor stubs to some of these signals as well.
Sometimes I find that a signal I want to probe is almost impossible to get to. I've worked on boards where the write line went to a via underneath the processor, then through a middle layer to a via that came up beneath another chip. Work closely with the PCB designer to make sure that these signals are accessible. If a signal doesn't already go to a via that's easy to get to, add one. And make sure a label gets attached to the via so you can find the signal later. It may even make sense to bring these common signals to a header so you can easily hook a probe onto the pins. Just be sure the pins are labeled.
When it comes time to lay out your board, be sure it has a mating connector for a logic analyzer (the user's manual will list the part number and manufacturer). And don't let the hardware designer skimp on the signals! Make sure every single pin on that connector has something on it. First and foremost, put the data and address buses on the connector. Then add bus control lines like read and write. If your system has a serial bus like I2C or SPI, bring those signals to the connector as well. If every interesting signal has been routed to the connector and pins are still unused, start adding uninteresting signals. If it takes two connectors, insist on two. The connectors don't have to be populated during production, so they only cost you a little board space.
The first debugging technique programmers learn is to send debug data to a display (“Beginning of interrupt: X = 2”), but a lot of embedded systems either don't have a display or have a tiny one with no extra room for debugging information. Even if the system does have a display, writing to it can often be a clock-consuming exercise that changes the system's performance.
Debug data can be captured with a much smaller performance penalty using a logic analyzer. Ask the hardware designer to add an unassigned I/O address to the address decoder and to route the signal to the logic analyzer connector along with the data lines. When you want to capture some real-time trace data, set up the logic analyzer to capture data written to that address. Then add debug statements to write data to that address. Of course, this will still alter the system timing, but it will typically cost a lot less than writing to a display. See Stuart Ball's book Debugging Embedded Microprocessor Systems for more ideas on using a logic analyzer to capture data.
If you're using an in-circuit emulator (ICE), make sure the PCB designer hasn't placed any tall components or connectors near the processor. One project I worked on had a connector right next to the microprocessor, which placed it underneath the ICE. The mating connector and cable stood about two inches in height, but the ICE only had an inch or so of clearance from the board. We had to replace the connector with a right-angled version so that we could attach the cable while the ICE was in place. On another project, the board had a small LCD mounted just to the side of the microprocessor. When the ICE was in place, the cable coming off the ICE completely covered up the LCD.
If the board is very small and not too complicated, it may make sense to start with a special layout that doesn't meet the size requirements. With a larger board, it will be easier to convince the hardware and PCB designers to add the connectors and labels mentioned previously. Another benefit of using a special board layout is dealing with processor packages. I worked on a project recently where board space was critical, forcing us to use a TQFP package. We had an ICE for the part, but we didn't want to balance two pounds worth of emulator on top of a TQFP adapter soldered to the board. The microprocessor also came in a much larger PLCC package, but it wouldn't fit because of space constraints.
Our solution was to layout two boards side-by-side on the same PCB. One was designed to the production footprint with the TQFP package. The other board was two or three times larger and had a PLCC socket. We used the larger board with the ICE to debug the software, then we burned a TQFP part and soldered it to the smaller board for testing.
Another common problem when laying out a board for use with an ICE is that the microprocessor and the memory are often very close to each other. When the ICE is in place, probing the bus control lines can be difficult if not impossible. Ask the PCB designer to place the logic analyzer connector far enough away from the microprocessor that it won't interfere with the ICE. If you're not using a logic analyzer, ask if the memory chips can be moved or the signals can be routed to a set of vias or a connector away from the microprocessor.
I should add a note of caution about all of the layout suggestions I've made so far. If the board has high frequency signals, altering the layout and adding stub resistors can drastically alter the board's performance. Work closely with the hardware designer to ensure that the signal integrity isn't affected.
One last word about PCBs: order spares. Blank PCBs are great tools when you're trying to find a particular reference designator. They're also great for planning those inevitable cut-and-add modifications since it's much easier to follow traces when there are no parts on the board.
If you're lucky enough to have some spare I/O pins, make sure they're accessible. Hardware designers typically ground unused inputs and make unused outputs no connects.
One particularly nasty problem with grounding pins is that isolating the pin later can be a real pain. Often there are two or three pins in a row that get grounded, and the easiest thing for the PCB designer to do is to run a tiny trace between the pads of all the grounded pins, then go to a via (sometimes underneath the chip) to the ground layer. Isolating one of those pins involves cutting the trace on both sides of the pin, then running a jumper wire around the isolated pin so that the rest of the pins are still grounded. Of course, you can also try the more daring solution of just raising the pin off the pad and soldering directly to it. Just make sure the pin doesn't get pushed back down when you're probing it.
Unconnected outputs are easier to get to by soldering a wire directly to the chip, but this can be difficult to do cleanly on small parts with tight pitches.
Fortunately there's a much simpler solution. Ask the PCB designer to connect every unused input pin to a pull-up or pull-down resistor. Every unused output can be connected to a resistor that is not connected on the other side (see Figure 1). The resistor isolates the pin from other pins without having to cut any traces and provides a pad that can be easily soldered to. Having pull-ups or pull-downs may even be a requirement if the board is going to undergo any kind of in-circuit testing. Place the resistors during the first prototype build, and then remove them as needed during debugging. As an added benefit, soldering a wire to the resistor pad is easier and less error prone than soldering it to a pin on the microprocessor.
My last piece of advice on the hardware side is to add a reset switch to the board. There are a number of small, surface mount switches that can be used for this purpose, so space shouldn't be an issue, and a reset switch will almost certainly come in handy during debugging and testing. This is especially important when using an ICE, since resetting the ICE won't reset the rest of the board.
Embedded programmers spend a large portion of their debugging time in the lab looking at schematics and probing PCBs. Consequently, helpful changes to the schematics and PCBs can shorten the debug cycle. So the next time there's a hardware design review for your project, make sure your name is on the list of attendees. See how receptive the hardware designers are to a few of the ideas mentioned in this article. Try out different ideas on different projects to determine which are most helpful and easiest to sell. Then develop a design guideline that includes all the ideas that worked and add it to your standard set of hardware design review documentation. The ultimate outcome will be higher quality products, shorter project schedules, and a stronger team atmosphere.
Tony Gray holds a BS in computer engineering from Lehigh University and has spent the last 10 years developing software for embedded systems. He currently works for Invensys Climate Controls. You can reach him at .