I greatly enjoyed Jack Ganssle's “An Example of Foolishness.” During my nearly 10 years of experience in developing embedded systems, I have produced my own examples of foolishness. Once, after many days of complex debugging effort, a seasoned veteran of the PC world finally pointed out a basic bus protocol violation that was eventually identified as the root cause. He used the opportunity to tell me a story about how he had once torn his motorcycle completely apart looking for the reason it would not start. It was not until his bike was almost completely disassembled and the gas tank was “in the way” of what remained that he realized it was empty. This story and its essential lesson, “always check for gas,” has served as one of my guiding debug principles ever since.
Another aspect of debug is the quality of the team. I have had the great privilege to work on teams with some highly skilled and seasoned hardware and software engineers. These individuals, when put together, created a powerful cross-functional troubleshooting force that tackled some elusive bugs quickly. In an environment filled with professionalism, mutual respect, and trust, each individual brings fresh ideas and challenge assumptions. When allowed to run its course, this leaves nothing but facts, analysis, and solutions. I don't know exactly how this environment gets created, but I do know it is a product of the people on the team. When the discipline you describe is applied by a team in this type of environment, no problem stands a chance!
Chris St. John