A Question of Quality

February 01, 2002

MurphysLaw-February 01, 2002

A Question of Quality

Can quality be generalized across multiple industries? ISO 9000 and its promoters claim it can. Here's a look at the ubiquitous standard.

I have written a number of times in these pages about how to build quality into your code. Self-checks such as the assert macro help detect bugs, as do code inspections. At a higher level, the process by which software is designed, written, and tested affects the reliability of your product. While some techniques are appropriate to the software industry, other industries have their own means of guaranteeing quality. Accountants use double-entry bookkeeping to minimize errors, financial institutions perform audits, and so on.

Some organizations claim to provide a generic form of quality that is applicable across all industries. ISO 9000 certification is the formula promoted by many of these organizations. Some companies buy into ISO 9000 certification for marketing reasons and others are compelled by their customers. For example, numerous government agencies in many countries insist that their suppliers be ISO 9000-compliant.

Process

The ISO 9000 model is based on creating procedures for all of the activities of a business in such a way that all behavior is reproducible.[1],[2] Furthermore, the behavior must produce documentation to show that the procedure was followed. This documentation may or may not be of intrinsic value to the business. Its primary purpose is to allow the procedures to be audited. The great mantra of the ISO 9000 movement is, "If it is not documented, it was not done."

In many instances, a complete audit trail of all activities is desirable. A bank, for example, should be able to trace the money trail of a series of transactions. In some activities however, especially design, producing documentation solely for auditing purposes does not make sense. I do not always design in the same way. Some design issues require intensive customer consultation; others are best addressed with a textbook algorithm.

ISO 9000 encourages procedures in which documents define how the design and test should be done for all projects within an organization. Consistency across projects can be a good thing, but once this has been defined, it is difficult for engineers to use their own judgment to stray from the standard.

The most unpleasant side effect of this approach is the large number of documents that have to be maintained. But that is merely extra work, a problem that could, in theory, be solved by hiring more people. The more serious impact is that it stifles improvement or change.

In my experience, the procedures are written in one of two ways. The first is to observe current behavior and write the procedure accordingly. This is reasonable if we assume that the current behavior is correct. But whatever the quality of the current behavior, we have now provided an obstacle to anyone who wishes to improve it. The more common way to write the procedure is for the author of the procedure to write down how he thinks things should be done. When this ideal cannot be followed, the transgression must be documented so that we have a paper trail to audit.

Missing the point

Don't get me wrong, I believe in process improvement, and I also believe that external audits provide healthy input. But processes should be audited by watching what actually happens. ISO 9000 audits emphasize checking the paper trail after the work has been completed. If the paperwork is flawless, the audit will be succesful, even if the job was badly done. Similarly, if the work was perfect, but the worker didn't document it consistently, the audit would fail. The audit measures the quality of the paperwork, not the quality of the work.

Audits performed after the fact based on paperwork generated for auditing purposes achieve little. Quality must come from engineers, not from external auditors. The need to pacify external auditors does little to encourage engineers to achieve true quality.

When a team or company sets out to improve quality, it should pursue a distinct goal. Whether you aim for faster turn-around time for customer complaints or fewer bugs per thousand lines of code, the goal should be measurable. For companies engaged in the ISO 9000 model, the goal is certification. By setting certification as their goal, businesses neglect more important objectives.

I have no objections to documentation, but one simple test should be applied to all documents. If no one uses a document to produce something of value to the company, it should not be written. Auditing alone is not reason enough to create documents.

Winners and losers

The consulting firms selling certification are the big winners. They sell the audits, and if you become certified, you must be re-audited to maintain certification.

The most important difference between these auditors and the types who work for government bodies such as the FDA and the FAA in the U.S. is their mission. The FDA is mandated to keep drugs and equipment used in the medical industry safe; the FAA is mandated to keep the airways safe. In both cases, the goal is independent of how many companies conform, and these organizations do not seek out extra companies to regulate. Profit is not an issue.

On the other hand, the ISO 9000 auditors and consultants are businesses and their mandate is to ensure bottom-line profit. This means that the more companies in the game, the more money they make. Over a quarter of a million companies worldwide are registered to ISO 9000. The number will continue to grow as the ISO 9000 brand is marketed.

Other winners are big companies that can absorb the cost of devoting a department to maintaining certification. They always support processes that raise the cost of entry into their market.

Quality departments within large companies now have a definition of quality that can be applied without regard to the type of business in which the company is engaged. This is a perceived advantage in high-tech business, where hiring employees well versed in the technology is expensive and difficult. Quality staff with an understanding of ISO 9000, but not of the business or technology involved will only add a superficial layer of quality.

Among the losers are engineers who genuinely care about quality, but have the ISO 9000 version of quality forced on them. Software engineers are often subjected to audits by someone who audited, say, a candlestick maker the previous day. One consolation is that they rarely have to worry about failing an audit. Certified companies have to pay for the privilege of carrying the ISO 9000 brand. If an auditor flunks a company, they lose a customer. So it's not surprising that few companies ever lose their certification.

In many cases, the customer also loses when rigid processes reduce a company's flexibility and, therefore, its ability to adapt to customers' needs.

One of the ironies of the ISO 9000 movement is that it advocates measurement as a way of judging improvement. But the ISO 9000 industry can't produce any measurements to show that applying ISO 9000 has improved a company in any way. Something-productivity, share price, or customer satisfaction-should increase if the movement's claims are true, but no measurements exist to prove that this is the case.

Carrot vs. stick

Attempting to document every change to a project can reduce an interesting engineering job to drudgery. Some engineers will simply reject all the paperwork as a waste of their talent and find more satisfactory employment elsewhere. Losing smart engineers has a dramatic effect on product quality. For those that stay, the paperwork makes it all too easy to say "Look at all of the documents I created, I must be productive," when the end result may not contribute to quality or profits.

In many ways, the ISO 9000 approach reminds me of the skilled and non-skilled approaches to factory workers. In the skills-based approach, workers are considered a valuable and motivated resource. The alternative approach assumes that workers are lazy, and that procedures should be in place to catch them when they dodge work or steal. In some cases, this assumption will get good results. But negative results are also possible-if you assume your employees are lazy and force them to clock in, the workers may feel obliged to leave as soon as their shifts are over. A more flexible arrangement in which the worker is trusted often benefit both sides.

In some ways, ISO 9000 parallels the approach in which workers are assumed to be lazy. Perhaps quality by means of auditing is a direction to take if you believe that your programmers are lazy and self-serving, that they won't perform well unless everything they do is somehow recorded. I believe most programmers deliver more when they are trusted and given opportunites to improve.

Where did it come from?

ISO 9000 fails more miserably as it moves away from its origins. ISO 9000 was born in the bomb factories of World War II. The idea was to make bomb production on the assembly line a repeatable process. If each bomb was manufactured according to the same procedures, variations in the output would be reduced, and fewer of them would go off unexpectedly. In this context, a process that reduced variation made sense, even though it did not provide a path to improve the methods of production.

For a long period, the British government maintained a large staff of auditors who ensured that anyone who supplied it with equipment or services adhered to the appropriate standards. ISO 9000 gave them the opportunity to outsource all that work. The British government accepted that ISO 9000-certified companies met the appropriate standard. The ISO 9000 audits were then performed by private companies and not by the government itself.

The ISO 9000 was gradually applied to an increasing number of companies. Laid-off government auditors who were started consulting firms. One of the first items on the agenda of an audit was to check if the suppliers to the company being audited were also ISO 9000 certified. Pressure was exerted to ensure that all suppliers to the company under audit also had certification. ISO 9000 spread like a virus.

As with many rules that are applied too broadly, the ISO 9000 standard can lead to absurdities. In one case, a company of Morris dancers in England had to be registered in order to supply their services when hired for a festival by the local council. The council is a government body and could not hire anyone, including dancers, unless they were ISO 9000 certified.

As this certification was applied more broadly, the principle of reducing variation by using a repeatable procedure became less effective. A software engineer does not necessarily want to write every program in the same way. Principles that belonged to the manufacturing world were twisted to apply to software engineering, with no concern to whether they actually helped.

Indeed, software was considered different enough that it spawned the ISO 9000-3 standard and the Ticket system. While these standards were written in the context of software, they were based on the same principles: reduce variation, document everything, and record all variations from procedure.

I worked on a project in which I had to create a document for each test before the test code could be written or run. Writing the document itself meant applying for a document number, and controlling it under an automated version control system. Inputs and outputs of the test had to be specified in the document. As you can imagine, the data had to exist somewhere other than in a word processor to be useful for the test. This led to duplication of the data-one copy in the documentation and one copy in the test itself. Many more tests could have been performed had we been allowed to test without laborious documentation. The end result was fewer tests and, therefore, lower quality.

Not only were there fewer tests, but the tests we performed were of less value. Since all tests had to conform to a test procedure, their scope was limited. For example, using random input data in some cases might have been useful, and a little automation would have allowed a large number of cases to be tested with little effort.

But the procedures dictated that input had to be specified, so new procedures would have to be written to allow for random data, which required far more work than the tests themselves.

ISO 9000 achieved its goal of reducing variation. All of the tests were similar and addressed the same small family of bugs. Reducing variation in testing is not desirable. Software testing should be as diverse as possible. In this case, the quality of testing was measured by the amount of test case documents produced, instead of the real metric, which is the number of bugs found.

Who reads it all?

The motivation behind ISO 9000 is to produce documentation for audit purposes. This means that the majority of documents produced are never read, since auditing will only have time to locate a small percentage. This means that the quality of these documents will diminish over time. When an engineer realizes that he gets no feedback from a document-good, bad, or indifferent-he will not make the effort to ensure that the document is accurate.

Doing it right

The most common defense offered when ISO 9000 leads to a loss of productivity is that it was implemented incorrectly. I would accept this if someone could tell me how to know when it is being used in the wrong way. It can be used incorrectly in many ways and no guidance exists for limiting paperwork. The standards are so open to interpretation that it would be virtually impossible to eliminate flawed variations. There seems to be no agreed-upon method for detecting mistakes. Therefore, ISO 9000 does not offer a path to improving the standardization process itself.

Individual engineers in many companies are finding themselves railroaded by this standard and are often left baffled, wondering if it is their own fault that they cannot perform to their full potential in what is being presented as a quality-driven work environment. I would recommend anyone who is doing battle with this monster to read John Seddon's book The Case Against ISO 9000. Readers will be so well-versed in the flaws of ISO 9000 that they may at least avoid some of its worst side-effects.

Niall Murphy has been writing software for user interfaces and medical systems for ten years. He is the author of Front Panel: Designing Software for Embedded User Interfaces. Murphy's consulting business is based in Galway, Ireland. He welcomes e-mail at nmurphy@panelsoft.com.

References

1. www.iso.org

2. Seddon, John. The Case Against ISO 9000. Dublin, Ireland: Oak Tree Press, 2000.

Back

Return to February 2002 Table of Contents

Loading comments...