The beginning and end of effective process for static analysis could besummed up as “inspect every defect and fix all defects.” As simple asthis seems, most practitioners of static analysis will not execute onthis approach.
Because static analysis identifies root causes it is easy for aknowledgeable developer with an understanding of the correct designbehavior of a piece of code to assess the severity of a given defectwithin that code. Data suggests that this inspection takes less thanfive minutes per defect, on average.
This inspection time is orders of magnitude less than thedevelopment time required to fix a severe and difficult to detectdefect that has already reached the customer. The vast discrepancy intime spent between inspecting a defect of minor importance versusfailing to inspect a defect of major importance that impacts a customerargues for an approach of inspecting all the defects.
While most development organizations can rally behind the idea ofinspecting every defect, the idea of fixing every defect is much morecontroversial. This is understandable for two primary reasons:
* Traditional dynamic testing and customer reported defects oftenrequire substantial diagnostic effort to fix, regardless of theseverity of the issue.
* Static analysis tools will report both false positives, and issuesthat are of minor importance.
With modern static analysis tools that essentially eliminatediagnostic effort and boast false positive rates under 15%, theseobjections largely disappear. While there are a number of compellingbenefits to fixing all statically reported defects, the fact that eachsuch defect will be a show-stopping customer bug is not among them.Some reasons behind advocating a strategy of fixing all the defectsare:
* Pedagogical: By requiringdevelopers to fix all the defects, good habits are instilled.Inconsistent and potentially crash-causing code may have no customerimpact when written as part of the test suite, but that is not a goodreason to permit such practices in development.
* Human fallibility: Thereason why static analysis tools exist and are useful is that humansare not so great at writing perfect code. In choosing not to fix allthe defects, we must trust our judgment as to which defects are notimportant. The only way to be certain every critical defect has beenfixed is to fix every defect.
* Efficiency: This isprobably the most surprising, and most compelling reason to fix all thedefects; doing so will be faster in many cases than not fixing all thedefects. The fact is, when some defects are not fixed, a process mustbe developed to decide which defects to address and which to ignore.This process will time consuming, and developers will often spend moretime arguing why a certain defect should not be fixed than the timerequired to simply fix the bug.
Neal Stephensoncoined the term “metaphor shear” to describe the phenomena wherebycomputer users are surprised by the behavior of an application, due totheir subscription to a flawed metaphor. Users of static analysis oftensuffer from metaphor shear surrounding the notion of a defect. Staticanalysis reports defects. However, these defects are an entirelydifferent species from those reported by customers or identified bytraditional dynamic testing techniques.
The key difference between static defect reports and customer defectreports is that static reports will identify a root cause of aneventual program behavior, whereas customers report the symptoms ofsuch root causes. This means that statically reported defects are easyto understand and fix, but that they will be lacking an inherent senseof priority.
Due to this key difference, it is imperative that the management ofdevelopment organizations foster the adoption and instill a balancedsense of priority between the two kinds of reports.
When implementing static analysis tools, providing time in thedevelopment schedule for using and advocating the removal of all staticdefects is imperative. It is also important that a knowledgeabledeveloper, who is familiar with the design intent of the code inquestion, inspect each static analysis defect.
Once all statistically detected defects have been inspected, theargument for fixing all such defects is stronger for several reasons,efficiency surprisingly among them.
Matthew Hayward is Director ofProfessional Services for CoverityInc. Since he joined the company in 2005, he's worked withhundreds of development organizations to define and implementstrategies for effectively applying static analysis to improve softwarequality and security. He holds a M.S. in Computer Science, and B.S.degrees in Mathematics and Physics from the University of Illinois.