The Truth Behind Static Analysis Pitfalls -

The Truth Behind Static Analysis Pitfalls

Perhaps the most surprising thing about static analysis is not that itcan detect memory leaks, buffer overflows, or incorrect pointerassignments at compile time, but rather that users of static analysiswill often fail to fix such detected defects.

One of the most frequently misunderstood aspects of static analysisis that it is distinctly different from other bug finding techniques.Static analysis users are often aware of the advantages it provides indefect detection, while simultaneously failing to realize that adifferent approach must be taken for defect resolution.

Static analysis has never been “just another way” to detect defects,nor is it just a way to detect defects earlier in the development cyclethan was previously possible. Static analysis testing reports defectsin a different way than dynamic testing and user reported defects;these differences must be understood and appreciated to effectivelyimprove software quality with any static analysis tool.

Root Causes vs. Symptoms
When a defective application misbehaves, the user experiences a varietyof symptoms such as crashes and slow or incorrect response. Each ofthese misbehaviors is triggered by a root cause – some piece of code,which performs incorrectly.

Much of software defect remediation is based on the observation ofsymptoms, followed by debugging to identify the root cause. Typically,defect tracking tools and processes tacitly assume that the symptoms ofa defect are all a developer has to go on when fixing a bug. Indeed, itwould be unusual for a user to file defect saying:

Report 1: On line 145 of file.c, the array”description” is written to with an input buffer whose maximum lengthis 24, as can be seen from the test on line 143, however the size ofthe description array is only 16 bytes, as you can see in header.h line23.

Instead the user would be likely to file a defect that sayssomething like:

Report 2: Whenever I enter the description:”Longer than 16 bytes.” into the form, the program crashes.

Static analysis, however, reports issues in a manner of Report 1,not Report 2. It's easy to get excited about how much easier it will beto fix the underlying bug when given Report 1 than it would be to fixif given only Report 2, but consider for a moment – is Report 1universally better than Report 2?

In fact, Report 1 is missing something fairly critical as comparedto Report 2: the impact that this defect has at runtime.

Static analysis tools detect root causes; they pinpoint the locationand triggering conditions for a defect, but do not predict the symptomsthat will be observed at runtime. Contrast this with a user reporteddefect, where it may take considerable effort to identify the rootcause, but where the symptoms, and therefore the impact, will bycrystal clear.

Static Analysis and The SoftwareDeveloper
In my observation there are few cynical developers when it comes tostatic analysis. Absent of the concrete impact of a customer reporteddefect, many developers express substantial optimism that a staticallyreported defect will not occur in the field, with a plan to cross thatbridge when they come to it.

It's all to common an anecdote that a user of static analysis spendssignificant time debugging a customer reported issue, only to find outlater that the same issue was previously reported by their staticanalysis tools.

Sometimes I suspect that developers are not really optimistic, butrather that they are overworked, and as such, can't engage inspeculative behavior (such as fixing statically detected defects), overfixing a customer reported issue.

In reality, for a development organization to reap the quality andproductivity benefits of static analysis, management needs to supportthe effort by helping to ensure sufficient time is budgeted for theinspection and resolution of statically reported issues.

Matthew Hayward is Director ofProfessional Services for CoverityInc. Since he joined the company in 2005, he's worked withhundreds of development organizations to define and implementstrategies for effectively applying static analysis to improve softwarequality and security. He holds a M.S. in Computer Science, and B.S.degrees in Mathematics and Physics from the University of Illinois.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.