Avoiding the most common software development goofs - Embedded.com

Avoiding the most common software development goofs

Finding defects in code has been the bane of developers' existencesince the earliest days of computer programming. Maurice Wilkes, theBritish computer scientist best known for his work on the EDSAC, said in 1949:

“As soon as we startedprogramming, we found to our surprise that it wasn't as easy to getprograms right as we had thought. Debugginghad to be discovered. I can remember the exact instant when Irealized that a large part of my life from then on was going to bespent in finding mistakes in my own programs.”

This keen observation from more than 50 years ago still resonateswith anyone tasked with developing software. But why do we makemistakes? And what are some of the ways that we can avoid makingmistakes in an attempt to diminish the task of debugging software afterit is written? In this paper, we use our years of experience fromdeveloping and commercializing static source code analysis to helpanswer these questions.

During this decade, we have analyzed hundreds of millions of linesof code, seen programming errors from the very simple to the mostcomplicated and heard first hand accounts of the bugs that killeddevelopment organizations. While it is an impossible task to relate allof the relevant and interesting anecdotes in this type of discussion,our aim is to convey the general impression of what mistakes keepdevelopers and managers awake at night.

This article is excerpted from a paper ofthe same name presented atthe Embedded Systems Conference Boston 2006. Used with permission ofthe Embedded Systems Conference. For more information, please visit www.embedded.com/esc/boston/

As a means for communicating our experience, we first discuss thecost of mistakes in software development and hypothesize as to whydevelopers make mistakes. Then, in an attempt to help developersidentify their most common mistakes as they write their code, weexamine some of the categories of these mistakes, both from a puresource code perspective as well as from a higher level programmingmethodology perspective. Finally, we make the case for automatictechnology to help weed out these mistakes earlier in the developmentprocess.

The cost of software defects
It is a well known fact that software defects are a very costlyproblem. According to a study commissioned by the NationalInstitute of Standards and Technology (NIST), softwareerrors are costing the U.S. economy an estimated $59.5 billionannually. The study also reports that more than one-third of thesecosts could be eliminated by an improved testing infrastructure thatenables earlier and more effective identification and removal ofsoftware defects.

Drilling into the problem further, it has been shown that the costof discovering a defect increases drastically the later it is found inthe development lifecycle. A defect found during the coding phase of aproject is very inexpensive to fix. This makes sense intuitively sincethe developer responsible for the defect is working on the questionablecode, has all of the context of that code in his head at the time thedefect is discovered, and as such, can make a reasonable fix in a smallamount of time.

When that same defect slips into the QA or system integration phaseof the development lifecycle, it now can become an order of magnitudemore expensive to address. Now the defect must be discovered as theprogram is being executed and the person who discovered the defect mustreproduce the defect and communicate the errant behavior with thedevelopment organization.

Then the development organization must determine which part of thecode was likely to cause that particular fault, assign the appropriatedeveloper or developers to investigate further to determine the rootcause in the faulty code, then finally fix the defect withoutintroducing other problems into the code.

Another order of magnitude in cost is added if a defect slips passedthe QA organization and reaches the field. Not only does anorganization have all of the above issues in removing that defect, theorganization must now deal with the additional cost of reproducing theissue through their support organization, not to mention the cost ofbad public perception surrounding their “buggy product.”

Software defects end up costing organizations millions of dollarsevery year. But the problem is not because the cost of discovering adefect in the field is high; it is because organizations arediscovering defects in the field. The distribution of defects acrossthe development lifecycle (from coding to testing to release) is whatdetermines the actual cost of those defects to the organization.

If two organizations each have one thousand defects in their codeand the first finds them all in the coding phase but the seconddiscovers them all after the product has been released, the firstorganization is in much better shape financially. Therefore, we mustfocus on discovering more defects earlier in the process.

Why do developers make mistakes?
If it's clear to everyone that software defects are an expensiveproblems (and we assume that it is), why do developers make mistakes?Or rather, why do they make as many mistakes as they do to the pointwhere NIST performs studies and shows that it is costing businessessixty billion dollars a year? Based on our experience in developingsoftware as well as interacting with thousands of software developersand seeing the types of bugs that come out of the software developmentprocess, we view the following as the top reasons developers makemistakes.

Ignorance. The reader mightthink from this header that we are taking a shot at the educationalsystem that trains our software developers, but that is not the thrustof this argument. Developers are ignorant of the systems that theydevelop. A single developer can keep thousands, maybe even tens ofthousands of lines of code in his or her head for the purpose ofperfectly understanding how different pieces of the code interact.

However, today's systems are in the hundreds of thousands, if notmillions or tens of millions of lines of code. A single developerworking on that type of system will be calling functions or methods ofwhich they are quite ignorant. The pieces of the code that he is forcedto interact with may have been written years ago by someone who is nolonger available to explain their intent or nuance. So the developerdoes his best, quickly reading though the implementation or thecomments (potentially incorrect!) provided when he needs to interactwith another piece of the system. And this leads to errors.

Stress. We mentioned abovethat the developer does his best to “quickly” read through theimplementation of a piece of code that he must interact with. If youare a developer, you probably didn't think twice about the phrasing ofthat sentence (nor did we when writing it) because that is the realityof any software development process. Managers put pressure ondevelopers to generate code quickly ” deadlines come fast and thisleads to hasty coding and that leads to mistakes. Often these mistakesare not necessarily in the most common case of the code (since that iswell tested), but on edge cases. When time is of the essence anddevelopers are stressed, the parts of the code less traversed suffer.Yet these defects can be just as costly as mainstream bugs.

Boredom . Not all coding isrocket science. In fact, a good number of coding projects, once thedesign is complete, would be classified by most developers as “boring.”Of course, if a developer is bored, he is much less likely to producegood code than if he is excited about his work.

Pounding out those last few cases in a switch statement when thefirst few took dozens of minutes can be just mind-numbing enough toswitch off the brain and make the simplest of mistakes. Boredom alsoleads to shortcuts ” if you are bored with any given task, you areprobably looking for ways to eliminate your boredom as quickly aspossible. And unfortunately, a shortcut in coding often translates to adefect in the code.

Human Frailties. Certainlythe above points play into this last point about the very nature ofhuman beings. Humans are creative and intelligent and able to solvedifficult problems through reason. However, we are not robots. We arenot so good at repeating the exact same operation thousands of timeswithout some variance. If you doubt this, pull out a piece of paper andsign your name ten times.

Signing your name is probably something you've done thousands oftimes in your life, yet each time is a little different. This variancemeans that even if a developer understood every interface in a systemperfectly, had all the time in the world, and were programming the mostinteresting project computer science has ever known, he would stillmake a mistake in the translation from the design in his head to thecode that he writes. That is just a fact of life.

Common goofs
When discussing common programming defects, we have (at least) twochoices for categorization. We can either categorize based on rootcause in the code (e.g., nullpointer dereference, failure to unlock after acquiring a lock, bufferoverrun, etc .) or based on a higher level reason for the mistake(e.g., improper error handling,typo, copy and paste , etc.).

Having a hybrid of these two categorizations is difficult in thisformat, so we choose the latter because we feel it gives a better sensefor why a particular defect is introduced. However, we acknowledge thatthis higher level categorization is very subjective. We're not here toforge new territory in defect classification, but rather want to shedlight on why we believe these defects are made.

The examples below are admittedly toy fragments meant only tohighlight the particular issue in the discussion. Bear in mind thatthese problems do manifest themselves over hundreds or thousands oflines of code within and across functions and methods in real systems.

Ignorance. If you were toask most developers, “should youreturn a pointer into data on the stack? ” they would answer aresounding no. However, from time to time, we see the following type ofcode in programs:

The function looks simple enough ” it is putting a name into acharacter array and then returning that array presumably for the callerto use. However, once the stack is popped upon return from thisfunction, that pointer is no longer a reliable piece of data. Onceother functions are called, the data containing that name will belikely overwritten. To make this function work correctly, we shouldallocate the memory dynamically so that it persists past the end of thefunction:

Now the caller of the function can trust that the pointer points tovalid data for as long as that memory is not freed. Imagine a potentialcaller:

This code will work just fine in printing the name. However, noticethat with the change to the get_name function, we now have introduced aresource leak in calls_get_name! If the developer implementing calls_get_name does not realize that the implementation changed, there is a defect dueto the developer ignorance of that changed interface.

Copy and paste. Now supposeour developer is tasked with writing a function similar to get_name , butthat instead duplicated the name of an incoming parameter, thedeveloper would likely copy and paste the original code. Copying andpasting code is a common practice and often stems from developerboredom (since the task is not seen as interesting) or from time stressin not having sufficient time to code a function from scratch. So, thedeveloper copies get_name as follows:

And then he changes the name and adds a parameter:

Then he just changes the part that does the strncpy to call strdupsince he knows that's a good way to duplicate a string:

And now the function works as desired. However, the astute readernotices that in the midst of the copy and pasting, the developer hasleft the original call to malloc in the code, thus causing a resourceleak on the very next line when he reassigns the temp_name pointer:

Error handling. One of themost common problems we see in code is in the handling of errorconditions. Programmers tend to program for the common case leaving theoutliers, from a path execution standpoint, largely untested. However,these outliers are exactly the scenario that the end user is likely tohit as the load becomes high or the application has been running fordays or weeks at a time. Examine the following piece of code, pulleddirectly from Linux:

Here a lock is being acquired near the beginning of the functionwith the call to spin_lock_irq .And on the common case, right before the end of the function, thecorresponding unlock function is called. However, notice that there isan error case in the middle of the function depending on the returnvalue of vortex_adb_allocroute .If this function fails, the calling function returns without unlockingthe acquired lock! This can lead to deadlock causing the kernel tohang. In this particular case, failing to handle the error casecorrectly lead to a concurrency type problem, but this bad behavior canalso lead to other coding defects like resource leaks.

Off by ones. Similar to thecase of returning pointers from the stack, if you were to ask adeveloper “How do you index arrays inC/C++ code? ” most would appropriately respond that arrays are0-indexed and the maximum value that should be used to index into arrayis the size of the array minus one. However, we still see this type ofcode more often than we'd like:

In this case, depending on how the stack is arranged, it is likelythat ptr will be overwritten by the buffer overrun caused by the off byone error in indexing the array. What's worse, this pointer is nownull, and as such, the caller of the function may inadvertentlydeference a null pointer. If you were to catch this type of problem intesting, it may seem very strange that the pointer is null if you knowthat the something_very_important function can never return a nullpointer!

Typos. From time to time, adeveloper simply omits some punctuation. Unlike in English, where thereader can likely “figure out what you meant,” a computer will blindlyexecute code as is, causing the functionality to be incorrect. In thisexample below, the developer clearly meant to break if the elementfound in the array was greater than 100. But because he forgot the { and } , thebreak will occur on the first iteration of the loop:

And finally, the following typo was discovered in the X.org code thatcontrols root access in a certain piece of the system:

Notice that the second “call” to geteuid doesnot have parenthesisfollowing the identifier. As such, it is treated as a function pointerand its value is compared against 0. This test always succeeds allowinga normal user of the system to have root access when this piece of codeis triggered. Yes, this piece of code is in a real system that tens ofthousands of users are probably still using.

Avoiding the goofs
Unfortunately, we do not have a silver bullet for guaranteeing thatdevelopers will not make some of the common mistakes that lead to veryexpensive defects.

There's no way to make code less complex or give them more time todevelop it. However, there is technology that helps alleviate theproblem of human frailties in the software development process.Research in static source code analysis has made tremendous strides inthe past decade ” gone are the false positive ridden days of Lint and other light weight codescanning tools.

All of the goofs listed in this paper are easily detected by stateof the art static source code analysis technology. Compared withtesting tools (e.g., purify), static source code analysis has thebenefit of analyzing all of the paths through a given code base and isnot tied to the particular test suite of the application. Compared withmanual code audits or developer debugging, static source code analysistechnology isn't hindered by the human frailties discussed previously.

There is no ignorance of the numerous interfaces in the code sinceit can analyze the whole program, keeping billions of contexts inmemory simultaneously. Also, static source code analysis never suffersfrom stress or boredom or typos. Computers are very good at performingthe same operation thousands of times in a row without variance. If youwant to avoid the most common development goofs, augment yourdevelopment process to include the latest technology to help finddefects earlier in the lifecycle.

Ben Chelf is CTO and Founder of Coverity.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.