Teaching Testing - Embedded.com

Teaching Testing

Click here for reader response to this article

An article in the April 15, 2004 issue of SD Times suggests that computer science majors get little or no exposure to software testing in their college classes. A researcher at Gartner Labs complained “The myth is that if you're not coding, you're not working.” Several academics said that testing is low-paid work no one wants to do, so why spend good college dollars preparing people for such dead-end jobs?


So I surfed the computer science catalogs from a few universities. Some offer cool courses like artificial intelligence, programming robots (what fun!), and cryptology. Testing? None, zilch, nada. Debugging? Ditto.

The University of Maryland's catalog is typical. Only a single undergraduate course mentions “design.” The word “requirements” appears only in terms of class prerequisites. Yet these subjects represent the holy grail of software engineering. Given solid requirements and a detailed design, anyone can write the program. Given solid requirements and a detailed design, the coding can be easily offshored.

Q/A and acceptance testing will generally remain a local effort; it's the last quality hurdle the software passes before the offshore vendor gets his final payment. So it seems colleges are equipping students with skills that can be easily replaced by low-wage competitors.

Fabulous software engineering can greatly reduce the amount of testing required. But the U of MD, like so many other schools, has only a single class on the subject. The bulk of available courses cover languages, data structures, and the like. The curriculum essentially skips all of the up-front and post-coding efforts in the software lifecycle. It's startling naïve.

Various studies and my own observations confirm that testing and debug consume about half of a typical project's schedule. (I combine the two, because debugging is the first set of checks used to isolate the most obvious problems. It's the beginning of the test process). Coding eats only around 20%.

Testing makes most of us yawn. And that's a tragedy. Though Deming taught us that you can't test quality into a system, it's still a critical part of building great code. Today there are all sorts of cool things happening in the test world. The DO-178B standard for safety critical software, for instance, requires branch and decision coverage, which may mandate the use of special — and interesting — tools. The SPARK language builds verification statements directly into the source code. SPARK is an Ada subset that's gaining some traction in the military and space work, where failures cost megabucks and lives. eXtreme Programming, a novel if wacky approach to programming, brilliantly integrates test into the entire engineering process.

Why is software so bad? It suffers from rushed timetables, lousy design, and inadequate testing. I doubt we'll ever solve the schedule problem, but the second two effects are both tractable and teachable.

If only they were taught.

Did college teach you coding or real software engineering?

Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at . His website is .

Reader Response

While I totally agree with the points you bring up in this article, I alsofeel that another major problem. Anyone can “declare” himself a software engineer. Whilea civil engineer is certified with a PE licence, an electrician is licensed by the statehe operates in and even the mechanic that works on your car is ASE certified, but thereis nothing to certify software engineer. As a result there are many folks doing”software engineering” have little or no training in the field.

– Steve Tymoszuk

So true! I am adjunct faculty at a local university teaching introductory CScourses. My assignment grades incorporate proper operation of a particular softwareassignment. And, though some grumble, this the way the real world grades your software.For today's software, almost right is always wrong.

– Tom Freund

I couldn't agree more.

Throughout my whole computer science studies there was almost no mention of softwaredesign or testing. The only academic who was pushing this cause was considered a hereticwithin the department. He was the subject of derision by the other academics.

The teaching program covered all sorts of neat theory. There was a great deal of leadingedge Research being conducted, in many interesting areas. We all had a great time.

I then entered the real world of military software projects, large teams, formalmethods…The computer science course had not covered any of these issues.We had been taught a lot of theory – at one extreme – and coding at the other. Thelargest project we had dealt with was ridiculously small and never extended beyond a teamof two.

I am not criticising the quality of the teaching or the relevance of the course material- it was all extremely valuable. What was required was additional course work in “realworld” software processes.

With hindsight it is clear that the University environment is fairly “self sustaining” most of the tutors and Academics have moved straight from being undergraduates to PhDs toresearch/teaching they know of no other life. It is not surprising that things are theway they are.

After about 25years, There have been some successes and quite a few failures. I amolder and wiser.

I now understand that design and analysis (something which was not covered at all in theComputer Science course) is the most critical stage. The seeds of failure are sownentirely during this stage and management never see it.

The next most important issue (although they are related) is designing for testability,testplans, specifications and verification.How easy life gets when you can feel confident that sub-systems are functioning correctlyand reliably.When you can re-run automated sub-system tests whenever you feel unsure of things.What a boon it is to access an internal interface and quickly verify its operation.

With the design and test issues in hand – coding and debugging are such trivialoperations (the bits that are so easy to “farm out”). Ironically, it is when things arefarmed out that specification and verification become essential…

Interestingly there is often an attitude that coding is where the product is producedand we should just skip all of the other time consuming overhead and just get into it.In my experience the overhead pays for itself many times over (during unplanned lastminute requirement changes and the like).

Sorry if this has wandered a bit off subject but there is so much to say!

– Perry Brown

At MIT, the “Laboratory for Software Engineering” class focussed heavily ontesting. Excellent class. I remember finishing that class with an impression of aproject taking 40% design, 20% implementation, and 40% testing.

– Kevin McMahon

As always, you hit the nail on the head. “Glamorous” courses help fill the vacancies incolleges, and that is a real tragedy. Having been working in software for 10 years now, Ialso find that the attitude of grads fresh out of college thus also turns to looking for”cutting-edge” stuff and areas like testing, or even development on a not-in-fashionprocessor finds no favour. I also think that the fact that Univs are filled with peoplewith esoteric Ph.Ds but often having little real-world industry experience is to blame asthey have not seen the immense problems that inattention to basic software engineeringcan create in real projects. Mostly it is the industry that has to teach grads theimportance of and basics of processes, testing and all parts of the life-cycle and Ithink it becomes very person specific then, in terms of who your mentor is and whatexperiences he or she had.

– Vivek Mohile

Yes of course you are right; design and coding are nothing compared tofighting the tools, fighting multiple levels of shorts / IC infant mortality problems /silicon errata, and fighting overpromises / infeasible requirements generated bycolleagues whenever you miss a meeting.

And the problem re lack of “testing” or “debugging” training isn't limited to colleges &universities. Clients will NOT budget any time to “debugging” because they ask, “Whywould you put in bugs??? Plan for success!” So… what I do is to try for lots of”integration” time. This “integration” time is when debugging can occur, and can involvemysterious props such as logic analyzers that make interesting squiggles and thus buyprecious time!

– Robert Knapp

I agree with you. People are taught Coding, but no one teaches debugging andtesting. In fact people are taught that Testing is a bad word on their Resumes.

I try to change the viewpoint whenever I get a chance. for example, we could tell apotential candidate, that based on the lack of his experience, we are putting him oncoding and not on testing. That will over time, in my opinion, hopefully, change theperception.

All of us in the industry know that Testing is very important and indeed takes asignificant part of the Project Schedules. Hence, a flurry of newer tools andmethodologies. But I think there is still a lot to be done. For one, I have never foundsan efficient way to do Unit level testing of a module, that is going to be buried in thechip with so much hardware around it. The code may have timing dependencies with respectto hardware events, which are very difficult ot debug.

– Sandeep Bhatia

This article really struck a chord with me. As a freshman undergraduate EEback in 1990, we were required to take an introductory software engineering course. Thecourse was intended to present an alternative for a career path that we may wish tochoose. I don't believe that the university had a computer science program yet as theconcept of a software engineer was pretty new. The concepts of software quality andproduct testing requirements were not even mentioned After I graduated, in my firstengineering job, the testing was all done by some of the lowest paid workers in thefactory. These people were responsible for ensuring that our systems were bug free butthey had little education and less on the job training.

– Matt Flyer

At Georgia Tech, the claim is that programming is taught, not programminglanguages. Design is covered in the intro CS classes (mostly OOP).

The CS department doesn't do much testing, but for at least one class, if your programdumped core you got a 0. end of story.

I was CmpE, so I learned testing in all of those EE classes I had to take. I think CmpEis a much more rounded approach than the CS department.

– Buddy Smith

There is a growing push from some teachers to move more testing into college.Florida Institue of Techology under Kaner and Wittaker (and others) is a leader on thisand has offered workshops to those of us who try to teach software testing. Some ofprofessional, like myself, are working with local schools to try to increase the amountof testing and “real” engineering, but many students do not want to hear about it becausethey have bought into the “coding” side of life and just want the “answer” (code is oftenthe answer). Real software engineering does not often have “the answer” and students andtext books find this hard to deal with. Some of the books on agile development are doingbetter jobs of dealing with this, but there is not the “scientific” data to show they arevalid from a teaching perspective.

– Jon Hagar

So software engineering is still not a discipline? To someextent I agree with you. In my mind, there are two types of coders; ones who design safety, time critical systems and those who write software for “coffee brewers”. Perhapsit's time for a shake-up in the comp-sci community – software engineering for PhDcanidates and software engineering for the rest of us! (One who design safety, timecritical systems.)

– Bill Toole

Jack, few people test because they don't have the deep insight that testingrequires. People who have that sort of insight go into research. Me, I'm an idiot wholeft research at Georgia Tech and went into industry. What an eye opener.

For years I've built software by building small tests around most of the complicatedparts. Recently, when I was forced to wait for some custom hardware to be done, Istarted to write a memory based message logging buffer. Along with the code I wrote amain function to test all sorts of edge cases. Though the code was less than 200 linesit took me many tries (~12) over two days to make all eight edge cases work as I wished. That tool took two days to make reliable, but it has cut three months of debugging workout of my development because without it I could not have quickly found all of the timingbugs of the quirky hardware I was given.

It is difficult to make the decision to test stuff diligently, because you KNOW that youare adding time. The trick is to know when you WILL save time. Because of my insightsinto testing and validating stuff, my boss doesn't let me do design anymore. I spend alot of time finding big mistakes and even more time proving it to the people who onlycode/design hardware for a living. It is sort of like being a modern day prophet, no onelikes you very much, but they keep you around because when the going gets tough, someonehas to know where to look.

– John Bond

Colleges don't teach “real-life” software development because it's notinteresting to young people who aren't creating real-life software. Process, design,test, maintenance, etc. are definitely learned on the job, and only after a few years ofexperience (and getting burned a few times). Thus, older, mature, experienced softwareengineers finally gain the maturity to create well designed, reliable products. But, howmany “older, mature, experienced” software engineers do you see around anymore? That's awhole set of issues in itself…

– Jay Nevin

Computer Science is to Software Engineering as Physics is to ElectricalEngineering; Physics is a pure science while Electrical Engineering is applied science. Computer Science is the study of the science, algorithms and language systems, that makecomputer systems possible. Early Computer Science departments of the 1960s and 1970sgrew out of the Math departments of large universities. Software Engineering is theapplication of algorithms, languages, and processes (requirements analysis, design,implementation, and test) to produce computer systems that perform useful work. MostSoftware Engineering departments of the 1990s grew out of the Electrical/ComputerEngineering departments of large universities.

What the orignal discussion highlights is that what the world probably needs is moreSoftware Engineers and possibly fewer Computer Scientists; I personnaly see a need forboth.

For an example of a Computer Science curriculum versus a Software Engineering curriculumsee http://www.cs.rit.edu versus http://www.se.rit.edu .

– Terry O'Brien

I have recently returned to graduate school after 6 years in the field. I wasshocked by the lack of real world programming knowledge of the professors. One of myprofessors asked me to stay after class one day. He wanted me to explain my program tohim so he could grade it! This same professor asked me what the std:: was in front of mycout statement (and what a namespace was). Another professor once asked me what thefunction assert did! I don't know of a single one of the professors that know that newthrows an exception when it fails. In the case of this college, I feel the problem isthat most of these professors have never had a real job outside of academia. One daythey were PhD candidates the next they were teaching. These professors are veryresistant to change. C++ has been standardized since 1998 I tell them. They do not likeit when I point out the problems with the computer science department and its staff. Consequently I am not very popular amongst the professors. Even worse though is thatthese same professors are the ones teaching C++ to undergrad students. These poorstudents are being taught pre-standard C++ simply because the professors are too lazy toupdate their own skills. I was recently working with a group of 3 undergrad seniors someof which are graduating in May. None of the three knew how to program a derived class orhow to implement a virtual method. As far as design, there is one undergrad course likethe college in your article (required). There are 2 graduate courses in softwareengineering (electives) which I have not had the pleasure yet to take. I feel that atleast one course on software engineering should be mandatory for all students in the CSdepartment. As far as debugging goes you are on your own in this school. Most studentshere think that cout/printf is the only way to debug. You may ask why I stay in thisschool? It is near my home, my job pays for it, and I just want the degree for mycareer.

– Jon Anglin

I got my bachelor's degree in the 70's and there were no courses in softwaretesting or software engineering to speak of. However, I am completing a master's insoftware engineering at S.M.U. in Dallas and courses in testing are part of the corerequirements for the degree, along with software project planning, quality engineering, process improvement, safety critical systems, and so forth. It is an excellent program.You might check it out at “engr.smu.edu”.

– Steve Hanka

My university is typical. I work with guys from different universities in thearea. They had the same experience. Testing was often 10% of a first year assignment. Itwas never covered in lecture and most students had no idea how to do it. After first yearthere was nothing on testing.

Three universities in my area (one is my old college) have recently being giving courseson Software Engineering and Quality Assurnace. Unfortunately, they are electives, theyare in third or fourth year and most students choice not to take them.

– Darrell

My college only taught coding. They did offer one course called softwareengineering but it was not widely attended (myself included). I learned about SWengineering (along with testing) on the job.There is a university that is starting to address these issues/disciplines, seehttp://www.rit.edu/~932www/ugrad_bulletin/colleges/ccis/softeng.htmlThanks for the article, keep up the fight!

– Brad Randolph

We don't expect college to teach you everything but the concept ofrequirements and testing there-of should be an important part of the curriculum. Did youcheck out any schools offering “Software Engineering” degrees? RIT, etchttp://www.rit.edu/~932www/ugrad_bulletin/colleges/ccis/softeng.html

– Mark Taube

Jack, you really hit on a sore spot with me. One company I worked for hadexecutives charging all kinds of perks to a government contract. They got caught, thecompany got fined, 110 million dollars. this meant there was a hiring freeze – – No stafffor testing software, and the staff we had testing software part time that had systemlevel product knowlege, got transfered to make in-flight telephone systems for executiveaircraft. We muddled through, but it took a lot longer than it would have with adedicated testing staff, and well written test plans.

There's ton's of good stuff on testing code related to the RTCA/FAA DO-178B, and relatedguidelines, that usually does not get covered in college.

I don't know how you teach it all in just 4 years — maybe Engineering has to reallychange and to go to a 6 year program like the Pharmacy programs.

Maybe the MBA needs to be an 8 year program with 2 years worth of ethics tests thrown infor good measure as well.

– Bill Murray

Inherently most of us think of testing as being separate from development. Designers/programmers will do some testing on their benches, but it is not complete andoften does not tie the design/code back to the requirements.

Until we start thinking of the requirements in more detail, the designs will never berobust. After 15 years I have seen many method and tools, none of which have been asilver bullet. We don't need a silver bullet, we need more discipline in therequirements and design.

Requirements work in two ways. First they should specify exactly what the softwareshould do, but additionally they imply what it should not. Defects in a system arerelated to a piece of the system operating in a way it was not designed. If therequirement excluded that operation, the defect would not exist.

Requirements start with an initial requirements document, but they are not finished untilthe final design is complete and shipped. The first version of the document, like thedesigns often just cover the obvious and absurd. It is the development to the finalrequirements and design where the valuable work happens, yet it is treated in mosteducation programs and company's R&D facilities as a black box without process.

How does this relate to testing? First in order to meet the requirements, the designersmust provide evidence, such as unit test results. I agree quality cannot be inserted bytesting, especially at the end of a project. However, testing to detailed requirementsas part of development can help dramatically.

– Robert Brown

My background and training is electronics, not CS at all. Interestinglyenough, the only software portions of my engineering education was labs that wererequired as part of other classes. The labs counted for nothing except you received nograde unless the lab was completed.

Until the late 90's, CS people could expect to earn about 20% less than their EEcounterparts. A frequent comment made during the interview process was that EE's tendedto have more rounded backgrounds than CS people, and were worth more to the company.

I “got into” software based on board and systems level test. Manually testing more andmore complex systems got to be “manually” impractical. I remember testing spacecraftsubsystems with a manual test procedure. One bad case was 1200 pages of steps to performmanually. Test computers like HP's 9830/9825/9826 series helped out with IEEE-488 bus andserial ports running instruments to get the data for the test, record the data, checkagainst pass/fail criteria, and even prepare the test reports.

Discussions with associates over time indicated that “there must be a better way” to puttogether software packages, so we all gradually learned techniques for writing softwarewe could maintain and adapt to ever changing requirements.

Some of the guys I met later never really “left” electronic engineering. Their softwarewas developed with labVIEW from ni.com. You draw schematics of software and execute them!

Testing, for me, has always been associated with testing a board or a circuit or asystem. Gradually the software moved from outside the unit under test to actuallyresiding inside the unit on the unit's CPU.

I test hardware board/box/system requirements with software.

– Don Doerres

I am a consultant with a CS degree and 21 years industry experience, includingevery aspect of the software development process (and work which includes mil-spec andbiomedical work). From what I have seen, industry is not interested in testing, so it ishardly surprising that it is not taught. To develop adequate tests for software (whichis far more than just automated code coverage testing that some people consider adequate)requires the same degree of skill and knowledge as was required to write the code, andyet for testing, industry pays from 25% to at most 75% of what they would pay thedeveloper who wrote the code, and most companies are either not willing or not able topay for formal testing at any price unless forced to by FDA or contractual requirements. Ultimately industry's holy grail is whatever gets the product out the door faster andcheaper (whether or not software is involved), testing will always be viewed assomething which costs money and delays shipment of the product, and therefore to beavoided unless some outside threat forces better testing on industry (governmentregulations, fear of lawsuits, loss of customers because of a history of poor quality). Good software processes are always faster and cheaper in the long run, and usually (butnot always) slower and more expensive in the short run, so until businesses start takingthe long term view there is little chance of a change in corporate attitudes towardstesting.

– Shannon C. Dealy

How can you assure that a zilion transistors chip works as you want? Testingit. And how do we test it? Sounds difficult, even impossible?If test engineers are being underpaid and/or schedule don't offer a test window, we areabout to get more blue screens on our desktop computers and, in a severe case (Therac25)have a letal dose of radiation from a buggy software machine. It's time to every engineerstart thinking how to test the next project.

– Darcio Prestes

It is true that testing is extremely important in software development. I have spent a few years in an environmentdealing in 178B software testing. I find that software testing for embedded systems is extremely difficult. Developers usuallyhave to research thoroughly the hardware they are coding for. This research usually leads to Zen-like understanding of thedevice and the way it operates. The hardware changes from project to project and sometimes is relatively new on the embeddedscene. Unless the tester has this same Zen-like understanding of the device, software testing does not catch flawed code.Module testing and Software Integration testing could both “pass” a module or code package based on testing to 178B standards,but still might not work correctly on the target. Only Hardware/Software Integration can begin to detect these types of errors. Hardware/Software Integration might be the responsiblity of an academic program, but how could they keep up with, or even offera course on all the different types of devices encountered in embedded development? I don't believe it is the job ofUniversities to teach people how to do a job, it is to educate them.

– Michael Sullenger

This article's reference to DO-178 is all fine and good, but I've seen and heard of all to many cases where thecriticality analysis and patitioning analysis said there would be no issues, but there later turned out to be a fatal systemflaw. A prime example involved the toilet system on a certain airliner which was computerized. It was only connected to themainainence computer, and therefore was not deemed as critical as for example the auto-pilot. The system had a bug thatsometimes caused the blue water to overflow a certain tank. The blue water was corrosive, and affected the wiring in adifficult to inspect area. This corroded wiring caused erronious autopilot signals in some cases. An airliner crashed due tothis problem, before the true criticality of the toilet computer was realized. Had the toilet computer been designed and testedto higher standards, this tragedy likely could have been avoided.

– Bill Murray

My experience must have been atypical, but then my education was somewhat atypical. Two quarters of SoftwareEngineering were required for all CompSci majors at my alma mater, Weber State University. (This was not all that suprising,given that as much as half of each graduating class immediately went to work on NASA or USAF engineering projects.) Allupper-division classes emphasized design and research to some extent, including a required class in “Advanced ProgrammingTopics” where classic problems in computability were presented and students encourage to either research and implement classicsolutions to the problesm, or to tackle a design on their own.

I still fondly recall Assitant Professor Russell Anderson admonishing students to spend a few minutes thinking before “rushinginto the lab to begin bugging code.”

I can't say how effective the courses were for the average student, but they did at least give each and every student someexposure to the concepts of designing and documenting software before they graduated. And yes, I believe this is a positiveimpact, turning out students who expect to put some thought into their software before the fact.

– Wes Peters

Terry O'Brien hits the nail exactly on the head. The problem is that while there are many Computer Scienceundergraduates, most of those go into work that is more accurately described as Software Engineering and they find themselvesunprepared for it.

SE is a discipline unto itself and is slowly making its way into undergraduate curriculums having long been the province ofgrad. school.

I was lucky enough to have benefit of different approaches: Bachelors in EE, Masters S/W Engr.

– Lyndon Walker

My background is pure software, both undergraduate and graduate. My 10+ year job experience is pure embedded systems.I took several software engineering courses during my graduate program. I was writing real-life firmware for cell phone before Iwent to graduate school. I thought I could bring software engineering to embedded systems. But I was wrong. The concept to”design” software is so alien that people stop listening to it. Unfortunately, that includes people who write software forembedded systems. I think it is even harder to introduce software “design” to embedded system developers than to IT developers.

– Vincent Wang

Computer Science, as taught in schools, is heavy on the science part. It's about developing algorithms, O(n) issues, and theory. It is not meant to be a degree program that teaches how to program in the Real World. It is unfortunate that colleges and universities continue to deceive students about that. I knew many CS majors that thought that they would become expert programmers. Meanwhile, at the career fair, EEs were being snatched up for the “premier” p

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.