An article in the April 15, 2004 issue of SD Times suggests that computer science majors get little or no exposure to software testing in their college classes. A researcher at Gartner Labs complained "The myth is that if you're not coding, you're not working." Several academics said that testing is low-paid work no one wants to do, so why spend good college dollars preparing people for such dead-end jobs?
So I surfed the computer science catalogs from a few universities. Some offer cool courses like artificial intelligence, programming robots (what fun!), and cryptology. Testing? None, zilch, nada. Debugging? Ditto.
The University of Maryland's catalog is typical. Only a single undergraduate course mentions "design." The word "requirements" appears only in terms of class prerequisites. Yet these subjects represent the holy grail of software engineering. Given solid requirements and a detailed design, anyone can write the program. Given solid requirements and a detailed design, the coding can be easily offshored.
Q/A and acceptance testing will generally remain a local effort; it's the last quality hurdle the software passes before the offshore vendor gets his final payment. So it seems colleges are equipping students with skills that can be easily replaced by low-wage competitors.
Fabulous software engineering can greatly reduce the amount of testing required. But the U of MD, like so many other schools, has only a single class on the subject. The bulk of available courses cover languages, data structures, and the like. The curriculum essentially skips all of the up-front and post-coding efforts in the software lifecycle. It's startling naïve.
Various studies and my own observations confirm that testing and debug consume about half of a typical project's schedule. (I combine the two, because debugging is the first set of checks used to isolate the most obvious problems. It's the beginning of the test process). Coding eats only around 20%.
Testing makes most of us yawn. And that's a tragedy. Though Deming taught us that you can't test quality into a system, it's still a critical part of building great code. Today there are all sorts of cool things happening in the test world. The DO-178B standard for safety critical software, for instance, requires branch and decision coverage, which may mandate the use of special and interesting tools. The SPARK language builds verification statements directly into the source code. SPARK is an Ada subset that's gaining some traction in the military and space work, where failures cost megabucks and lives. eXtreme Programming, a novel if wacky approach to programming, brilliantly integrates test into the entire engineering process.
Why is software so bad? It suffers from rushed timetables, lousy design, and inadequate testing. I doubt we'll ever solve the schedule problem, but the second two effects are both tractable and teachable.
If only they were taught.
Did college teach you coding or real software engineering?
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. He founded two companies specializing in embedded systems. Contact him at email@example.com. His website is www.ganssle.com.
While I totally agree with the points you bring up in this article, I also feel that another major problem. Anyone can "declare" himself a software engineer. While a civil engineer is certified with a PE licence, an electrician is licensed by the state he operates in and even the mechanic that works on your car is ASE certified, but there is nothing to certify software engineer. As a result there are many folks doing "software engineering" have little or no training in the field.
- Steve Tymoszuk
So true! I am adjunct faculty at a local university teaching introductory CS courses. My assignment grades incorporate proper operation of a particular software assignment. And, though some grumble, this the way the real world grades your software. For today's software, almost right is always wrong.
- Tom Freund
I couldn't agree more.
Throughout my whole computer science studies there was almost no mention of software design or testing. The only academic who was pushing this cause was considered a heretic within the department. He was the subject of derision by the other academics.
The teaching program covered all sorts of neat theory. There was a great deal of leading edge Research being conducted, in many interesting areas. We all had a great time.
I then entered the real world of military software projects, large teams, formal methods... The computer science course had not covered any of these issues. We had been taught a lot of theory - at one extreme - and coding at the other. The largest project we had dealt with was ridiculously small and never extended beyond a team of two.
I am not criticising the quality of the teaching or the relevance of the course material - it was all extremely valuable. What was required was additional course work in "real world" software processes.
With hindsight it is clear that the University environment is fairly "self sustaining" most of the tutors and Academics have moved straight from being undergraduates to PhDs to research/teaching they know of no other life. It is not surprising that things are the way they are.
After about 25years, There have been some successes and quite a few failures. I am older and wiser.
I now understand that design and analysis (something which was not covered at all in the Computer Science course) is the most critical stage. The seeds of failure are sown entirely during this stage and management never see it.
The next most important issue (although they are related) is designing for testability, testplans, specifications and verification. How easy life gets when you can feel confident that sub-systems are functioning correctly and reliably. When you can re-run automated sub-system tests whenever you feel unsure of things. What a boon it is to access an internal interface and quickly verify its operation.
With the design and test issues in hand - coding and debugging are such trivial operations (the bits that are so easy to "farm out"). Ironically, it is when things are farmed out that specification and verification become essential...
Interestingly there is often an attitude that coding is where the product is produced and we should just skip all of the other time consuming overhead and just get into it. In my experience the overhead pays for itself many times over (during unplanned last minute requirement changes and the like).
Sorry if this has wandered a bit off subject but there is so much to say!
- Perry Brown
At MIT, the "Laboratory for Software Engineering" class focussed heavily on testing. Excellent class. I remember finishing that class with an impression of a project taking 40% design, 20% implementation, and 40% testing.
- Kevin McMahon
As always, you hit the nail on the head. "Glamorous" courses help fill the vacancies in colleges, and that is a real tragedy. Having been working in software for 10 years now, I also find that the attitude of grads fresh out of college thus also turns to looking for "cutting-edge" stuff and areas like testing, or even development on a not-in-fashion processor finds no favour. I also think that the fact that Univs are filled with people with esoteric Ph.Ds but often having little real-world industry experience is to blame as they have not seen the immense problems that inattention to basic software engineering can create in real projects. Mostly it is the industry that has to teach grads the importance of and basics of processes, testing and all parts of the life-cycle and I think it becomes very person specific then, in terms of who your mentor is and what experiences he or she had.
- Vivek Mohile
Yes of course you are right; design and coding are nothing compared to fighting the tools, fighting multiple levels of shorts / IC infant mortality problems / silicon errata, and fighting overpromises / infeasible requirements generated by colleagues whenever you miss a meeting.
And the problem re lack of "testing" or "debugging" training isn't limited to colleges & universities. Clients will NOT budget any time to "debugging" because they ask, "Why would you put in bugs??? Plan for success!" So... what I do is to try for lots of "integration" time. This "integration" time is when debugging can occur, and can involve mysterious props such as logic analyzers that make interesting squiggles and thus buy precious time!
- Robert Knapp
I agree with you. People are taught Coding, but no one teaches debugging and testing. In fact people are taught that Testing is a bad word on their Resumes.
I try to change the viewpoint whenever I get a chance. for example, we could tell a potential candidate, that based on the lack of his experience, we are putting him on coding and not on testing. That will over time, in my opinion, hopefully, change the perception.
All of us in the industry know that Testing is very important and indeed takes a significant part of the Project Schedules. Hence, a flurry of newer tools and methodologies. But I think there is still a lot to be done. For one, I have never founds an efficient way to do Unit level testing of a module, that is going to be buried in the chip with so much hardware around it. The code may have timing dependencies with respect to hardware events, which are very difficult ot debug.
- Sandeep Bhatia
This article really struck a chord with me. As a freshman undergraduate EE back in 1990, we were required to take an introductory software engineering course. The course was intended to present an alternative for a career path that we may wish to choose. I don't believe that the university had a computer science program yet as the concept of a software engineer was pretty new. The concepts of software quality and product testing requirements were not even mentioned After I graduated, in my first engineering job, the testing was all done by some of the lowest paid workers in the factory. These people were responsible for ensuring that our systems were bug free but they had little education and less on the job training.
- Matt Flyer
At Georgia Tech, the claim is that programming is taught, not programming languages. Design is covered in the intro CS classes (mostly OOP).
The CS department doesn't do much testing, but for at least one class, if your program dumped core you got a 0. end of story.
I was CmpE, so I learned testing in all of those EE classes I had to take. I think CmpE is a much more rounded approach than the CS department.
- Buddy Smith
There is a growing push from some teachers to move more testing into college. Florida Institue of Techology under Kaner and Wittaker (and others) is a leader on this and has offered workshops to those of us who try to teach software testing. Some of professional, like myself, are working with local schools to try to increase the amount of testing and "real" engineering, but many students do not want to hear about it because they have bought into the "coding" side of life and just want the "answer" (code is often the answer). Real software engineering does not often have "the answer" and students and text books find this hard to deal with. Some of the books on agile development are doing better jobs of dealing with this, but there is not the "scientific" data to show they are valid from a teaching perspective.
- Jon Hagar
So software engineering is still not a discipline? To some extent I agree with you. In my mind, there are two types of coders; ones who design safety, time critical systems and those who write software for "coffee brewers". Perhaps it's time for a shake-up in the comp-sci community - software engineering for PhD canidates and software engineering for the rest of us! (One who design safety, time critical systems.)
- Bill Toole
Jack, few people test because they don't have the deep insight that testing requires. People who have that sort of insight go into research. Me, I'm an idiot who left research at Georgia Tech and went into industry. What an eye opener.
For years I've built software by building small tests around most of the complicated parts. Recently, when I was forced to wait for some custom hardware to be done, I started to write a memory based message logging buffer. Along with the code I wrote a main function to test all sorts of edge cases. Though the code was less than 200 lines it took me many tries (~12) over two days to make all eight edge cases work as I wished. That tool took two days to make reliable, but it has cut three months of debugging work out of my development because without it I could not have quickly found all of the timing bugs of the quirky hardware I was given.
It is difficult to make the decision to test stuff diligently, because you KNOW that you are adding time. The trick is to know when you WILL save time. Because of my insights into testing and validating stuff, my boss doesn't let me do design anymore. I spend a lot of time finding big mistakes and even more time proving it to the people who only code/design hardware for a living. It is sort of like being a modern day prophet, no one likes you very much, but they keep you around because when the going gets tough, someone has to know where to look.
- John Bond
Colleges don't teach "real-life" software development because it's not interesting to young people who aren't creating real-life software. Process, design, test, maintenance, etc. are definitely learned on the job, and only after a few years of experience (and getting burned a few times). Thus, older, mature, experienced software engineers finally gain the maturity to create well designed, reliable products. But, how many "older, mature, experienced" software engineers do you see around anymore? That's a whole set of issues in itself...
- Jay Nevin
Computer Science is to Software Engineering as Physics is to Electrical Engineering; Physics is a pure science while Electrical Engineering is applied science. Computer Science is the study of the science, algorithms and language systems, that make computer systems possible. Early Computer Science departments of the 1960s and 1970s grew out of the Math departments of large universities. Software Engineering is the application of algorithms, languages, and processes (requirements analysis, design, implementation, and test) to produce computer systems that perform useful work. Most Software Engineering departments of the 1990s grew out of the Electrical/Computer Engineering departments of large universities.
What the orignal discussion highlights is that what the world probably needs is more Software Engineers and possibly fewer Computer Scientists; I personnaly see a need for both.
For an example of a Computer Science curriculum versus a Software Engineering curriculum see http://www.cs.rit.edu versus http://www.se.rit.edu .
- Terry O'Brien
I have recently returned to graduate school after 6 years in the field. I was shocked by the lack of real world programming knowledge of the professors. One of my professors asked me to stay after class one day. He wanted me to explain my program to him so he could grade it! This same professor asked me what the std:: was in front of my cout statement (and what a namespace was). Another professor once asked me what the function assert did! I don't know of a single one of the professors that know that new throws an exception when it fails. In the case of this college, I feel the problem is that most of these professors have never had a real job outside of academia. One day they were PhD candidates the next they were teaching. These professors are very resistant to change. C++ has been standardized since 1998 I tell them. They do not like it when I point out the problems with the computer science department and its staff. Consequently I am not very popular amongst the professors. Even worse though is that these same professors are the ones teaching C++ to undergrad students. These poor students are being taught pre-standard C++ simply because the professors are too lazy to update their own skills. I was recently working with a group of 3 undergrad seniors some of which are graduating in May. None of the three knew how to program a derived class or how to implement a virtual method. As far as design, there is one undergrad course like the college in your article (required). There are 2 graduate courses in software engineering (electives) which I have not had the pleasure yet to take. I feel that at least one course on software engineering should be mandatory for all students in the CS department. As far as debugging goes you are on your own in this school. Most students here think that cout/printf is the only way to debug. You may ask why I stay in this school? It is near my home, my job pays for it, and I just want the degree for my career.
- Jon Anglin
I got my bachelor's degree in the 70's and there were no courses in software testing or software engineering to speak of. However, I am completing a master's in software engineering at S.M.U. in Dallas and courses in testing are part of the core requirements for the degree, along with software project planning, quality engineering, process improvement, safety critical systems, and so forth. It is an excellent program. You might check it out at "engr.smu.edu".
- Steve Hanka
My university is typical. I work with guys from different universities in the area. They had the same experience. Testing was often 10% of a first year assignment. It was never covered in lecture and most students had no idea how to do it. After first year there was nothing on testing.
Three universities in my area (one is my old college) have recently being giving courses on Software Engineering and Quality Assurnace. Unfortunately, they are electives, they are in third or fourth year and most students choice not to take them.
My college only taught coding. They did offer one course called software engineering but it was not widely attended (myself included). I learned about SW engineering (along with testing) on the job. There is a university that is starting to address these issues/disciplines, see http://www.rit.edu/~932www/ugrad_bulletin/colleges/ccis/softeng.html Thanks for the article, keep up the fight!
- Brad Randolph
We don't expect college to teach you everything but the concept of requirements and testing there-of should be an important part of the curriculum. Did you check out any schools offering "Software Engineering" degrees? RIT, etc http://www.rit.edu/~932www/ugrad_bulletin/colleges/ccis/softeng.html
- Mark Taube
Jack, you really hit on a sore spot with me. One company I worked for had executives charging all kinds of perks to a government contract. They got caught, the company got fined, 110 million dollars. this meant there was a hiring freeze - - No staff for testing software, and the staff we had testing software part time that had system level product knowlege, got transfered to make in-flight telephone systems for executive aircraft. We muddled through, but it took a lot longer than it would have with a dedicated testing staff, and well written test plans.
There's ton's of good stuff on testing code related to the RTCA/FAA DO-178B, and related guidelines, that usually does not get covered in college.
I don't know how you teach it all in just 4 years -- maybe Engineering has to really change and to go to a 6 year program like the Pharmacy programs.
Maybe the MBA needs to be an 8 year program with 2 years worth of ethics tests thrown in for good measure as well.
- Bill Murray
Inherently most of us think of testing as being separate from development. Designers/programmers will do some testing on their benches, but it is not complete and often does not tie the design/code back to the requirements.
Until we start thinking of the requirements in more detail, the designs will never be robust. After 15 years I have seen many method and tools, none of which have been a silver bullet. We don't need a silver bullet, we need more discipline in the requirements and design.
Requirements work in two ways. First they should specify exactly what the software should do, but additionally they imply what it should not. Defects in a system are related to a piece of the system operating in a way it was not designed. If the requirement excluded that operation, the defect would not exist.
Requirements start with an initial requirements document, but they are not finished until the final design is complete and shipped. The first version of the document, like the designs often just cover the obvious and absurd. It is the development to the final requirements and design where the valuable work happens, yet it is treated in most education programs and company's R&D facilities as a black box without process.
How does this relate to testing? First in order to meet the requirements, the designers must provide evidence, such as unit test results. I agree quality cannot be inserted by testing, especially at the end of a project. However, testing to detailed requirements as part of development can help dramatically.
- Robert Brown
My background and training is electronics, not CS at all. Interestingly enough, the only software portions of my engineering education was labs that were required as part of other classes. The labs counted for nothing except you received no grade unless the lab was completed.
Until the late 90's, CS people could expect to earn about 20% less than their EE counterparts. A frequent comment made during the interview process was that EE's tended to have more rounded backgrounds than CS people, and were worth more to the company.
I "got into" software based on board and systems level test. Manually testing more and more complex systems got to be "manually" impractical. I remember testing spacecraft subsystems with a manual test procedure. One bad case was 1200 pages of steps to perform manually. Test computers like HP's 9830/9825/9826 series helped out with IEEE-488 bus and serial ports running instruments to get the data for the test, record the data, check against pass/fail criteria, and even prepare the test reports.
Discussions with associates over time indicated that "there must be a better way" to put together software packages, so we all gradually learned techniques for writing software we could maintain and adapt to ever changing requirements.
Some of the guys I met later never really "left" electronic engineering. Their software was developed with labVIEW from ni.com. You draw schematics of software and execute them!
Testing, for me, has always been associated with testing a board or a circuit or a system. Gradually the software moved from outside the unit under test to actually residing inside the unit on the unit's CPU.
I test hardware board/box/system requirements with software.
- Don Doerres
I am a consultant with a CS degree and 21 years industry experience, including every aspect of the software development process (and work which includes mil-spec and biomedical work). From what I have seen, industry is not interested in testing, so it is hardly surprising that it is not taught. To develop adequate tests for software (which is far more than just automated code coverage testing that some people consider adequate) requires the same degree of skill and knowledge as was required to write the code, and yet for testing, industry pays from 25% to at most 75% of what they would pay the developer who wrote the code, and most companies are either not willing or not able to pay for formal testing at any price unless forced to by FDA or contractual requirements. Ultimately industry's holy grail is whatever gets the product out the door faster and cheaper (whether or not software is involved), testing will always be viewed as something which costs money and delays shipment of the product, and therefore to be avoided unless some outside threat forces better testing on industry (government regulations, fear of lawsuits, loss of customers because of a history of poor quality). Good software processes are always faster and cheaper in the long run, and usually (but not always) slower and more expensive in the short run, so until businesses start taking the long term view there is little chance of a change in corporate attitudes towards testing.
- Shannon C. Dealy
How can you assure that a zilion transistors chip works as you want? Testing it. And how do we test it? Sounds difficult, even impossible? If test engineers are being underpaid and/or schedule don't offer a test window, we are about to get more blue screens on our desktop computers and, in a severe case (Therac25) have a letal dose of radiation from a buggy software machine. It's time to every engineer start thinking how to test the next project.
- Darcio Prestes
It is true that testing is extremely important in software development. I have spent a few years in an environment dealing in 178B software testing. I find that software testing for embedded systems is extremely difficult. Developers usually have to research thoroughly the hardware they are coding for. This research usually leads to Zen-like understanding of the device and the way it operates. The hardware changes from project to project and sometimes is relatively new on the embedded scene. Unless the tester has this same Zen-like understanding of the device, software testing does not catch flawed code. Module testing and Software Integration testing could both "pass" a module or code package based on testing to 178B standards, but still might not work correctly on the target. Only Hardware/Software Integration can begin to detect these types of errors. Hardware/Software Integration might be the responsiblity of an academic program, but how could they keep up with, or even offer a course on all the different types of devices encountered in embedded development? I don't believe it is the job of Universities to teach people how to do a job, it is to educate them.
- Michael Sullenger
This article's reference to DO-178 is all fine and good, but I've seen and heard of all to many cases where the criticality analysis and patitioning analysis said there would be no issues, but there later turned out to be a fatal system flaw. A prime example involved the toilet system on a certain airliner which was computerized. It was only connected to the mainainence computer, and therefore was not deemed as critical as for example the auto-pilot. The system had a bug that sometimes caused the blue water to overflow a certain tank. The blue water was corrosive, and affected the wiring in a difficult to inspect area. This corroded wiring caused erronious autopilot signals in some cases. An airliner crashed due to this problem, before the true criticality of the toilet computer was realized. Had the toilet computer been designed and tested to higher standards, this tragedy likely could have been avoided.
- Bill Murray
My experience must have been atypical, but then my education was somewhat atypical. Two quarters of Software Engineering were required for all CompSci majors at my alma mater, Weber State University. (This was not all that suprising, given that as much as half of each graduating class immediately went to work on NASA or USAF engineering projects.) All upper-division classes emphasized design and research to some extent, including a required class in "Advanced Programming Topics" where classic problems in computability were presented and students encourage to either research and implement classic solutions to the problesm, or to tackle a design on their own.
I still fondly recall Assitant Professor Russell Anderson admonishing students to spend a few minutes thinking before "rushing into the lab to begin bugging code."
I can't say how effective the courses were for the average student, but they did at least give each and every student some exposure to the concepts of designing and documenting software before they graduated. And yes, I believe this is a positive impact, turning out students who expect to put some thought into their software before the fact.
- Wes Peters
Terry O'Brien hits the nail exactly on the head. The problem is that while there are many Computer Science undergraduates, most of those go into work that is more accurately described as Software Engineering and they find themselves unprepared for it.
SE is a discipline unto itself and is slowly making its way into undergraduate curriculums having long been the province of grad. school.
I was lucky enough to have benefit of different approaches: Bachelors in EE, Masters S/W Engr.
- Lyndon Walker
My background is pure software, both undergraduate and graduate. My 10+ year job experience is pure embedded systems. I took several software engineering courses during my graduate program. I was writing real-life firmware for cell phone before I went to graduate school. I thought I could bring software engineering to embedded systems. But I was wrong. The concept to "design" software is so alien that people stop listening to it. Unfortunately, that includes people who write software for embedded systems. I think it is even harder to introduce software "design" to embedded system developers than to IT developers.
- Vincent Wang
Computer Science, as taught in schools, is heavy on the science part. It's about developing algorithms, O(n) issues, and theory. It is not meant to be a degree program that teaches how to program in the Real World. It is unfortunate that colleges and universities continue to deceive students about that. I knew many CS majors that thought that they would become expert programmers. Meanwhile, at the career fair, EEs were being snatched up for the "premier" programming jobs, especially for embedded products in the defense industry. CS people couldn't understand why only Microsoft was willing to talk to them. (Correlation between Microsoft quality and CS hires can be drawn by the reader if desired.)
Science is about theory, engineering is about practice.
One of my favorite sayings (attribution unknown): In theory, there is no difference between theory and practice, but in practice, there is.
- John Patrick
I sort of disagree with the views expressed here. The realm of design, code and test is true only for product companies. The number of engineers employed by the services companies far outnumbers the number of engineers employed by products companies. The design-code-test cycles have been true only for product companies. My friends in services companies have required completely different skills, mainly for succeeding through the vicious maintenance cycles: debug and test. But, oddly enough none of the CS courses focus on these aspects of software development.
Should CS courses really focus on every aspect of the software development and maintenance? I don't think so. I went to engineering school not to learn the specifics, but to learn the general. Engineering school gave me a lot of chance for analytical thinking. Engineering school gave some chances for doing large enough software/hardware projects. But, all of those were toy projects!
I think I learnt more in the industry than I could have ever learned during those four years at engineering school.
So, I think engineering schools should provide just the foundations. But, engineering schools should probably provide more industry/student interactions to ensure that students come out learning a lot more than the school can ever imagine teaching those bored souls.
We don't need no education. We just need more industry interaction.
Freshman and sophomores should be made to work on product testing. Juniors and seniors should be made to work on product maintenance. Graduate students should be made to work on actual product development (writing useful industrial grade code, rather than writing some useless network simulator code, e.g.). And PhDs should be made to participate in design efforts in the industry.
I think the more the industry interacts with the students, and the earlier it does, the faster we can produce better software engineers.
- Mahesh S