Building a software test and regression plan
Defining the scope of the testing
At Mirabilis Design, we create detailed flow diagrams and design documents before scheduling the development. During development and after the code has reached a critical stage, we run the tests every night after all development has completed. Around the clock operation allows us to immediately identify the bugs.
The primary purpose for testing is to detect software failures so that defects may be uncovered and corrected (Figure 2, below). This is a non-trivial pursuit. Testing cannot establish that a product functions properly under all conditions but can only establish that it does not function properly under specific conditions.
The scope of software testing often includes examination of code as well as execution of that code in various environments and conditions as well as examining the aspects of code: does it do what it is supposed to do and do what it needs to do.
 |
| Figure 2. Major Testing Categories (Source: protocoltesting.com) |
In the current culture of software development, a testing organization may be separate from the development team. There are various roles for testing team members. Information derived from software testing may be used to correct the process by which software is developed.
We have a round-table where each member shares their code with the team. The team then does a functional inspection and quizzes on the logic and code flow. In effect, multiple team members are aware of the detail of each code block.
For the graphical editors, we assemble models using the documented features to utilize all the features. We also attempt features that are not documented but could cause incorrectly operation. During this process, we try to bridge error messages to any operation that might create unexpected operations.
Defects and Failures
Not all software defects are caused by coding errors. One common source of expensive defects is caused by requirements gaps, e.g., unrecognized requirements that result in errors of omission by the program designer.
A common source of requirements gaps is non-functional requirements such as testability, scalability, maintainability, usability, performance, and security. Software faults occur through the following processes. A programmer makes an error (mistake), which results in a defect (fault, bug) in the software source code.
If this defect is executed, in certain situations the system will produce wrong results, causing a failure. Not all defects will necessarily result in failures. For example, defects in dead code will never result in failures. A defect can turn into a failure when the environment is changed.
Examples of these changes in environment include the software being run on a new hardware platform, alterations in source data or interacting with different software. A single defect may result in a wide range of failure symptoms.
Our product, VisualSim, is built entirely in Java. So, the testing on multiple platforms has not been a major focus. Nevertheless, we occasionally find issues that can be major showstopper like illegal directory separators or unsupported image formats. Interfaces to VisualSim need to be tested on different platforms. These include MatLab, C, C++, Verilog, SystemC and Satellite Toolkit.
Compatibility issues
A frequent cause of software failure is compatibility with another application, a new operating system, or, increasingly, web browser version. In the case of lack of backward compatibility, this can occur (for example...) because the programmers have only considered coding their programs for, or testing the software upon, "the latest version of" this-or-that operating system.
The unintended consequence of this fact is that: their latest work might not be fully compatible with earlier mixtures of software/hardware, or it might not be fully compatible with another important operating system.
In any case, these differences, whatever they might be, may have resulted in (unintended...) software failures, as witnessed by some significant population of computer users. This could be considered a "prevention oriented strategy" that fits well with the latest testing phase.
Java has ensured the backward-compatibility between versions of applications at the underlying level and between Operating Systems (OS). We compares results of models built with previous generation tools and determines the reason for any change in statistics output. Java reduces the backward compatibility testing but does not eliminate it.
Loading comments... Write a comment