Planning for product quality through better inspection, evaluation, and testing
Ensuring product quality is not accomplished solely through testing and verification activities. Testing is but a fraction of the techniques that are at an organization’s disposal to improve their development quality. Good planning of the product incarnations; that is, a phased and incremental delivery of the feature content, makes it possible for an organization to employ test, inspections, and evaluation as tools for competitive advantage. To really improve (and prove product quality), a more comprehensive approach is required (Figure 1).
Figure 1: Key elements in supporting product design and delivery are the use of test, inspection and evaluation at every step of the process.
Described in this article is the Test, Inspection & Evaluation Master Plan Organized (TIEMPO) which extends the Test and Evaluation Master Plan (TEMP) specified in IEEE 1220 and MIL-STD-499B (Draft) to support product quality.
TIEMPO expands the concept of a Test and Evaluation Master Plan by focusing on staged deliveries with each product/process release being a superset of the previous release. We can consider these iterations our development baselines closely linked to our configuration management activities. Each package is well defined; likewise the test, inspection and evaluation demands are well defined for all iterations. Ultimately, the planned product releases are coordinated with evaluation methods for each delivery. Under TIEMPO the inspections are:
Philosophy of the Master Plan
At its core, TIEMPO assists in coordinating our product functional growth. Each package has a defined set of contents, to which our entire battery of quality safeguarding techniques will be deployed. This approach—defined builds of moderate size and constant critique—has a very agile character, allowing for readily available reviews of the product.
In addition, TIEMPO reduces risk by developing superset releases wherein each subset remains relatively untouched. Most defects will reside in the new portion (the previously developed part of the product or process is now a subset and proven defect-free).
Should the previous iteration contain unresolved defects, we will have had the opportunity between these iterations to correct these defects. Frequent critical reviews are utilized to guide design and to find faults.
The frequent testing facilitates quality growth and reliability growth and gives us data from which we can assess the product readiness level (should we launch the product). Experience suggests the following benefits arise:
- Well planned functional growth in iterative software and hardware packages
- Ability to prepare for test (known build content), inspection and evaluation activities based upon clearly-identified packages
- Linking test, inspection and evaluations to design iterations (eliminate testing or inspecting items that are not there)
- Reduced risk
- Identification of all activities to safeguard the quality—even before material availability and testing can take place
- Ease of stakeholder assessment, including customer access for review of product evolution and appraisal activities
Experience also indicates that at least 15% of the time associated with downstream trouble shooting is wasted in unsuccessful searches for data simply due to lack of meaningful information associations with the developmental process baselines. The TIEMPO approach eliminates this waste ferreting out issues earlier in the process and allowing more dollars for up front product refinement.
How TIEMPO works
This article will describe how each of these pieces fit together. TIEMPO need not be a restricted only phase-oriented product development, but any incremental and iterative approach will be the beneficiary of a constant critique including entrepreneurial activities.
System Definition The TIEMPO document and processes begin with the system definition. Linked to the configuration management plan it will describe the iterative product development baselines that build up to our final product function entirety. In other words, we describe each of our incarnations of the products in each package delivery. We are essentially describing our function growth as we move from little content to the final product. Each package will have incremental feature content and bug fixes from the previous iteration.
By defining this up front, we are able to specifically link the testing, inspection and evaluation activities to not only make an iteration but specific attributes of that iteration and capture it in an associative data map in our CM system.
In the example of testing, we know the specific test cases we will conduct by mapping the product instantiation with the specifications and ultimately to test cases. We do this through our configuration management activities. We end up with a planned road map of the product development that our team can follow. Of course, as things change we will again update the TIEMPO document through our configuration management actions.
Test or Verification Test or verification consists of those activities typically associated with determining whether the product meets the specification or original design criterion. If an incremental and iterative approach is applied, prototype parts are constantly compared against specifications. The parts will not be made from entirely production processes but will increase in level of production content as the project progresses and we approach our production intent.
Though prototype parts may not represent production in terms of durability, they should represent some reasonable facsimile of the shape and feature content congruent with the final product. We use these parts to reduce the risk by not jumping from idea to final product without learning along the way.
We should learn something from this testing to use to weigh the future quality of the resultant product. It is obvious how testing fits into TIEMPO. However, there are some non-obvious opportunities to apply TIEMPO. We could use inspection as a form of test plan. Did we get the testing scope correct? We can also use this inspection technique on our test cases, analyzing if we will indeed stress the product in a way valuable to our organization and project, not to mention that we can inspect software long before it is even executable.
The feedback from this inspection process will allow us to refine the testing scope, test cases, or any proposed non-specification or exploratory-based testing. The testing relationships in a typical HW/SW release plan are shown below in Figure 2.