Maximizing the benefits of continuous integration with simulation
The role of software is becoming more important than ever in embedded systems. It provides the opportunity for competitive differentiation and can significantly accelerate and increase revenues. However, companies are often not willing to wait a year or two for new and updated software, so there is increased pressure to release faster. The way developers work is fast evolving, regardless of the type of code or system in development. Market observation shows that new practices such as Agile development, continuous integration, continuous deployment and cross-functional teams are being established for the development of embedded software. Embedded systems development is clearly very different from developing a web or an IT-based application; and target hardware does not always lend itself to adopting these new development methodologies.
Most of the challenges can be categorized as being related to access, collaboration or automation. Let investigate the first of these challenges: access or the availability of the hardware required to move to continuous integration and testing. Continuous integration is often considered to be a useful means to achieve maximum velocity in embedded development. Although it can deliver many benefits, continuous integration can also highlight hardware limitations that prevent its effective attainment.
Due to market needs, the traditional lifecycle – design, platform development, application development, test and integration, deployment and maintenance – is changing in order to achieve greater access. Test and integration is now a continuous and ongoing activity that happens throughout the product lifecycle: it occurs during the development of a product and after deployment when the product is in the maintenance phase.
Figure 1 – Product lifecycle and continuous testing
Software is also often updated in deployed products, so it is being continuously tested and integrated, which means that the traditional post-development test and integration phase is changing and even disappearing in some cases. It has not been entirely eliminated but it’s largely been encompassed in Quality Assurance (QA), which is perhaps less about doing integration and basic tests and more about ensuring that proper testing is done before product shipment. In this area, there are two opposing views: either production quality is part of the continuous test and integration system, where it is the final step in that process, meaning that the end-goal of the continuous integration is to have production quality; or there is a dedicated QA phase that is a specific activity with specific equipment and comes after and is separate from the continuous test and integration system.
The basic ideas behind continuous testing and integration include: provide faster feedback to developers; build and test the system piece by piece to avoid big-bang integration and big failure; assess most changes; reduce the waiting time to test systems; find errors in the smallest possible context; and find unique errors at each level. This is often difficult to achieve because there is limited access to often-fragile and scarce target hardware, which causes long latency to run tests that makes the fixing of bugs more expensive. A long time to run tests means it takes time to get feedback and regressions creep back in, and using only two levels of test increases the likelihood that bugs in intermediate levels are not caught.
Figure 2 – Typical time frames of continuous practices
Target hardware can be too costly or impractical to obtain enough of it. Developers may have to wait for access to a test lab or specific equipment, or may have to wait for a result to be reported back from a test lab, or may lose time with setup or configuration changes after they gain lab time, all causing a loss of velocity, thought flow and momentum. This access bottleneck also causes an inability to scale out the test and integration; and it may cause quality problems if a lack of access to hardware causes developers to reduce their test matrix to what possible rather than what should be tested. This may cause late stage integration problems that should have been avoided by introducing continuous integration.