Agile embedded software development
|April 28 was last day to register for ESC Silicon Valley. Registration is now closed. You may register on site at conference. Click here for more info.|
Why should you consider adopting Agile software development practices? Not because it is the latest buzzword (actually a 10-year-old buzzword). You should consider adopting it because you want to improve. The plan-driven approach hides a lot of problems until it is too late. Ad hoc development does not provide the business with the needed visibility to confidently plan product releases and rollouts.
Software development projects often suffer from long development cycles, late delivery, unpredictable schedules, poor quality, missed customer expectations, and developer burnout. These problems often interact to become a positive feedback loop. Unpredictable delivery leads to schedule pressure, and unrealistic plans. Schedule pressure leads to long hours and shortcuts. Long hours lead to burnout. Shortcuts lead to defects, defects lead to more long hours debugging. Bug removal is an inherently unpredictable activity leading to even more schedule pressure, shortcuts, defects, etc.. Figure 1 shows some of the interactions software development problems.
Click on image to enlarge.
Vague requirements You can see a few positive feedback loops in this vicious cycle. There are scheduling, defects, and requirements vicious cycles. These are important problems to solve for which the iterative approach of Agile Development has been shown to be effective. The specify-design-test-build-test-deploy approach used by waterfall-based software development lifecycles sounds appealing but has proven time and again to give less than adequate results.
Iterative development, one of the core practices of Agile development has been around for decades. As far back as 1987 Fred Brooks' as chairman of the Defense Science Board Task Force on Military Software recommended that the waterfall process be replaced with iterative development due to waterfall's history of failure on large DoD contracts. Iterative development has been used successfully on some very high profile projects from the 1950's to present day including: the X-15 rocket plane, project mercury, trident missile submarine control systems, the space shuttle avionics, and the Canadian Automated Air Traffic Control System, to name a few.[LARMAN]
Over the last ten years, agile has been mostly associated with software development other than embedded. Even though embedded does have its unique and special challenges, we can and should benefit from Agile development approaches. Yes you are special, but there is nothing special about some of the problems we share with non-embedded development. We can learn, benefit and apply agile to embedded.
What is Agile development?
A good place to start in describing agile development is to see what a group of respected software developers (which I am lucky enough and honored to be associated with), that coined the term, had to say about it. The Agile Manifesto says:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.[AGILEMAN]
Individuals and interactions over processes and tools
The first point stresses the importance of human interaction and teamwork. Many development processes try to take the human element out of software development, but the agile manifesto's leading statement is about leveraging the people and their interactions. Tools are needed, but it is good people, working in teams who build successful software products. This point is often misconstrued to say that processes do not matter. Processes and discipline do matter, but people matter more.
This statement could be misconstrued to suggest that Agile developers get things done by sitting in a circle and singing Kumbaya. However, there is another interpretation. An increase in teamwork does not automatically lead to a decrease in discipline. Consider, for example, the teams at the Skunk works, or Burt Rutan's team who built SpaceShipOne to win the X-Prize. These teams capture the intent of the Agile Manifesto well. While they deeply value discipline and process, they value teamwork even more.
Working software over comprehensive documentation
The second point stresses the importance of having working software as a measure of progress. Documents may be valuable, but working software is a more meaningful gauge of software development progress. I have heard this misinterpreted as, "We're doing agile, so we aren't doing documentation". That's baloney! Documents are often invaluable. Those that are, must be produced. However, documentation is expensive to create and maintain so it is important to create only those documents you truly need. In document-centered development, I've heard more than once that the reason for the document is "our process requires it". This is wasteful. Agile developers articulate and validate the reasons for documents. They know who the customer of the document is. Then produce documents if, and when, they are needed and try to find an economical way to create them.
Remember also that most documents don't execute (some do, like test cases); so they cannot be used as effective measures of project completeness. Agile developers believe they are 50% complete when 50% of the features of the system have been demonstrated.
Even though embedded software is often only delivered once, along with the hardware, does not mean we cannot track our progress by demonstrating progress through working features. This can be challenging, but necessary to reduce risk and build a feature rich and robust system.
Customer collaboration over contract negotiation
This point addresses the need to work closely with the customer. By customer we meant the person or persons that are specifying the product and can make trade-off decisions on features and dates. Ideally the customer is the person using the product, but in mass marketed products the customer role is internal and indirect at best. Customer interaction is favored because software is very difficult to completely specify up front.
Requirements and market needs change over time. The customer has to be part of the team to help make trade-offs and to see what they have asked for.
Responding to change over following a plan
The forth and final point deals with the reality of any complex endeavor. Plans are important, but situations change and that require constant adaptation. Plans cannot be viewed as static, even though specific delivery dates are. This point is often misunderstood to mean there are no dates or commitments in Agile development. On the contrary, dates and commitments are taken very seriously in Agile development. Agile developers create working software in very short cycles in order to measure compliance to the plan.
Principles supporting the Manifesto
The Agile Principles back up the manifesto.[AGILEP] The principles are:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
- Business people and developers must work together daily throughout the project.
- Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity—the art of maximizing the amount of work not done—is essential.
- The best architectures, requirements, and designs emerge from self-organizing teams.
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Agile development is based on iterative and incremental development (IID). IID provides regular feedback by breaking the project into iterations that are generally one oir two weeks in length. The output of each iteration is working software. Each iteration is like a stand-alone project ending after fixed amount of time, and delivering some executable version of the product. In early iterations the software might only run in a test environment or prototype.
There are two main roles in agile development, the customer role, and the developer role. Two groups of people usually represents each role. In a few words, the customer defines and tests the product and the developer builds the product. I will cover these roles more completely later.
Embedded software engineers should understand feedback. The control systems we design always have feedback mechanisms to keep the systems under control. An agile project is based on iterations that provide feedback to the critical variables of the project: schedule, requirements, and design. Think of agile as a control system for software development.
The team estimates, plans and organizes work into iterations. The regular cadence of delivery allows a team to establish a velocity that can be measured. Velocity is used to calibrate the development plan and monitor progress giving valuable management data. Work is estimated in effort points. For example, if the team is completing about 20 points per iteration and there are 200 points in the backlog, it will take about ten iterations. If plan only allows eight more iterations, we have some managing to do. Skillfully, we are not in the 11th hour and have options. Maybe its not too late to add a couple people, move the date out, or remove functionality.
Requirements are the broken into smaller demonstrable units called stories. They are the estimate-able, testable, and deliverable units. They are small enough so that many can be completed within a single iteration. When the developers complete a story the customers get feedback on the requirements by seeing and touching what has been developed. In the early iterations, prior to hardware availability some of these stories are demonstrated through tests and simulations; but the demos are based on real working code.
By building the software incrementally the developers get feedback on the design as it evolves. The stories cut across elements of the design provide early integration of simplified versions of the subsystems giving the developers valuable experience with the architecture. Something that looks great in UML does not always look so great once it is coded, and it is better to figure that out sooner than later.
The incremental or evolutionary design approach is central to Agile development. Software requirements are constantly evolving. Priorities change. Software is expected to be used year and after year and evolve along with the market and changing hardware and technology. So it is essential that code has to be built to last, and that means being built to change. Designs and code will continue to be changed throughout the life of the product and consequently there is the ever-present risk of side-effect defects. I'll describe alter how agile developers use automated tests that help to lock in the existing behavior as new features are added.
Developing iteratively gives the business great power. The approach can be used to either manage to a specific delivery date, or to manage to specific feature content. The team's track record is input used to adjust the plan based on facts rather than wishful thinking
In the single-pass waterfall situation, considerable time is spent up front on items that may never make the final product. In Agile development effort is first expended on the highest priority features and capabilities, requirements are elaborated in parallel with development and consequently little time is wasted. Scope is managed at the detail level. When you have 10 must have features, they are broken into smaller stories, and the product content is managed at the detail level, deferring less important parts of the critical features.
Imagine trains running on parallel tracks, one train represents requirements definition and the others hardware and software development.
Click on image to enlarge.
The requirements train leaves the station a little before the development train. Requirements are discovered, refined and handed off while both trains are moving forward toward the final destination.
In phased development only one track is needed and the development trains would not leave the station until the requirements train arrived and wired back the requirements.
Click on image to enlarge.
Maybe, just maybe, the development train will go faster knowing all the requirements, but it will never make up for all the time spent waiting for departure.
Concurrent engineering is a strategy designed to shorten development time-to-market by doing development activities in parallel that might have been done serially. Concurrent engineering is core to Agile, and Agile teams have to be skilled at working incrementally, without detailed knowledge of the whole picture, to be successful. This takes some getting used to after having waterfall based project management in the limelight for last twenty or so years.
Time to market critical is so we need to find ways to finish development sooner. A way to finish development sooner is to start building the product sooner. We need to begin development before all the requirements are known or we will unnecessarily delay the product. If you think about it, the requirements are never all known up front so what are you giving up! Have you ever been on a project where there were no surprise requirement changes? The world of changing requirements is the world we have to live in, so lets master it.
I've seen too many teams paralyzed by not knowing all the requirements. There is a fear of making mistakes and a belief that we can figure it all out, and then design the perfect architecture with no false steps or rework. This sounds great but is not practical in the complex world of product development. So, at the beginning of a product development effort, we have to identify some of the core features, ones that are important and well enough understood, and begin development work immediately. Beginning development will lead us to confront our requirements misconceptions and design flaws. In addition, and maybe more importantly, the feedback will help us find our blind spots, the unknown risks and weaknesses we cannot anticipate. Getting development started buys time for the requirements team to work in parallel on the remaining requirements. Requirements details are delivered just in time. We can make suer we work on the most critical features first and get them rock solid.
Initial implementations explore the requirements and the design thus improving the team knowledge, clarifying the requirements and solidifying the software design alternatives. The early development of these high-value/high-risk features provides feedback to the customer as well as developers. The feedback helps to mitigate the risks of building the wrong product, or building the wrong architecture. We get executable feedback on requirements and architecture.
The uncertainty and risk the team is trying to manage is not limited to software, but also to hardware and the hardware/software boundary. One way to deal with hardware uncertainty is for the software developers to wait until the hardware design is complete and then start the software design. WAIT! Just kidding! I am not recommending waiting! So, please don't quote me out of context!
Embedded developers don't have to wait for hardware train to get to the end of the track either; the trains do not have to be coupled. Hardware abstractions are created in the software that decouples software from hardware. The abstractions define an interface for interacting with the hardware by defining the service the hardware will provide, without getting bogged down in volatile implementation details. Hardware abstractions enable concurrent hardware/software engineering by allowing software development and testing to start prior to hardware availability. This important practice can also provide input into the hardware requirements and help define and refine the hardware/software boundary.
The iterative approach to requirements gathering, risk reduction and development practices means that changes in customer needs, project goals and hardware architecture are more naturally accommodated. Time to market can be improved by eliminating wasteful serialization in the development process by engineering the requirements, software and hardware trains on three parallel tracks.
The sequential approach may appear to be less wasteful, allowing people to focus on their part and not starting development until we know everything about what we want top build. We strive to eliminate rework.
Click on image to enlarge.
It is important to consider what we want to optimize. Is it the requirements phase, the design phase? Neither, we want to optimize product time to market. What if by working in parallel we could deliver the product sooner, even with some rework as shown in this next diagram.
Click on image to enlarge.
I call the inefficiency hypothetical because most development efforts finish the requirements and the design the day the product ships. These phases never really end when we pretend they do.
The complexity and evolutionary nature of software development means there are many opportunities to break existing working software. A simple one-line change, carefully thought out, could bring the system crashing-down months in the future and leave no evidence of the crash. Side effect defects are common and often do not get discovered until long after the defect is injected.
Most product teams rely on manual testing in the real hardware to prove out their system. Manual tests have a few problems. First and foremost manual tests take a lot of clock time and are labor intensive. They might require special lab equipment. These realities mean that the tests will not be run often enough. What we would really like is a way to verify each change that is made to the software. If you had a magic button that you could press that would tell you if your software was operating to specification, how often would you press that button? I would press it after every change, and I do.
When you manually test you have to select the test to run and that leaves you with a growing untested code gap as shown in this diagram.
Click on image to enlarge.
Pick any coefficients you like for new-feature test effort and regression-test effort and you find that a manual test strategy is unsustainable. This means that unless you automate the bulk of your tests, you will have an evergrowing untested code gap.
Agile teams automate unit and acceptance tests making rerunning tests very cheap. So cheap that we can run all the unit tests whenever any change is made to the source code. This is hard to do, especially when you are first learning, but the time saved will outweigh the time you normally would waste manually testing and debugging. Automated tests support the concept that new features should not break existing features. What a concept! I can just hear marketing now, "please add <new feature goes here> system, and its OK if you break any other random feature, no problem." Marketing does not want that, but they might get the idea that software developers think it's no problem.
If you think of the tests as a critical asset of the system, even though they are not shipped with the system, when you run the tests the system basically reports when it is broken. Bob Martin (author of numerous good books on design and agile) tells us having tests is like double entry accounting, if the sum of the debits does not equal the sum of the credits, we have a problem. Accountants don't just sum the credits to save time, they make sure the books balance with their built in tests. They don't save time by only doing the credits, or the debits. They would prefer not to go to jail for messing up the books.
Where do these tests come from? Who writes them? When are they written? Automated unit tests are written incrementally by the software developers in a tight feedback loop with the production code using a technique called Test-Driven Development.[BECK] The unit tests tell developers if their code does what they intended. My book, Test-Driven Development for Embedded C, can help you apply TDD to the challenging world of embedded C or C++.[GREN-TDD] For a shorter introduction to TDD see my Embedded Systems Conference Paper.[GREN-ET]
Automated acceptance tests provide evidence that the system meets its requirements. The developers and test engineers working with the customer team write automated acceptance tests. The acceptance tests are written in advance of the iteration where the development is to be done. This practice changes the role of test engineers in a very profound way. Rather than drowning at the bottom of the waterfall, at the end of the project by a deluge of untested software pounding down on them, the test engineers adopt a proactive role by specifying the behavior of the system in the form of automated tests. These automated tests provide an unambiguous definition of done.
These tests are a very valuable investment. They make testing a repeatable process, that can be run with every change. The tests double as an executable specification. Unit tests provide examples of how a given module is used at a very detailed level, and provide feedback to the developer that the code behaves as expected. Acceptance tests demonstrate how larger groups of modules work together to deliver the product's requirements. Acceptance tests are written in a domain specific language so that non-engineers can read and possible write tests. An open source tool called FitNesse is often employed for automating acceptance tests. You can find more about this in my Embedded Systems Conference Paper "Executable Use Cases". [GREN-XU]
I expect that the automated test description sounds impractical to many of you. If you read this as requiring the automation of the entire environment around your embedded system, I agree, it is impractical except in cases where the cost of failure is very high (you be the judge). Rather than using expensive physical instrumentation to support automated test, we use virtual instrumentation where possible. Virtual instrumentation uses the hardware abstractions as the interface point for event injection and detection during unit and acceptance testing, the same abstractions we talked about that facilitate concurrent engineering.
A hardware abstraction defines an interface to the hardware services. Figure 2 shows an example of part of a security system. The SecuritySystem shows two operations: Arm and HandleSensorActive. In the production system the arm button is sensed by some device driver and as a reaction to that event, the Arm() function is called on SecuritySystem. Likewise, HandleSensorActive is called when a specific sensor has changed state. The SecuritySystemImplementation can tell the HardwareControl interface to TurnOnAlarm as well as other operations not shown. When the automated tests are run, the incoming events don't come from the real hardware, they come from the test itself. Outgoing instructions to the hardware are intercepted by the test through the Hardware Fake-out layer. For example the test simulates that the Arm button was pressed and then generates the HandleSensorActive event just like the HardwareImplementationLayer will do in the production code. The SecutiySystemImplementation's reaction, sounding the alarm, is verified by the test by checking that the Fake-out layer was told to TurnOnAlarm.
Click on image to enlarge.
As a side note, a very exciting thing about this approach to testing is that our design has built in hardware isolation from the start. Without the automated test influence, there would not have been an immediate need for isolating the core software form the hardware. Often good structuring like this is left out of embedded designs because of worries about performance. Please see my other conference paper on design for more discussion.[GREN-DES]
Working in iterations
The team delivers working software every iteration. In non-embedded agile development software may be deployed each iteration every iteration. In embedded incremental development it may not be practical to release these small increments, especially when there is concurrent hardware development and high deployment costs. Because value cannot be delivered each iteration some say that agile cannot be used on embedded software development. My opinion is different. Instead of delivering value, we provide visible progress. I don't mean doing show and tell on what might be build, but rather a demonstration of real working running software. If real hardware is not available, the demos are done in the simulation, or evaluation environment.
Iterations are fixed-duration time boxes. A time box ends when the time is up, it is not extended when some selected stories are not complete, and it is not ended early when work gets done ahead of schedule (yes, this happens about as often as getting behind schedule in a well functioning agile team).
Iterations are short because people are not very good at estimating long activities. People are pretty good at a two week planning horizon. So, the long-term plans are made up of a series of smaller plans with more precision in the current and next couple iterations. There's more uncertainty and freedom in the further out iterations.
Because iterations are short, the work has to be broken into small pieces of functionality that can be completed within the iteration. We call them stories. The goal of the iteration is to complete all the planned stories and make the stores' automated acceptance tests pass. The stories and their estimates are used in tracking progress against the plan.
We will have to look at stories in more depth because one of the big challenges of Agile development is breaking the system into the stories that allow incremental and visible progress on the product. Embedded agile development has even more challenges in defining stories because of the added complexity of hardware/software interactions.
Stories are usually referred to as User Stories. I prefer to call them Product Stories. Often the work we do in embedded development is not visible to the end user. the name Product Story seems to fit better.
A product story delivers value, shows progress or reduces some risk and can be completed within one iteration. Usually a story is considered a concise description of system behavior. In that sense stories are similar to a use case
Page 1 of 2Next >