Even if you're not involved in a startup, these best practices for running a small organization on a tight budget may help you critique your own organization. How to hold meetings, what tools to buy, and what to skipit's all here straight from one who's learned the hard lessons.
Although software-development processes are numerous and well documented, they don't always fit the time and resource constraints of an embedded systems startup company. Often a startup's software team follows a leaner path to achieve their goal: get a product out on time with no major defects. These teams frequently use public-domain tools and a mix of hybrid software development life cycle techniques, but there are as many nuanced processes as there are embedded systems startups.
What works? This article provides a set of software-development processes that work well in the context of an embedded systems startup. These include:
- The early stage setup of the source control and build policies
- Activities during start of the development stagefrom the functional requirements document (FRD) to first customer ship (FCS)
While some of these steps may seem obvious, each step I describe is vital to achieving the stated goal.
Minimal tools requiredStarting with the assumption that startups are notoriously cheap organizations, I've compiled a list of tools that startups would do well to have at their disposal:
- Source-code control (SCC) system, such as CVS and CVSweb
- Scripting languages for builds and reports, such as Perl and make
- Bug-tracking or problem-tracking database
Your bug-tracking databases should preferably have a web interface. Free bug-tracking databases include Bugzilla (www.bugzilla.org) and GNATS (www.gnu.org/software/gnats), but licensed databases such as FogBUGZ (www.fogcreek.com/fogbugz) have compelling features such as reports that make them worth considering, too.
Before starting to develop the product, the engineering organization needs a foundation on which to reliably build and release software. This foundation should include, at a minimum, support for documentation, source-code control (SCC), and release check-ins.
Document in SCC and on the web
You should make sure that all documentation is source-code controlled and available via the company's intranet. This step makes it easy to track, change, and find the latest design and module information. The documentation should include some means of displaying source changes and histories so that interested parties (for example, the quality-assurance and documentation groups) can review and note code changes. The CVS web tool, CVSweb, does just this. It provides a web interface that displays source history and differences between code revisions.
You should also have a repository online that contains the overall structure of each major module within the product. The repository should point out the changes to be added to a module in each release. This information reduces architectural and release-oriented questions that might otherwise be foisted on engineers during the cycle, thus freeing them up from low-priority interrupts. One of the major jobs of engineering managers is to protect their engineers from external information and feature requests. By having all pertinent information online, a manager will reduce some of these requests; others can usually be handled by the manager himself.
Plan the release and the merge
A major release should be represented as a branch off of the current CVS trunk. Minor and patch releases should be branches off of existing release branches. Spell out your policies regarding merges back to other releases or to the main branch on a web page and via e-mails to the engineers. If possible you should plan to check merge progress using an automation tool; with CVS you can peruse the CVSROOT/commitlog to check comments for bug reference and branch targets.
Include a reference to a bug ID in release check-ins
It's relatively easy to tweak the CVS loginfo script to call a Perl script that can munge the check-in information into a webified version using the MIME::Lite CPAN Perl module. The script can then make the source-file reference and new revision-number links into CVSweb's pages for the file and the revision-difference report respectively. Additional points are possible for parsing out the bug reference and making it a link into the web interface of the bug database.
The net effect is that the MIME e-mail is sent to your software organization for every check-in. If you have a policy of code reviewing check-ins, the revision-difference report makes that task much easier. Figure 1 shows an example of a source change report.
Figure 1: Revision-difference report for reviewing check-ins
The comment field has a link to the bug or problem report (1621 in this case), and below that are the CVSweb links to the source file and a revision-difference report (from 22.214.171.124 to 126.96.36.199). Finally, readers can see that the check-in was a file modification for the 3.1 release branch (with the name “Release_3_1_branch” as it exists in CVS). Note that if someone fails to add the bug ID on a check-in, you can go back and use “cvs admin” to fix the comment so that later reports work properly.
With the source controlled, it's time to tame the build. How often will you build? What is the format for the version of the release? What type of notification will take place each time a build is completed (or fails)?
Create a versioning format
You're going to be firing up many builds over the course of a release lifecycle. At the same time, your team might create other release builds or top of tree (trunk) builds. You will most assuredly need a versioning scheme of some sort. To maintain a logical versioning system, you could use a scheme that includes major, minor, patch, QA, and nightly build numbers. For instance, if a developer wants to check the 2.5.0 release nightly build #3 off of QA build 2, he would download build version 188.8.131.52.3. The build directory should be named something akin to its version, so that in the last example the build directory might be named “020500-002-003.” In this way nightly, QA, and release candidate (RC) builds can all peacefully coexist in a logical fashion.
Mandate e-mail notification of builds
For any build completed, an e-mail notification should be sent to the entire software group. This e-mail should once again be a MIME-enabled web page that includes all of the files that have changed since the last build of the same type (nightly or QA). The e-mail should include a similar listing of all files checked into the branch and their new revisionand once again, this e-mail should have links to bugs, source files in CVSweb, and a revision difference section in CVSweb. In this way, engineers know what parts of the system have changed or what bugs were potentially fixed. To increase the report's usefulness to QA especially, the generating script should pull differences between build tags for QA or RC builds.
Figure 2: An e-mail notification for release 2.5.0 QA build 15.0
Figure 2 is an example e-mail notification for release 2.5.0 QA build 15.0. Note that a CVS tag with this version information has also been created (“QA_BTAG_V2_5_0_15_0”). The first nightly build after this build milestone would have the version 184.108.40.206.1. The QA department, on receipt of this build, can tell that the pesky foobar device has a fix applied against an outstanding bug and that the state of the bug is now “fixed.”
Schedule nightly builds for current releases and the trunk
You should schedule nightly builds for all releases that are currently in the works until they reach beta. This process ensures that first the release has not been broken (breaking a build is a serious offense with a dreaded penalty: the offender has to buy doughnuts for the rest of the organization), and allows for a nightly “smoke test” or automated-test procedure to run. In addition to releases in the works, a nightly build of the main trunk (or HEAD branch in CVS) should be done to ensure its health. Nightly builds can devour a great deal of disk space, so pruning the builds to keep the latest five can maintain your free space. If no check-ins are detected for the branch or trunk, the build shouldn't run (and the e-mail notification should indicate that no check-ins were found so the build was not done).
QA and RC builds are usually done out-of-band to the nightly builds, but maintaining a valid, smoke-tested build can go far toward deriving a product that's ready for testing or release.
Engineering and QA policies
Developers are buzzing away on features and fixes, you've got the nightly builds firing, so now it's time to engage the QA people. Some organizations promote builds to QA fairly early in the cycle; others wait for a preordained code-freeze milestone before sending a build in for testing. Your mileage may vary, and you might be short a tire if QA is working on other releases concurrently. Once you do decide to send QA a release, here are some tips on what to expect.
QA and later-cycle RC builds
Engineers use the nightly builds when testing embedded systems on their target processor. The builds can be loaded onto the target and debugged from that point. QA builds are done on a regular basis so that QA may perform their test suites and validate and fixed bugs. As a rule of thumb, a QA build is appropriate once a week so that QA has enough time to run their suites and the queue of fixed bugs doesn't grow too large. The frequency of QA builds depends on the size of the project.
RC builds, like QA builds, should be designated by an increment in the QA number that appears in both the version and the build-directory name. RC builds are created the same way that QA builds are, minus whatever internal QA designations that might appear in the build information. Avoid deleting RC and QA builds (especially if they aren't backed-up), since they're extremely useful for testing older builds for the existence of some classes of difficult bugs.
Finally, both QA and RC builds should be tagged in CVS so they can be reliably reproduced. Having tags also helps generate build-difference reports so that engineering team and QA can agree on what fixes made it into a build.
Turning knobstime vs. resources
Which knob are you going to turn? Do you have the luxury of time in your schedule to take your project to zero defects, or will you be forced to settle for zero fatal and major defects? Can you move resources either within QA or into QA to help the time crunch? The more challenging part of a release is the tradeoff when confronted with a time-constrained release or resource-constrained release. While there are no short answers to problems such as these, once you know where you are and where you should be, you can gauge what knobs to turn. The key is understanding your current position.
Hold regular meetings
Hold regular meetings to consider the state of the release. You should require two types of meetings: the daily bug meeting and the weekly product review meeting.
The first of meeting goes under different names at different companies but for our purposes I'll call it a daily bug meeting. Whatever you call it, the meeting's goal is defined as taking a regular and frequent (daily is the norm) snapshot of the current bug state of a release. Early in the cycle numerous bugs may be on the list so you should concentrate on a those with a high priority.
The first part of the daily bug meeting should be devoted to the bug review. The meeting coordinator (either the project manager or the engineering manager in charge) runs a projector that displays a summary and individual bug reports. The latter part of the meeting should look at the bug queues (new, open, fixed) and outstanding issues to determine how the release is progressing along. The benefits of a regular review are many:
- All of the responsible parties are up-to-date on the state of the releasethus it makes sense to include a representative from each of the stakeholders for the releaseengineering, QA, product marketing, program management, manufacturing, and documentation if possible
- Bug priorities can be assignedlater in the cycle this becomes a critical determinate of whether a bug will be fixed before shipping
- Late feature requests can be judged for merit
The second required meeting is a product review meeting, during which all of the stakeholders are required to meet and review the progress of the product's release as a whole. This weekly meeting should be called by the marketing product manager and should be attended by the same people as the daily bug meeting plus some senior marketing and manufacturing heads. The purpose is to ensure the release stays on schedule and that the disparate groups coordinate their efforts (is the web site up, is the marketing collateral in shape, will the product launch at any trade shows?).
Toward the end game
So you have these wonderful policies implemented and your process is humming, new source developed, and at some point the QA builds start. What now? There should be some time in the schedule for a code freeze. That point may be determined by some heuristic like a graph showing open bugs vs. closed over time, or it may be the hard-and-fast requirement of a predetermined schedule. It all depends on what knobs you're able to turn. At code freeze, the bar is set highersoftware engineers are advised to check-in only the bugs at a threshold priority. QA must turn around and verify bug fixes with a high degree of reliability for both the fixes themselves and for the time that it takes to test them.
Open vs. fixed graph
A graph that displays open vs. fixed issues can provide insight on stability of the release (number of open issues) and QA loading (number of fixed bugs waiting close). For example, consider the graph in Figure 3; the open issues are being handled even though there are still five waiting for a fix. The to-be-fixed queue is rising, indicating that QA is lagging closing out bugs. A decision maker might look at this and decide to move some QA resources to help close out this release.
Figure 3: Graph of open versus fixed problems
To get to this stage and have a product ready for customer trial, only minor issues should be outstanding (or at least one or two major issues with workarounds). The open vs. fixed graph should be trending downward, and the frequency of bug meetings should be reduced to once or twice a week, and the majority of the discussion should be on coordination issues with the documentation group on how to characterize any lingering issues. Most of the engineering group should have already started on the next release with some possibly working on a follow-up maintenance or patch release and others on the next major release.
First customer ship
Ah, you are done. Or are you? Congratulate yourself and your team and then get ready to start pushing the rock uphill once more. Often the release team should host a post-release meeting to go over what did and did not succeed during the cycle. This is a great time to fine tune your release process.
Along the way you might consider some additional questions. Have you started on a maintenance release? Have you checked with your sales or support engineer (quite a valuable and underutilized resource for internal status checks) on how the release is being accepted? Has anyone considered a customer survey to see if the release produced as it promised or if it missed the mark?
Many anxious moments will occur during any release; some may seem as bad as NASA Fight Director Gene Krantz had it during Apollo 13. He heroically rose above the issues and led a seemingly doomed space mission back to Earth. He and the rest of Mission Control were able to detach themselves emotionally from the problems to search for solutions.
Engineering release managers must be made of the same mettle and must summon some introspection during the tough times to be able to wade through a mess and see your organization's “finest hour.” The recommendations within this article cannot in and of themselves help forge your mettle, but they can provide you with the tools that give you release status information, process, and controls in place.
Osborne Hardison works in a senior embedded engineering role at a small startup. He previously served as a director of core engineering at AvantGo (now Sybase) where he was responsible for delivery of enterprise client and server software. He's also worked as director of engineering at ePocrates, a startup specializing in medical information technology. You can contact him at .
Process for a small startup? I can't imagine what you think a heavy process would be.
I have spent 15 of my 20 years of medical software development in small startups. Typically, we don't even have a QA department, let alone someone to whom to delegate builds. Right at the moment, we are in the position in R&D where we don't even have a marketing guy.
Your policies seem more to apply to activities at GE Medical, hardly a startup. Don't get me wrong, I heartily agree with what you have to say, and found your terminology will be useful. The application to startup just seems outside of my realm of experience.
MedQuest Products, Inc.
Osborne Hardison Responds
Thank you for your response.
Your response seems to indicate that your company endures the same time and resource constraints that are common to most startups. If there is no formal QA or build department, there must be someone tasked with owning QA or builds. With minimal effort you could (if you aren't already) at the least enjoy the benefits of some of the recommendations contained in the article (check in emails, bug meetings).
Larger organizations tend to involve well more than what I've included in the article (http://www.usdoj.gov/jmd/irm/lifecycle/table.htm). Full blown SDLC includes numerous documents, more meetings, and more procedures than you need. I really was trying to get at the essence of what should be required to get quality software out the door ignoring unnecessary intermediate steps.
Thanks for the article. As a startup we initially started off without much of a process in place, similar to Mr. Geisler. However we were fortunately smart enough to learn from our initial stumbling blocks to come up with a process very similar to the one described above. As the company grew in size, our process has been instrumental in providing stability to our core features as well allowing resources to be shuffled around for new feature additions. I particularly feel “documented” communication to be an important aspect of the process, as a lot of critical Ideas tend to get discussed….and forgotten. This is a huge waste of time and energy, which startups cannot afford.