Last year a friend had a St. Jude pacemaker attached to his heart. When he reported an unexpected low battery reading (displayed on an associated digital watch) to his doctor a month later, he learned this was the result of a firmware bug known to the manufacturer. The battery was fine and would last on the order of a decade more. His new-model pacemaker's firmware didn't include a bug fix that was provided the year before to wearers of old-model.

Another friend owns a Land Rover LR2 SUV with back-up sensors. When the car is in reverse and nearing an obstacle or another car, the driver is alerted via a beeping sound. Except that the back-up sensors don't always work. Some "reboots" of the SUV don't seem to have this feature enabled. He suspects there is a "race condition" during the software startup sequence.

Yet another friend has driven a Toyota Prius hybrid over 100,000 miles. He reports that the brakes very occasionally have an odd/different feel. But his older model Prius is not expected to be subject to the 2010 model year recall.

These are just a few of the personal anecdotes behind the headlines. Embedded software is everywhere now, with over 4 billion new devices manufactured each year. Increasingly the quality and safety of products is a side-effect of the quality and safety of the software embedded inside.

One important question is, can we trust future software updates any more than we can trust the existing firmware? How do we know that the Toyota Prius hybrids with upgraded braking firmware will be safer than those with the factory firmware?

It's nice to see that more people are taking security issues seriously. A release crossed my desk last week from Dell, saying that they've hooked up with Integrity Global Security to provide a secure operating system on desktop PCs for government applications. Integrity Global Security is a spin-off of Green Hills Software, the group that was tasked with selling the company's secure OS to government and military outlets.

The agreement between Dell and Integrity Global Security makes Dell the exclusive provider of the Integrity separation kernel for general-purpose secure computing to government agencies.

It's nice to see a company like Dell, who has its roots in PCs and servers, make a move like this one. From all the documentation I've seen, there's no disputing that Integrity really is a secure OS. It's been certified by NIAP to EAL6+1 and High Robustness. Coupled with the servers offered by Dell, it's a nice combination.

This will simplify the process for government workers, who will be able to operate a PC more like PC, knowing that their data will remain secure. Hence, there's little to no training required for the user.

The hardware-software combination that'll be offered by Dell includes the Integrity-178B separation kernel and Dell's OptiPlex line of desktop PCs, which are already used by government customers.

I believe that 2010 will be the year of embedded virtualization, all the signs point in the right direction. It always takes a while for new technology to grab the imagination of embedded device developers. Embedded developers are traditionally a conservative bunch, however, the benefits of virtualization can not be ignored, even by them.

I would love to give blow-by-blow examples of real customer scenarios, but for obvious reasons I can not. Instead I will use a fictitious example: a digital camera. Everybody is familiar with a digital camera and knows a lot of the capabilities that it offers. A digital camera has several really obvious requirements such as quick boot. Remember the first crop of digital cameras? By the time the camera had booted the moment had slipped away. Some of the other real-time requirements have to do with lens control, light measurements, focus and so forth.

Today's devices however are more and more integrated. A mobile phone is more and more also a capable camera, it's first purpose is to be a good phone, it's second purpose is to take pictures and share (on Twitter, Facebook and other social media sites). What I have not seen yet though is a device that has "digital camera" as its first purpose, but integrates some aspects of a mobile phone, WiFi and a GPS into it.

Maybe these devices exist and are simply out of the reach of my toy budget, nevertheless, I think the integrated camera makes a great example to use when discussing virtualization. Cameras exist and a lot of software content has been written for it, typically for real-time operating systems. Now all we have to do is to add the 'integrated' part to it.

The type of content that we need to add could be things like networking (TCP/IP probably), browsing, file sharing, WiFi, Bluetooth, GPS, a more advanced GUI (though screen!), you name it. Wouldn't it be great if your camera could record GPS coordinates when you take the picture (ok, some can do this), that you can tag pictures on your camera while on the road, then sync when you get home, which would upload certain pictures to your Facebook profile or on-line photo album as well as back up all pictures and remove them from the memory stick.

Many of the building blocks of these features are readily available on open-source operating systems, such as Linux. Wouldn't it be great if our device could run a real-time OS for the real-time tasks, re-using the code that we have and that we know works and run Linux for new content?

Well, this is exactly where virtualization comes in, virtualization allows partitioning of single and multicore devices to run multiple operating systems on a single processor efficiently, thereby enabling this type of innovation.

Virtualization also allows for the development of a low-end pocket camera (which are the ones that I typically end up buying) that could have this functionality on a low-power single core processor, and re-use it on a top-of-the-line SLR and add more features and processing power by using a dual core processor.

Again, the example is fictitious, but I wouldn't mind seeing a nice pocket-camera that can display its pictures on an iPad, laptop or netbook for editing and commenting while I wait in an airport-lounge and then automatically syncs with iPhoto over 802.11 when I get home and uploads them to my blog for good measure, preparing my camera for the next day of picture taking.

Let's not argue whether the camera, the iPad or iPhoto is responsible for uploading or syncing, I really shouldn't need to worry about it.

Hmm, in hindsight, maybe I should submit this for a patent application :)

Last month I received an interesting e-mail in response to a column I wrote for Embedded Systems Design called The Lawyers are Coming! My column was partly about the poor state of embedded software quality across all industries, and my correspondent was writing to say my observations were accurate from his perch within the automotive industry. Included in his e-mail was this interesting tidbit:

I read something about the big Toyota recall being related to floor mats interfering with the accelerator, but I was told that the problem appears to be software (firmware) for the control-by-wire pedal. Me thinks somebody probably forgot to check ranges, overflows, or stability properly when implementing the "algorithm."

As background for those of you who have been working in SCIFs or other labs, the "big Toyota recall" was first announced in September 2009. It was said to concern removable floor mats causing the accelerator pedal to be pressed down. Some 3.8 million Toyota and Lexus vehicles were involved and owners were told to remove floor mats immediately.

This week several related major news events have transpired, including:

• Toyota recalled millions of additional vehicles in the U.S.


• Under pressure from the U.S. NHTSA, Toyota halted production and sales of eight models


• Avis, Hertz, and Enterprise pulled affected Toyota models from their rental fleets


• Toyota's recall spread to Europe and China


• Ford stopped production of a full-size commercial vehicle after discovering that the gas pedal came from the supplier involved in the Toyota recall

But none of the articles I've read have talked about software being a cause. And it's not clear if the affected models are drive-by-wire. However, at least one article I read yesterday suggested that one fix being worked on is a software interlock to ensure that if both the brake and the gas pedal are depressed, the brake will override the accelerator. On the one hand, that seems to mean that software is already in the middle; on the other, I would be extremely surprised to learn that such an interlock wasn't already present in a drive-by-wire system.

So what's the story? Are embedded software bugs to blame for this massive recall? Do you know? Have you found any helpful articles pointing at software problems? Please share what you know in the comments below, or e-mail me privately.

I've been writing about the practice of embedded software development--in the form of books, articles, columns, conference papers, and blog posts--for nearly 15 years. (How time flies...) I also served as editor-in-chief of Embedded Systems Design magazine for about 3-1/2 years in the middle. But it wasn't until August of last year that it occurred to me to write an e-mail newsletter.

My newsletter is called Firmware Update, and it is published about every 3 weeks. The stated mission of Firmware Update is to spread the word about the firmware development best practices I have learned in my career as an engineer, consultant, and trainer. In addition to connecting my past, present, and future writings into a coherent storyline, I am using the newsletter to link to articles and papers by others who influence my thinking.

In less than six months, the newsletter is up to more than 11,000 subscribers. We've placed a helpful archive of all past issues at FirmwareUpdate.net. And I'm working hard to make the format as easy and fun to read as it is informative. If you develop embedded software, I'm certain you will find it valuable. If you're not already a subscriber, you can join the mailing list at http://visitor.constantcontact.com/email.jsp?m=1101728959593.

Note that each issue of Firmware Update is Copyright 2009-2010 by Netrino, LLC. However, you may reprint the newsletter for non-commercial purposes. Indeed, I encourage you to forward it to colleagues who may benefit from the information.

Rate Monotonic Analysis (RMA) is a way of proving a priori via mathematics (rather than post-implementation via testing) that a set of tasks and interrupt service routines (ISRs) will always meet their deadlines--even under worst-case timing. In this blog, I address the issue of what to do if two or more tasks or ISRs have equal priority and whether round robin scheduling is necessary in an RTOS to deal with that special case.

First a little background. In order for the schedulability analysis portion of the RMA mathematics to provide meaningful results, the following assumptions must hold:

• A preemptive real-time operating system (RTOS) is used for scheduling


• Each task or ISR must be assigned a fixed priority (relative to the others) that is not changed while the system runs


• Unbounded priority inversions must be prevented

Under RMA, the relative priorities are assigned according to a simple rule: "The more often a task or ISR runs (in the worst-case), the higher its priority." Put another way, the task or ISR with the longest period between iterations (interarrival time, if you prefer) is least important. This is because an infrequent but high-priority task could prevent a more frequent task from missing an entire iteration.

So what happens if you are using RMA to assign priorities and you wind up with two (or more) tasks or ISRs assigned equal priority? (Translation: they have the same worst-case interarrival times). Must they be assigned equal priority in the real system? What if the RTOS (in the case of tasks) or hardware (in the case of interrupts) doesn't support round-robin scheduling--or even equal priorities with run-to-completion?

Interestingly, it turns out not to matter a bit whether you:

1. Merge the two tasks into one (i.e., executed code for Task A then Task B)


2. Give them equal priority with round robin behavior, or with run-to-completion behavior


3. Give them adjacent unequal priorities (in either relative order)

If you run through the timing diagrams for each of the above scenarios, you'll see that all three are equivalent. Except that the equal priority with round robin potentially suffers a performance impact from unnecessary additional context switches.

Several people have asked me to write about my new car and post some pictures here. If you’re tired of hearing about it, be sure to stop reading now. I’m having a whole lot of fun with this vehicle, and it doesn’t take much to get me to talk about it.

Chevrolet’s engineers designed an amazing automobile that is in very high demand. People have been waiting 6 months or more for their orders to be delivered from the GM factory in Oshawa, Canada. For a period of time last year, the 2010 Camaros were as difficult to get as the new Priuses. I was lucky enough to wait only 7 weeks.

Two Saturdays ago I picked up my brand new 2010 2SS Camaro. It’s a limited Transformers edition “Bumblebee” – right out of the movie – with special emblems and logos that charm kids and adults who are kids at heart (like me). It has just about every feature you can imagine, and under the hood is a 6.2 liter 400hp V8 engine. (Believe it or not, it’s rated at 25 mpg on the highway.) There must be a quite a large number of ICs performing all sorts of tasks to make my Camaro safe and powerful. And to the delight of my fellow blogger, Eric Huang of “To USB or Not to USB”, there’s a USB port in the console.

This time, however, I didn’t want a standard. I chose the automatic transmission. A few of my friends scoffed. I did give up a little bit of horsepower, but 400 is plenty for me! If I feel like shifting manually, I can do it with the buttons on the steering wheel that remind me of a video game controller. Plus, the automatic transmission came with a remote starter. I can’t wait to surprise parking lot onlookers. ;)

Enough talk – here are a few pictures of my Bumblebee on the showroom floor, right before I drove it home.

When Texas Instruments suggested I try the eZ430 Chronos wireless development kit in response to my Twitter call for an interesting project idea for my son's scout den, they piqued my curiosity. When they sent along a pre-launch version, I was hooked. At least at first.

Code developers beware: I got a call from Baruch Evenor the other day. He turned me on to an automatic, interactive code generator for embedded controller systems that outputs assembly code--without you ever having to write a single line of code. Too good to be true? Read on!

Of course, good or bad depends on where you sit. If you're an engineer, programmer or college professor who would like to quickly get a design up and running or would like to do a demo, this could be perfect. However, if you're a programmer dependent upon system developers short on coding skills to throw work your way, this may not be so good. Time to move up the food chain.

That may all be down the road. Right now, the Embedded Controller Code Maker, or ECCM for short, does allow even experienced programmers to get up and running quickly and not only with new code. It also automatically integrates existing code with the ECCM module code.

The site has a bunch of examples already set up, with tons of code for controllers, appliances, motor control, multi-channel processing systems and military applications.

Check it out. I'd be curious to hear what you think: Just a simple, useful tool, or the start of 'autocoding' in earnest. Email me at patrick.mannion@ubm.com.

Congratulations to Accellera’s Verification IP Technical Subcommittee (VIP-TSC) for reaching yet another milestone on its journey to achieve harmony among verification standards. The near-unanimous desire and commitment to create a Universal Verification Methodology is an indication of the still growing need for collaboration among verification engineers, verification IP vendors, service providers, and tool suppliers – and their faith in Accellera to do so as an open standards organization.

In my October 1, 2009 blog post, A milestone completed towards verification standards, I wrote:

Now it’s time for the group to start working on their “long term” standard. Their efforts will produce a common base class library that can be used in simulators from multiple design automation tool vendors. The common base class library will foster a broad (universal) verification methodology to benefit verification engineers and developers of verification IP.

Again I’m optimistic that the VIP-TSC will provide the industry with an effective verification standard. Hmm. Maybe they will call it the Universal Verification Methodology (UVM).

According to the status report from the VIP-TSC, the next phase of their work is indeed called the Universal Verification Methodology (UVM)!! I’m pretty sure the working group didn’t refer to my post when deciding on a name for the standard, but it’s fun to see a prediction come true nevertheless.

And now it’s time for the group to start working on their “long term” standard. Their efforts will produce a common base class library that can be used in simulators from multiple electronic design automation tool vendors. The common base class library will foster a broad (universal) verification methodology to benefit verification engineers and developers of verification IP.

The VIP-TSC working group that will now tackle UVM appears to be focused on a critical aspect of standardization – delivering not only a specification but also a usable reference implementation. In the short-term phase of their work, they created an interoperability guide, and now they will work on providing a single UVM library that will reflect the best of VVM and OVM. This is what I like about an industry collaboration that’s focused as much on deployment of a standard as it is on the creation of it.

I’m glad to see this open, inclusive, and timely standard coming to life with support from a wide-ranging verification community. Synopsys strongly endorses this UVM effort under Accellera. I encourage the committee to ensure that UVM not only meets immediate requirements but also builds the foundation of an industry-wide verification methodology for years to come.

Overall, big kudos to the working group for their focus on the long term goals, their dedication, and their hard work. It’s a great way to start 2010!