Furthermore, the fact that the KVM hypervisor is a complete Linux kernel means that it is huge by embedded standards. Few embedded systems could afford increasing memory size by dozens if not hundreds of Megabytes just to accommodate the hypervisor. And the large size completely rules out running it in on-chip memory.

Finally, KVM does not provide the real-time capability required for many embedded systems. While real-time support in Linux has made significant progress over the years, it is not a suitable replacement for an RTOS in many cases. If it were, why would one run an RTOS concurrently with a rich OS in the first place? In summary, KVM doesn't look like a promising hypervisor for embedded systems. How about Xen?

A port of Xen to ARM and Power have been done at Samsung and IBM respectively; presumably a MIPS port (if it doesn't already exist) would be possible too. However, Xen was not designed for real-time use. The Linux experience shows that it is hard, expensive and time-consuming to retrofit real-time support into a system that was not designed for it. Xen is unlikely to be an exception.

How about communication? The authors of Xen stated that fast inter-VM communication was not a design goal, so it would have to be retrofitted. This brings up memories from a related area--early microkernels suffered from poor communication performance and attempts to improve it failed. Jochen Liedtke, the creator of the L4 microkernel, finally showed that fast communication must be designed into the system from the beginning. Any attempt to beef up inter-VM communication in Xen is likely to go through the same experience.

How about size? The Samsung port of Xen to ARM fits into 2MB. This is impressively small compared to the typical footprint of an enterprise-style hypervisor, but it is still far too big to fit into on-chip memory. Furthermore, Xen relies on a privileged VM, called dom0, to provide device drivers. That domain contains a complete Linux guest OS, and is part of the system's TCB. On ARM, dom0 adds another 14MB to the memory footprint. Hence, even a significantly slimmed-down Xen hypervisor is still huge by embedded standards.

Now let's look at the hypervisor designed for embedded systems. Like Xen, OKL4 supports a number of architectures, including ARM, x86 and MIPS (related L4 ports support Power). The system's real-time capability is demonstrated by its use in an estimated 250 million mobile phones.

The memory footprint of OKL4 is less than 64kB, more than an order of magnitude less than Xen's, and certainly more in line with the resources of typical embedded systems. Specifically, this is small enough to fit into on-chip RAM on many SoCs, leaving some space for application code.

What may be the most impressive comparison is the size of the source code. Xen is of the order of 100 kLoC, plus all of Linux in dom0, which puts the total into many 100s of k-LoC, while all of OKL4 is around 10 kLoC. The Samsung researchers report that for the port of Xen to ARM they had to modify or add 23 kLoC. This is more than twice the size of the complete ARM source of the OKL4 hypervisor! If a port is twice the size than doing it properly from scratch, why bother?

Conclusions
Enterprise and embedded domains have strongly differing characteristics and this is reflected in the requirements put on virtualization solutions in the two domains. A one-size-fits-all approach that is implied by porting an enterprise hypervisor to embedded systems will short-change embedded-systems designers. Embedded systems need their own optimized solutions.

About the Author
Gernot Heiser, cofounder of Open Kernel Labs, is the technical leader of the firm. Prior to co-founding OK, Dr. Heiser created and led the Embedded, Real-Time and Operating Systems (ERTOS) research program at NICTA. He can be contacted at gernot@ok-labs.com.