ARM-based devices are seeing tremendous growth across smart- phones, netbooks, and embedded computers. While ARM CPUs have benefited from their advantages in power efficiency in these markets, ARM CPUs also continue to increase in performance such that they are now within the range of x86 CPUs for many classes of applications. This is spurring the development of new ARM-based microservers and an upward push of ARM CPUs into traditional server, PC, and network systems.
To help bring the benefits of virtualization to ARM-based devices, ARM CPUs now include hardware support for virtualization. Although virtualization is commonly used on x86-based systems, there are key differences between ARM and x86 virtualization.
First, ARM and x86 virtualization extensions have important differences such that x86 hypervisor designs are not directly amenable to ARM. These differences impact hypervisor performance and design, especially for multicore systems, but have not been evaluated with real hardware.
Second, unlike x86-based systems, there is no PC-standard hardware equivalent for ARM. The ARM market is fragmented with many different vertically integrated ARM platforms with non-standard hardware. Virtualizing ARM in a manner that works across the diversity of ARM hardware in the absence of any real hardware standard is a key challenge.
In this paper, we describe our experiences building KVM/ARM, the ARM hypervisor in the mainline Linux kernel. KVM/ARM is the first hypervisor to leverage ARM hardware virtualization support to run unmodified guest operating systems (OSes) on ARM multicore hardware.
Our work makes several contributions. First, we introduce split-mode virtualization, a new approach to hypervisor design that splits the core hypervisor so that it runs across different privileged CPU modes to take advantage of the specific benefits and functionality offered by each CPU mode.
This approach provides key benefits in the context of ARM virtualization. ARM introduced a new CPU mode for running hypervisors called Hyp mode, but Hyp mode has its own set of features distinct from existing kernel modes. Hyp mode targets running a standalone hypervisor underneath the OS kernel, and was not designed to work well with a hosted hypervisor design, where the hypervisor is integrated with a host kernel.
For example, standard OS mechanisms in Linux would have to be significantly redesigned to run in Hyp mode. Split- mode virtualization makes it possible to take advantage of the benefits of a hosted hypervisor design by running the hypervisor in normal privileged CPU modes to leverage existing OS mechanisms without modification while at the same time still using Hyp mode to leverage ARM hardware virtualization features.
Second, we designed and implemented KVM/ARM from the ground up as an open source project that would be easy to maintain and integrate into the Linux kernel. This is especially important in the context of ARM systems which lack standard ways to integrate hardware components, features for hardware discovery such as a standard BIOS or PCI bus, and standard mechanisms for installing low-level software.
Third, we demonstrate the effectiveness of KVM/ARM on real multicore ARM hardware. Our results are the first measurements of a hypervisor using ARM virtualization support on real hardware. We compare against the standard widely-used Linux KVM x86 hypervisor and evaluate its performance overhead for running application workloads in virtual machines (VMs) versus native non-virtualized execution.
A standalone bare metal hypervisor would need to be ported to each and every supported hardware platform, a huge maintenance and development burden. Linux, however, is supported across almost all ARM platforms and by integrating KVM/ARM with Linux, KVM/ARM is automatically available on any device running a recent version of the Linux kernel.
By using split-mode virtualization, we can leverage the existing Kernel-based Virtual Machine (KVM)hypervisor interface in Linux and can reuse substantial pieces of existing kernel code and interfaces to reduce implementation complexity.
KVM/ARM requires adding less than 6,000 lines of ARM code to Linux, a much smaller code base to maintain than standalone hypervisors. KVM/ARM was accepted as the ARM hypervisor of the mainline Linux kernel as of the Linux 3.9 kernel, ensuring its wide adoption and use given the dominance of Linux on ARM platforms. Based on our open source experiences, we offer some useful hints on transferring research ideas into implementations likely to be adopted by the open source community.
Our results show that KVM/ARM achieves comparable performance overhead in most cases, and significantly lower performance overhead for two important applications, Apache and MySQL, on multicore platforms. These results provide the first comparison of ARM and x86 virtualization extensions on real hardware to quantitatively demonstrate how the different design choices affect virtualization performance. We show that KVM/ARM also provides power efficiency benefits over Linux KVM x86.
To read this external content in full, download the complete paper from the author open online archives at Columbia University .