My experiments with ARMv8 server software stackIn the last couple of weeks, I had the pleasure to get my hands on a virtual prototype featuring an ARMv8 Cortex-A57 core.
The ARMv8 architecture introduces two new execution modes:
- AArch32, which is backwards compatible with ARMv7-A. and
- AArch64 with a completely new instruction-set and supports 64-bit registers + 64-bit memory accesses.
AArch64 also comes with a new exception model with four Exception Levels (EL0-lowest to EL3-highest). This exception model is supporting hardware assisted virtualization and security, which are very important hardware features for server software stacks. In the exception level EL3, the core typically runs a secure monitor and EL2 is used to host a hypervisor or virtual machine. The operating system and application software typically use EL1 and EL0.
An operating system which is supporting ARMv8, such as Linux, requires heavy ground work to be completed before the basic OS services are enabled. Considering the scope of changes in ARMv8, this is much more effort than required when moving from v6 to v7. Since hardware is not available for the most recent ARMv8 architecture specification, the software community is using virtual prototypes for the software porting. With that, an initial version of the Linux kernel, coming from Linaro, is already available for AArch64. Also, a lot of libraries have been already ported to AArch64.
One of our products, the so-called Virtualizer Development Kit (VDK) for ARMv8 and DesignWare provides a virtual prototype that embeds a Cortex-A57 (ARM's FastModel) on a virtual motherboard, which is oriented at the ARM Versatile Express uATX specification. Our engineering team has also packaged a recent v3.9 Linaro Linux kernel for AArch64 and ported various Synopsys DesignWare IP drivers (USB 3, GMAC, eMMC) to 64bit. Also included with the package is a complete AArch64 build environment.
My personal goal has been to get a realistic server software stack running on this VDK and demo is at the Linaro Connect event in Dublin. I have been especially interested in the virtualization aspects. Ideally I would be able to run multiple guest operating systems in parallel and run applications such as Apache, MySQL, and PHP. I started on this project with only the limited knowledge about ARMv8 that I described in the first paragraph.
Building an AArch64 host Linux kernel with kernel-based virtualization support, I have started researching the feasible options, hoping that most of the required software pieces have been already ported to ARMv8, or at least somebody is providing some information about it. I have concluded that Linux plus the so called Kernel based Virtual Machine (KVM) would be a good starting point for my work. Marc Zyngier has been providing a Linux kernel git repository with a patch that includes the KVM port for AArch64. Getting this to work on our VDK was easy. Just compile the kernel and load the image into the NOR flash of the VDK is enough. The VDK automatically generates the device tree information to configure the host kernel for the underlying hardware platform. No porting effort was needed, as it was before the ARM Linux kernel supported device trees. The kernel initially got stuck with some problems with the VGA console driver. The VDK’s capability of making the kernel debug messages visible (via catching strings sent to the printk function immediately) even before the UART-, or framebuffer-console are working has been of great help.
Building and debugging a Qemu-based virtual machine
As a next step, I had to find a virtual machine that supports both the KVM and ARM. The 1st choice here is Qemu-kvm for ARM. This version of Qemu uses the virtualization services provided by KVM to execute the guest software. It does not emulate the instructions as we know it from typical processor instruction-set simulation. I found a Linaro page that helped me to accomplish the bring up of Qemu-kvm. This was a lengthy process. For the cross compilation, I have been using an Ubuntu Linux running inside an Oracle Virtual Box. Setting up a complete bootstrap cross-compilation environment which is needed to build a 64bit ARM executable can take some time, and requires quite some search for extra packages etc. Qemu-kvm depends on AArch64 libraries such as zlib, glibc, bison, flex, and the device tree compiler.
After building Qemu-kvm successfully on Linux 3.10+KVM for AArch64, I ran into the issue that Qemu was bailing out at its initialization. For debugging such integration issues, the non-intrusive stop mode debugging of VDKs is of help. Using ARM’s DS-5 for v8, which is Linux aware, I debugged from the Qemu initialization, via the system call, into the kernel’s kvm ioctl handler. I found out that the HW target identifier returned by Linux was not expected by Qemu. Thus, I had to register my Versatile Express based VDK as a supported target for Qemu. In this process, I have developed a small OS-awareness kit which allows our VDK to perform a function trace (integrated in ARM’s DS-5 via an Eclipse plugin) for the different exception levels and kernel/user threads. Having OS-aware tracing was of great help to understand the sequencing of actions in the software.
Click on image to enlarge.
Building the guest kernel and guest filesystem
The next step has been to prepare a guest kernel and guest filesystem. I could simply re-use the host AArch64 Linux kernel also as a guest kernel. I have tailored the guest kernel device tree configuration for the simple Qemu-kvm virtualized hardware (v8 core, GIC, timer, UART, Ethernet). I have added an Ethernet model to the default Qemu hardware to emulate network traffic. For my initial work, I have not looked at the more performing virtio- (using KVM networking API) or the highest performing pass-through- (using HW assisted virtualization) peripheral virtualization.
For the guest filesystem, I have been leveraging the buildroot project. A configuration for ARM v8 already exists here. It was as simple as clicking two checkboxes to add PHP and MySQL to the filesystem. A bigger hassle has been Apache (httpd) which is not included in the buildroot package. It has general issues with cross-compilation, especially the configuration files and Makefiles are not ready for AArch64 and have dependencies on further libraries (prce, apr, apr-util). But, I also got this to work finally.
Reaching the ARMv8 Apache webserver from my MS Internet Explorer
The next step has been to configure the webserver and test whether I can successfully reach it from the outside world (e.g. another PC in the network). For this purpose, the VDK’s Ethernet model supports real world I/O capabilities, which allow the VDK to be connected to the real Ethernet. At the initial testing stage I had some initial problems with the webserver. It did not respond when I was contacting it from the outside (my host PC IE browser) world. The VDK’s Ethernet model supports analysis of the Ethernet traffic in our analysis tools as well as in the famous Wireshark analyzer. It quickly became clear that the problem was due to a wrong configuration of the Apache’s http version.
After three weeks of work, we have been able to demonstrate our system at the Linaro Connect event in Dublin. We have been showing a complete LAMP (Linux, Apache, MySQL, PHP) webserver running on an AArch64 guest Linux, on top of a virtual machine, on top of an AArch64 host Linux. All of which are simulated using virtual prototype. The webserver has been communicating with the real world through real world I/O and I could show a website through Internet Explorer on my host PC.
Click on image to enlarge.
I have started with no knowledge on ARMv8 and ended up with a complete LAMP software stack running on an ARMv8 architecture. In the first place, this would not have been possible without the work done by Linaro and the open source community. But, I would also not have been able to accomplish this without the easy, extensive and extensible debugging capabilities of a virtual prototype. Using a virtual prototype helped me to understand the ARMv8 architecture, the general operation of AArch64 Linux and assisted in solving problems that cross the boundaries of multiple software layers (host Linux+KVM, Qemu VM, guest OS, guest applications).
Achim Nohl is a technical marketing manager for virtual prototyping at Synopsys. He actively publishes technical articles, conference papers and blog posts around the topic of developing embedded software using virtual prototypes. His specialty is bringing up firmware, drivers and operating systems for the most recent CPUs and IPs. Achim holds a diploma degree in Electrical Engineering from the Institute for Integrated Signal Processing Systems at the Aachen University of Technology, Germany. Before joining Synopsys, Achim has worked in various engineering and marketing roles for LISATek and CoWare.