The architecture of ARMv8-based firmware systems
Another initialization path may be used in certain ARMv8-based server systems: ATF is started during the UEFI PEI phase, after which the transition to the UEFI DXE phase occurs.
The ARMv8 UEFI differs significantly from that on x86. The PEI and DXE (Driver) phases are used for both x86 and ARMv8. However, on many ARMv8 systems, the PEI phase is significantly reduced and no hardware initialization is performed during it. This stage consists of setting up the MMU translation tables, configuring the interrupt controller and the CPU timer (according to the UEFI specification, the only interrupt processed for this environment is the timer interrupt), building the EFI Hand-off block (HOB), and execution of the DXE Core. At this stage, native UEFI modules tend to use the platform-specific SMC calls described above.
The bulk of UEFI work is performed at the DXE phase. First, this involves loading and the startup of hardware drivers – both for on-chip peripherals and external devices connected via PCIe, USB, SATA, etc. interfaces.
It should be noted that ARMv8-based systems differ significantly from similar systems based on the x86 architecture in terms of configuration, device detection mechanisms, etc. For example, the main device detection mechanism for x86 is scanning the PCI configuration space and assigning memory addresses to devices, which they have to decode. In the case of ARMv8-based systems, built-in peripherals almost always have fixed addresses in the memory space (ports are unused, as they are unsupported by the CPU architecture) and in some cases are not visible in the PCI configuration space. For such systems, there is a hardware description composed of a Flattened Device Tree, a tree-like description of device connections, which also describe resources such as memory ranges and interrupt numbers associated with these devices.
In more advanced systems, SoCs support access through the PCI configuration space and contain controllers implementing access to this space via the Enhanced Configuration Access Mechanism (ECAM). Since the memory addresses of such units are fixed, the common PCI device configuration mechanism is not applicable. Specifically, for the systems with fixed PCI device address windows, the Enhanced Allocation PCI capability has been developed, which resolves this conflict. A separate article could be written on the unique properties of this capability. In brief, it can be described as a set of alternative registers containing information about memory addresses, bus numbers (for built-in PCI-PCI bridges), etc.
UEFI cannot be separated from another method of passing on information about the platform configuration – ACPI. At the moment, the development and refinement of ACPI specifications for the improved support of the ARMv8 architecture is ongoing. According to available information, ACPI should become the key method of describing essential information about the platform (primarily the number and configuration of processor cores, PCI/PCIe controllers) and its management for ARMv8. Some of the ARMv8 OSes planned for release only support the ACPI mechanism.
The DXE stage includes device detection and their initialization and registration in UEFI, as well as preparation for OS booting. The latter consists of preparing the system memory map and configuration information – that is, loading, generating, and publishing ACPI tables, the modification of these to reflect the current configuration of the platform, making similar changes to the FDT, and checking and generating checksums. The modules loaded at this stage may implement UEFI Runtime Services – functions available for calling from the OS in runtime. It should be noted that in all the systems that have ever been used by the authors of this article, device detection was implemented via the PCI ECAM mechanism.
Upon completion of this stage, the Boot Device Selection (BDS) commences. A separate module is usually used at this stage that processes the values of the “BootOrder,” “BootNext,” and other related variables. Often, this module implements a (pseudo) graphical user interface. At this point, there are many commonalities with x86-based systems: the same boot methods are used – PXE, iSCSI, block devices (such as SATA/SAS/USB drives, SSDs, and NVME devices), etc.
It is necessary to call the reader’s attention to the drivers of external devices (usually PCIe devices) for ARMv8 UEFI. They can be implemented in the form of modules located in storage devices (with the FAT32 file system) as well as reside directly in the device (Option ROM). Adding ARMv8 to the list of supported architectures causes problems for vendors in some cases. The simple recompilation of the source code for ARMv8 is not always sufficient, because some modules are not designed to function in a full 64-bit address space. Difficulties may also arise due to the fact that the translation of the PCI bus to the processor addresses and vice versa is widely used in ARMv8 systems. This is caused by the decision to abandon legacy "windows" located within the lower 32 bits of memory address spaces. In terms of support enhancement, drivers compiled in the EBC bytecode may provide the required level of compatibility. However, when this article was written, the EBC interpreter for ARMv8 was at an early stage of development.
The transfer of control to the module loaded into the memory (boot loader or directly into the OS kernel) is performed in accordance with the UEFI specification: the UEFI handle of the module in the X0 register, the system table pointer in X1, and the return address in X30 (LR).
The OS kernel takes some preparation steps using the UEFI services, then sets its own translation tables and calls the UEFI methods ExitBootServices() and SetVirtualAddressMap(). This is necessary because the UEFI code executes in the same address space as the OS kernel. In addition, timer interrupts and any possible DMA transfers have to be disabled. There is a notable aspect of the ARMv8 Linux OS design: the main kernel code executes in the EL1 mode, whereas the EL2 mode is reserved only for a part of the KVM hypervisor code. Thus, the kernel drops its privilege level from EL2 to EL1 during initialization. After that, only the Runtime Services, a subset of all UEFI services, are available to the kernel. The Linux kernel on ARMv8 uses the PSCI interface extensively when it is implemented in one of the ATF modules, as mentioned earlier. This is especially characteristic of multicore systems. The interface itself and the process of secondary CPU core initialization can be briefly described as issuing SMC calls with the PSCI function number and the entry point of the initialization function as parameters. As a matter of fact, calls to UEFI and SMC services are currently the main means of interaction between the OS and the firmware. There are draft specifications of other firmware event notification facilities, but there have been no reports of any completed implementations to date (2015).
To sum up, it should be mentioned that this article does not provide an exhaustive description of the functioning and interaction of ARMv8-based firmware components. Moreover, the development of the architecture is undergoing constant refinement and refactoring, which are likely to provide material for new articles.
Sergey Temerkhanov is a Principal Software Engineer at Auriga, Inc. He has over 14 years’ experience in the field of embedded software development, including deep expertise in bootloader/firmware porting and development, Linux kernel, Linux device driver development, embedded Linux distribution building and deployment. Sergey has extensive experience in board bring-up, low-level software and driver development, hardware-assisted debugging, real-time control software development, and the design of special-purpose processor system architectures. Sergey Temerkhanov holds a Master of Technology (MTech) in radio engineering from Bauman Moscow State Technical University.
Igor Pochinok is the Head of Mobile and Embedded Software Systems Laboratory of the MSU RCC (Research Computing Center at Lomonosov Moscow State University). He is an author and co-author of multiple scientific papers, studies, and articles related to embedded software development and testing. Igor Pochinok holds a PhD in Computer Science.