Once you have a real-time Linux platform of your very own, you'll need to know how to establish communications between real-time tasks and non-real-time Linux application code.
Last month, the first installment of this two-part series explored the key concepts and architectural characteristics that form the foundation of real-time Linux (“Real-Time Linux,” p. 35). It also showed how to interface a data-acquisition driver with that operating environment. This installment concludes the series by digging into the details of how to develop a real-time application suited for real-time Linux. Note that this month, just as in Part 1, the examples use RTLinux for the sake of consistency. Other implementations of real-time Linux are available, most notably RTAI. Although these two implementations differ in a few syntactical details, they both operate in a roughly similar fashion, so the principles and concepts are valid for both.
Riding a clock with real-time Linux
Let's begin this installment by reviewing how a real-time task communicates with a user process. Some real-time applications can run silently in the background without any user interface. Increasingly, though, real-time apps do need a user interface as well as other system functions such as file operations or networking-all of which must run only in the user space. The difficulty, though, is that user-space operations are non-deterministic and incompatible with real-time operation. Fortunately, real-time Linux supplies a mechanism to decouple real-time and non-real-time operations with respect to time. This mechanism comes in driver form and is called a real-time FIFO.
When insmod inserts the rtl_fifo.o driver into the Linux kernel, that driver registers itself as part of RTLinux and as a Linux driver. Upon insertion into the Linux kernel, real-time Linux FIFOs become available both for user space processes and real-time tasks.Before jumping deeply into the details of real-time FIFOs, let's review some aspects of real-time application architecture (Figure 1). A popular and effective way to design an embedded application is to decouple the real-time parts from the inherently non-real-time functions (see guidelines in Table 1). If any part of an application-such as the user interface, graphics, database, or networking-needs only soft real-time performance, it's better to write that part in the user space. Then write only that part of the application that must meet all timing deadlines as a real-time task.
|Table 1: Guidelines for splitting an application into real-time and non real-time parts
Functions to place in non-real-time section
Functions to place in real-time section
Note that some of the very latest developments in RTLinux (PSC, the Portable Signal Code) and in RTAI (LXRT, the Linux Extension for Real Time) have introduced a way of performing soft and hard real-time in the user space.
Any hard real-time task runs under control of RTLinux. That task typically performs a periodic job, processes interrupts and communicates with I/O device drivers to acquire or output analog and digital information. When the real-time task needs to inform the user process that an event has happened, it puts a message into a real-time FIFO. Each FIFO can transfer data in one direction: either from the real-time task to a user process or the reverse. Thus you need to use two FIFOs for duplex communications. Any reads or writes from or to the real-time side are non-blocked operations, so rtf_put() and rtf_get() return immediately regardless of the FIFO state.
From the application side, the FIFO behaves like a regular file. By default, the RTLinux installation creates 64 real-time FIFO nodes in the /dev directory, but you can create additional nodes if necessary. For example, to create /dev/rtf80, use:
mknod c 150 80;
chmod 0666 /dev/rtf80
where 150 is the real-time FIFO major number and 80 is rtf80's minor number.From the standpoint of a user process, a real-time FIFO can perform standard file operations. Viewing a FIFO from a real-time task, you can choose two ways of communicating with it: either call RTLinux FIFO functions directly, or use the FIFO as a RTLinux device driver and use open(), close(), read(), and write() operations. To use a FIFO as a device you should set the configuration variable CONFIG_RTL_POSIX_IO to 1 in rtl_conf.h.
The set of functions that controls the FIFO is relatively simple. It includes rtf_create(), rtf_destroy(), rtf_put(), and rtf_get(). A final function, rtf_create_handler(), allows you to set up handler functions. Then the rtl_fifo driver calls this handler every time a Linux process reads or writes to the FIFO. Note that this handler resides in the Linux kernel context, so it's safe to make any kernel calls from this handler when it's called by Linux. The best way to communicate from this handler to a real-time task is to use a flag or thread-synchronization functions. Finally, note that the FIFO driver must allocate and deallocate kernel memory. Thus you shouldn't call rtf_create() inside a real-time thread. Instead, call the rtf_create() function in init_module() and rtf_destroy() in cleanup_module().
For example, Listing 1 shows the real-time part of a simple data-acq application that uses two FIFOs. You create both in init_module(), which assigns them minor numbers 1 and 2. Before calling rtf_create(minor, size) this program calls rtf_destroy(minor) just in case some other program has already created this FIFO, one possibility being another module that failed to load during development. Then make a call to rtf_create_handler(ID, &pd_do_aout) to register the data-acq analog output function pd_do_aout() with this real-time FIFO. Note that the author created the real-time thread pp_thread_ep() such that it is periodic with an interval equal to 1/100 sec.
Every time the periodic thread gets control of the system, it calls rtf_put(ID, dataptr, size) to insert data into the FIFO with the minor number 2. The Linux process opens /dev/rtf2, reads from the real-time FIFO, and displays the acquired data. The process opens /dev/rtf1 to write data to the other real-time FIFO. When a user moves a screen slider to change the analog output voltage, the process writes the new value to that FIFO. In turn, RTLinux calls the pd_do_aout() handler, whereupon pd_do_aout() gets the value from the FIFO by using rtf_get() and calls the actual hardware driver to set the voltage on analog output. As you can see, the real-time task and user process use the FIFO asynchronously.
Sharing memory among tasks
FIFOs provide a convenient mechanism for interfacing a user process and a real-time task, but it's more appropriate to use them as message queues. For instance, a real-time thread can use a FIFO to log test results, which a user process can then read and store in a database file.
Most data acquisition applications involve the transfer of a significant amount of data between the kernel and user spaces. Linux kernel v. 2.2.x doesn't provide a mechanism for sharing data between these spaces, but v. 2.4.0 is expected to include the kiobuf framework for this purpose. To get around this shortcoming in the current stable kernel, RTLinux includes the mbuff driver, which can allocate named memory regions in the virtual kernel memory using vmalloc(). It employs the same memory allocation and page locking techniques used in the bttv frame-grabber driver included in most Linux distributions.
In more detail, mbuff goes page by page and locks virtual memory to actual physical memory pages. Then any real-time or kernel task or a user process can access this memory at any time. By locking pages of virtual memory to physical ones, mbuff guarantees that allocated pages reside in physical memory permanently and that a page fault exception won't occur. In other words, it guarantees that the VMM won't be called when a real-time or kernel process accesses allocated memory. Note that because real-time Linux freezes standard kernel execution while a real-time task is executing, any call to the VMM causes the system to halt. Moreover, even the normal Linux kernel driver causes a system fault if it tries to access a virtual memory page that isn't located in physical RAM.
Because mbuff is a Linux driver, its functions are accessible through the device node /dev/mbuff, which exposes several entry points, including mmap() to map the kernel space address into the user space. It also uses the ioctl() entry point to control it. Fortunately you don't need to fill extensive structures and makes calls to ioctl directly. Instead, mbuff supplies a wrapper for ioctl() calls, and you can allocate and free a shared memory buffer merely by calling two simple functions.
Certainly, you can't make calls to the mbuff driver from a real-time task because that driver calls functions for virtual-memory allocation, which is an inherently non-deterministic operation. The time needed to allocate shared memory depends on amount of memory installed on the host system as well as CPU speed, disk-drive performance, and the current state of memory allocation. Thus you can allocate shared memory only from the Linux kernel side of your module, for example from init_module() or an ioctl() request.
A related question you might be asking is how much memory can you allocate for a shared buffer. I recommend that you leave at least 8MB of memory for Linux if you don't run heavy server or graphical applications. To find the optimal configuration, try to measure your real-time application's performance while limiting memory size and thereby ascertain how much memory your apps need.
Listing 2 shows how to access shared memory from both a real-time task and a user process. The kernel module and user task employ the same set of functions. Of course, you should insmod mbuff.o into the Linux kernel to use it. For instance, mbuff_alloc(“buf_name”, size) allocates a buffer with symbolic name buf_name, and mbuff_free(“buf_name”, mbuf) frees it.
|Debugging with RTLinux
You might know how difficult it is to debug kernel modules-but such isn't the case with RTLinux v. 2.3 and later. This package includes a debugger suited for real-time tasks. Unlike many other kernel debuggers, it supports source-level debugging on one PC, crash protection, and thread and SMP support. Following installation of the rtl_debug.o module, you can communicate with the debugger through real-time FIFOs. Further, rtl_debug can work in tandem with a standard gdb or graphical debugging tools such as DDD, xxgdb, or insight. The RTLinux package provides a Getting Started guide that's comprehensive enough to let you start working with rtl_debug immediately.
When you first call mbuff_alloc() with a symbolic buffer name, mbuff performs the actual memory allocation. When you call this function again from either a kernel module or a user process, it simply increases a usage counter and returns a pointer to the existing buffer. Each call to mbuff_free() decreases the usage counter, and when it reaches zero mbuff deallocates the buffer with that symbolic name. This approach eliminates problems with getting a pointer to the same shared buffer from multiple kernel modules and user processes. It also guarantees that a shared buffer remains valid until the last application frees it. Please notice that either a real-time module or a user process can do the actual allocation/deallocation of buf1 depending on which of them receives control first.
There is another “poor man's” way to share memory between a real-time application, the kernel module, and a user application, a method that's acceptable for embedded applications. For example, if you have 128MB of RAM installed on a PC you can add the line append=”mem=120m” into the lilo.conf file (Listing 3). Then, when you boot up a system with the Linux kernel and RTLinux 2.3, Linux uses only 120MB out of the 128MB. The OS doesn't use the remaining 8MB of memory (physically located from the addresses 0x7F00000 to 0x7FFFFFF), which remain reserved for sharing by various tasks running under the OS. To get memory addresses and thereby access reserved memory from the user process, you must open the /dev/mem driver with the O_RDWR access mode as one parameter and then use mmap() to reserve memory (Listing 4). Later, from a real-time module or the kernel driver side you must use ioremap(0x7F00000, 0x100000); to obtain access to those 8MB (0x100000 bytes) of reserved memory.
There are pluses and minuses to this method. You have control over neither ownership of reserved memory nor who reads or writes into it. No mechanism is defined to allocate and free chunks of memory correctly. Also, this memory becomes excluded from Linux use regardless of whether your real-time process needs it or not. Perhaps the only situation where poor man's memory sharing is favorable is in a small-footprint embedded system tailored to a particular application where you're willing to trade off the use of the mbuff driver for a smaller footprint.
This article wouldn't be complete if it failed to address real-time interrupts. Interrupts drive RTLinux like a puppeteer drives a marionette. Each timer clock causes the OS to call the RTLinux interrupt handler, which in turn kicks in the scheduler, which executes various tasks-including Linux itself-based on their priority. RTLinux introduces two types of interrupts: hard and soft. Soft interrupts are simply regular Linux kernel interrupts, which on average offer good latency but do poorly with worst-case values. The beauty of a soft interrupt handler is that it can use Linux kernel calls without restriction. This class of interrupts is useful as the second half of the processing for a hard interrupt. (See Reference 5 for more details about interrupt processing under Linux.)
Hard (or real-time) interrupts are the reason you installed real-time Linux to begin with. To install an interrupt handler, call rtl_request_irq(…), and later, to free it, call rtl_free_irq(). Depending on the system, a hard real-time interrupt under real-time Linux has a latency in the order of 15s. This value is typical for a 133MHz Pentium-based system. Faster processors show better latency. If you want to process the same IRQ from a device in both a real-time handler and in a regular Linux driver, you must request an IRQ for each of them separately.
Listing 5 shows how to install a real-time interrupt handler. RTLinux disables IRQs while it executes a real-time interrupt handler. Please notice that this code calls rtl_hard_enable_irq() before exiting from the real-time interrupt handler to re-enable them.
Two issues prevent you from calling Linux kernel functions directly from a real-time interrupt handler: the kernel disables all interrupts, and it doesn't define the execution context. Also recall that you can't perform floating-point operations there, either. A good technique to avoid these problems is to use a real-time interrupt handler to control thread execution. This example uses the function pthread_wakeup_np() to wake up a real-time thread. The interrupt handler performs the immediate work and this thread does the rest.
Advantages of SMP architecture
From the very beginning, real-time Linux was oriented toward multiprocessor support. There are only a few advantages of a symmetric multiprocessor architecture (SMP), and one of them is very important to real-time operation. Specifically, SMP introduces the advanced programmable interrupt controller (APIC). Pentium-class processors have a local APIC on-chip that delivers interrupts to a local processor. SMPs (and even some single-processor motherboards) have an I/O APIC that collects interrupt requests from peripherals and delivers them to local APICs. The old 8259 PIC is slow, handles an insufficient number of interrupt vectors, and thereby forces devices to share interrupts, slowing down interrupt processing even further. The APIC solves these problems. A system can dramatically reduce interrupt latency by requesting a unique IRQ line for each device. The APIC also speeds up synchronization code.
Real-time Linux takes advantage of the APIC. On SMP systems, the real-time scheduler uses the APIC for timing instead of the antiquated 8254. Due to PC compatibility, an 8254 sits on every ISA bus, and every call to reprogram the device is expensive in terms of processor cycles. A gigahertz CPU wastes several hundred processor cycles waiting for an 8MHz timer (approximately 2.5s). The APIC works at bus frequency and performs all timer operations almost immediately. This means that you can achieve higher periodic frequencies on SMP machines using local APIC clock. (My dual P-III-500 runs a periodic real-time thread at 100kHz without visible loss of performance.)
Real-time Linux works well with multiprocessing. It implements separate processes for each CPU. When you call pthread_create() it creates a thread that runs on the current CPU. You can assign that thread to a particular CPU by using pthread_attr_setcpu_np() to modify thread attributes. Before you call this function you must initialize thread attributes as shown in the previous installment's Listing 1.
RTLinux v. 3 includes the reserve_cpu functionality that can reserve one of the CPUs on an SMP platform for the exclusive use of RTLinux. It runs on 2.4x kernels. RTAI has almost the same features.
If you're thinking about ways to dedicate tasks to a specific CPU, make note of the “pset” project (ftp://ftp.ueidaq.com/pub/software/pset-utils-0.65.tar.gz). Using this kernel patch you can always dedicate one SMP processor to a user application or even remove one processor from a Linux processor set to dedicate it entirely for real-time tasks.
In the early days of real-time Linux, synchronization primitives were unavailable. Now POSIX-type semaphores, mutexes, and signals are starting to appear in the latest real-time Linux releases. While using such synchronization primitives is questionable in real-time designs, the most interesting is to synchronize or signal real-time tasks and user applications. However, this requires advanced expertise on the part of the software developer and is beyond the scope of this article.
The best way to learn quickly how to use synchronization functions such as pthread_mutex_init(), pthread_mutex_lock(), pthread_mutex_trylock(), pthread_mutex_unlock(), and pthread_mutex_destroy() is to look into ./examples/mutex/mutex.c. Spec-ifically, the file ./examples/mutex/sema_test.c could be a good starting point for working with semaphores.
Where real-time Linux is headed
Real-time Linux, just like Linux itself, is on the move. Every version brings more features and functionality. Real-time Linux is moving toward better POSIX 1003.x implementation. The latest developments include real-time support for user-space processes, mutexes, signals, semaphores, real-time memory management, and extended SMP support. If you're still considering which real-time system to choose for that next project, download one of the real-time Linux packages available and take a look. You'll find that Linux is a mature OS and with real-time extensions, a good choice for embedded applications.
Alex Ivchenko , PhD, is R&D engineering manager at United Electronic Industries and is one of the major developers of that firm's PowerDAQ II family of PCI-based data-acquisition boards. He has most recently spent his time writing Linux drivers for this card family. You can e-mail him at .
1. Ivchenko, A. “Real-Time Linux,” Embedded Systems Programming, May 2001, p. 35.
2. Marsh, David. “Understand Linux Device Drivers,” Test & Measurement World, April 15, 2000.
3. Johnson, M.K. and E.W. Troan. Linux Application Development. Reading, MA: Addison Wesley Longman, 1998.
4. Mantegazza, P., E. Bianchi, L. Dozio, S. Papacharalambous, S. Hughes, and D. Beal. “RTAI: Real-Time Application Interface. An introduction to RTAI for deterministic and preemptive real-time behavior for Linux,” Linux Journal, April 2000.
6. RTAI sites: www.rtai.org
7. Other good sources of driver information come from “The Linux Documentation Project” (www.linuxdoc.org), especially “The Linux Kernel” by David A. Rusling and “The Linux Kernel Hackers Guide” by Michael K. Johnson (http://kernelbook.sourceforge.net/). Finally, it's advisable to keep track of the latest kernel updates (kernelnotes.org).
- Listing 1: Using a real-time FIFO
- Listing 2: Sharing memory by using the mbuff driver
- Listing 3: Setting amount of memory kernel can use in lilo.conf file
- Listing 4: Setting up and using “poor man’s” shared memory
- Listing 5: Getting an interrupt vector