Device drivers in user space

Hemant Agrawal and Ravi Malhotra, Network Processing Division, Freescale

November 19, 2012

Hemant Agrawal and Ravi Malhotra, Network Processing Division, Freescale

Constraints of user-space drivers
Direct access to network devices brings its own set of complications for user-space applications, which were hidden by several layers of kernel stack and system calls.

a. Sharing a single network device across multiple applications.

b. Blocking access to network data.

c. Lack of network stack services like TCP/IP.

d. Memory management for packet buffers.

e. Resource management across application restarts.

f. Lack of a standardized driver interface for applications.

Sharing devices
Unlike the Linux socket layer which allows multiple applications to open sockets--TCP, UDP, or raw IP--the user-space network drivers allow only a single application to access the data from an interface. However, most network interfaces nowadays provide multiple buffer descriptor rings in both receive and transmit direction. Further, these interfaces also provide some kind of hardware classification mechanism to divert incoming traffic to these multiple rings. Such a mechanism can be used to map individual buffer descriptor rings to different applications. This again limits the number of applications on a single interface to the number of rings supported by the hardware device. An alternate to this is to develop a dispatcher framework over the user-space driver, which will deal with multiple applications.

Blocking access to data
Unlike traditional socket-based access, which allows user-space applications to block until data was available on the socket, or to do a select()/poll() to wait on multiple inputs, the user-space application has to constantly poll the buffer descriptor ring for an indication for incoming data. This can be addressed by the use of a blocking read() call on the UIO device entry, which would allow the user-space application to block on receive interrupts from the Ethernet device. This also provides the application with the freedom of when it wants to be notified of interrupts--in other words, instead of being interrupted for each packet, it can choose to implement a polling mechanism to consume a certain number of buffer descriptor entries before returning to other processing tasks. When all buffer descriptor entries are consumed, the application can again perform a read() to block until further data arrives.

Lack of network-stack services
The Linux network stack and socket interface also abstract basic networking services from applications such as route lookup and ARP (Address Resolution Protocol). In the absence of such services, the application has to either runs its own equivalent of a network stack or maintain a local copy of the routing and neighbor databases in the kernel.

Memory management for buffers
The user-space application also needs to deal with the buffers provided to the network device for storing and retrieving data. Besides allocation and freeing of these buffers, it also needs to perform the translation of the user space's virtual address to the physical address before providing them to the device. Doing this translation for each buffer at runtime can be very costly. Also, since the number of TLBs in the processor may be limited, performance may be hit. The alternative is to use Huge-TLB to allocate a single large chunk of memory and carve out the data buffers out of this memory chunk.

Application restart
The application is responsible for allocating and managing device resources and current state of the device. In case the application crashes or is restarted without being given control to perform cleanup, the device may be left in an inconsistent state. One way to resolve this could be to use the kernel space's UIO component to keep track of application process state and on restart, to reset the device and reset any memory mappings created by the application.

Standardized user interface
The current generation of user-space network drivers provide a low-level application programming interface (API), which is often very specific to the device implementation, rather than conforming to standard system-call API like open()/close(), read()/write() or send()/receive(). Device-specific APIs imply that the application needs to be ported to use each specific network device.

Freedom with limitations
While the UIO framework provides user-space applications with the freedom of having direct access to network devices, it brings its own share of limitations in terms of sharing across applications, resource and memory management. The current generation of user-space network drivers works well in a constrained use-case environment of a single application tightly coupled to a network device. However, further work on such drivers must address some of these limitations.

Note: this article was first published in the International Journal of Information and Education Technology; Hemant Agrawal and Ravi Malhotra, Member, IACSIT, "Device Drivers in User Space: A Case for Network Device Driver," International Journal of Information and Education Technology vol. 2, no. 5, pp. 461-463, 2012.

References:
1. Dragan Stancevic, "Zero Copy I: User-Mode Perspective"; Linux Journal, Jan 2003, pp. 105

2. Moti N. Thadani et al., "An Efficient Zero-Copy I/O Framework for UNIX"; Sun Microsystem Inc, May 1995.

3. Matt Welsh et al., "Memory Management For User-Level Network Interfaces"; IEEE Micro, Mar-Apr 1998, pp. 77-82

4. Hans-Jürgen Koch, The Userspace I/O HOWTO, http://www.kernel.org/doc/htmldocs/uio-howto.html

Hemant Agrawal is a software architect for the Networking Processor Division of Freescale working on QorIQ product line. He holds a bachelor's degree in electrical engineering from Institute of Technology, BHU, India.

Ravi Malhotra is a software architect for the Networking Processor Division of Freescale working on QorIQ product line. He holds a bachelor's degree in electrical engineering from Institute of Technology, BHU, India.