Embedding TCP/IP

Thomas Herbert

Last month, the author introduced you to the basics of TCP/IP. In this second installment, he discusses the details of putting TCP/IP into a resource-constrained embedded system.

Embedded systems have inherited the programming practices used in larger systems. Network protocols, and TCP/IP in particular, incorporate programming practices used in larger systems. As discussed in the first part of this article, the history of TCP/IP is one of adapting and modifying the original sources written at the University of California at Berkeley to embedded systems. The Berkeley stack is the basis for most of these ports and is the basis of most of the commercial TCP/IP stacks for embedded systems. Of course, real-time and embedded systems face many issues that are unique. A straight port of the Berkeley stack is not the best implementation for the particular needs of an embedded and real-time system.

Most vendors have modified the Berkeley code over the years to improve the performance of the stack in embedded systems. Any ports or modifications of the original Berkeley sources should address the following issues. This rule applies to commercial canned stacks, as well as home-grown porting jobs. Certainly, if you're purchasing a TCP/IP stack, you will want to verify that your vendor has taken the following things into account:

Buffer management. The TCP/IP mbuf buffer management should be able to use pre-allocated buffers rather than allocating them from the global heap at run time (via malloc).

Timers. The timers used in the protocols for connection management, timeouts, and retries should be managed by the RTOS. They should not be a separate implementation that will secretly steal bandwidth from the CPU or cause concurrency problems.

Latency. If an RTOS is present, it should not add any additional latency. Interrupt-handling interfaces should be fast and deterministic. The RTOS should not add any latency to the interrupt processing required with the physical transmission and reception of a frame. The large amount of context switches and CPU processing required in dealing with a packet increases the importance of using an OS with minimal thread context switch time.

Concurrency. All buffering mechanisms should have semaphore protection to allow higher performance potential in real-time systems. The first TCP/IP protocol implementations were on Unix systems and depended on problems. Semaphore protection should be available to the timers to reduce concurrency problems.

Minimized data copying . The TCP/IP implementation should minimize the amount of data copying. The data within each frame can be maintained in the same buffer so it doesn't need to be copied and re-copied by the CPU at each stage of the protocol. The networking chip's DMA places the packets directly in the managed buffer pool where the packet is passed up through the stack by manipulating pointers and not by copying data. Also, some vendors have extended the mbuf mechanism to allow the data to be shared between mbufs and mblocks where there are STREAMS protocols also present in the system.

Link layer multiplexing. Protocol implementation requires a framework with mechanisms for queueing and buffer management. Also, modern protocols require more flexible device driver interfaces and more flexible multiplexing. This is particularly true where serial point-to-point protocols such as PPP are now extended to support IP tunneling and Virtual Private Networks (VPN). The original Berkeley implementation isn't sufficiently flexible to meet all of these needs. The better protocol stack implementations use a framework that allows the stack to be extended as new protocols and interfaces are developed. This can be accomplished by extending the basic Berkeley driver interface scheme, or the protocols can be rewritten to use a different framework.

CPU bandwidth. Each embedded system application has different requirements for its TCP/IP stack. For example, a TCP/IP stack in most Internet appliances probably would not be considered real time. Also, if the network is used for control and management functions, the hard bandwidth requirements will be fairly low. On the other hand, if the application is streaming video or voice, the faster packet rates would qualify the application as a real-time application.

Link layer interfaces and device drivers

As I've described, the OSI model shows an interface between the physical and data link layers. In actual implementation, this interface is implemented in several ways.

BSD 4.3. Most of the examples in this article show the BSD 4.3 type of structures and interfaces. The traditional Berkeley stack could multiplex between multiple interfaces if they were using a common IP stack. Originally, it could only interface cleanly with link layers that were compatible with Ethernet and only among IP and its related protocols, such as ARP and RARP. Subsequently some implementors have hacked the BSD code to allow it to be used with serial interfaces such as SLIP (Serial Line Interface Protocol) or PPP. This was generally a somewhat kludgy way to make them look like Ethernet and as such, was not particularly efficient. Also, each vendor of a TCP/IP stack for embedded systems has put its own nuances in the interface mechanism as well.

The BSD 4.3-compatible stacks uses the ifattach() function and the ifnet structure. This structure and its associated attachment mechanism were inherited by most ports of the Berkeley stack used in embedded systems. Typically, the network device driver initialization function is specific to a particular OS. When the device specific initialization function is called, it first allocates space for its own internal data structures, often called a "softc" structure. This data structure generally includes space for the ifnet structure. It determines its own MAC address by reading it from the hardware. Then it fills in the fields in the ifnet structure. It sets the device-specific information such as the MTU, the MAC address, and the if_name and if_unit. It then fills in the function pointer fields with pointers to the driver's interface functions. Once this structure is appropriately initialized, the initialization code calls the if_attach() function with a pointer to the ifnet structure as an argument.

The ifnet structure is illustrated in Table 1. The ifnet structure contains, among other things, a pointer to a list of address structures for each interface. This address structure, called ifaddr, contains the interface's MAC address and the broadcast address.

BSD 4.4. BSD 4.4 extends the traditional interface in a few significant ways. Inside the stack, BSD 4.3 had hardcoded address lengths in its sockaddr structure. MAC addresses are also stored in the sockaddr and similar structures, and in BSD 4.4, these were extended to support variable length addresses to allow for PHYs other than Ethernet. Also, the device driver interface was generalized to make it less specific to Ethernet. In addition, a pointer to the interface, netif, is passed along by the link layer when incoming data is queued to IP or another protocol in the network layer.

Data link provider interface. The data link provider interface (DLPI) is found in most implementations of STREAMS. The DLPI interface is not specific to TCP/IP and can be generalized for almost any protocol. It does, however, require that the protocols be implemented as STREAMS modules and the link layer be implemented as a STREAMS driver. It also requires 802.2-type framing to properly multiplex between the link layer and the network layer of the protocols bound to the link layer.

The legacy BSD interface between the link layer and the network layer doesn't incorporate any mechanism for interfacing specifically to connection-oriented or connectionless link layers. On the other hand, DLPI provides support for both connectionless or connection-oriented link layer binding. The connection-oriented service would be available if the Service Access Provider (SAP) is capable of supporting connection-oriented transmission. Also, DLPI allows the data link SAP (DLSAP) to identify itself as a promiscuous SAP, that is, one that can grab all the packets on the net, not just those directed to it. See the sidebar called "Promiscuity" for more information.

Fundamental to DLPI is the concept of the SAP. In Figure 1 you can see how the SAP identifiers are incorporated in the LLC framing. The network layer above the interface is a data link service user (DLSU) and the link layer driver is a DLSAP. DLPI uses a set of request primitives passed as messages from the DLSU to the DLSAP. In response, the DLSAP passes a set of acknowledgment primitives back to the DLSU.

Figure 2 shows the relationship between the SAP and the DLSU. DLPI manages the state of the relationship between the DLSU and the DLSAP with a state machine. Table 2 lists some of the common primitives likely to be used for TCP/IP connectionless link layer service and their responses.

Table 2: Primitives used in STREAMS DLPI
Message	Response	Purpose
DL_ATTACH_REQ	DL_BIND_ACK	Used to tell driver to initialize hardware and associate physical layer with a STREAM. Required for Style 2 providers to initialize themselves
DL_INFO_REQ	DL_INFO_REQ	Asks the provider to send information about the interface including MAC type, service mode, QOS (quality of service)
DL_BIND_REQ	DL_BIND_ACK	Requests DLP to bind an SAP to a STREAM. Used to indicate the type of service, connection oriented or connectionless SAP
DL_ENABLEMULTI_REQ	DL_OK_ACK	Requests the provider to enable a muticast address for the specific DLSAP
DL_PROMISCON_REQ	DL_OK_ACK	Asks provider to enable promiscuous mode for this SAP
DL_PROMISCOFF_REQ	DL_OK_ACK	Asks provider to disable promiscuous mode for this SAP
DL_UNITDATA_REQ	DL_UNITDATA_IND	Requests connectionless datagrams be passed to the DLS user from the SAP. The datagrams are passed in a message block preceded by DL_UNITDATA_IND primitive

Extended multiplexing interface. Extended multiplexing interfaces have been developed by some commercial RTOS vendors to extend the BSD 4.4 interface to include link layer multiplexing and multiprotocol capability. This allows some of the capabilities found in the STREAMS DLPI interface to be available to TCP/IP-based networking. A good implementation will do this while still maintaining compatibility with the traditional Berkeley-type TCP/IP stacks and drivers.

Traditionally, various attempts have been made to allow drivers to support simultaneous interfaces to both Berkeley and STREAM-type stacks. This was done with a layer of glue code that would copy the messages from STREAMS mblocks to Berkeley-type mbufs. A better mechanism should allow the actual data in the frame to be shared between the mblock and mbuf header structures.

A good extended multiplexing interface mechanism gives you a DLPI-type binding mechanism for binding a stack to an arbitrary link layer interface. It should do this while providing as much backward compatibility as possible with Berkeley-style implementations. As in DLPI, this advanced mechanism should allow multiple stacks to be bound to the same link level interface. It provides a generic mechanism for address resolution between MAC addresses and protocol addresses. This address resolution capability should still support the BSD ARP protocol previously discussed under the section about BSD 4.3. As with DLPI, the advanced interface has a state machine to keep track of the relationship between the stack and the link layer. It also provides a means of enabling multicast addresses on an interface. This type of mechanism was absent from BSD 4.3 and only partially present in BSD 4.4.

Hosting TCP/IP in your embedded system

Since the BSD stack has been available in source code form for many years, most people implementing TCP/IP for embedded systems have used it as a base. Hardly anyone implements his or her own protocol suite from scratch. You'll have a number of fundamental choices if you want to put TCP/IP in a product for the first time, whether you want to use a commercial of TCP/IP or do your own port.

You may want to look at a number of factors before you decide which path to take for incorporating TCP/IP. You'll want to ask yourself basic questions about the connectivity requirements in your design. For example, how sensitive is your project to unit manufacturing cost? Ask about the future of your design. What is its reuse potential? Is the project to be a platform for launching future projects? Following are a few broad categories of products used in connected embedded systems. Each of these categories suggests a different direction for your implementation of TCP/IP in your embedded design.

1. The embedded product is based on a legacy system. There is only a limited need for remote access. So far, remote access has only been available through a serial port. TCP/IP allows access to this serial port. Potential increase in manufacturing cost is not the determining factor when adding TCP/IP

2. The product should be relatively easy to maintain. A network connection is needed for remote management of the product

3. The networking connection is required for data capture and analysis

4. The product is a router, gateway, switch, broadband modem, or a similar product in which networking is a fundamental element

5. The product is a consumer device with a graphical display, such as a personal digital assistant. The ability to browse the web is a fundamental part

6. The device is associated with a measurement and control system. An easy method must exist for reaching the device from a PC with a web browser. An embedded web server is a fundamental requirement

Below I list some of the choices you have today for incorporating TCP/IP in your product.

Total hardware implementation
A number of companies are developing completely self-contained cards in a small form factor that add network connectivity. The stack is implemented as part of an opaque system and is generally packaged as a PCM card which can be interfaced to your system's serial port. Your system is made to think it is talking to a serial port. This solution may be best for a category 1.

Advantages

• Vendor owns all networking problems
• Easy to specify a working solution
• Very little product redesign required for legacy products
• You can concentrate on your product's added value

Disadvantages

• Not a flexible solution
• Can't be configured to give remote users an elegant interface
• Lack of networking integrated into your product can make the solution seem clunky
• Higher added manufacturing cost

Roll your own stack with no RTOS
Using no RTOS involves snarfing a free (public domain) or source-licensable TCP/IP stack and porting it straight to the hardware without the benefit of a commercial RTOS. At minimum, you will need to implement a basic scheduler, a timer mechanism, and a buffer allocation mechanism. This solution may be appropriate for categories 1 and 2, but only if the requirement domain is well defined and there is little need for reuse.

Advantages

• Low manufacturing cost because of reduced royalties
• High engineering cost
• You own all the sources
• It might be fun to do

Disadvantages You own all the problems

• No RTOS to help allocate CPU bandwidth between the network stack and other parts of your application
• Inflexible and doesn't allow for future product growth
• Can't be easily brought up to date with new networking standards
• Compliance and interoperability testing is an enormous burden
• Upgrading is difficult because of incompatibility with other implementations
• High engineering cost
• Bad for time-to-market-constrained projects

Third-party integration
With this method, you would purchase an RTOS from one vendor and a TCP/IP stack from another.

Advantages

• You can use any RTOS, even one that is fairly low in cost and/or has no royalties

Disadvantages

• It is up to the you to select and integrate your own TCP/IP stack
• You may have to design and implement your own transport and link layer APIs
• Not time-to-market friendly
• You own all the interface problems
• Little or no support available

All in one
With this option, you would purchase a bundled product from a vendor who has a complete solution that includes an RTOS, development tools, and a TCP/IP stack. This solution can be used for all the product categories above. Of course, with category 1 you would have to incur some redesigning to incorporate the new RTOS into your design.

Advantages

• Best for design cycles with critical time-to-market constraints
• Scaleable to meet evolving needs
• Should give you access to added value upper-layer protocols, such as embedded web servers and clients, network management, and other applications

Disadvantages

• May add some royalty cost to your product
• Some redesign of legacy products may be required to add the networking feature
• RTOS may be more than you need

Selection criteria for commercial RTOS and TCP/IP products

The following is a list of items you may want to consider when selecting an RTOS and network stack vendor:

• Choose a vendor who offers a variety of interfaces and available device drivers. It should be able to smoothly support interfaces other than LAN PHYs, such as Ethernet
• The vendor should have stable and tested support for your target CPU and networking interfaces
• If you're building a platform to be used as a base for a variety of products, make sure that your vendor can provide you with an upgrade path as your requirements grow and change
• The TCP/IP should be BSD 4.4-compatible (not 4.3)
• The stack should have the hooks to allow it to be managed remotely via SNMP or RMON
• Zero copy buffers should be an available option for performance. This is accomplished in conjunction with some kind of enhanced buffer management
• Link layer multiplexing should be available to allow the TCP/IP stack to co-exist with other protocols, as well as support multiple interfaces

An established solution

As a technology, TCP/IP is close to 20 years old. It was developed as part of one of the first attempts to establish an internetwork. The protocol suite has outlasted more recent LAN and WAN networking protocols, and by now many of these other protocols have been dropped or forgotten. TCP/IP and the concept of the Internet is elegant in its simplicity. Largely because of its ubiquity and its simplicity, this old TCP/IP-based Internet has become the global fabric uniting much of modern life and is now the basis for much of the hot new "high technology" of today.

Obviously, I shouldn't try to dictate the choices you make as an embedded engineer when you move your legacy design to the Internet-enabled world or as you develop your brand new Internet appliance. As an embedded engineer, you will follow your own dictates given the business and technical constraints particular to your project. As I discussed above, there are many choices for selecting and integrating TCP/IP in your design. Many of the new ASICS designed today have extra gate capacity as well as extra bandwidth in the CPU core, and often memory is not as constrained as in previous designs. After making the right choice for your design, you may find that in today's world most designs have enough headroom to incorporate Internet ability. Even on many small embedded systems, adding TCP/IP probably won't be a large draw on your system for CPU and memory.

Thomas Herbert is a field consulting engineer with Wind River Systems. Before that, he worked in embedded systems in various applications for many years for such companies as Xerox Corp., Eastman Kodak, and as an independent consultant. Tom has contributed several articles to Embedded Systems Programming over the past few years and holds a patent in pattern recognition. He can be reached at tom.herbert@windriver.com

Further Reading

Many excellent books have been written about TCP/IP internals, application programming using TCP/IP, and maintenance and configuration of IP networks. I would like to suggest a few references below which I have found valuable for explaining the internals of TCP/IP that may be useful to the embedded engineers who want to bone up on networking.

The first and most important references for TCP/IP are the following two volumes. If you really need both to see how the protocols work and how they are implemented, these books are must-haves:

• Stevens, W. Richard. TCP/IP Illustrated, Vol. 1, The Protocols. Reading, MA: Addison-Wesley, 1994.
• Wright, Gary R. and W. Richard Stevens, TCP/IP Illustrated, Vol. 2, The Implementation. Reading, MA: Addison-Wesley, 1995.

For an authoritative work on BSD 4.4, I suggest the following book:

• McKusic, Marshall Kirk, Keith Bostic, Michael J. Karels, and John S. Quaterman. The Design and Implementation of the 4.4BSD Operating System. Reading, MA: Addison-Wesley, 1996.

For general information on STREAMS:

Herbert, Thomas F., " Implementing Network Protocols and Drivers with STREAMS," Embedded Systems Programming, April 1997, p. 28.

Or you may want to read the definitive reference on SVR4 STREAMS. Even if you aren't specifically interested in STREAMS, chapter 11 on DLPI (Data Link Provider Interface) is a good look at what functionality needs to be in a good multiplexing data link interface. Unfortunately, at the time of the writing of this article, this book is out of print.