I have defined the header length and flags as two single-byte values for simplicity, but it should be pointed out that the standard defines these as a 4-bit header length, a 6-bit reserved field, and then six code bits.

Window size. The window size indicates the amount of receive buffer space that is available, and is used for flow control. It can be set to a fixed size on transmit, and ignored on receive. We can safely assume that any client contacting our server has sufficient buffer space for our humble pages. (If they don't, it was pretty stupid of them to send the request in the first place!)

Urgent pointer. This can be ignored, since all data can be treated with equal priority.

Options. In addition to a variable-length data field, there is a variable-length header options field. Mercifully, we don't have to generate any options, and can safely discard any incoming options.

Checksum. The only awkward point to note about this header is that it must include a valid checksum value, which is computed across the whole TCP segment, plus a pseudo-header containing parts of the IP header. See Figure 5.

The usual checksum computation technique is to scan the TCP segment image in memory, but we don't have sufficient RAM to do this. If the checksum came at the end of the data, it would be easy to compute it on-the-fly as the header plus data was being sent out, then append the resulting value to the data. TCP checksum generation is a major issue in our small implementation, particularly when attempting to include dynamic data on the web pages.

Long segments
If we are to make any headway creating a web server, we'll have to handle TCP segments that are larger than the available RAM. In the case of transmitted segments, the bulk of data will reside in external ROM, and so will be copied directly from there to the RS-232 output. We'll also be receiving long HTTP requests, where the only items of interest are in the first few tens of bytes.

Transmit. The IP and TCP headers must be created in RAM, so that their checksums can be computed. For normal (short) segments, these images are then SLIP encoded (by inserting escape sequences) while they are being sent down the serial line. If the segment is long (that is, includes ROM data), then a flag is set such that the ROM-to-SLIP transmission takes over when the RAM-to-SLIP transmission stops. This depends heavily on the TCP checksum being known in advance, that is, pre-computed for the ROM image, and added in when the TCP header and pseudo-header checksum is being calculated.

Receive. We're only interested in the start of an HTTP request, and can happily discard the rest. TCP doesn't possess any mechanism for discarding data; if we don't acknowledge it, it will simply be resent until we do! There are two possible solutions: we could reduce the TCP window size so that the request is sent in two or more chunks, and discard all but the first chunk. A simpler method is to only store the start of each segment data in RAM and discard the rest, and this is what we'll do. It is tempting to ignore the checksum on the incoming TCP segment, and assume it is correct, but this is rightly frowned on in the TCP community. Instead, we could compute a checksum for the discarded portion of the segment, and add it on after the complete segment is received. The approach I have adopted is a minor variation of this, whereby the checksum of all the incoming TCP data is computed separately, irrespective of how much is stored in RAM or discarded. This is added to the value computed from the TCP header and pseudo-header, which are always stored in RAM.

IP

To convey the TCP segments between hosts, the Internet Protocol (IP) is used. After the difficulties of TCP, IP is relatively easy to implement.

Datagram format
An IP header plus data block is known as a datagram. See Figure 6.

Version and header length. We're using IPv4, and the default header size (measured in 32-bit words) is 5, so we'll be assuming a value of 0x45 for both transmit and receive.

Service. This field is used to prioritize datagrams, and is set to zero, which is normal precedence.

Length. The total datagram length in bytes, including the IP header.

Ident. A value that is incremented for each datagram sent.

Fragmentation. IP allows a large datagram to be split into two or more smaller datagrams, using a process called fragmentation. Considering the acute lack of RAM on our microcontroller, it is impossible to support fragmentation. This is unlikely to be a problem in practice, since it carries a very significant performance penalty, so is generally avoided wherever possible.

Time to live. An expiry time for the datagram, to prevent it from endlessly circulating the Internet. A constant value generated on transmit, and ignored on receive.

Protocol. Indicator of which protocol is used in the data area of the datagram. We'll only be using the following values: ICMP and TCP.

Checksum. A simple checksum of the IP header only.

Source and destination addresses. These are IP addresses, expressed as 32-bit values. An important question is what IP address to assign to our system, and how to program it with that address. This issue can be side-stepped by making the assumption that as we're using a point-to-point serial link, there can only be one intended recipient for all the network traffic, namely our web server. Hence we can disregard the destination IP address value, but must be careful to use this value in the source address field of our outgoing datagrams.

Options. Header options are occasionally used to give tighter control over datagram routing; for simplicity, we won't be accepting or transmitting any options.

Long datagrams
To accommodate long TCP segments, we have to accept IP datagrams that are longer than the available RAM. Unlike TCP, there are no checksum problems, since the IP checksum does not include the data area, so we can discard excessive input data or add extra output data, without any checksum problems.

ICMP
Internet Control Message Protocol (ICMP) is very useful for performing network diagnostics. An ICMP message is contained within the data field of an IP datagram.

Ping
The most commonly used facility is the ICMP Echo Request, or "ping." We don't have to implement this on our web server, but it will be very useful to check the lower protocol layers prior to implementing the web server itself. The echo request is type 8 code 0, and the reply is type 0 code 0. The checksum covers the complete ICMP header and data area. The ident and sequence numbers, and all the data, are echoed back to the sender as a check of network integrity.

Buffer Size
The default data size for a Unix ping is 64 bytes, which is too large for our available buffer RAM. Fortunately, the ping utility has an argument to specify the data size, so it can be reduced to, say, 32 bytes, which is the default size for DOS systems. This requires a buffer size of 60 bytes (including 20-byte IP header and 8-byte ICMP header), which is more realistic.

SLIP
This is a simple method of converting a stream of serial data characters into a defined block, which we're calling a frame. It's easy to implement: a delimiter-character is put at the end of each frame (and also, by convention, at the start). If the delimiter character is encountered in the data stream, a two-character escape sequence is substituted.

#define SLIP_END	0xc0
/* SLIP escape codes	*/
#define SLIP_ESC	0xdb
#define ESC_END	0xdc
#define ESC_ESC	0xdd

/* Start a transmission */
void tx_start(void)
{
    putchar(SLIP_END);
}

/* Encode and transmit a single SLIP byte */
void tx_byte(BYTE b)
{
    if (b==SLIP_END || b==SLIP_ESC)
    {
        putchar(SLIP_ESC);
        putchar(b==SLIP_END ? ESC_END
            : ESC_ESC);
    }
    else
        putchar(b);
}

Modem driver
The implicit assumption in most PC communications software is that serial networking should be configured for access via a modem and telephone line. We would like to be able to link a PC directly to our PIC server's serial port, so the server will have to impersonate a modem to keep the PC happy.

Fortunately, this only involves accepting modem command strings, which are prefixed by "AT" and delimited by a Carriage Return character, and returning an "OK" string. This is usually sufficient, though it is wise to assert the Data Carrier Detect (DCD) hardware handshake line to the PC as well, in case its software uses this to check that the (emulated) phone link is still functioning correctly.

Table 1: A typical PC-to-modem interaction
PC	Modem
AT<CR>	OK<CR><LF>	Check that modem is responding.
ATE0V1<CR>	OK<CR><LF>	Disable command echo, enable text-message responses.
AT<CR>	OK<CR><LF>	Check that modem is responding.
ATDT12345<CR>	OK<CR><LF>	Tone-dial telephone number 12345.

A typical PC-to-modem interaction is shown in Table 1. Some modem scripts look for a CONNECT message after dialling, but this does not appear necessary when using the standard Windows modem types. Disconnection follows a similar pattern, as shown in Table 2.

Table 2: Disconnection
PC	Modem
ATH<CR>	OK<CR><LF>	Disconnect from line.
ATZ<CR>	OK<CR><LF>	Reset modem.

Software techniques
Having discussed the limitations of microcontrollers, we need to work out how we can squeeze the complete TCP/IP stack into one of them.

RAM limitation
The most acute problem, by far, is the lack of RAM. The usual assumption is that the incoming and outgoing frames are stored in RAM, and structures are overlaid onto this RAM, so that specific values can be checked, read, or modified, for example:

/**** IP packet ('datagram') ****/
typedef struct
{
    IPHDR    i;            /* IP header */
    BYTE        data[MAXIP];     /* Data
                        area */
}    IPKT;<

If we were to use this technique, the only way of cramming these structures into 368 bytes of RAM is to severely restrict the maximum frame size we can send or receive. At best, we might be able to accommodate a 128-byte frame, which is unacceptable. To permit the use of full-size frames, they must be decoded on the fly as they are received, and created on the fly as they are transmitted.

To achieve a good response time, the incoming frame must be decoded on the fly as it is received, and the outgoing frame must be prepared on the fly as it is being transmitted.

Creating protocols on-the-fly
Consider the code fragment shown in 1. It isn't too hard to imagine the same data being created on the fly as it is being transmitted, using code such as that in Listing 2, where the "put" functions send the values via a SLIP driver to the serial port. The abolition of the structure reduces RAM consumption significantly, and also reduces the code size slightly; the complicated indexed-addressing assignment operation is replaced by a single-word function call. Unfortunately, the new code is harder to debug, since there isn't a convenient memory image to browse when we want to check the last frame sent. To compensate, we may have to employ the services of a protocol analyzer to monitor the external communications when debugging.

Listing 1: Packet assembly using structures

/* ***** IP (Internet Protocol) header ***** */
typedef struct
{

	BYTE	vhl,	/* Version and header len */
		service;	/* Quality of IP service */
	WORD	en,	/* Total len of IP datagram */
		ident,	/* Identification value */
		frags;	/* Flags & fragment offset */
	BYTE	ttl,	/* Time to live */
		pcol;	/* Protocol used in data area */
	WORD	check;	/* Header checksum */
	LWORD	sip,	/* IP source addr */
		dip;	/* IP dest addr */

} IPHDR;
...
IPKT *ip;
...

ip->i.vhl = 0x40+(sizeof(IPHDR)>>2);		/* Version 4, header len 5 LWORDs */
ip->i.service = 0;		/* Routine message */
ip->i.len = len + sizeof(IPHDR)		/* Data length */

...

Listing 2: On-the-fly packet assembly

put_byte(0x40+(sizeof(IPHDR)>>2);		/* Version 4, header len 5 LWORDs */
put_byte(0);		/* Routine message */
put_word(len + sizeof(IPHDR));		/* Data length */

Checksums
Another problem occurs with the calculation of protocol checksums. Usually a memory image is scanned to compute these, but we don't have a memory image to scan, so we'll have to get the "put" functions to do the job, for example:

/* Send a byte out to the SLIP link, /*then add to checksum */ void put_byte(BYTE b) {     putchar(b);     check_byte(b); }

It would have been really helpful if the checksum was the last word transmitted, because then we could just send out the calculated value as part of the end-of-frame sequence, as shown in Listing 3.

Listing 3: Checksum after data (wishful thinking)

put_byte(0x40+(sizeof(IPHDR)>>2);		/* Version 4, header len 5 LWORDs */
put_byte(0);		/* Routine message */
put_word(len + sizeof(IPHDR))		/* Data length */
...
put_word(sum);		Checksum value */

Unfortunately, the TCP checksum is in the header, where the data to be checked hasn't been transmitted yet. If the data is being fetched from ROM, then its checksum can be pre-computed and stored in the file directory, and just added on to the TCP header fields to produce the final value.

Reception
Just as the transmit structures can be replaced by a string of function calls, so can those used for receive. Rather than storing the complete frame and then using structure references to pick out the elements of interest, the frame is scanned on input, with only the important information being stored (for example, the code in Listing 4 scans the start of the IP header).

Listing 4: On-the-fly packet decoding

if	(match_byte(0x45) && skip_byte()		// Version, service
	get_word(iplen) && skip_word() &&		// Len, ID
	skip_word() && skip_byte() &&		// Frags, TTL
	...

I'm using three basic types of input functions:

• match: ensures that the specified value is present
• skip: checks that the byte(s) are present, then discards them
• get: checks that the byte(s) are present, and saves them for re-use later

These functions are surprisingly versatile, in that they allow us to indicate which values are unimportant, those that must be checked but need not be retained, and those that must be saved for later use. If these values are separated out when the frame is received, we can free up a significant amount of storage space.

All these functions return a Boolean true/false value, so the surrounding if statement will only return "true" if all the required data has been obtained and matched correctly. The obvious disadvantage with this method is that the processor will tend to lock up if communications fail in mid-message; it will wait forever for a byte that never comes. This can be avoided by inclusion of a timeout in the function that fetches the bytes from the serial link, which makes all subsequent input calls fail (return "false") until communications is restored.

Source code

In this article, I have reviewed the key areas associated with creating a microcontroller-based web server implementation. The implementation has sufficient resources to support the creation of useful web pages, including dynamic data. Full source code for this project (and various PC-based network utilities) is included in my book TCP/IP Lean: Web servers for Embedded Systems (CMP Books, 2000).

Jeremy Bentham cofounded the industrial networking company Io Ltd., as well as the software consulting offshoot IoSoft Ltd., where he develops embedded TCP/IP solutions. He was software manager at Arcom Control Systems, a UK manufacturer of boards and systems for industrial applications. Contact him at jb@iosoft.co.uk.