Wes Howl

A record-and-playback test methodology lets you exercise your system with realistic test data. But making a system playback-ready offers some challenges of its own.

Stimulating other systems with a playback-equipped system. All on-board mission-critical subsystems had to pass a cursory pre-flight test to demonstrate flight-test readiness. Failure to do so could cause the flight to be scrubbed or delayed. Although scanning for nearby FAA radar emitters was an easy way to show that our system was working, use of playback files from previous flight tests provided more realistic outputs to other on-board systems. In this mode, we were able to see that other systems were processing our data correctly. The test officer knew that we were using playback files to do this and approved because he knew that it was "real" data from a recent flight.

Range test. The range test was the moment of truth, not only for our application software but also for our playback technology. Obviously, it was a critical moment for our application software to show its stuff. But, less obvious, it was a key moment for our data recording software. On the range, we could collect the data that we could neither simulate nor succeed without: actual radar pulses diffracting over mountain ridges and reflecting from mountain slopes and spinning propeller blades.

The entire flight test was made up of one or more "runs," each of which consisted of a series of points, called "way-points." A test involved flying, at a specified speed and altitude, from one way-point to the next while our system looked for the test emitters that were being switched on and off, according to a test procedure, by an independent test team on the ground. Record mode was enabled only during a run since that was the only time we had a known standard (that is, the test procedure) against which to compare our performance and because conserving disk space was a high priority. The recording of the sensor data did not depend on our application software generating correct answers. Even if we did less well than we hoped during the test, we would still bring back the recorded pulse and navigation data and that would allow us to succeed on the next flight.

Performance verification. Test range playback files served a dual purpose with respect to customer verification. First, we could literally re-fly the flight test in the lab and show, via a target display, that the system was performing correctly. Second, the customer would input our playback files to their own analysis software for further verification as to the accuracy of our results.

Replay in the lab. After the flight, the target system was physically removed from the aircraft and set up to run in a lab environment. At this point, we had access to the test procedure and so knew precisely when the emitters were turned on and off, their geographic location, and the frequency and modes at which they were operating. The target system would then be run in replay mode against the very same emitter data we had flown against hours before. This gave us the opportunity to debug our system until its playback performance was correct. This ability was key to our succeeding in our flight tests.

As mentioned earlier, we started flight testing with neither a complete understanding of the physics of the range nor the ability to generate simulated target data. However, saying that something can't be simulated is not to say that it can't be anticipated. Our analysts had designed algorithms to deal with multi-path as they understood it. The algorithms were heavily parameterized and these parameters were adjusted until the correct values were determined. Also, new algorithms were developed as playback sessions deepened our understanding of the emitter environment.

Flight tests, like most field trials, are heavily scheduled and the culture is "fix and fly" until the money runs out or the aircraft or ship or other test platform leaves for its next scheduled duty station. After a field trial you will typically have only a few days to resolve problems and be expected to go again. It is imperative that you make the most of the time between trials. Playback technology greatly improves the efficiency with which you use this time.

Our system as an environmental probe. Toward the end of our flight-test schedule, the customer used our EW system as an environmental probe. This was done in order to improve our understanding of how the multi-path signal environment was perceived by our system.

A probe version of an embedded system is simply a version of the system capable of recording the environmental data required for playback but with little or no application processing functionality. Consider implementing a probe version of your system in parallel with the design and implementation of the application-processing portion of the software. The probe version must be able to exercise all hardware devices related to environmental sensing plus record all environmental data required for playback. This could be useful in any of the following circumstances:

• You have access to a realistic target environmental simulator
• You have access to the target environment
• Completion of application processing software is behind schedule

If your target environment is outer space then, for the purposes of playback, an environmental simulator is as close as you will ever get to the target environment. Access to such simulators can be very expensive, inconvenient, and heavily scheduled. Running a probe version of your system in a simulated environment would provide the processing software development team with valuable inputs without the trouble of actually being connected to the environmental simulator.

If you have access to the target environment then, as with the environmental simulator, running a probe version of your system in that environment will yield useful real-world data to your development team. If the target platform for your system is an aircraft or naval vessel or other expensive platform, you'll find that they are scheduled very tightly to maximize utilization. However, a little negotiation with the test director may allow you to operate your probe system in a passive mode to collect valuable test range data. This data can be taken back to the lab and given to the developers to test the application processing software.

Evaluating your system's need for playback
Having used playback on three separate and very different EW systems, I can't see why the builders of virtually any embedded system wouldn't want to use it. However, making a system playback-ready does not come without some effort. You should perform a cost/benefit analysis before continuing down this path. This section presents the criteria for evaluating whether or not your embedded system could profit from playback technology. An affirmative answer to any of the questions below indicates you could benefit from equipping your system with playback.

Are you unable to produce or obtain simulated data that very accurately represents the entire scope of environmental characteristics of interest to your system? This is the fundamental question you need to answer. If the answer is "no" or "we're not sure" then you will benefit from using playback. A negative answer implies that you must submit your system to a field trial to completely verify its performance. Playback is an excellent way to verify system performance prior to going into an expensive field trial.

We knew that we could not model all the propagation paths between a radar emitter and a C-130 flying below the ridgeline on the opposite side of a mountain. It simply wasn't possible, given our schedule and budget. Our analysts said it wasn't possible at any cost because we simply didn't know enough about how our receiver truly perceived the radar signals in that environment.

Is the target environment in a remote location? Perhaps your system is going to operate on the North Pole. How often do you want your system or yourself to go there? With playback, you go can go there once, record the necessary environmental data and return to the warm lab to test until things are working. Our flight test was over a military test range in California. The ground and air space there is controlled by the military and access to it is by invitation only.

Is the target environment expensive to access? A location needn't be remote to be expensive to access. A naval sea trial may take place right off the coast of your city, but it will still cost plenty and be very inconvenient to transport your engineering staff and equipment on-board and maintain them there for the duration. And if you forget something or something breaks after you put to sea the Navy will have to fly whatever or whomever to and from an aircraft carrier deck. For our flight test, we had to put our system and ourselves aboard the aircraft. An empty C-130 has an operating cost of about $2,000 per hour. Add in the salaries of the engineers, and the sumptuous in-flight meal, and you're talking serious money. The test range itself costs about $1,500 per hour and we usually spent from two to three hours on range. So an average flight costs about $15,000. Playback can save you thousands of dollars in just this area alone.

Is your time in the target environment extremely limited? If the target environment of your system is outer space or an ocean trench, it's likely that acceptance testing on your system will be done in a completely simulated environment. The mechanism by which this simulation will be accomplished will be fantastically expensive and heavily scheduled. The more expensive the test platform and the more systems that must be integrated together, the less time any one company will have on that platform. You must maximize every minute you have in the test environment; playback is the best way of doing that.

Is the target environment a hostile place for humans and/or equipment? The interior of a C-130, flying low and slow over desert terrain, is no place for good engineering work to be done. The temperature of the cargo/engineering area was certainly in the 90s and the decibel level equaled or exceeded the temperature. The heat and noise, G-forces, and constant maneuvering, climbing, and diving gave nearly all but the flight crew moderate to severe motion sickness.

Playback data requirements
In any system design effort, knowing the requirements beforehand is a great benefit. Playback is no different. The following sections address the source of most playback requirements.

Identify what portion of the software you intend to test with playback. Most embedded systems with multiple sensor inputs have at least the following three processing layers or phases: acquisition of raw sensor data from hardware interfaces; pre-processing of sensor data; and core processing of pre-processed sensor data. ( Figure 4 depicts these layers, using our EW system as an example.)

Phase one is usually comprised of software that interfaces directly to the hardware. This software configures the sensor devices to acquire data, allocates buffers for the received data, copies the data into the buffers, and queues the raw data buffers onto the pre-processing programs.

Pre-processing transforms raw data into a form where the core algorithms of the system can process it. Examples of pre-processing are as follows:

• Application of calibration correction values
• Discarding of invalid data
• Associating data from one sensor with data from another
• Separating raw data into sets meaningful to the core algorithms
• Searching for data requested or required by core algorithms

The core processing software performs those functions or computations on the pre-processed data that meet your system's primary functional requirements. The specific software layer or layers to be tested with playback will determine from which data extraction (DX) point to obtain the sensor data for playback recording (see Figure 4) .

Playback is not well suited for testing the outer layers of software that handle hardware devices. The nature of the processing doesn't meet any of the evaluation criteria for an embedded system's need for playback. The recorded data itself will add nothing to using simulated data. The issues at this layer have to do with the software/hardware interface, such as understanding how the device actually operates, meeting response time limits in reacting to hardware interrupts or other notifications, conditioning sensors to acquire the required data, and hardware error detection.

Realistically, there will probably be only one or two target candidates for playback testing and they may be in the pre-processing or, almost certainly, in the core processing layer. The candidates for playback testing are, by definition, those that would most profit from the use of real-world inputs to be fully tested and produce the most important outputs. These functions are at the heart of your system and represent the central reason that the system was created. You must record the data these functions require.

There was only one portion of our EW system software that was a candidate for playback: the emitter signal processing software, the core processing layer. Our software design had all the data required by the signal processing software placed in a single data structure we called a "dwell buffer" (a dwell is defined as a particular configuration of the radar receiver hardware). Obviously, recording the dwell buffers was mandatory. The decision to target the core processing layer meant that we could use DX point 6 (see Figure 4) where the dwell buffer exited the core processing layer, and log every dwell buffer that passed that DX point. This simplified the job of recording since all the data of interest was in the dwell buffer.

Regardless from which DX point you decide to obtain the target environmental data, you should still log events of interest that occur in any layer. Just because you don't intend to present playback data to a particular software layer in no way rules out the idea of logging events of interest that occur anywhere in the system. These events can still be very useful to off-line analysis of system behavior.

Identify the off-line analyses. What off-line analyses will be performed on the recorded data files? Suggested off-line analyses topics are as follows:

• System input validity
• Input data rates
• Error rates
• System output validity

However, you can analyze the data for any characteristics that are of interest to you. You may want to scan the recorded data files for errors in data supplied by another system. This could be especially useful during system integration.

An off-line program to evaluate the timing of significant events sometimes produces surprising results and may pinpoint areas where your system is not meeting its hard real-time requirements. Also, examining the type of errors that are logged may point to a faulty hardware device. Other curious anomalies such as excessive time to complete certain operations may be revealed with this technique. The things you can search for off-line are limited only by your budget and the data present in the playback files. But, if you didn't record it, you can't analyze it.

Some errors are not visible during normal operations. For example, our system was required to locate an emitter to within a specified percent error. Except for large miscalculations, an observer of our emitter location display couldn't say with confidence that an emitter was within the specified limit or not. Answers such as these were available only by running an off-line analysis program against the recorded data.

Playback design topics
You need a design that allows the embedded system to record and replay the data and the off-line software to analyze it. There are three primary aspects to designing playback into an embedded system:

• Playback file structure and record formats
• Playback file recording
• Presentation of playback data to embedded software

Whether you're designing from scratch or retrofitting an existing system, the issues are the same. The following sections offer some guidance in these design areas.

Playback data design
A database design is needed that captures every aspect of system operation required by both the embedded system and the off-line software.

Data record identification and versioning. A playback file for a system of even moderate complexity will have tens or possibly hundreds of record types stored within it. Obviously, each record class will be assigned a unique type code. Something not quite so obvious and much easier to ignore is the fact that, over time, the format of some of these records may require change. Therefore, each record must also have a version number. This allows software reading the record to adapt to multiple versions of the same class of record. Failure to provide this mechanism will result in records in playback files being misinterpreted, perhaps in terribly subtle ways, and lots of wasted engineering time and data.

Capture the hidden context. Translate contextual information into explicit message contents. For example, the fact that a particular message is sent over a certain inter-process queue implies the identities of the two processes on either end of the queue. If the identities of the sending and receiving processes are important in interpreting the logged message then you must design in the ability to record a process' identity. The method by which you encode contextual data must allow meaningful interpretation at a time or by persons far removed from the development phase of the project. In other words, recording a random number the RTOS assigned as the process ID is not sufficient. You must log data, the use of which permits translation of that process ID into something meaningful, like a process name.

Playback file space budget. Estimate the disk space requirements of the playback files based on the size of individual data records, frequency of recording various record types, and duration of record mode. Don't skip this step. Playback file sizes can easily grow to hundreds of megabytes and beyond. You must have a fairly accurate estimate of the upper bound of these file sizes early on in order to provide sufficient recording capacity. Failure to do so may result in a system that fails to record sufficient data and can invalidate all your playback efforts.

Time synchronization data. In many cases, data collected from multiple sensors can only be processed correctly if the exact time of each sensor collection is known. For example, in our EW system we obtained aircraft navigation data and radar pulse data from two different sensors. The pulse data couldn't be processed accurately unless the precise times of both the pulse and navigation measurement were known. This requirement would be present with or without playback. However, if playback is to succeed on your system then you must make sure that the data that is the basis for time synchronization is recorded.

The typical solution to this problem is to adopt a universal measure of time (like Coordinated Universal Time) as the "system time" and time stamp every message with the system time at the instant the event occurred. Note, that the time stamp indicates when the event occurred, not when the message was sent, received, or queued.

Playback file header data. A playback file may contain thousands of recorded events but if you can't remember the larger context of the field trial it may prove difficult to get maximum use of the file. Also, on a project of long duration you may have tens, if not hundreds, of playback files and you won't remember what the file "big_test.dat" contains or why it was such a big deal. Each playback file should contain information such as the following:

• Test session identification (for example, flight 10 test sequence 3)
• Date and time of test start and end
• Location of test
• Purpose of test
• Outcome of test
• Miscellaneous remarks

This sort of information may have to be added after the field trial but some mechanism for recording this data should be designed in to your file structure.

Record calibration data. Identify hardware calibration values that are input to the software and make sure that these values are output to the playback file. Failure to do so will virtually guarantee that you will never achieve the same results from a replay session as compared to normal operation.

Recording playback files
The design for recording playback files must address the following issues:

• Identify DX points within the embedded software
• Specify a detailed message format for each message type
• Design a mechanism for transferring messages from the DX points to the recording software
• Design a way to enable and disable the recording of all or selected message types
• Design a mechanism for recording the logged messages on permanent storage (disk, CD)

The set of DX points are determined by the requirements developed in the preceding "Playback Data Requirements" section. As mentioned before, the DX points determine the nature of the data to be recorded and are a function of what sections of your embedded software are to be tested and what off-line analysis will be done. Each DX point will have one or more message types available to it. The format of each message must be specified in detail, in accordance with the principles set down in the "Playback Data Design" section. Estimate the number of messages of each type recorded per second for each DX point. Use these figures, along with duration of record mode for an estimate of your recording bandwidth and storage requirements.

Obviously, messages from the DX points must somehow be sent to a central point for actual logging to permanent storage. Your choices depend on the sophistication of your RTOS and system architecture. In most cases, simply creating an inter-process queue will suffice. More complex systems may elect to use sockets as a transport mechanism.

You will want a way to enable or disable the recording of messages from the various DX points. At a minimum, you will want the scope of this function to include all DX points and all messages. Without this control your system will always be recording when operating. This almost certainly is not desirable. Despite the quantum leaps in disk storage capacity of recent years, it is not prudent to waste resources. And, regardless of disk capacities, you'll have to read everything you record to get any use of it.

Finally, a means by which the extracted data is actually recorded on a permanent storage device must be found. The recording channel must be able to handle the sustained peak recording load and have sufficient capacity to hold all the data from field trials of the longest estimated duration. This is critical since failure of your recording channel or media to meet these requirements will undermine your efforts by being unable to generate a complete and reliable playback file.

Use a debug channel for playback. Your target platform may never be deployed with a network hardware device, but if you use one for debugging then you're in luck. It is hard to imagine a solution simpler than using TCP/IP sockets over Ethernet or other compatible media. If you're using a TCP/IP-based debugger then your target RTOS and development station almost certainly support socket programming (the API to TCP/IP). You'll need to host the playback file on the development station and modify your embedded code to log the file via sockets or perhaps the network file server (NFS) if that seems a better choice.

Reuse an existing data channel for playback. Okay, so you don't have a network interface card and can't use TCP/IP. When the going gets tough, the tough get expedient. Cast around for a data channel that you could either share or otherwise utilize, perhaps a medium to high-speed bus or channel only used for backup or other rare events or modes. We once used a MIL-STD 1553 bus to log DX data. It was slow (only 1 Mbit/s) but it was good enough. We equipped a PC with a MIL-STD 1553 PCMCIA card and were able to use the channel for both playback and record modes.

Flush often. Imagine this: you've just executed your field trial, collected 20 minutes worth of very valuable target environment data and your system crashes because of a power glitch on the aircraft (or whatever platform you're testing on). You reboot and, to your horror, the precious playback file has zero blocks allocated. Why? The file was neither closed nor flushed so the file allocation table wasn't updated. Suggestion: periodically issue a flush to the file during record mode. The period should be a little shorter than the maximum amount of data you can afford to lose.

Substituting playback data for sensor inputs
The sensor data from a playback file must be substituted for sensor device inputs and presented to the embedded software, while running in the target system. This is a fundamental requirement for a playback-equipped system. It can also be one of the more interesting playback design challenges. If you're lucky, you can solve this problem in an elegant manner, but be prepared to do otherwise.

First, you need an input channel. The suggestions in the preceding "Recording Playback Files" section on reusing debug or other data channels apply equally to playback channels. Second, a way must be found to substitute recorded data for normal sensor inputs with minimum disruption to your existing software architecture.

Our EW system had been tested via simulated data files and so we already had a tested way of bypassing the sensor inputs and substituting the simulated data. Our embedded code ran under VxWorks while our simulation software ran under Unix. The two CPU boards were co-located in the same chassis. We had a TCP/IP channel implemented via shared memory and so socket communication was the answer. Our simulator sent its data to the embedded software via socket connections.

We leveraged the existing simulation channel for replay in the following manner. Figure 5 shows a block diagram of our EW system in normal operation. The central point of the pre-processing software was to join the radar emitter pulse data with the navigation data sampled during the dwell and put it all in one dwell buffer. Note that although the navigation data is continuously coming in at a 64Hz rate, it is only stored for processing during a dwell.

Figure 6 shows our system configured for replay. In this configuration the live 64Hz navigation data was replaced by the recorded navigation data from the playback file. The recorded navigation data was written to the circular buffer as if it had been received from the 1553 bus. The recorded radar pulse data was substituted for the real thing by writing it to an auxiliary input channel on the receiver hardware. The pulses then appeared in the FIFO and generated an interrupt just as if they had been received from the A/D converter. The great benefit of this technique was that only one process was modified: the Receiver Control process. None of the other processes required modification because all the interfaces remained the same.

The time investment we made to equip our EW system with playback was repaid many times over. Playback allowed us to both meet our flight test goals and to collect and record valuable target environment data. We could not have succeeded without it. I hope your experience with playback is as good.