High Level Architecture (HLA) has emerged as a widely adopted middleware standard for simulator-to-simulator connectivity and data sharing, especially in the defense system market. <>

But until recently there was no viable open standard middleware solution able to address effectively the even more demanding requirements of real-time data distribution within the individual simulator where it is necessary to produce and control animated images, sound reproduction, and device feedback in a manner as realistic and responsive as the real world, and chasing this ideal has constantly pushed the industry forward in many different ways.

Individual simulators have adopted techniques such as multi-processor systems, high performance graphics cards and distributed sensors and actuators to approach the desired objective.

Increasingly such systems consist of many different high performance processing sub-system units that need to communicate in real-time. Although COTS-based open-standard hardware such as VME and Unix/PC systems has been commonly used, until recently the only way to provide software connectivity between applications has been with proprietary systems solutions.

And as such systems get larger and more distributed, the performance issues of latency, determinism and system bottlenecks are becoming ever more important in maintaining the simulation experience.

A further issue is the critical need to ensure investment in application software can be re-used effectively across multiple projects. The combination of these vital but difficult issues is driving the need for more formalized software structures and a growing move towards COTS middleware adoption.

These issues have to be tackled at two levels: both inside (between the system components of the simulator) and outside (for distributed simulator to simulator connectivity) such systems.

This article will examine the problem in greater detail, and introduce an integrated design approach based on open standards middleware that addresses these issues at both levels.

Optimizing "Man-in-the-loop"
One of the key issues in achieving the best performance in a simulator design is optimization of the 'man-in-the-loop' function. This control loop has to respond in real time to the changing environment and operator input, possibly within a network of simulators.

The demands on the network throughput will vary, from a dynamics model running possibly as fast as 1000Hz exchanging information with an I/O device at up to 100Hz, with a resultant potential network throughput requirement of 10ms per data packet; or from the dynamics model to an image display device that needs data input at anything up to 60-80 Hz to match the screen refresh.

If data appears outside of this time interval it will have to be ignored; a common response is to reduce the simulation fidelity to allow state recovery, a situation that simulator developers always seek to avoid.

As mentioned previously, the issue of simulator-to-simulator data sharing has already been addressed by the development of an open standard middleware for linking simulators promoted by the US Department of Defense and implemented by the Simulation Interoperability Standards Organization (SISO).

The emergence of HLA
This is the High Level Architecture (HLA) and its application programming interface, which is already used by a number of simulator developers to connect systems together.

HLA is a publish/subscribe architecture where elements publish data onto the bus to be picked up by other units that subscribe to that data, commonly referred to as 'federated data'. This allows a system to be distributed, avoiding the bottlenecks of a client-server architecture and allowing the system to be more easily scalable.

The HLA Run Time Infrastructure (HLA-RTI) is most commonly used to link simulator systems together in a wide area network, and in some cases it is even being seen as a way to link the processing elements within individual simulators.

This approach has the advantage of avoiding having to translate the data from one format to another. Unfortunately, the HLA-RTI wasn't designed to provide the speed and detailed control of real-time performance required by systems that need consistently low latencies and deterministic responses.

As a result, simulator developers often end up creating proprietary protocols and methodologies to enable data distribution with adequate real-time performance within the simulator.

However, there is another open standard that also uses a publish/subscribe architecture that provides a much better fit for our real-time requirements. Data Distribution Service (DDS) is an open standard data oriented middleware optimized for hard real time systems with the low latency and quality of service capabilities to provide the required speed and level of control of real-time performance built in. Its similar structure allows it to work very well alongside, and in co-operation with, the existing HLA standard.

What is DDS?
The Data Distribution Service (DDS) is a newly adopted open specification from the Object Management Group (OMG), a group of around 800 members, for data-centric publish/subscribe communications in real time systems. These include applications in aerospace and defense, distributed simulation, industrial automation, distributed control, robotics, telecom, and networked consumer electronics.

The publish-subscribe (P-S) model connects anonymous information producers (publishers) with information consumers (subscribers). The overall distributed application is composed of processes, each running in a separate address space and even on different computers. The API and Quality of Service (QoS) are chosen to balance predictable real-time behavior and implementation efficiency/performance.

The specification provides a platform independent model (PIM) that can then be mapped into a variety of platform specific models (PSMs) and programming languages.

DDS draws upon common practice in existing publish/subscribe architectures including HLA, OMG event notification service, Java Messaging Service (JMS), and experience with Real-Time Innovations NDDS product. Many enhancements have been specifically designed to provide the higher latency and determinism required by real time distributed systems.

Like HLA, DDS uses a publish/subscribe model where data dissemination between producers and consumers may be from one-to-one, one-to-many, many-to-one, or many-to-many. The communication model is decentralized, with publishers and subscribers loosely coupled and having no knowledge of each other.

This means that publishers and subscribers can join and leave dynamically, providing an ideal platform for a flexible and scalable simulation system architecture. This data-centric development approach, enabled through a standard API, also enables modularization of simulator development and thus the potential for significant application code re-use.

DDS builds on the definition of HLA to add support for object modelling, and ownership management. It addresses a number of performance-related issues not dealt with by HLA, such as a rich set Quality of Service (QoS) policies, a strongly typed data model, and support for state propagation including coherent and ordered data distribution.

In particular, the QoS capability of DDS allows the designers to maintain levels of priority for data ensuring a fine level of control which ensures the minimum latency requirements are met across the distributed system.

An enhanced version of DDS, called NDDS 4.0, is designed to maximize benefits of the deterministic data environment provided by the standard (see Figure 1, below). The latest version provides pluggable transports that support any media, from wireless to switched fabrics with programmable parameters, programmable Quality of Service (QoS) and customizable data types.

Figure 1: The NDDS 4.0 architecture

Pluggable transports have been incorporated into RTI's implementation of the DDS standard. When combined with the QoS mechanisms of DDS this capability provides performance "tuneability" of data paths to the underlying simulator connectivity fabric. This also means that any transport medium can be used, from the widely used TCP/IP protocols to a switched fabric for more high performance applications.

NDDS 4.0 also uses direct end-to-end messaging, which eliminates context-switch delays. Combined with the support for message prioritization, this ensures a predictable and low level of data latency, while pre-allocated memory prevents allocation and fragmentation delays that would otherwise contribute to a lack of deterministic operation in the responsiveness of the sub-systems.

Enhancements to the memory handling further enhance the deterministic performance of NDDS, as there are no shared threads or memory that requires locking. It also provides dedicated buffers to prevent the different processes having corrupted shared memory.

Another advantage of NDDS 4.0 for the simulator developer is that there is no server process to crash, and applications do not share address space through the middleware, so operating system processes can be isolated and this fully protects all applications.

This open-standards approach is already in use by several simulator developers, including CAE for flight simulators, where NDDS is used to link subsystems via high speed IEE1394 Firewire links in its SimXXI product line. Simulator developer Nextel Engineering Systems in Madrid, Spain, is also using NDDS to link together the different elements within its simulator architecture in a way that did not compromise performance or reduce scalability for its Simware kernel.

Dr. Rajive Joshi is a Principal Engineer at Real-Time Innovations, Inc. specializing in the design of distributed and real-time systems, emphasizing object-oriented and component-based software architecture.