Building a effective real-time distributed publish-subscribe framework: part 3
In Part 1 and Part 2 in this series of articles we examined the functional similarities and differences between DDS and JMS, and their implications for real time distributed application design. Beyond this, a number of practical factors come into play when choosing a middleware technology for building a real time distributed system: DDS, JMS or some combination. What you must deal with when it comes time to implement an actual design involve several implementation-related issues:
(1) the most appropriate
(2) whether or not you need point-to-point delivery and what kinds;
(3) the most appropriate underlying software platform;
(5) the transport mechanisms you will be using;
(6) the middleware performance you will need;
(7) scalability; and
(8) the right balance between support for real-time operation and for the very real needs of the enterprise applications your system will be interfacing to.
However, such decisions are not just either/or considerations. Often, it will require that you use some combination of the new protocol frameworks.
JMS APIs are described in terms of a client/provider interaction; the client being the user application, distinct from the JMS provider or the middleware server. Most popular JMS provider implementations have centralized server-based architecture; some use a cluster of servers for fault-tolerance and load balancing. In a centralized server-based architecture, a message must pass via the server, which introduces extra latency and a potential resource bottleneck.
DDS APIs are described in terms of a peer-to-peer interaction; data is transferred directly from a DataWriter to a DataReader. DDS implementations generally use a decentralized peer-to-peer architecture. For example, the "RTI Data Distribution Service" from Real-Time Innovations, Inc. has a completely symmetric architecture. There is no single point of failure and data transfer latency is minimized. Participants can freely join or leave a domain, without needing special configuration. This is in keeping with the goals of DDS to enable robust, high-performance, low latency distributed applications.
In the JMS PtP messaging domain, a destination (Queue) may have multiple consumers (QueueReceivers) and producers (QueueSenders). A message is processed by exactly one of the attached consumers. Thus, a message is delivered point-to-point, from the producer to one of the many available consumers. The policy for selecting a consumer is left up-to the middleware provider. Upon message redelivery (if any), a message may get dispatched to a different consumer.
The point-to-point delivery mechanism in the JMS PtP messaging domain makes it very easy to distribute processing load across multiple identical consumers, thus providing a simple means for load balancing. However, since PtP behaves as if the messages are put in a single logical queue and handed over to one of the available consumers, this messaging domain will generally be less scalable that the Pub/Sub domain.
DDS does not support a point-to-point delivery mechanism. All the matching consumers associated with a topic will receive updates. The PARTITION QosPolicy may be used to partially achieve point-to-point delivery. A PARTITION QosPolicy, specifies a set strings that introduce a logical partition among the topics visible by a Publisher and a Subscriber.
A DataWriter within a Publisher only communicates with a DataReader in a Subscriber if (in addition to matching the Topic and having compatible QoS) the Publisher and Subscriber have a common partition name string. A change of this policy can potentially modify the "association" of existing DataReader and DataWriter entities. It may establish new "associations" that did not exist before, or break existing associations.
However, Point-to-point delivery may be accomplished by: (1) assigning a unique partition name to every consumer (Subscriber, DataReader pair); and (2) switching a producer (Publisher, DataWriter pair) among those partition names. The consumer selection policy can be configured in a variety of ways, at the application level.
JMS, as the name implies, was developed to provide a portable vendor neutral Java API for a wide range of message oriented middleware (MOM) implementations. JMS requires the Java platform. Some vendors provide JMS like APIs for other programming languages, but there is no established standard. JMS vendors may also provide proprietary APIs in other languages, native to the underlying MOM implementation. Also, note that JMS applications require non JMS APIs to bootstrap the client application. The standard practice is to use JNDI APIs, which are well established for Java EE programming.
DDS is an Object Management Group (OMG) standard defined in a language and platform neutral manner. OMG defines standard platform specific mappings to create language specific bindings. Therefore, standardized DDS APIs are available for all OMG supported programming languages; C, C++, and Java being the most popular. Since DDS is a standard in C, C++, and Java, it is available on a wide variety of platforms, including popular real-time operating systems (RTOS), desktop and server operating systems, and Java platforms.
JMS is an API only standard, and does not define an on-the-wire interoperability protocol. JMS only requires limited message portability on the client side: a Message created by provider A should be usable with provider B. Beyond this, a producer written using provider A cannot be expected to deliver messages to a consumer written using provider B.
Currently DDS is also an API only standard. However, there is active progress being made at the OMG towards standardizing on a DDS on-the-wire interoperability protocol.
JMS being an API only specification does not specify a transport model. However, since JMS message delivery is reliable (messages are not dropped unless the provider fails) and ordered, a JMS implementation can benefit from a reliable transport such as TCP. Being connection-oriented, TCP also fits naturally into the client/provider scheme.
Centralized server based JMS implementations generally use TCP. They rely on TCP to guarantee reliable and ordered delivery required by the JMS APIs. Some UDP based implementations do exist; they implement the reliable and ordered message delivery semantics on top of UDP.
Like JMS, DDS being an API only specification does not specify a transport model. However, DDS does not depend on reliable and ordered delivery of messages. In fact, the "best-effort" delivery QosPolicy is naturally suited to an unreliable low-latency transport such as UDP, whereas "reliable" delivery may benefit from the use of a reliable transport like TCP.
However, since DDS middleware must support both delivery schemes, it cannot make any assumptions about the reliability properties of the underlying transport. Therefore, DDS middleware is less reliant on the capabilities provided by a particular class of transport, and may be able to work well with a variety of transport classes. As an example, the RTI Data Distribution Service implementation provides a pluggable transport architecture, wherein any kind of transport can be plugged in, including UDP, TCP, shared memory, and various specialized transports.
JMS has a provision for optionally specifying a (usernme, password) when creating a Connection. Beyond this, security issues are left up to the JMS middleware vendor and the client application.
DDS provides an extension mechanism that can be useful in creating secure applications. For example, a consumer application can use a DataReader's USER_DATA QosPolicy to present security credentials to a DataWriter in the producer application. The producer application can authenticate the security credentials; these may potentially contain authorization rights.
If the consumer's security credentials are not acceptable, the DataReader entity can be permanently ignored using the DomainParticipant.ignore_subscription() method. A secure transport provided by the middleware vendor can ensure that the data is transferred securely.
JMS implementations minimally require the use of an external means for configuring and administering Destinations and ConnectionFactories. It is common practice to use JNDI to access these objects. Vendor provided proprietary tools must be used to configure the JNDI registries, before they can be used by an application. Evolving an application over its lifetime requires coordination with JNDI administration.
DDS implementations are expected to spontaneously discover each other; the DDS APIs do not rely on external means for establishing the discovery information. For example, with RTI Data Distribution Service, the user application only needs to link in a middleware library; no additional configuration is required to discover and establish dataflows with peer applications. As a result, DDS applications are plug-n-play, and require "zero" system administration.
Middleware performance can be characterized along several aspects including: (1) the end-to-end latency, i.e. the time required to send a message from a producer to a consumer; (2) the throughput, i.e. the maximum amount of data per unit time that can be transferred from a producer to a consumer.
While it is impossible to make any specific comments about performance---this can vary significantly from one middleware implementation to another (regardless of the supported APIs)---it is possible to make some general observations regarding potential middleware performance, based on the different choices made by the JMS vs. DDS APIs.
As compared with JMS, DDS has several features that can potentially minimize the end-to-end latency. These include:
(1) a "best-efforts"
delivery mode that does not require acknowledgements;
(2) reduced message overhead, since meta-data such as message headers and properties are not specified per message, but rather on a per endpoint basis;
(3) ability to use arbitrary data types which eliminates the need for converting back-and-forth between user and middleware provided types; (4) notification of data availability does not include the actual data, thus avoiding the overhead in setting this up;
(5) ability to support "zero-copy" data access so that an application can access the received data directly in the middleware internal buffers without requiring a copy;
(6) direct peer-to-peer data transfer from a DataWriter to DataReader without needed an intermediary. Thus, DDS middleware can potentially have better (lower) latency performance,
As compared with JMS, DDS has several features that can potentially maximize the throughput. These include: (a) reduced message overhead as in (2) above; (b) reduced processing in the data path as a result of (3) and (4) above; (c) direct data transfer as in (6) above, which eliminates a potential resource bottleneck and loading point. Thus, DDS middleware can potentially have better (higher) throughput performance as well.
Independent studies have observed DDS implementations that provide a factor of ten performance improvement of over JMS implementations.
Scalability refers to the ability to maintain performance levels as more nodes are added to a distributed system. For example, publish-subscribe scales better compared to "remote-procedure-calls (RPC)", due to the loose coupling between participants. As with performance, it is impossible to make any specific comments about scalability---this can vary significantly from one middleware implementation to another (regardless of the supported APIs). We some general observations regarding potential middleware scalability, based on the different choices made by the JMS vs. DDS APIs.
As compared with DDS, JMS has certain features that can potentially limit its scalability compared to DDS. These include:
(1) PtP messaging domain,
which specifies that a message be delivered to exactly one consumer,
and behaves as if the messages are put in a single logical queue and
handed over to one of the available consumers;
(2) centralized server-based architectures, generally used by JMS implementations will be less scalable than decentralized peer-to-peer architectures that support direct data transfer between endpoints.
DDS and JMS vary in their support for the needs of both real-time and enterprise applications. DDS has a variety of features that directly meet the needs of real-time applications, and have no counterparts in JMS. This is not surprising since DDS was developed while keeping real-time requirements in mind. The real-time specific features of DDS include:
(1) a low-latency best-efforts delivery mechanism;
(2) QoS policies for predictable delivery;
(3) QoS<> policies for resource management;
(4) status notifications; and
(5) potential for lower latency and higher throughput as discussed earlier.
In addition, the availability of DDS on high-performance RTOSes and the ability to utilize low latency transports (for example UDP instead of TCP) can further minimize end-to-end latency and support predictable operation. Combined together, they make possible DDS implementations that enable high-performance real-time distributed applications.
JMS has a variety of features that directly meet the needs of enterprise applications, and have no counterparts in DDS. This is not surprising since JMS was originally developed to provide a Java adaptor for a variety of enterprise messaging middleware implementations.
The enterprise specific features of JMS include: (1) full transaction support; and (2) explicit user application message acknowledgements. Combining message acknowledgements with persistent and durable delivery allows enterprise applications to guarantee message delivery.
In addition, Java EE, widely used in enterprise environments, supports a Message-driven Bean. A message-driven bean is a data consumer integrated into the Enterprise Java Beans (EJB) framework. Producers are written directly using the messaging API. While the message-driven bean specification does not assume the use of JMS, it is the most commonly messaging technology supported by Java EE vendors.
JMS implementations generally also support for JTA, so that a message application can participate in a distributed transaction. Also, the JNDI APIs generally used by JMS applications are included in the Java EE specifications.
Thus, JMS is well integrated into enterprise application frameworks; given its legacy this is hardly surprising. However, DDS can also be used in enterprise environments; a message-driven bean using DDS can potentially simplify Java EE integration.
Using DDS and JMS together
It should be obvious that the choice of DDS or JMS as the middleware technology has a significant impact on a data-centric design. While DDS and JMS offer some capabilities that are similar, there also offer some unique capabilities. Thus, a data-centric design may employ both in complementary ways. There is nothing precluding the use of JMS and DDS together in the same application.
In developing a distributed system using both JMS and DDS, certain capabilities can facilitate development and integration. These include (1) DDS-JMS bridging; (2) JMS/DDS bindings; and (3) DDS for JMS discovery.
JMS-DDS bridging. DDS-JMS bridging involves creating a "bridge" that is both a DDS and JMS application. A bridge allows JMS and DDS applications to interoperate.
A bridge forwards JMS messages as DDS data updates, and DDS data updates as JMS messages. A "bridge configuration" file can specify the mapping between DDS and JMS topics, types, and QoS. Such a bridge will incur data conversion and mapping overhead, and introduce a single point of failure between the DDS and JMS domains. It will be limited to supporting the "least common denominator" i.e. only the overlapping capabilities of DDS and JMS. It may not be always possible to provide end-to-end data delivery semantics. However, it can be useful for integrating disparate sub-systems using JMS or DDS.
JMS/DDS bindings. JMS/DDS bindings wrap a DDS middleware with JMS APIs. JMS/DDS bindings can be useful for porting applications written to a JMS API to a DDS domain, or for developing JMS applications that are interoperable with DDS applications, or simply to enable a potentially higher performance JMS implementation.
Since DDS provides finer grained data distribution and management of data flows with many more QoS, it is possible to efficiently map and implement JMS APIs on top of DDS. JMS features not directly provided by DDS, such as PtP messaging semantics, full transactional semantics, and client acknowledgement, can be implemented on top of the DDS APIs. The remaining JMS features can be mapped into DDS APIs.
Note that implementing a DDS API on top of JMS is not a practical idea, since JMS does not provide the fine granularity and low-level primitives needed to provide an efficient DDS implementation.
DDS for JMS discovery. JMS clients rely on non-JMS APIs for creating the ConnectionFactory and Destination objects. Typically these objects are obtained by performing a JNDI lookup; these must have been already registered with the JNDI directory.
An application can alternatively utilize DDS to discover these objects. A JMS provider, they could use DDS to announce the configured ConnectionFactory and Destination objects to the client applications. A JMS client application would receive the available objects, select the ones it is interested in, and bootstrap the JMS APIs. This approach is useful in a mixed DDS and JMS environment, where DDS and JMS are used simultaneously to distribute different types of information. JMS applications can benefit from the use of DDS's spontaneous discovery mechanism.
DDS and JMS differ in their ability to cater to the key data-centric design requirements. DDS is newer standard based on fundamentally different paradigms than JMS, with regards to data modeling, dataflow routing, discovery, and data typing. However, these differences provide applications designers with powerful new architectural possibilities.
If you are designing or integrating distributed data-centric
applications, DDS and JMS merit careful consideration. Using one or
both can considerably simplify a data-centric design and integration,
and help maintain the focus on application issues, rather than becoming
bogged down by communication and data delivery concerns.
Rajive Joshi, Ph.D., is principal engineer at Real-Time Innovations, Inc.
1) Data Distribution Service for Real-time Systems, v1.1,