Developing an object-oriented database for J2ME-based embedded devices

Java 2 Platform, Micro Edition (J2ME) is thewidely used Java application platform for embedded devices. It providesa runtime environment for software on a broad range of devices, such asmobile phones, PDAs and TV set-top boxes.

These applications increasingly require sophisticated data management,to support software such as Electronic Programming Guides in set-topboxes, mobile business applications in PDAs, and music indexes indigital audio players.

Most applications store and manipulate data using a relationaldatabase system. In fact, the relational, client/server approach todata management can be justified when an embedded application needs toaccess data remotely.

In this scenario, on-device resource demands are reduced by usingsimplified protocols and by moving most complex request-processinglogic to the server.

However, embedded applications often need to store and process datalocally (the database and application reside on the same device),either because no connection is available, or because communicatingwith a remote server would entail unacceptable network transportoverhead. In such cases, relational databases become impractical tointegrate within device-based software, in large part due to thesophisticated, resource-consuming SQL optimizersneeded to execute queries.

In fact, the level of query processing provided by an Oracle, or even a MySQL database is often not required for software deployed on a device,because the queries are simple and are known when the embeddedapplication is compiled. This permits specialized resource-conservingdata management solutions to be developed for embedded devices.

Object-oriented databases canimprove performance in resource-constrained embedded devices, and cangreatly simplify development. The main advantage of object-orienteddatabases is their seamless integration with object-orientedprogramming languages. With a relational database, the application musttranslate the relational representation of the application's data tothe object representation required by the host language.

Some modern IDEs and modeling tools generate this “translation” codeautomatically, saving programming time. However, this extra layer ofcode still constrains performance and increases the overall size of theapplication.

Neither attribute is desirable in resource-constrained J2MEenvironments. In contrast, object-oriented databases access objectsdirectly, for much simpler and more efficient queries. Consider, forexample, the following class (Figure 1below) :

classPerson { public String firstName; public String lastName; public int age; public long salary; };

Figure1 .

The relational database approach to fetch the data is illustrated inFigure 2, below:

//RDBMS apporachPersongetPersonByName(string lastName) {Statement stmt = con.createStatement();stmt.setString(1, lastName); ResultSet cursor = stmt.executeQuery(“SELECT * FROM Person where lastName like ” + lastName + “'%'”); if (!cursor.next()) {cursor.close();return null; }Person p = new Person();p.firstName = cursor.getString(“firstName”); p.lastName = cursor.getString(“lastName”);p.age = cursor.getInt(“age”);p.salary = cursor.getLong(“salary”);return p; }

Figure2.

The object-oriented approach eliminates the database-to-languagetranslations (Figure 3 below ):

//OODB appoachPersongetPersonByName(string lastName) {return personIndex.prefixSearch(lastName); }

Figure3.

J2ME Limitations & Perst Lite'sSolutions
However, despite the promise of object-oriented database systems fordevice-based embedded systems, few such databases have emerged forJ2ME. The reason is that nearly all existing object-oriented databasesdepend on Java features that are limited or nonexistent in J2ME. Thefundamental concept of the J2ME platform is the Connected LimitedDevice Configuration (CLDC).

This configuration defines the base set of application programminginterfaces and virtual-machine features that must be present in eachimplementation of a J2ME environment. CLDC Version 1.0 (JSR 30) is thefirst release of the CLDC specification, which defines a compactvirtual machine and basic libraries for resource-constrained devices.Among other things, the specification doesn't support reflection,floating point arithmetic and weak references.

CLDC 1.1 is a revised version of the CLDC 1.0 specification, andincludes a number of new features, such as the floating point and weakreferences support. However, CLDC 1.0 is still used widely in the J2MEdevelopment community, and is the only configuration available for manyhardware platforms.

This article describes how developers of the opensource Perst Lite object-oriented J2ME database – released inSeptember, 2006 – were able to deliver full object-oriented databasefeatures on J2ME by “engineering around” the absence of standard Javafeatures on the platform.

In fact, some of J2ME'slimitations might be considered blessings in disguise: certaintechniques that enabled the database to conform to CDLC 1.0specifications, also contributed to greater efficiency and to a smallercode footprint that better meets the resource constraints of embeddedsystems. These techniques are illustrated below.

Perst Lite is a J2ME version of the open source, object-orientedPerst C# and Java database. Because both databases are open source,the code used in the examples cited below, as well as all other sourcecode for both databases, is available to freelydownload  and fully inspect.

Reflection
Usually, an object-oriented database's interface with the hostapplication is based on reflection capabilities provided by the hostlanguage. Reflection is used to inspect the object format, to get andset object fields, create class instances and invoke class methods atruntime. Thus, reflection gives the database runtime “knowledge” of thestorage format and access methods of the object. Figure 4 below illustrates how anapplication can pack an object using the reflection mechanism.

final int packObject
(Object obj, ClassDescriptor desc, int offs,
ByteBuffer buf, IPersistent po, boolean finalized)

throws Exception { // This is a list of field descriptors prepared by the //OODBMS using reflectionClassDescriptor.FieldDescriptor[] flds = desc.allFields; //Loop through all fields in the classfor (int i = 0, n = flds.length; i < n; i++) {ClassDescriptor.FieldDescriptor fd = flds[i]; // java.lang.reflect.Field contains field information field and //provides access to the fieldField f = fd.field; switch(fd.type) {case ClassDescriptor.tpByte: buf.extend(offs + 1); // get the value of a byte field buf.arr[offs++] = f.getByte(obj); continue; case ClassDescriptor.tpBoolean: buf.extend(offs + 1); // get the value of a boolean fieldbuf.arr[offs++] = (byte)(f.getBoolean(obj) ? 1 : 0); continue; case ClassDescriptor.tpShort: buf.extend(offs + 2); // get the value of a short integer fieldBytes.pack2(buf.arr, offs, f.getShort(obj)); offs += 2; continue;

Figure4

The routine illustrated here entails performance overhead because ofits use of the reflection mechanism – but it does the job of giving thedatabase the information it needs to pack and unpack an object.

Due to the implementation complexity, the reflection mechanism isomitted in the J2ME CDLC 1.0. Thus, a database needs to operate withoutreflection support. One solution, borrowed from the C/C++ world, is forthe application to provide pack and unpack routines to serialize andde-serialize objects, and for the database runtime to use theseroutines to fetch and store objects. The following code in Figure 5, below illustrates thisconcept.

publicclass Guess extends Persistent {
public Guess yes;
public Guess no;
public String question;

// Object serialization routine. Writes all object fields to the stream
public void writeObject(IOutputStream out) {
out.writeObject(yes); // serialize reference to the persistent object
out.writeObject(no);
out.writeString(question); // write string field
}

// Object de-serialization routine.Restores object fields from the stream public void readObject(IInputStreamin) {
yes = (Guess)in.readObject();//restores a reference to the object no = (Guess)in.readObject(); question = in.readString();// reads astring value from the stream
}

Figure5

In the following code, the Perst Lite database runtime uses thesefunctions to fetch data (Figure 6 below ):

final byte[] packObject(IPersistent obj) {
// create stream for writing serialized object data
PerstObjectOutputStream out = new PerstObjectOutputStream(obj);
// write the object to the stream
obj.writeObject(out);

// return bytearray with the serialized object data returnout.toArray(); }

Figure6

This approach eliminates reflection's overhead and replaces thatmechanism with direct access to data, which significantly improves theapplication's performance (or enables equivalent performance usingfewer CPU cycles, which is often just as important for an embeddeddevice).

The downside is that it can be less convenient and more error-pronesince the packing and unpacking routines are written manually by aprogrammer. To eliminate this unwanted labor and risk, Perst Litegenerates the packing/unpacking procedures via a special utility, whichreads the class definitions and creates the serialization andde-serialization methods automatically.

Object Caching
Object caching is another challenge to object-oriented databases ingeneral, and to embedded Java databases in particular. An OODBMS'sobject cache keeps frequently used objects in memory, to avoidexcessive access to non-volatile device storage.

Fundamentally, object-oriented and relational databases access theirdata differently. In a relational setup, a client application sends adata query to the server and receives a result set, usually containingfewer entries than the number of records in the tables from which theresult set has been selected.

In contrast, an object oriented application needs to select theobjects by itself (there is no separate server process to sort thedata) and accesses objects frequently. If each object access requiredfetching the object from non-volatile storage, triggering an I/Ooperation, application performance would degrade unacceptably.

However, an application's “working set” is usually not excessivelylarge: the number of objects accessed frequently is rather small andcan fit into a main memory cache. Therefore, the database provides anobject cache that reduces storage I/O overhead and allows anapplication to access persistent objects with performance similar towhen it fetches normal, transient objects.

<>Dealing with garbage collection
Java's automatic memory de-allocation or “garbage collection” presentsa challenge when interacting with the object cache. If the object is”pinned” into the cache using a reference “a “strong reference”—to theobject, the garbage collector won't de-allocate it. As a result, theobject cache grows, and can either take all available memory, or evenoverflow memory.

To counter this and to provide applications with the ability tointeract with the garbage collector, Java 2 introduced a special way toreference objects–“weak references” that allow a program to maintain areference to an object, while also allowing that object to be reclaimedby the garbage collector. This enables object-oriented databases tocache objects, with the benefits described above.

However, weak references, and especially “finalizer” methods, dogreatly complicate garbage collection (because of the possible”resurrection” of the de-allocated object). Java 2 Platform, MicroEdition is by necessity a simplified Java platform, and as a result,weak references are omitted from J2ME CDLC 1.0.

What is an object-oriented database for Java to do about this lackof a key Java capability? To conform to the specification, the databaseruntime needs to control the size of the object cache explicitly—forexample, by clearing the cache upon transaction commit. The problemwith this approach is that transactions become limited in size by theamount of memory available to the application.

The database runtime needs to keep all objects modified by thetransaction in memory as it has to use strong references to preventthem from being de-allocated by the garbage collector. Counterbalancingthis difficulty is the fact that, in an embedded setting, transactionsizes are usually known in advance, which allows for adequateapplication structuring to meet these challenges.

Perst Lite Experience
The open source Perst Lite project has evolved from the main-streamPerst open source, object-oriented embedded database system to servethe data management needs of embedded Java applications based on JDK1.1.

Perst Lite is missing some of the main-stream Perst functionalitydue to the JDK 1.1 and CDLC limitations. Most of the functionality thatis based on reflection is eliminated from Perst Lite. That includescompound indices for random access and alternative B-Treeimplementations, XML import/export, garbage collection, SubSQL, valueand raw types support, automatic schema evolution and scalable map.

However, Perst Lite works around many of the missing features: thegarbage collection is replaced with explicit de-allocation, compoundindexes are built using “normal” indexes, and scalable map is replacedwith a standard map or a vector. These solutions enhance the featureset provided by the database runtime with the ability to deploy intolimited-resource embedded systems settings.

Other limitations imposed by JDK 1.1 include the absence of the filelocking API, standard socket API, compression support and theClassLoader mechanism. In some instances, these features areimpractical to compensate for in a general purpose J2ME database, sodatabase functions such as database replication and databasecompression, which depend on these platform features, are notimplemented in Perst Lite.

That is not to say that adding these database functions to PerstLite is impossible ” just that the existing Perst Lite software wasviewed as providing the optimal balance of database capabilities, onone hand, and minimized complexity and resource demands, on the other.

<>Future directions
If future users deem additional database features to be worthdeveloping, Perst Lite's open source standing is a boon: there is noneed for complicated licensing and source code is available up front,enabling developers to extend the database system much more easily thanwith a commercial product. Adding a feature such as databasereplication to Perst Lite will increase both database complexity andfootprint. But developers will find that tradeoff to be worthwhile insome cases.

In order to be truly useful, an object-oriented embedded J2MEdatabase must not only adhere to CDLC specifications, but alsoaccommodate development in specific environments, such as the popularBlackberry platform used by wireless carriers throughout the world.Balancing the feature set provided by the database runtime with theneed for a highly reduced footprint and the ability to fit intorestricted embedded settings is imperative.

Assumptions made in the mainstream Perst embedded database have beenrefined and even replaced to mesh with J2ME embedded environments'pared down file systems and limited memory and CPU resources.

By avoiding the one-size-fits-all approach, the open source PerstLite provides the benefits of the object-oriented programming model aswell as such inherent database strengths as transaction support,optimized algorithms, and significantly faster time-to-market, todevelopers working with J2ME.

Andrei Gorine is Co-Founder andCTO, and Konstantin Knizhnik is a software engineer, at McObject. Formore information plus links to freely downloadable open source Perstand Perst Lite software, go to  www.mcobject.com/perst/.

EmbeddedDatabase resources on Embedded.com

1) EnsuringDatabase Quality
2) Designingdata-centric software
3) Providingreal-time embedded to enterprise connectivity with DDS and DBMS
4) XML,SQL, and C
5) Buildinga effective real-time distributed publish-subscribe framework
6) Tacklingmemory allocation in multicore and multithreaded applications
7) Designingdata-centric software
8) Reducecomplexity of network systems development with data-centric software
9) Telematicsoftware needs data-centric development approaches
10)
Theproper care and feeding of object databases in embedded systems
11)
UsingC++ to view Objects
12)
Howto add relational database search to your embedded device application

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.