Java 2 Platform, Micro Edition (J2ME) is the widely used Java application platform for embedded devices. It provides a runtime environment for software on a broad range of devices, such as mobile phones, PDAs and TV set-top boxes.

These applications increasingly require sophisticated data management, to support software such as Electronic Programming Guides in set-top boxes, mobile business applications in PDAs, and music indexes in digital audio players.

Most applications store and manipulate data using a relational database system. In fact, the relational, client/server approach to data management can be justified when an embedded application needs to access data remotely.

In this scenario, on-device resource demands are reduced by using simplified protocols and by moving most complex request-processing logic to the server.

However, embedded applications often need to store and process data locally (the database and application reside on the same device), either because no connection is available, or because communicating with a remote server would entail unacceptable network transport overhead. In such cases, relational databases become impractical to integrate within device-based software, in large part due to the sophisticated, resource-consuming SQL optimizers needed to execute queries.

In fact, the level of query processing provided by an Oracle, or even a MySQL database is often not required for software deployed on a device, because the queries are simple and are known when the embedded application is compiled. This permits specialized resource-conserving data management solutions to be developed for embedded devices.

Object-oriented databases can improve performance in resource-constrained embedded devices, and can greatly simplify development. The main advantage of object-oriented databases is their seamless integration with object-oriented programming languages. With a relational database, the application must translate the relational representation of the application's data to the object representation required by the host language.

Some modern IDEs and modeling tools generate this "translation" code automatically, saving programming time. However, this extra layer of code still constrains performance and increases the overall size of the application.

Neither attribute is desirable in resource-constrained J2ME environments. In contrast, object-oriented databases access objects directly, for much simpler and more efficient queries. Consider, for example, the following class (Figure 1 below):

class Person {
public String firstName;
public String lastName;
public int age;
public long salary;
};

The relational database approach to fetch the data is illustrated in Figure 2, below:

// RDBMS apporach
Person getPersonByName(string lastName)
{
Statement stmt = con.createStatement();
stmt.setString(1, lastName);
ResultSet cursor = stmt.executeQuery
("SELECT * FROM Person where lastName like " + lastName + "'%'");
if (!cursor.next()) {
cursor.close();
return null;
}
Person p = new Person();
p.firstName = cursor.getString("firstName");
p.lastName = cursor.getString("lastName");
p.age = cursor.getInt("age");
p.salary = cursor.getLong("salary");
return p;
}

The object-oriented approach eliminates the database-to-language translations (Figure 3 below):

// OODB appoach
Person getPersonByName(string lastName)
{
return personIndex.prefixSearch(lastName);
}

J2ME Limitations & Perst Lite's Solutions
However, despite the promise of object-oriented database systems for device-based embedded systems, few such databases have emerged for J2ME. The reason is that nearly all existing object-oriented databases depend on Java features that are limited or nonexistent in J2ME. The fundamental concept of the J2ME platform is the Connected Limited Device Configuration (CLDC).

This configuration defines the base set of application programming interfaces and virtual-machine features that must be present in each implementation of a J2ME environment. CLDC Version 1.0 (JSR 30) is the first release of the CLDC specification, which defines a compact virtual machine and basic libraries for resource-constrained devices. Among other things, the specification doesn't support reflection, floating point arithmetic and weak references.

CLDC 1.1 is a revised version of the CLDC 1.0 specification, and includes a number of new features, such as the floating point and weak references support. However, CLDC 1.0 is still used widely in the J2ME development community, and is the only configuration available for many hardware platforms.

This article describes how developers of the open source Perst Lite object-oriented J2ME database - released in September, 2006 - were able to deliver full object-oriented database features on J2ME by "engineering around" the absence of standard Java features on the platform.

In fact, some of J2ME's limitations might be considered blessings in disguise: certain techniques that enabled the database to conform to CDLC 1.0 specifications, also contributed to greater efficiency and to a smaller code footprint that better meets the resource constraints of embedded systems. These techniques are illustrated below.

Perst Lite is a J2ME version of the open source, object-oriented Perst C# and Java database. Because both databases are open source, the code used in the examples cited below, as well as all other source code for both databases, is available to freely download  and fully inspect.

Reflection
Usually, an object-oriented database's interface with the host application is based on reflection capabilities provided by the host language. Reflection is used to inspect the object format, to get and set object fields, create class instances and invoke class methods at runtime. Thus, reflection gives the database runtime "knowledge" of the storage format and access methods of the object. Figure 4 below illustrates how an application can pack an object using the reflection mechanism.

final int packObject
(Object obj, ClassDescriptor desc, int offs,
ByteBuffer buf, IPersistent po, boolean finalized)

throws Exception
{
// This is a list of field descriptors prepared by the
//OODBMS using reflection
ClassDescriptor.FieldDescriptor[] flds = desc.allFields;
//Loop through all fields in the class
for (int i = 0, n = flds.length; i < n; i++) {
ClassDescriptor.FieldDescriptor fd = flds[i];
// java.lang.reflect.Field contains field information field and
//provides access to the field
Field f = fd.field;
switch(fd.type) {
case ClassDescriptor.tpByte:
buf.extend(offs + 1);
// get the value of a byte field
buf.arr[offs++] = f.getByte(obj);
continue;
case ClassDescriptor.tpBoolean:
buf.extend(offs + 1);
// get the value of a boolean field
buf.arr[offs++] = (byte)(f.getBoolean(obj) ? 1 : 0);
continue;
case ClassDescriptor.tpShort:
buf.extend(offs + 2);
// get the value of a short integer field
Bytes.pack2(buf.arr, offs, f.getShort(obj));
offs += 2;
continue;

The routine illustrated here entails performance overhead because of its use of the reflection mechanism - but it does the job of giving the database the information it needs to pack and unpack an object.

Due to the implementation complexity, the reflection mechanism is omitted in the J2ME CDLC 1.0. Thus, a database needs to operate without reflection support. One solution, borrowed from the C/C++ world, is for the application to provide pack and unpack routines to serialize and de-serialize objects, and for the database runtime to use these routines to fetch and store objects. The following code in Figure 5, below illustrates this concept.

public class Guess extends Persistent {
public Guess yes;
public Guess no;
public String question;

// Object serialization routine. Writes all object fields to the stream
public void writeObject(IOutputStream out) {
out.writeObject(yes); // serialize reference to the persistent object
out.writeObject(no);
out.writeString(question); // write string field
}

// Object de-serialization routine. Restores object fields from the stream
public void readObject(IInputStream in) {
yes = (Guess)in.readObject(); //restores a reference to the object
no = (Guess)in.readObject();
question = in.readString();// reads a string value from the stream
}

In the following code, the Perst Lite database runtime uses these functions to fetch data (Figure 6 below):

final byte[] packObject(IPersistent obj) {
// create stream for writing serialized object data
PerstObjectOutputStream out = new PerstObjectOutputStream(obj);

// write the object to the stream
obj.writeObject(out);

// return byte array with the serialized object data
return out.toArray(); }

This approach eliminates reflection's overhead and replaces that mechanism with direct access to data, which significantly improves the application's performance (or enables equivalent performance using fewer CPU cycles, which is often just as important for an embedded device).

The downside is that it can be less convenient and more error-prone since the packing and unpacking routines are written manually by a programmer. To eliminate this unwanted labor and risk, Perst Lite generates the packing/unpacking procedures via a special utility, which reads the class definitions and creates the serialization and de-serialization methods automatically.

Object Caching
Object caching is another challenge to object-oriented databases in general, and to embedded Java databases in particular. An OODBMS's object cache keeps frequently used objects in memory, to avoid excessive access to non-volatile device storage.

Fundamentally, object-oriented and relational databases access their data differently. In a relational setup, a client application sends a data query to the server and receives a result set, usually containing fewer entries than the number of records in the tables from which the result set has been selected.

In contrast, an object oriented application needs to select the objects by itself (there is no separate server process to sort the data) and accesses objects frequently. If each object access required fetching the object from non-volatile storage, triggering an I/O operation, application performance would degrade unacceptably.

However, an application's "working set" is usually not excessively large: the number of objects accessed frequently is rather small and can fit into a main memory cache. Therefore, the database provides an object cache that reduces storage I/O overhead and allows an application to access persistent objects with performance similar to when it fetches normal, transient objects.

<>Dealing with garbage collection
Java's automatic memory de-allocation or "garbage collection" presents a challenge when interacting with the object cache. If the object is "pinned" into the cache using a reference "a "strong reference"—to the object, the garbage collector won't de-allocate it. As a result, the object cache grows, and can either take all available memory, or even overflow memory.

To counter this and to provide applications with the ability to interact with the garbage collector, Java 2 introduced a special way to reference objects--"weak references" that allow a program to maintain a reference to an object, while also allowing that object to be reclaimed by the garbage collector. This enables object-oriented databases to cache objects, with the benefits described above.

However, weak references, and especially "finalizer" methods, do greatly complicate garbage collection (because of the possible "resurrection" of the de-allocated object). Java 2 Platform, Micro Edition is by necessity a simplified Java platform, and as a result, weak references are omitted from J2ME CDLC 1.0.

What is an object-oriented database for Java to do about this lack of a key Java capability? To conform to the specification, the database runtime needs to control the size of the object cache explicitly—for example, by clearing the cache upon transaction commit. The problem with this approach is that transactions become limited in size by the amount of memory available to the application.

The database runtime needs to keep all objects modified by the transaction in memory as it has to use strong references to prevent them from being de-allocated by the garbage collector. Counterbalancing this difficulty is the fact that, in an embedded setting, transaction sizes are usually known in advance, which allows for adequate application structuring to meet these challenges.

Perst Lite Experience
The open source Perst Lite project has evolved from the main-stream Perst open source, object-oriented embedded database system to serve the data management needs of embedded Java applications based on JDK 1.1.

Perst Lite is missing some of the main-stream Perst functionality due to the JDK 1.1 and CDLC limitations. Most of the functionality that is based on reflection is eliminated from Perst Lite. That includes compound indices for random access and alternative B-Tree implementations, XML import/export, garbage collection, SubSQL, value and raw types support, automatic schema evolution and scalable map.

However, Perst Lite works around many of the missing features: the garbage collection is replaced with explicit de-allocation, compound indexes are built using "normal" indexes, and scalable map is replaced with a standard map or a vector. These solutions enhance the feature set provided by the database runtime with the ability to deploy into limited-resource embedded systems settings.

Other limitations imposed by JDK 1.1 include the absence of the file locking API, standard socket API, compression support and the ClassLoader mechanism. In some instances, these features are impractical to compensate for in a general purpose J2ME database, so database functions such as database replication and database compression, which depend on these platform features, are not implemented in Perst Lite.

That is not to say that adding these database functions to Perst Lite is impossible " just that the existing Perst Lite software was viewed as providing the optimal balance of database capabilities, on one hand, and minimized complexity and resource demands, on the other.

<>Future directions
If future users deem additional database features to be worth developing, Perst Lite's open source standing is a boon: there is no need for complicated licensing and source code is available up front, enabling developers to extend the database system much more easily than with a commercial product. Adding a feature such as database replication to Perst Lite will increase both database complexity and footprint. But developers will find that tradeoff to be worthwhile in some cases.

In order to be truly useful, an object-oriented embedded J2ME database must not only adhere to CDLC specifications, but also accommodate development in specific environments, such as the popular Blackberry platform used by wireless carriers throughout the world. Balancing the feature set provided by the database runtime with the need for a highly reduced footprint and the ability to fit into restricted embedded settings is imperative.

Assumptions made in the mainstream Perst embedded database have been refined and even replaced to mesh with J2ME embedded environments' pared down file systems and limited memory and CPU resources.

By avoiding the one-size-fits-all approach, the open source Perst Lite provides the benefits of the object-oriented programming model as well as such inherent database strengths as transaction support, optimized algorithms, and significantly faster time-to-market, to developers working with J2ME.

Andrei Gorine is Co-Founder and CTO, and Konstantin Knizhnik is a software engineer, at McObject. For more information plus links to freely downloadable open source Perst and Perst Lite software, go to  www.mcobject.com/perst/.