Data Management - Embedded.com

Data Management

When I heard that Encirq Corporation was peddling data management software for embedded systems I rolled my eyes. “Just what I need,” I reflected, “SQL for 8051s.”

Recently, though, Encirq's Eugene Buechele gave me a different perspective.

“Data management” does not necessarily mean “database.” It includes all sorts of data manipulation issues, from managing lists, arrays, indexes, buffers and all the stuff we struggle with every day. Eugene claims that 55% of most embedded apps are concerned with these sorts of activities.

Though I don't know how accurate that statement is, we sure do write a lot of lines of code for sucking data in from networks and peripherals, and for storing and manipulating it. Consider the non-embedded space: the bane of security seems to be buffer overflows, and the nuisance of Windows stems in no small part from memory leaks. Would that PC folks used a tool to manage their allocations! Or at least to flag a forgotten free().

He also claims the 89% of “serious” bugs are in the data-handling part of our programs. So it would seem logical that developers would seek out some canned solution to data woes; a product that makes it easier to write this 55% of our code, reliably.

Encirq's product compiles data-manipulation commands written in a tuned version of PL/SQL into C. A small runtime library provides support services. Many operations look like database manipulation, but as the product is designed for the embedded world implementation is based on streams and is much, much faster than standard database accesses. But is there an actual database? Where is the data stored?

The answer seems to be: who cares? Let the tool take care of these implementation details, just as we rely on malloc() to find chunks of heap.

If you're building an MP3 player, to insert a song into a playlist the PL/SQL might look like “INSERT INTO playlist (“Rocky Racoon”, “The White Album”, “The Beatles”)” instead of a few hundred lines of complicated C code. It's a very high level of abstraction that delegates all the data tedium to the tools.

Which sounds pretty good to me.

I'm still trying to get my hands around the technology and applications, but am fascinated by the different approach to thinking about the way we build firmware. What do you think? How much effort goes into dealing with all the various forms of data in your systems?

Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at . His website is .


Two points

1. SQLITE is a light weigth embedded SQL data base and is totally open source with a foot print of 100-300K. Just curious how this product compares with the product in the article.

2. I like the concepts in the article and have been considering implementing RTOS features in such a DB for non real time event processing. This would be useful where embedded systems had to interface with enterprise work flow systems. Also, useful for keeping state information when power is lost.

– Glenn Edgar


89%???

That is a bit high. I can see something like 75%. That is because that is about the percentage of our programs that is data handling/manipulation. About the other 25% is the stuff we usually worry about. (threads, synchronization, timing, interrupts etc.). We tend to just 'plow' through the data manipulation.

Data manipulation is very important though. However, try and get a group of software types to agree on how best to do this! Linked Lists? Arrays? Hashtables? Search algorithms? Indices? Keys?

heh…

Alot of us stay with the old switch-case statements and generate a ton o' code! From what I can see with SNMP MIBS, this appears to be the preferred method.

I am gradually seeing a shift towards more dynamic methods … and a glimmer or two of folks trying to embed XML into their projects!

There are some signs of hope for some real clever engineering that departs from the old switch-case statement!

– Ken Wada


I think the data centric idea is good. An alternative to RDMS's are scripting languages such as Python and Rubby. These languages are build upon a hash table data structure much as LISP is built upon a list structure. In languages like this the data structures are built into the language. Python code can ususally be written in 5 times less statements than typical C code. Also, the bigger the system, the more leverage is obtained with scripting languages like Python.

The problem with Python is the memory foot print. A language called Lua is Python like and has a memory foot print of less than 100K. I have used this language for a couple of years in embedded systems and has have good performance. Lua is used extensively in the game industry to do many of the thinks that are advacated in this article.

– Glenn Edgar


Per Jack's story, “'Data management' does not necessarily mean 'database.'” So, when we talk about how Encirq's DeviceSQL stacks up against SQLite, we are looking at a narrow implementation of DeviceSQL.

That notwithstanding, for the majority of embedded DB applications search times are critical. In the case of DeviceSQL, benchmarks show it to process SELECT statements 30 to 50 times faster than SQLite depending on the platform. And the DeviceSQL services library footprint is just 24K.

Importantly, while SQLite, like other embedded DBs, supports storage, search, and retrieval of data, its does not optimally address the full spectrum of data management needs of embedded devices.

Conversely, developers who use DeviceSQL are implementing custom, device data management solutions optimized around performance, footprint, scalability, portability and control while realizing the productivity gains of using a high-level, domain specific language for data definition and manipulation.

Developers use DeviceSQL to implement traditional DB functions (including in-memory, memory mapped, and page indexed DB models); they use it to concurrently search data in multiple files stored on multiple memory media (while SQLite and other DB products store data in a single file on a single storage medium). And they use DeviceSQL to handle dynamically streaming data (like a GPS signal or sensor feed) as if it were table data, without having to first store then process it as when using SQLite or another embedded DB.

When combined these uses entail true “device data management.”

– David Cowan

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.