The DBMS challenges ahead for embedded IoT
The near ubiquitous connectivity of the Internet of Things (IoT) is buffeting virtually every embedded software building block and tool, from RTOSs to compilers, debuggers, code analysis tools and languages. Embedded database management systems and tools, small footprint versions of their cousins on desktop and enterprise systems, are also being forced to change drastically.
In the 1990s and early 2000s, embedded databases were used to manage the data flows required for many media-intensive applications over the wired Internet. In some designs, especially on network switches, this involved sorting and managing the various data flows. This key role continued as the world moved toward wireless consumer media players and, ultimately, smart phones. Those kinds of applications still exist, and traditional embedded database programs are still playing a pivotal role and also extending their influence in high reliability automotive, industrial and military/aerospace designs.
But in the Internet of Things, life is becoming much more complicated. All IoT transactions, not just media-intensive applications, will require data management services. As a result, embedded database vendors such as McObject, Ittia, Raima, and others are facing a set of data management challenges never conceived of until recently. Developers who want to develop applications for the embedded IoT will need to integrate a new set of design parameters into their thinking. IoT database management apps will have to become as second nature to designers as considerations of interrupt service routines (ISRs), callbacks, and function pointers are now in traditional embedded designs. Some of these new considerations include:
Schema: The database structure your device will need, such as relational, object, networked, or structured.
Persistence: the degree to which an application retains information about devices it is connected to. In the current wireless environment, this depends on the amount of flash available and the efficiency with which it is used.
ACID: Acronym for Atomicity, Consistency, Isolation, and Durability, key features any embedded database design will need in a wireless IoT environment.
- Atomicity requires that each transaction be "all or nothing" such that if the transaction fails, the database state is left unchanged.
- Consistency ensures that any transaction will bring the database from one valid state to another.
- The isolation property ensures that the concurrent execution of transactions will result in a system state that would be obtained if transactions were executed serially.
- Durability ensures that once a transaction has been committed, it will remain so, even in the event of power loss or system crashes.
Database query languages. In addition to C, C++, Java, and a few Web scripting languages, developers of IoT apps will have to be knowledgeable about database query languages, particularly the most common one, the Structured Query Language (SQL), the lingua franca of most cloud-based systems that IoT systems will interact with. Developers will also have to be aware of when and where to use NoSQL, a broad term applied to data storage and retrieval schemes other than those used in relational databases based on SQL. NoSQL serves the needs of large cloud service companies for managing big data and real-time web applications.
Key value storage. Unlike relational databases used in the cloud environment, a key value database stores, retrieves, and manages data structures in the form of a dictionary or hash. Records are stored and retrieved using a key that uniquely identifies the record, and is used to quickly find data within the database.
In-process data management. An embedded database technique in which the database and the application software it manages execute within the same address space, reducing latency by through elimination of unnecessary client-server communications.
In-memory data management. Often employed in IoT systems that use flash, in-memory databases rely primarily on main memory for computer data storage, versus external disk storage as on cloud-based servers. In-memory is faster since the internal optimization algorithms are simpler and execute fewer CPU instructions.
IoT DBM challenges
A designer who wants to build an embedded IoT based system has to deal with two different kinds of data management. First are those procedures and functions necessary to the operation of the field devices. Second are operations for managing data flow to and from remote aggregation points, usually on a server in the cloud, or in an intermediary layer in the near-user "fog" closer to the edge devices.
The first data management chore is fundamentally simple: collect small subsets of data, analyze them, and respond with automated, real-time control actions in each case. Because an embedded device is designed for a limited range of jobs, the developer can know quite a lot about its data, including the types, relationships, volume, and velocity of data, all of which make it easier to assess and direct what actions are to be taken. What is different now is the sheer number of devices that must be managed.
And with the cloud infrastructure of the IoT rolled into the equation, things get much more complicated, especially as far as the second operation is concerned. An aggregation point on the cloud or in communication with it must collect and manage data from a large number of field devices, analyze it, and determine if it has actionable intelligence that must either be responded to quickly or passed along for analysis or action in the future. For example: aggregating vending machine data and determining where and when to take an immediate corrective action, or place a request for later repair, or just to refill or change what the machine is dispensing.