Data formats change over time. To make upgrades easier, track these changes with version numbers.

Have you ever upgraded a package on your PC, only to find that all of the data files generated by previous versions of that product are no longer readable? With so many devices using flash memory to allow software upgrades in the field, a similar issue can arise on your embedded system. If data is stored in persistent memory (NVRAM or EEPROM usually), the format of that data may change from one version of the software to the next, to support new features. This month's column will look at how we can organize that data for a smooth transition.

Last month we discussed the use of checksums to confirm that data remained valid during a reset or power cycle.^[1] We also introduced the idea of breaking up the data into multiple blocks, each of which has its own checksum, to allow better management of data.

In a C program, the layout of the data in persistent store will usually be represented as a structure. The structure may be copied to/from serial EEPROM, or, if the persistent storage is mapped directly into the program's memory map, the structure simply lives in that area of memory. Listing 1 shows an example of such a structure. Because we want to be able to trust our calibration data even if the user settings become corrupt, we have broken the data into two groups: calibration and user settings. For simplicity, I do not show a second copy of the data as would be needed for double buffering, as discussed last month, but that technique could be combined with everything we will discuss in this column.

Listing 1: Structures to describe the layout of persistent store

typedef unsigned short uint16;

struct CalibrationData {
    float sensor1Offset;
    float sensor1Slope;
};

struct SettingsData {
    int speed;
    int alarmLimit;
};

struct PersistentStore {
    struct CalibrationData calib;
    uint16 calibCRC;
    struct SettingsData settings;
    uint16 settingsCRC; };

struct PersistentStore * persistent = (void *) 0x00FF;

The pointer, persistent, is declared to point to the place in the memory map where the persistent storage is located. If the space is memory mapped, there is no need to declare an instance of this structure; we can simply point at the space that always exists on the NVRAM chip. If this were a serial EEPROM implementation, you would probably declare an instance of the structure as a global variable, then copy that structure through the serial interface when required.

We want to be able to add other data to this structure in future versions. The first thing we want to ensure is that as the structure grows, it does not become too big for the physical medium on which it is being stored. In Listing 2, I check that the size of the structure is not greater than the number of bytes available, which in this case is 100. The CASSERT macro is a check that leads to a syntax error if the condition being checked is false. Thanks and credit to Bill Emshoff for telling me about this macro in response to a previous column.^[2]

Listing 2: How to check that your structures are not too big for the size of persistent store

#define CASSERT(ex) { typedef char cassert_type[(ex) ? 1 : -1]; }

int main(void)
{
    CASSERT(sizeof(struct PersistentStore) <= 100);
...

If we add a data member to CalibrationData, we face the problem that the location of all of the members of SettingData will move, probably by the size of the item that was added. If we load the software onto a device that was previously running an older version of the software, any of the fields after the new one will be corrupted. A problem will be detected immediately, since the CRCs will have moved, and the misread calibCRC value will not match the value calculated from the calib structure. Similarly, the check of settingCRC will also fail. This is fine if we accept that every time the software is upgraded, all of the persistent data will be lost and we will have to recalibrate the device.

In some environments, we want to do a bit better than this. This is only possible if we plan for it up front and leave some padding for the new fields. Listing 3 shows the same structures with padding inserted. By using the size of the structure in the calculation of the size of the padding, we have an array that gradually shrinks as the size of the structure grows. We also have to be sure that the padding is always included in the CRC calculation. Assuming we have a function called calcCRC() that calculates a CRC for a set number of bytes, Listing 4 shows a function to update the CRC after the value of a data member has been changed, and to check that the CRC is sane after a reboot. Note that we have set 20 bytes aside for the calibration section, and so we use all 20 bytes for the CRC or checksum, regardless of how many of those bytes are actually being used in the current version.

Listing 3: Using padding to ensure that the position of the CRCs doesn't change across versions of software

struct CalibrationData {
    float sensor1Offset;
    float sensor1Slope;
};

struct SettingsData {
    int speed;
    int alarmLimit;
};

#define CALIB_SIZE 20
#define SETTING_SIZE 30

struct PersistentStore {
    struct CalibrationData calib;
    char calibPadding[CALIB_SIZE - sizeof(struct CalibrationData)];
    uint16 calibCRC;
    struct SettingsData settings;
    char settingPadding[SETTING_SIZE - sizeof(struct SettingsData)];
    uint16 settingsCRC;
};

Listing 4: Setting and checking the CRC

void setCalibCRC(void)
{
    persistent->calibCRC = calcCRC(&persistent->calib, CALIB_SIZE);
}

int checkCalibCRC(void)
{
    if (persistent->calibCRC==calcCRC(&persistent->calib, CALIB_SIZE))
    {
        return true;
    }
    else
    {
        return false;
    }
}

Now each time we add a field, the padding will shrink, but the CRC and any following structures remain in the same place. The main problem with this scheme is that you have to guess how many extra bytes you are going to need. If we have broken up the persistent store into many separate blocks, for the reasons discussed last month, then we have to guess the amount of padding for each block. In the example shown here, if the padding for the calibration section were completely used up, it wouldn't be possible to borrow any space from the settings section, because that would cause the CRC and other fields to move. This can lead to cases where fields are put into the wrong block because there was no room in the block where it really belonged. This issue should be considered when you are deciding how many divisions you want to create in your persistent store.

Versioning

We now have a structure that will pass its CRC test after the new software has been installed. However, we still want to know that the upgrade happened. We need this information because we have fields living in locations that used to be part of the padding. The padding was probably all zeros at the beginning of the product's life so we may need to put some sensible initial values in those fields as soon as we start running a version of software that uses that area.

In some cases there might be an ad-hoc way of detecting the change. For example, if the value of a new field is zero, when zero is not a legal value for that field, we can assume that this is the first time running with the updated software. Such schemes quickly turn ugly when a number of upgrade paths are possible-you have to assume that some customers out there will have all the previous versions and they might upgrade to any version newer than the one they have.

A better option is to put a version number in the persistent store. There is a bit of a dilemma about where to place this version number. In some cases, in spite of the method just discussed, the CRCs might move. Reading the version number might tell us where they are for this particular version. If the version number is inside a block that has a CRC, then you do not know how to CRC the block before reading the version number, because we do not know the size of the block. Because of this possible catch-22, a reasonable approach is to store the version number in the first byte and the inverse of it in the second byte, as a check. Changes to the structures that follow will never reposition the version number, and the version number will never depend on the size of the blocks that we are checking against a CRC.

The version number of persistent store is different from the version number of software. It is possible that many releases of the software will be made that do not change the layout of the persistent store and, so, the version number stored there does not need to be upgraded. Any time that the layout of our structure is changed, we must update the constant in software that says which version of persistent storage is in use. We will call this constant the persistent store version number (PSVN). It is important to be aware that any version of software will have a PSVN defined as a compile time constant. The PSVN stored in persistent storage is the same as the PSVN of the program that wrote the data to that persistent store.

When the program boots, it needs to check the PSVN in persistent storage. If it doesn't match the compile time constant, we know that this version of software is being run for the first time on this device. It will be necessary to set any new fields to sensible defaults. The number of new values in the structure depends on how many versions there are between the current PSVN and the PSVN found in the persistent store.

Once you have set up reasonable values in all fields, update the PSVN stored in the persistent store to match the number defined in the current revision of software. In addition to setting the new values, the conversion process may involve massaging some of the old ones. On one project, the programmers changed the range of values that the user was allowed to enter. The old range was 0 to 60. The new range was 0 to 50. If the setting was 55 when you did the upgrade, then the first time the new version of software is run it will have an illegal setting, which, on this system, would cause an assert to fire, since there was no legal way for the user to set a value of 55 in the new version. The solution was to check the setting, and if it was above the new maximum, 50, then set it to be equal to 50. This meant that we were not preserving all of the settings exactly, but it was a reasonable compromise.

Other problems are caused by values that change their meaning. For example, if an enumerated type is reordered. For these types of changes, whcih require the persistent store to be updated, it is necessary to increase the PSVN for the new version of the program, even if the actual layout has not changed.

Pointers

In most cases you would not store pointers in the structures that are persistently stored. Pointers into RAM lose all meaning after a reset. Occasionally, pointers to ROM can be usefully stored. They may point to constant strings. In other cases there may be a set of structures in ROM. Each structure represents a mode of operation. Storing a pointer to the structure represents the current mode of operation. You can avoid pointers by storing the structures in an array and simply storing the index into the array. In that case you are paying the price of an array access every time the structure is used.

If you are one of the brave souls who decides to store pointers in persistent store, you should take note of the following warnings:

• Any recompilation may change the location of pointers in ROM, so the PSVN needs to be increased for every compile.
• The first time a new version of software is run, all of the pointer fields need to be set to defaults.