The Mirai botnet attack that enslaved poorly secured connected embedded devices is yet another tangible example of the importance of implementing security before bringing your embedded devices online. (A more recent strain of Mirai caused network outages to about a million Deutsche Telekom customers due to poorly secured routers). A secure and robust deployment approach for software updates to your connected device fleet is clearly one of a myriad security considerations that have to be taken into account before you go into production. This article covers specific requirements on deploying software updates to mitigate growing security risks.
In Code Complete: A Practical Handbook of Software Construction , Steve McConnell states that there are 1 to 25 bugs and vulnerabilities per 1,000 lines of code. The ability to deploy new features over-the-air (OTA) without having to recall the entire device fleet can save manufacturers substantially on costs. A good example of the problems associated with not having this capability was demonstrated by Fiat Chrysler recalling at least 1.4 million of their vehicles in 2014 in order to install software to prevent hackers from gaining remote control of the engine, steering, and other systems. Since Fiat Chrysler did not have the capability to deploy this software update OTA, the vehicles had to be updated “by hand” via USB drives.
Deploying updates could help avoid serious breaches, such as the large DDOS Mirai botnet attack, by securing devices remotely once a vulnerability has been found. Specifically, in the case of the Mirai botnet attack, video camera and digital video recorder devices had simple default passwords and many of them had the password hardcoded into the firmware. Thus, the only way to change them was to deploy an update and install a patch.
There are two primary approaches to updates. One is an image-based approach in which the entire image is updated, including the root filesystem. The other is package-based, which allows the user to implement changes at the file-level.
We conducted a survey of embedded developers and found that 45.5% preferred image-based updates, while 18.2% preferred the package-based approach. The remaining 36% were indecisive as to their preferred technique.
The most cited reason image-based updates were chosen was the confidence in device consistency: It helped lower the risk of device downtime and the developers claimed they were certain that what was being evaluated in their test environment was exactly the same as the code being deployed to devices in the field. This approach also eases the design burden for atomic installations as many use a failover partition if there is a problem mid-update such as a power loss or a network connectivity issue.
Package-based updates have the benefit of being more targeted, only modifying the small part of the device software that has actually changed with the update. This helps save bandwidth consumption and minimizes the time required to transfer and install. However, critics to this approach say the risk is too high for device inconsistency and it can be an enormous challenge to properly test with many moving pieces that increase the likelihood of errors.
A common embedded environment will have one or more of the following challenges that should be considered when choosing your update strategy:
- Difficult and/or expensive to reach/access physically.
- A long expected lifetime of 5-10 years.
- Unreliable power.
- Intermittent network connectivity.
These factors should be put into context with the aforementioned update approaches in order to choose which method makes the most sense.
Additionally, we have found there are four key considerations for one's software updater as follows:
- Being robust and secure.
- Integration into existing environments.
- Ease of getting started.
- Bandwidth consumption and downtime during updates.
Being robust and secure is an obvious requirement, which includes having an atomic and consistent deployment and installation; also, ensuring that the test environment is equal to the production environment. There also needs to be a “sanity check” after the update has taken place to ensure another update can be applied; “It boots” is not good enough. This also means that custom checks are required, such as the devices can reach the servers, the services are running, etc. Finally, there needs to be a way to verify the authenticity of the update. This can be achieved by utilizing digital signatures and RSA or elliptic curve cryptography (ECC).
The updater needs to integrate into existing environments as few projects have the luxury of starting from scratch; instead, they have existing development tools and build/test environments. Other aspects of this include the hardware (e.g., storage type(s), network type(s)) and operating systems (e.g., Yocto Project based, FreeRTOS), as well as potentially having devices already deployed in the field.
In order to integrate with existing deployments, the updater solution should offer a standalone mode where it supports updates via USB/SD. This is because the transition to remote updates may take time given there are many existing installations lacking connectivity.
Extensibility is another important consideration. The updater should allow custom update actions, as well as sanity checks and pre- and post-install scripts. What types of embedded systems will you have to support in the future?
A generic workflow for an embedded system update (Source: mender.io)
The ease of getting started is also important. How long does it take to test the updater on a reference board. Is the updater of high quality, well-tested, and well-documented? Campaign management is another consideration, as performing 1:1 software updates to each individual embedded system is too painful and error-prone of a process. Is there a way to group devices easily and a way to obtain reliable reports on device health.
A good measure to consider is how successful is the adoption of the updater by one's peers. Also, if the person responsible for the updating process leaves the team, will the process be easily executed by other team members? It is also important to consider whether the updater technology is a community-backed (open source) project or proprietary software and the implications that come with each.
For both bandwidth and downtime during the update, lower is clearly better, but requirements can vary widely between embedded systems. In the case of a safety-critical embedded device, does that system need to always be up? What about the maintenance window, and how long and frequent will be the updates? The network type — WiFi, cellular, SigFox, etc. — also needs to be taken into account.
Delta-based updates are an easily understood requirement, given the savings of bandwidth and related costs, as well as the time taken to perform the actual update. We have seen some interesting approaches to this, including using containers such as Docker to safely deploy delta-based updates. Container technology in the embedded world is still in its infancy, and most embedded developers we've spoken with are unfamiliar with this approach and are hesitant in deploying containers to devices.
Security is fast becoming the top concern when it comes to deploying IoT devices, and the software update process is a critical component of this. It's 2017 — there must be a way to efficiently and safely deploy software updates to remote devices. This article presented some high-level points, but there are many more aspects to consider and we'd love to hear your thoughts on all of this.
Eystein Stenberg has over eight years of experience in security and systems management as a developer, a support engineer, a technical account manager, a product manager, and now CTO at mender.io. He has been in the front lines of some of the largest production environments in various roles and has in-depth knowledge of the challenges in systems security in a real-world context. Eystein has spoken at various conferences, including the Embedded Linux Conference, the Embedded Systems Conference, the Automotive Linux Summit, and the Internet of Things World. He holds a Master's degree in cryptography and his writing credits include Distributing a Private Key Generator in Ad Hoc Networks.