There are many differences between some of today’s embedded systems andthose of days gone by, and some of the underlying changes havesignificant unexpected consequences. Perhaps the greatest change isthat increasingly modern embedded systems are based not on closedproprietary platforms but on commodity OS platforms. As a consequence,such embedded systems share all the security vulnerabilities of generalpurpose systems built on the same OS.
In Part1 and Part2 of this three part series, we showed the feasibility of usingsystem diversity techniques to insulate embedded systems from thesecurity vulnerabilities of an underlying open platform. But what doesis take to actually diversify a set of systems in practice? It turnsout that “managing change” is the tricky bit in diversification. Oncethe software of an embedded system needs to change, how do we expect tochange it?
Three broad types of embedded systems can be defined from theperspective of how the vendor intends to handle software upgrades tothe fielded systems, with each type requiring a slightly differentapproach:
1. systems that are notintended to receive field upgrades;
2. systems that will receivefield upgrades via the vendor physically accessing the systems; and
3. systems that are intended toreceive controlled field upgrades over the Internet or other network bya service technician.
No field upgrades
An embedded system may be a “fixed-function device,” such as amanufacturing process controller, where the product definitionintentionally excludes field upgrades. Note that this does not implythat the system is incapable of being modified in the field. Quite thecontrary, the fact that the system is built upon an open platform meansthat there are many ways in which the software can be modified. It issimply the vendor’s business model that excludes field upgrades.
The vendor expectation is that changes to the systems are only forthe repair of field defects, not for product upgrades, and repairs andreplacements are to be handled by the systems returning to the vendor.Since field defects are expected to be rare, there is no need to planand provide an interface for large scale change in the field. However,the reality is that the open platform allows the systems to drift andbreak in the field, causing the number of returned systems to climbsubstantially higher than expected.
Open platforms do allow myriad ways to modify the system, includingaccidental changes by customer staff, well-intentioned but harmfulchanges by administrative staff, malicious hacking, automatic updates,etc. One real-world example applies to a range of basic network storagedevices that are essentially storage management software running on anopen OS, with the automatic update features of the OS disabled by thevendor to prevent change in the field. In a customer network, however,the updates can be turned on, and the updates happen automatically, theembedded software misbehaves, and the device goes back to the vendorunder warranty.
The flip side of the above real example occurs when the OS vendor(or vendors of other third party software in the embedded system)provide a patch for a newly discovered security vulnerability. Sinceno-field-upgrade systems are only modified in cases of breakage, theyremain vulnerable to attack, until and unless an attack occurs thatcauses visible breakage that appears to be a field defect. It’s badenough that the embedded systems can break, but while they remainvulnerable they are a threat to other customer systems well, despitethe customer’s most diligent efforts to patch the systems as a responseto the discovered vulnerability.
It’s not that the vendors are careless or lazy. The patch issue isactually not simple. A vendor may have multiple different embeddedsystem products on a given open platform, with various versions andpatch levels. Every issued patch has to be analyzed to see whichproducts it may apply to, and tested to determine if thegeneral-availability patch might impair the embedded software, as isoften the case especially if the patch is applied to a system that isnot at the exact patch level for which the patch itself was defined.
For relatively low-cost embedded systems for which field upgradeswere not anticipated, it is simply infeasible for the vendor to engagein complex patch management – even if it were feasible to patch everyfielded system.
So, can diversity techniques help here? The answer is yes. Since thecode base of the embedded system is not intended to change in thefield, the white list mechanism works very well, with the addedconvenience that the list itself requires no maintenance. Furthermore,the fact that the code base of the embedded system is notfield-upgraded means that once a vulnerability is discovered, thefielded systems remain exposed as long as they remain in the field.
Therefore, it is highly desirable to implement memory executecontrols and memory layout randomization mechanisms on the systemsbefore they ship, so that when future vulnerabilities are discovered,they cannot be exploited on the embedded system even if the systemremains unpatched indefinitely.
Field upgrades with physical access
Another type of embedded system is one where the vendor expects toupgrade the device in the field, from time to time, in order to providecustomers with desirable additional features. This upgrade modelleverages the facilities of the open platform to give vendors theconvenience of field upgrades.
An automated teller machine (ATM) is a good example of such asystem, with upgrades applied in the field by a technician withphysical access to the system. The downside is that when a patch isreleased for a vulnerability in the platform, a technician must travelto the field to apply the patch. And as is the case withno-field-upgrade devices, platform patch announcements must result insome kind of manufacturer activity, regardless of whether the systemsend up getting patched or not.
This patch analysis, triage, test, and re-package activity isexpensive, does not scale well with the number of fielded systems, andcontributes to customer risk exposure. Nevertheless, today manyembedded systems vendors pay this cost of using open system platforms,but with misgiving as the cost rises over time with increasing threats.But once again we can apply diversity techniques.
A white list implementation can be used to lock down the code baseof the embedded system. The white list itself must allow modification,because field updates to the embedded system must be accompanied bymatching updates to the white list. This in turn requires a lockingmechanism for the white list in order to prevent tampering andinclusion of arbitrary (and possibly malicious) code.
Also, as in the previous case, memory execute control and memorylayout randomization techniques can be used to further reduce risk byensuring that future vulnerabilities are not exploitable. In thesecases, application of patches can be deferred until the next scheduledmaintenance, while also eliminating the window of vulnerability betweenpatch announcement and hasty patch deployment that can requireunscheduled downtime and risk breakage from patch incompatibility.
In these cases, diversity techniques can overcome many of thegrowing challenges of patching today: For example, there is no easyaccess to some embedded systems, such as geographically dispersedsystems on board of aircraft and ships and distributed devices inphysically hard-to-reach locations.
Other embedded systems must be recertified after a patch in order tomaintain regulatory compliance. It is simply more cost effective andbetter business practice to apply preventative diversity techniques andremove the need for reactive field patching altogether, and insteadupgrade when it benefits the business, not when it is dictated byforces outside the system vendor’s control.
Field upgrades over a network
In this upgrade model, the embedded systems vendor further leveragesthe open platform facilities to do upgrades without having tophysically access the system.
Networked embedded systems such as many safety critical medicalinstruments use just such a product model, as do some sensitivemanufacturing or financial systems. Field upgrades typically requirephysical access to a local network and hence physical access to thefacility (though not to each individual embedded system at thefacility).
In addition, customer effort may still be required for a networkupgrade, since many customers restrict network access as a matter ofpolicy and therefore may need to grant special access to vendor staffon site, or may even insist on performing the upgrades themselves.
Such customers often regard the upgrade methods – rightly or wrongly– as cumbersome remedies for suboptimal embedded system design. But theconvenience of network upgradeability does help considerably, since thephysical travel is painful enough and in many cases it is simply notpractical to walk the halls of the customer’s facilities in order tophysically find and access all the embedded systems.
On the far end of the spectrum, some vendors are starting toexperiment with network-only upgrades via leased lines orInternet-based virtual private network links, eliminating the need tovisit customer facilities. Unfortunately, this approach raises thecustomer-perceived risk to the embedded systems. The dilemma here isthat the more an embedded systems vendor tries to mitigate the costs ofdoing reactive patching, the more risk (real or perceived) is shiftedto the customer. tools
Such perceived risks include exactly those that can be mitigatedwith diversity: susceptibility to known and unknown remote partiesattempting to execute unauthorized code or modify the system over thenetwork, with possible results ranging from system unavailability tomalfunction to loss of critical data.
Once again system diversity techniques can be applied to retain thebenefits of open platform development while at the same time enjoyingthe security and stability of a closed system. The same diversitytechniques described above with the physical-access-upgrade model canbe applied here, with similar results and benefits. In addition, sincethe embedded system code base and the white list are network upgraded,additional communications and control software is needed for theupgrade staff to interface with the white list locking and maintenancemechanisms on the embedded system.
Life can be simplified by adopting the practice of developing on opensystems, then deploying it as a stable and secure closed system. Andtheway to do this is by leveraging known system diversity techniques totransition from open to closed. If you are lucky, you can get somesystem diversity benefits in each of the above embedded systemscategories without an undue level of technical effort, with an OS thatpermits the inclusion of the described mechanisms without actuallyhaving to directly modify OS code.
|Figure1 – Embedded system vendor managing own diversity tools|
For a variety of types of embedded systems and open OS platforms,these benefits can even be obtained with reasonable system engineeringefforts, where the embedded systems vendor manages the combination ofthe base OS together with code that implements diversity techniques(see Figure 1, above) .
So is there a catch? Yes, but a paradigm shift rather than a catch:you must not fall into the trap of adding system diversification as yetanother expertise to develop in order to remain competitive. The beautyof diversification is that it shifts the frame on security and riskaway from systems and security technology to defining the fieldlifecycle of your embedded products.
To readPart 1, go to “ Safetyrequirements: vulnerabilities, patching, and band-aids.”
To read Part 2, go to “ Safetythrough openness and diversification.”
E. John Sebes is the CTO of SolidcoreSystems, a provider of a line of system diversity products tailoredfor a wide range of hardware, operating systems and deploymentenvironments.