Managing the addition of oil to a diesel engine's fuel-oil mix turns out to be a tricky problem. Here's one design that worked well.
Welcome to the first edition of Inside Look, a periodic column that takes an in-depth look at the design of a successful, unsuccessful, or just plain interesting embedded system. This isn't a marketing pitch. We're scrutinizing the technical aspects of the design, to find out what went right, what went wrong, and what would go differently if the designers had the opportunity to do it all over again.
This month features a device called Centinel, a system that works to eliminate oil changes for big diesel trucks and other diesel powered equipment. This product has been an amazing success, and chances are you've passed or been passed by trucks with this hardware built in. Enjoy the inside look.
The Centinel Advanced Oil Management System is an embedded system that extends oil change intervals on electronically controlled diesel engines by periodically removing a small amount of used oil from the engine's crankcase and replacing it with fresh oil. The used oil is sent to the engine's fuel tank, where it is blended with the fuel and burned during normal combustion.
Centinel allows diesel-powered trucks, tractors, generators, and other equipment to spend more time at work, and less time in the shop for oil changes and other routine maintenance. And by burning the used oil as fuel, Centinel also saves money and reduces environmental damage by eliminating the need to dispose of used oil. This idea is not new. Mechanical systems with similar functionality have been around for years, but today's sophisticated engine controllers and strict air quality standards make an all-mechanical solution impractical. Centinel is better than mechanical systems at maintaining engine oil quality because it contains a sophisticated algorithm that replaces oil at a rate proportional to the engine's workload, and also because it adds fault detection logic that cannot be implemented mechanically.
Problems to solve
From the beginning, Centinel's designers knew that the device needed to be reliable. Significant damage can occur to an engine if the oil quality deteriorates or the oil level drops, and since Centinel removes oil from an engine, it must take precautions to ensure that the oil level and quality are maintained throughout the process. This problem was trickier than it sounds, because inexpensive oil level sensors capable of surviving the harsh conditions found inside of a diesel engine did not exist at the time. Centinel also had to be durable. On-highway diesel trucks routinely travel as many as 200,000 miles a year in all types of climates and conditions, and industrial machinery, like construction equipment, often goes without maintenance for months at a time (although this is not generally considered a good idea). Furthermore, commercial diesel applications seek to maintain the highest productivity possible, which means they cannot stand idle while an "accessory" like Centinel is down for repairs. Centinel's software quality was also an issue, since the microcontroller would be an one-time-programmable part: the firmware could not be updated once the system was manufactured. A defect in software found after production would result in the replacement of all existing Centinel units.
Finally, the product had to be affordable. The first Centinel units were to be offered as an aftermarket, owner-installed device, meaning its sales price had to be significantly less than the savings it could achieve. This made for an aggressive cost target.
Centinel is an electromechanical design featuring a TMS370 microcontroller, an SAE J1587 automotive datalink interface, a mechanical valve assembly to transfer oil in and out of the engine, sensors to measure oil levels, and a tank to hold fresh oil.
During normal operation, the microcontroller reads engine information from the J1587 datalink to determine the engine's current workload and to detect the occurrence of various engine- and oil-related system faults. This information is used to calculate an oil burn rate, which is then trnaslated into a series of electrical pulses that cause the oil transfer valve assembly to transfer used oil from the engine's crankcase to the fuel tank. The oil transfer valve's design combines the used and fresh oil pistons into the same mechanical package, yielding a device that reliably transfers a fixed, known quantity of used oil to the fuel tank, and replaces it with an identical quantity of fresh oil on the return stroke. The valve and related plumbing are heated by the engine, which helps keep the oil flowing even in the coldest climates.
A sensor indicates when the fresh oil reservoir is empty, and Centinel will postpone pulses until additional oil is provided. The engine operator must still monitor the dipstick in the engine's crankcase, and manually add oil as necessary to replace the small amount the engine burns internally during normal operation.
Texas Instruments' TMS370 microcontroller forms the foundation of the design. This controller was selected for its low price, high integration, and availability (at the time, competing chips from other vendors had lead times that were beyond the intended project completion date). Other chip selection criteria included temperature range, number of onboard timers, amount of memory (RAM, ROM, and EEPROM), and the chip's packaging options.
The variant of the TMS370 chosen sports 256 bytes of onboard RAM, 8KB of OTPROM, 256 bytes of EEPROM, a serial communications interface (SCI), two 16-bit counters, 23 digital inputs, an 8-channel analog-to-digital converter (ADC), and a 12MHz system clock. Additional components in the Centinel design include power conditioning and power failure detection circuitry, and an SAE J1587 datalink interface (similar to an RS-485 network) to connect to the engine's communications datalink.
To improve reliability and durability, all of Centinel's data inputs are extensively qualified in both hardware and software before the values are used. Clever hardware detects open and short circuits on inputs and outputs, and careful range checking of all input values prevents improper operation in the event of tampering or electrical failure. The primary control algorithm runs on a fixed 20ms interval, and combines engine workload, climate, and other information received from the engine datalink to compute the proper oil burn rate. The oil replacement rate, therefore, varies in real time, and seeks to maintain oil quality at a level that approximates a typical 20,000 mile oil change interval, regardless of the engine's actual duty cycle.
Once the proper oil burn rate is calculated, the value is passed to a second algorithm that computes the rate at which pulses must be sent to the oil transfer valve assembly in order to actually burn and replace the requested volume of oil. The logic at this stage is challenging, because pulses are postponed when the fresh oil tank is empty, the oil is cold, or a system fault is detected. When the operator refills the fresh oil tank, the oil warms, or the fault is fixed, pulses are sent at an accelerated rate in order to "catch up" to the requested volume.
To minimize air pollution concerns, EPA guidelines limit the amount of oil that can be burned as fuel in an automotive diesel engine. To prevent Centinel from exceeding this limit, a third algorithm restricts the maximum pulse rate sent to the oil transfer valve under all circumstances, so much so that in extreme cases, the recovery of delayed pulses can take hours, or even days to complete.
On startup, the software configures the microcontroller's I/O hardware, timers, serial communications port, ADC, and other peripherals. It then enters an endless do-nothing loop. Interrupt service routines contain the bulk of Centinel's code, a common approach in applications that need multithreaded behavior yet lack an operating system scheduler.
A periodic timer interrupt initiates the main control loop code, which resets a hardare watchdog timer only on successful completion. By resetting the watchdog timer at the end of the computation, a natural immunity to overruns is imposed: if the code ever takes too long to execute, the watchdog reset code gets interrupted by the next iteration of the algorithm, and eventually the watchdog timer expires.
Another interrupt service routine receives engine datalink information from the serial port. This code includes a simple state machine to parse the data and checksum fields of the incoming packets; packets with invalid checksums or out-of-range data are discarded. A timer measures the duration between successfully received packets and halts the burn rate computation if an excessive delay-indicating a cut datalink cable-is detected. This prevents Centinel from transferring oil when it lacks the engine information needed to determine what the proper burn rate should be.
Other interrupt service routines force periodic sampling of the TMS370's onboard ADCs, and notify Centinel of an impending power failure induced when the operator shuts down the engine (Centinel is powered by the ignition switch).
Unused interrupt vectors are filled with pointers to the startup function, essentially rebooting the system when an unexpected interrupt is detected.
Centinel's EEPROM is used to store important persistent data, like the number of oil pulses needed vs. the number actually sent. Unfortunately, these values cannot be updated on every iteration of the control loop, because doing so would surpass the EEPROM's write cycle limit well before the end of Centinel's expected lifetime. Instead, data is mirrored in RAM and written to EEPROM only at shutdown, during a 200ms holdup provided by capacitors built into Centinel's power supply circuitry.
The holdup time provides a good balance between reliability and cost (longer values would require more capacitance), but is critically close to the EEPROM's write cycle time, which means that there is virtually no room for unexpected delays during the shutdown process. And since the EEPROM contains vital information about the state of the oil burning process, its contents must be maintained under all real-world conditions other than outright device failure.
The shutdown and EEPROM updating process was reviewed and revised almost continuously during development. Particular attention was given to the logic that induced the writing process, the order in which bytes were written, checksums, and how much data was queued before writing began, to make sure that the amount of time required to perform the write operation was absolutely bounded, and to help assure that the system would reawaken in a safe state if an unanticipated error or early power failure somehow did occur.
One unexpected surprise during early field testing was that the excessively high-powered (in some cases illegally so) CB radios favored by many truckers could induce voltage swings of nearly 30V peak-to-peak on the truck's main power system (typically a 12V DC supply). These voltage spikes would trip Centinel's impending-power-failure circuitry, triggering excessive EEPROM writes and generally compromising system performance. To combat this, additional hysteresis code had to be added to Centinel's already extensive set of signal analysis logic, to prevent false power fail indications from initiating an avalanche of EEPROM updates when the radio was in use.
Process was key
One lesson that the Centinel development team learned was the benefits of good software design and process. Development tools were unexpectedly late in arriving, which gave the team further opportunity to refine Centinel's implementation on paper before coding began. Structured design tools were employed, along with use-case analysis and failure mode studies, so that by the time coding started the requirements were fairly complete.
By resisting the urge to start writing code too early, Centinel's software team produced an application that was remarkably free from defects and missed requirements, and even contained unspecified, supplemental logic to combat what were viewed to be the most likely bugs, such as the EEPROM overrun example mentioned previously. What might have been perceived as a costly delay actually produced a better quality product in the end, with minimal long-term impact to the project schedule.
A peer review process was also employed, and all work products like code and circuit designs were studied by other developers before they became part of the final solution. These "extra eyes" not only helped uncover bugs and missed requirements, it also fostered a sense of teamwork that let more people within the company participate in Centinel's success.
Centinel is one of Cummins Engine Company's most successful products, apart from their award-winning family of electronically controlled diesel engines. Even after more than five years of production, the Centinel system has yet to need major improvements or revisions.
The Centinel design won an OEMmie award from OEM Off-Highway Magazine, and is largely regarded within Cummins as an example of how their continuous improvements in system engineering practices are yielding superior products. In fact, the design is so successful, it has served as a platform for several other new products, some of which will undoubtedly be featured in future Inside Looks.
Bill Gatliff is an independent embedded systems consultant, a former Cummins employee, and a contributing editor to ESP. If you have a product you would like to see featured in a future Inside Look, please contact him at firstname.lastname@example.org
Paul Cantrell is a management consultant with Pittiglio, Rabin, Todd, and McGrath. He is a former employee of Cummins Engine Company, and was the chief architect and developer of Centinel's firmware.
The authors wish to acknowledge the efforts of Andy Pajakowski, Alex Knight, and Gary Gron, all of Cummins Engine Company, for their assistance in preparing this article.
Currently no items