Sensor fusion: Of sea captains, gyroscopes, and thermal cameras
For as long as I can remember, the term "sensor fusion" has always been the holy grail of data processing—a mythical unicorn mentioned frequently but (more often than not) in a confusing context. So, what is sensor fusion? The answer is not straightforward because this term can mean different things to different people. In this article, we will cover the general concept of sensor fusion, look at the historical perspective, and examine two detailed cases—inertial sensor fusion and image fusion.
Sensor fusion is one aspect of a larger field called data fusion which encompasses various ways of combining data (i.e. raw signals from sensors) to either a) refine the quality of information (i.e. analyzed data) received from a single source or b) derive new information which would not have been available otherwise. In the context of data fusion, a sensor simply acts as a source of data to be fused to produce information.
In everyday life, one could consider calculating monthly expenses to be a form of sensor fusion. Let’s say your data comes from several credit cards. The credit card company would become a sensor since fusing the data means assembling negative and positive values from various sources into a single table to create a unified picture and also derive a new piece of information—the monthly total. To take it further, we could derive other pieces of information from our credit card sensor fusion, such as a histogram that would categorize expenses by their type, or perhaps show a distribution of expenses across the month.
Although it is considered to be a modern (and even trendy) concept, sensor fusion has existed since ancient times in various forms. Sea captains traveling great distances relied on a navigation technique known as "dead reckoning," an approach that required the combination of a chronometer and a compass with speed estimation to derive a ship’s latitude and longitude. And even before the chronometer was invented, a sextant was used to determine a ship's position based on the angles between celestial bodies. Captains would fuse their sensor inputs to plot the course and find their way long before the first computer gave its first tick.
Fast-forwarding to a much nearer past, early color movies used a technique where every frame was exposed on two black-and-white strips, each behind a different red-and-green filter. As early as 1906, a technique was invented to unite those images by applying complementary tints, thus creating the colored picture, another form of sensor fusion.
Now let’s make a swooshing sound and quickly zoom into the modern age. A boy is out in a field, holding an iPhone. From time to time, he touches the screen with his finger. If we follow his gaze, we can see a small contraption hovering a few feet up in the air, making a buzzing sound. You and I, being creatures of the modern age, can easily recognize the small drone as it is being controlled by the boy via Wi-Fi. However—and here’s the wonder—the boy’s control over the drone is very imprecise, and his adjustments are crude and loosely timed. So, how does the drone keep a steady position in the air? How does it hover? How does it move in such a beautiful straight line? The answer is both simple and complicated at the same time, and it has to do with inertial sensors fusion.
Here’s a riddle: what do a dreidel, a kid’s toy, a fighter jet, and Put and Take (a gambling game from the First World War) all have in common? The answer is a spinning top—an object that follows the physical law of conservation of angular momentum. Simply, this means that a body rotating around its central axis strives to maintain its orientation along that axis. This is the same principle used in throwing an American football and also the reason why gun barrels are threaded—rotating bullets are more precise in maintaining trajectory.
A gyroscope (or gyro) is a kind of a spinning top combined with a measurement device that is used to measure angular changes; while earlier aircraft versions from not so long ago used actual spinning tops, modern versions could include a vibrating nano-block or even just a beam of light going round and round inside an optic fiber.
The gyro operates on the following principle: assuming the gyro rotates around its axis (i.e.the “main axis”), any angular change around an axis orthogonal to the main axis (i.e. the “input axis”) will produce a proportional change around a third orthogonal axis (i.e. the “output axis”). By observing any change around the expected output axis, we can measure the amount of change that happened around the input axis of the gyro (angle θ). The derivative of those changes over the time domain will produce a measurement of angular velocity (ω) and a second derivative will produce angular acceleration (α).
Inside a drone, there are at least three tiny gyroscopes, set to measure any angular change along the orthogonal axes. All three of them are most likely contained within a single package that’s the size of the tip of your pinky; this was made possible by the introduction of micro-electro-mechanical systems (MEMS) technology, which enabled the production of unbelievably tiny components thus reducing the size of the gyroscope. Why is it important, you ask? Quite simple: this is what allows us to keep the drone stable.