What's driving AI to the edge - Embedded.com

What’s driving AI to the edge


In 2020, we predict more than 750 million edge AI chips — chips or parts of chips that perform or accelerate machine learning tasks on-device, rather than in a remote data center — will be sold, representing US$2.6 billion in revenue. Furthermore, the edge AI chip market will grow much more quickly than the overall chip market. By 2024, we expect sales of edge AI chips to exceed 1.5 billion, possibly by a great deal. This represents annual unit sales growth of at least 20%, more than double the longer-term forecast of 9% CAGR for the overall semiconductor industry.

These edge AI chips will likely find their way into an increasing number of consumer devices, such as high-end smartphones, tablets, smart speakers, and wearables. They will also be used in multiple enterprise markets: robots, cameras, sensors, and other IoT (internet of things) devices in general.

The consumer edge AI chip market is much larger than the enterprise market, but it is likely to grow more slowly, with a CAGR of 18% expected between 2020 and 2024. The enterprise edge AI chip market is growing much faster, with a predicted CAGR of 50% over the same time frame.

Nevertheless, right now, in 2020, the consumer device market will likely represent more than 90 percent of the edge AI chip market, both in terms of the numbers sold and their dollar value. The vast majority of these edge AI chips will go into high-end smartphones, which account for more than 70 percent of all consumer edge AI chips currently in use. Hence in 2020 and for the next few years, AI chip growth will be driven principally by smartphones. We believe over a third of the 1.56 billion-unit smartphone market may have edge AI chips in 2020.

Because of the extremely processor-intensive requirements, AI computations have almost all been performed remotely in data centers, on enterprise core appliances, or on telecom edge processors — not locally on devices. Edge AI chips are changing all that. They are physically smaller, relatively inexpensive, use much less power, and generate much less heat, making it possible to integrate them into handheld devices as well as non-consumer devices such as robots. By enabling these devices to perform processor-intensive AI computations locally, edge AI chips reduce or eliminate the need to send large amounts of data to a remote location — thereby delivering benefits in usability, speed, and data security and privacy.

Keeping the processing on the device is better in terms of privacy and security; personal information that never leaves a phone cannot be intercepted or misused. And when the edge AI chip is on the phone, it can do all these things even when not connected to a network.

Of course, not all AI computations have to take place locally. For some applications, sending data to be processed by a remote AI array may be adequate or even preferred — for instance, when there is simply too much data for a device’s edge AI chip to handle. In fact, most of the time, AI will be done in a hybrid fashion: some portion on-device, and some in the cloud. The preferred mix in any given situation will vary depending on exactly what kind of AI processing needs to be done.

Figure 1: Locations where intelligence can be embedded (Image: Deloitte Insights)

The economics of edge AI in smartphones

Smartphones aren’t the only devices that use edge AI chips; other device categories — tablets, wearables, smart speakers — contain them as well. In the short term, these non-smartphone devices will likely have much less of an impact on edge AI chip sales than smartphones, either because the market is not growing (as for tablets) or because it is too small to make a material difference (for instance, smart speakers and wearables combined are expected to sell a mere 125 million units in 2020). However, many wearables and smart speakers depend on edge AI chips, so penetration is already high.

Edge AI chip market

Figure 2: The Edge AI chip market (Image: Deloitte Insights)

Currently, only the most expensive smartphones — those in the top third of the price distribution — are likely to use edge AI chips. Putting an AI chip in a smartphone doesn’t have to be price-prohibitive for the consumer.

It’s possible to arrive at a fairly sound estimate of a smartphone’s edge AI chip. To date, images of phone processors in Samsung, Apple, and Huawei show the naked silicon die with all its features visible, allowing identification of which portions of the chips are used for which functions. A die shot of the chip for Samsung’s Exynos 9820 shows that about 5% of the total chip area is dedicated to AI processors. Samsung’s cost for the entire SoC application processor is estimated at $70.50, which is the phone’s second-most expensive component (after the display), representing about 17% of the device’s total bill of materials. Assuming that the AI portion costs the same as the rest of the components on a die-area basis, the Exynos’s edge AI neural processing unit (NPU) represents roughly 5% of the chip’s total cost. This translates into about $3.50 each.

Figure 3: A die shot of the chip for Samsung’s Exynos 9820 shows that about 5% of the  total chip area is dedicated to AI processors. (Image: ChipRebel, Annotation: AnandTech)

Similarly, Apple’s A12 Bionic chip dedicates about 7% of the die area to machine learning. At an estimated $72 for the whole processor, this suggests a cost of $5.10 for the edge AI portion. The Huawei Kirin 970 chip, estimated to cost the manufacturer $52.50, dedicates 2.1% of the die to the NPU, suggesting a cost of $1.10. (Die area is not the only way to measure what percent of a chip’s total cost goes toward AI, however. According to Huawei, the Kirin 970’s NPU has 150 million transistors, representing 2.7% of the chip’s total of 5.5 billion transistors. This would suggest a slightly higher NPU cost of $1.42).

Figure 4: Apple’s A12 Bionic chip dedicates about 7% of the die area to machine learning. (Image: TechInsights / AnandTech)

Although this cost range is wide, it may be reasonable to assume that NPUs cost an average of $3.50 per chip. Multiplied by half a billion smartphones (not to mention tablets, speakers, and wearables), that makes for a large market, despite the low price per chip. More importantly, at an average cost of $3.50 to the manufacturer, and a probable minimum of $1, adding a dedicated edge AI NPU to smartphone processing chips starts looking like a no-brainer. Assuming normal markup, adding $1 to the manufacturing cost translates into only $2 more for the end customer. This means that NPUs and their attendant benefits — a better camera, offline voice assistance, and so on — can be put into even a $250 smartphone for less than a 1 percent price increase.

Sourcing AI chips: in-house or third party?

Companies that manufacture smartphones and other device types take different approaches to obtaining edge AI chips, with the decision driven by factors including phone model and (sometimes) geography. Some buy application processor/modem chips from third-party companies that specialize in making and selling them to phone makers, but do not make their own phones. Qualcomm and MediaTek are two prominent examples; combined, these two companies captured roughly 60 percent of the smartphone SoC chip market in 2018.

Both Qualcomm and MediaTek offer a range of SoCs at various prices; while not all of them include an edge AI chip, the higher-end offerings (including Qualcomm’s Snapdragon 845 and 855 and MediaTek’s Helio P60) usually do. At the other end of the scale, Apple does not use external AP chips at all: it designs and uses its own SoC processors such as the A11, A12, and A13 Bionic chips, all of which have edge AI.

Other device makers, such as Samsung and Huawei, use a hybrid strategy, buying some SoCs from merchant market silicon suppliers and using their own chips (such as Samsung’s Exynos 9820 and Huawei’s Kirin 970/980) for the rest.

Over 50 AI accelerator companies vying for edge AI in enterprise and industrial

If edge AI processors used in smartphones and other devices are so great, why not use them for enterprise applications too? This has, in fact, already happened for some use cases, such as for some autonomous drones. Equipped with a smartphone SoC application processor, a drone is capable of performing navigation and obstacle avoidance in real time and completely on-device, with no network connection at all.

However, a chip optimized for a smartphone or tablet is not the right choice for many enterprise or industrial applications. As discussed earlier, the edge AI portion of a smartphone SoC is only about 5% of the total area, about $3.50 of the total cost, and would use about 95 percent less power than the whole SoC does. What if someone built a chip that had only the edge AI portion (along with a few other required functions such as memory) that cost less, used less electricity, and was smaller?

Well, they have. In all, as many as 50 different companies are said to be working on AI accelerators of various kinds. The standalone edge AI chips available in 2019 were targeted at developers, who would buy them one at a time for around $80 each. In volumes of thousands or millions, these chips will likely cost device manufacturers much less to buy: some as little as $1 (or possibly even less), some in the tens of dollars. We are, for now, assuming an average cost of around $3.50, using the smartphone edge AI chip as a proxy.

Besides being relatively inexpensive, standalone edge AI processors have the advantage of being small. They are also relatively low power, drawing between 1 to 10 watts. For comparison, a data center cluster (albeit a very powerful one) of 16 GPUs and two CPUs costs $400,000, weighs 350 pounds, and consumes 10,000 watts of power.

With chips such as these in the works, edge AI can open many new possibilities for enterprises, particularly with regard to IoT applications. Using edge AI chips, companies can greatly increase their ability to analyze — not just collect — data from connected devices and convert this analysis into action, while avoiding the cost, complexity, and security challenges of sending huge amounts of data into the cloud. Issues that AI chips can help address include:

Data security and privacy. Collecting, storing, and moving data to the cloud inevitably exposes an organization to cybersecurity and privacy threats, even when companies are vigilant about data protection. This immensely important risk is becoming even more critical to address as time goes on. Regulations about personally identifiable information are emerging across jurisdictions, and consumers are becoming more cognizant of the data enterprises collect, with 80 percent of them saying that they don’t feel that companies are doing all they can to protect consumer privacy. Some devices, such as smart speakers, are starting to be used in settings such as hospitals, where patient privacy is regulated even more stringently.

By allowing large amounts of data to be processed locally, edge AI chips can reduce the risk of personal or enterprise data being intercepted or misused. Security cameras with machine learning processing, for instance, can reduce privacy risks by analyzing the video to determine which segments of the video are relevant, and sending only those to the cloud. Machine learning chips can also recognize a broader range of voice commands, so that less audio needs to be analyzed in the cloud. More accurate speech recognition can deliver the additional bonus of helping smart speakers detect the “wake word” more accurately, preventing it from listening to unrelated conversation.

Low connectivity. A device must be connected for data to be processed in the cloud. In some cases, however, connecting the device is impractical. Take drones as an example. Maintaining connectivity with a drone can be difficult depending on where they operate, and both the connection itself and uploading data to the cloud can reduce battery life. In New South Wales, Australia, drones with embedded machine learning patrol beaches to keep swimmers safe. They can identify swimmers who have been taken by riptides, or warn swimmers of sharks and crocodiles before an attack, all without an internet connection.

(Too) big data. IoT devices can generate huge amounts of data. For example, an Airbus A-350 jet has over 6,000 sensors and generates 2.5 terabytes of data each day it flies. Globally, security cameras create about 2,500 petabytes of data per day. Sending all this data to the cloud for storage and analysis is costly and complex. Putting machine learning processors on the endpoints, whether sensors or cameras, can solve this problem. Cameras, for example, could be equipped with vision processing units (VPUs), low-power SoC processors specialized for analyzing or pre-processing digital images. With edge AI chips embedded, a device can analyze data in real time, transmit only what is relevant for further analysis in the cloud, and “forget” the rest, reducing the cost of storage and bandwidth.

Power constraints. Low-power machine learning chips can allow even devices with small batteries to perform AI computations without undue power drain. For instance, Arm chips are being embedded in respiratory inhalers to analyze data, such as inhalation lung capacity and the flow of medicine into the lungs. The AI analysis is performed on the inhaler, and the results are then sent to a smartphone app, helping health care professionals to develop personalized care for asthma patients. In addition to the low-power edge AI NPUs currently available, companies are working to develop “tiny machine learning”: deep learning on devices as small as microcontroller units. Google, for instance, is developing a version of TensorFlow Lite that can enable microcontrollers to analyze data, condensing what needs to be sent off-chip into a few bytes.

Low latency requirements. Whether over a wired or wireless network, performing AI computations at a remote data center means a round-trip latency of at least 1–2 milliseconds in the best case, and tens or even hundreds of milliseconds in the worst case. Performing AI on-device using an edge AI chip would reduce that to nanoseconds — critical for uses where the device must collect, process, and act upon data virtually instantaneously. Autonomous vehicles, for instance, must collect and process huge amounts of data from computer vision systems to identify objects, as well as from the sensors that control the vehicle’s functions. They must then convert this data into decisions immediately — when to turn, brake, or accelerate — in order to operate safely. To do this, autonomous vehicles must process much of the data they collect in the vehicle itself. Low latency is also important for robots, and it will become more so as robots emerge from factory settings to work alongside people.

The bottom line: edge AI will be vital for data-heavy applications

The spread of edge AI chips will likely drive significant changes for consumers and enterprises alike. For consumers, edge AI chips can make possible a plethora of features — from unlocking their phone, to having a conversation with its voice assistant, to taking mind-blowing photos under extremely difficult conditions, and without the need for an internet connection.

But in the long term, edge AI chips’ greater impact may come from their use in enterprise, where they can enable companies to take their IoT applications to a whole new level. Smart machines powered by AI chips could help expand existing markets, threaten incumbents, and shift how profits are divided in industries such as manufacturing, construction, logistics, agriculture, and energy. The ability to collect, interpret, and immediately act on vast amounts of data is critical for many of the data-heavy applications that futurists see as becoming widespread: video monitoring, virtual reality, autonomous drones and vehicles, and more. That future, in large part, depends on what edge AI chips make possible: bringing the intelligence to the device.

Duncan Stewart and Jeff Loucks are with Deloitte’s Center for Technology, Media and Telecommunications. This article is based on an article originally published by Deloitte for its TMT Predictions 2020 report.

>> This article was originally published on our sister site, EE Times.


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.