Arm has launched its latest image signal processor (ISP) in the Mali family, which it said is the smallest and highest performance ISP from Arm to date.
Compared to its existing ISPs, the new Arm Mali-C55 offers upgraded image quality features, works under a wide range of different lighting and weather conditions, and is designed to enable maximum performance and capability in area and power constrained applications, making it suitable for smart camera and edge AI vision use cases in various markets. This includes surveillance and security cameras, which with this improved capability could then detect more critical detail, such as recognizing the exact information on license plates traveling at up to 75mph; home cameras and security systems can capture higher resolution images both inside and out; and smart home hubs can efficiently include advanced features like secure visual unlock.
For developers, a complete software package is also available for controlling the ISP, plus a full set of tuning and calibration tools to help partners achieve the desired image quality for their application.
The new ISP meets the needs of vision applications for wired or battery powered devices at different power envelopes in challenging lighting and weather conditions. Mali-C55 can be seamlessly integrated in SoC designs with either Cortex-A or Cortex-M CPUs as it includes industry standard AXI and AHB interfaces. A range of configurability options are also offered, providing an area-efficient implementation to support targeted use cases. For example, a computer vision-only application would not need the same level of image-processing as a full color display output. With Mali-C55, specific image processing functions can be left out or individually simplified in a modular fashion, resulting in an area optimized silicon footprint.
The ISP also comes with improved image quality due to updates to the color reproduction and noise reduction functions and has higher performance with up to 1.2Gpix/sec throughput.
With multi-camera capability for up to 8 separate inputs, support for image resolutions up to 8K and a maximum image size up to 48 megapixels (MP), the Mali-C55 offers an efficient combination of image quality, throughput, power consumption and silicon area. For applications that require greater than 48 MP capabilities, such as video conferencing, multiple Mali-C55 ISPs can be combined to enable larger image sizes.
Mali-C55 builds on the previous generation Mali-C52 ISP, to offer better image quality through features including improved tone mapping and spatial noise reduction, enhanced support for high dynamic range (HDR) sensors and seamless integration with machine learning accelerators to take advantage of neural networks for various de-noising techniques. Enhancements are provided for the following functions:
- Arm Iridix local tone mapping
- Arm Temper temporal noise reduction
- Arm Sinter spatial noise reduction
- High Dynamic Range (HDR) sensors support
Iridix local tone mapping is the process of applying intensity transformations to images to achieve better visualization by using information gathered from local regions within images. Iridix defines these local regions in an image as grids with equal sizes. It extracts statistics from each grid to apply the collected statistics to the corresponding local regions in the image. Compared to Mali-C52, Mali-C55 improves the Iridix local tone mapping algorithm by smoothing each local tone curve therefore enabling a more natural fall-off around bright light sources.
Temper is a temporal noise reduction algorithm that improves the quality of images in low light conditions by combining consecutive frames. Mali-C55 not only improves the image quality with updated noise reduction algorithms but achieves this with up to 50% reduced memory bandwidth compared to Mali-C52.
Sinter 2.6 is an improved spatial noise reduction technique that improves the detail and noise balance in color channels. Compared to Mali-C52, Mali-C55’s Sinter achieves better balance of detail by using specific registers for each color channel.
The Temper and Sinter functional blocks were designed to work together for significantly better image quality by sharing information between the modules to apply stronger noise reduction in various regions. The Temper and Sinter block order is switched in the pipeline compared with previous ISP designs. This way the input motion mask from Temper improves the overall motion-adaptive noise-reduction performance, while providing per-plane noise profiling.
For computer vision applications where high throughput and low latency is required, the Mali-C55 ISP can be configured by either enabling or disabling the above-mentioned features. Arm said Mali-C55’s configurability means silicon partners and OEMs can easily add or remove features in line with their application requirements. For example, a low-cost home camera system may require a simple, limited set of features whereas a commercial camera might demand more sophisticated capabilities such as high resolution, noise reduction and enhanced security.
For embedded and IoT vision applications, silicon footprint and cost are important factors, and with the Mali-C55 Arm said it has provided these enhanced features in almost half of the silicon area size of previous generations, significantly lowering power consumption for extended battery life, and in the process also lowering the cost of these devices.
ML processing addresses bandwidth issues
In modern SoCs, the output of an ISP is directly connected to a machine learning accelerator for further processing via neural networks or other ML algorithms. This usually means providing downscaled images for ML models to detect and identify objects, or even attempting more complex inferences such as pose estimation. Mali-C55 provides a second output pipe that can output downscaled images suitable for input to any machine learning accelerator. Pairing an ISP with a machine learning accelerator adds ML capabilities to the image-processing pipeline by being able to use other transformer models as denoising techniques, which further enhances image quality.
With the increasing demand for higher resolutions, memory bandwidth is expected to be a key bottleneck in upcoming vision orientated SoCs. Adding a machine learning accelerator to such an SoC can further increase the bandwidth requirements, thus exacerbating the issue. Arm said Mali-C55 enables integration with other products such as Arm Frame Buffer Compression (AFBC), that can address such bandwidth issue and provide great benefit at the system level.
Arm also said it expects Mali-C55 ISP to be a key component for the flexible and capable compute systems contributing to the new generation of ML-enabled or software defined cameras. Key hardware capabilities are provided up-front, but with sufficient ML and general-purpose processing to improve or enable additional software features as time progresses. Subscription and other revenue models are being developed around these dynamic feature capabilities.
A software package is available for Mali-C55 licenseesfor controlling the following functions:
- Auto White Balance, Auto-Focus, and Auto Exposure
- Bare-metal and Linux (Video4Linux framework – V4L2) support
ISP users also need the capability to tune both objectively and subjectively, so Arm said it provides a full set of tuning and calibration tools. Arm tools help developers to achieve the desired image quality, and together with software enable optimal designs and reduce time to market. Arm also offers specialized tuning training courses and support for tailored tuning to meet specific sensors and use cases. To speed up of SoC development with Mali-C55 before silicon availability, Arm also offers a bit-exact simulation model along with a reference platform, which enables the pre-built and pre-tuned evaluation of Mali-C55 image quality.
- Arm adds CPU, GPU and ISP for autonomous and vision safety
- AMD-Xilinx upgrades Versal ACAP for extreme signal processing
- The fundamentals of image quality tuning
- OmniVision shrinks medical image sensor for deeper endoscopy
- FPGAs displace ASICs in Subaru Eyesight vision-based ADAS
- Fourth-generation global shutter explained, and why embedded image sensors need better performance metrics