In recent years, the reduction in price and increase in quality of digital photos and video has resulted in an explosive proliferation of digital camera products. Multi-megapixel cameras are now used in a wide variety of applications, including digital still cameras, camcorders, camera phones and video surveillance equipment.

Increasing image resolution, however, comes with a corresponding increase in image processing complexity. The internal image pipeline, which takes the raw data produced by a camera sensor and generates a digital image that can then be viewed or undergo further processing, is a series of specialized algorithms that adjust image data in real-time. As the final image quality is determined by the performance of the image pipeline, many cameras implement the image pipeline in hardware. In this way, front-end image processing can be completed without consuming any cycles on the main application processor, leaving these cycles free for encoding and advanced processing, such as object recognition.

There is no standard tuning methodology or quality metric as the final image and video preferences vary between each camera user. Thus, it is important to understand the purpose of each block and how each stage of the pipeline can be optimized for a particular application. When combined, these modifications can significantly intensify the image and video quality to be sharper and clearer.

Breaking down the image pipeline
The image pipeline can be broken down into 10 distinct blocks (see Figure 1). In some cases it is possible and even desirable to disable certain blocks. This will not affect overall processing latency as latency is determined by the slowest block, typically color filter array (CFA) interpolation, which transforms sensor data into RGB data.


Figure 1: The image pipeline, which takes raw data produced by a camera sensor and generates a digital image which can then be viewed or undergo further processing, is a series of specialized algorithms that adjust image data in real-time. This figure shows the ten distinct blocks in which the image pipeline is implemented in a processor based on TI's DaVinci technology.

Given the parallel nature of image processing, each stage in the pipeline does not need to receive the entire image before processing can begin. Additionally, breaking the pipeline into these stages enables a camera to work with multiple pictures in parallel. Thus, the more efficient the pipeline, the faster pictures can be processed and taken.

The stages of an image pipeline are fairly standard, perhaps varying in order, but covering the same basic processing territory. Performance, however, is not the only factor that influences quality. In addition to being tuned to meet specific user preferences, many of the blocks need to be specifically tuned to match the particular sensor or lens combination of a camera, as well as to the various operating conditions in which the camera will be used.

Ideally, the image pipeline can be implemented in a configurable fashion allowing developers to adjust the filter and algorithm parameters but still benefit from the speed of hardware-based processing. While ASIC implementations provide excellent performance and low cost, they fail to achieve the best quality due to their fixed nature, which limits the ability to adapt to multiple applications. Programmable approaches, on the other hand, provide too much flexibility at the expense of requiring too many MIPS on the main application processor and consuming too much power.

Fortunately, the nature of the algorithms in the image pipeline is well-suited to a configurable approach, even those blocks that tend to be proprietary in the implementation. To achieve the fastest real-time video processing, for example, developers can tune the ten distinct blocks in which the image pipeline is implemented in a digital signal processor (DSP) based on Texas Instruments' DaVinci technology. Developers will then have the flexibility to fine-tune the images produced by the pipeline by adjusting specific parameters within each pipeline block to increase the quality of their final product.

The fine-tuning process is completed once for each design (i.e. each sensor and lens combination). If the camera is to be used for a fixed application, one set of parameters may be enough. For more versatile cameras, developers will want to take multiple measurements and calculate the appropriate parameters for each set of applicable lighting conditions. For example, the user may be able to select between parameters optimized for cloudy, sunny or nighttime conditions. The camera itself may even automatically evaluate the current lighting conditions and pick the appropriate set of parameters for the particular image being captured.

Next: Fine tuning the image pipeline blocks