Part I
Part II

The advent of wafer-scale manufacturing techniques has made possible the production of extremely compact camera modules at incredibly low cost. For reasons of physics, a small camera module will have inferior performance to a larger camera module, but the deficiencies can be corrected by exploiting the novel lens structures that become possible by the switch to wafer-scale manufacturing. Nevertheless, this merely preserves the status quo in terms of image quality and does nothing to enhance the user experience.

Designers of higher resolution camera phones have, until recently, been able to sell to consumers solely on the basis of the headline pixel count number. With the proliferation of camera phones (more than 80 percent of models now possessing one or more cameras), consumers have come to realize that picture quality and pixel count are not strongly connected. Indeed, the stunning pictures sent back by the Mars Rover vehicles were taken by ~1Mpixel cameras. Likewise, the designers of professional-grade cameras have known for a long time that to obtain the highest quality digital images requires the combination of optics with software.

As discussed in Part 1 of this article series, customers demand improved image quality and a range of embedded features in new camera phones, the most desirable of which are optical zoom, focus and low-light sensitivity, i.e., photography without flash. All of these features are relatively straightforward to implement provided the camera module is permitted to increase in height and cost, particularly in the case of zoom. Conventional zoom requires two lenses to move independently along the optical axis of the camera. This can be accomplished with miniature actuators, but the final product is large, power-hungry, has slow response time and is somewhat incompatible with the harsh survival envelope of portable electronic devices, particularly the "drop test." So, how can the camera module designer provide all of the features consumers desire and improve picture quality without affecting the form factor, reliability and above all, the cost of the camera module? The answer is software-enhanced optics.

Software-Enhanced Optics
Software-enhanced optics--or "smart optics"--is the technique of correcting for known optical effects by image processing. The optical effect can be an intrinsic defect that must be corrected or a deliberately introduced artifact that provides a feature or special effect. If the objective is merely to boost the picture quality, rather than invest in high-quality, precision optics, it is possible to use software to correct for known aberrations introduced by a lower-cost optical train. For example, if the constraints on size and cost mean that the picture corners will always be blurred to the same degree, software-enhanced optics can apply edge-sharpening algorithms to just those regions. The user is then happy with the picture because the inherent deficiencies in the camera module have been corrected or masked and the picture appears to be good in all areas. To be effective, the adjustments should be completely transparent to the user and require no intervention to use.

Having embraced the concept of software-enhanced optics, a world of new opportunities opens. The basis of the approach is to use a specialty lens that manipulates the optical rays during their passage through the camera to provide an intensity distribution on the imager with desired features. The manipulated image is not used as is; it needs further software correction. However, because the image was manipulated in a known manner, it can be digitally restored so high-quality output can be extracted. It is possible to implement many features with this approach, including full optical zoom with no moving parts, extended field depths, and small F-number optics for low-light environments.

A software-enhanced optics solution for zoom exploits the phenomenon that in a conventional optics train the density of information is not uniform over the field of view. The central region contains more data than the periphery. However an image sensor has a regular, two-dimensional pixels array. This means that a scene is under-sampled in the center of the imager while being over-sampled at the edges. The software-enhanced optics solution uses a specially designed fixed lens that provides intentionally non-uniform optical information density over the image area to match the quantized format of the solid-state imager. This is, in effect, the converse of the approach taken by nature. Many animals with single-aperture eyes, particularly birds of prey, have a standard lens, but a non-linear distribution of rods and cones in the retina. In both cases the resulting image is distorted, but can be rectified because the lens design and pixel distribution of the imager (or retina) are known.

For viewing at unity magnification, the algorithm has to compress details in the central portion of the field of view, where the special lens increases magnification and resolution. Thus, the compression does not degrade image quality and indeed, the software-enhanced lens solution is designed so that, in this mode, the picture quality is as good as in a conventional camera. When zoom is selected, the image borders are cropped off, and the already magnified center is retained. The image is then corrected for distortion. This is fundamentally different than digital zoom because magnification is the result of the lens action and is fixed at image capture, so that the zoomed image retains its high resolution. Software-enhanced optics can achieve up to 3x zoom.

An example of software-enhanced optics providing zoom is shown in Figure 7. This solution has no moving parts, is physically compact, rugged, virtually instantaneous, consumes negligible power and can be implemented at relatively low cost. It is greatly superior to digital zoom. Digital zoom involves cropping and expanding the image to fill the field of view. This decreases resolution because the available information has to be spread over a larger area. In a 3x digital zoom, approximately 90 percent of the information quantity in the captured image is lost, which is why digital zoom can only provide a small amount of magnification before the picture quality becomes unacceptable. The image enhancement solution for zoom is accomplished by a fixed lens and a simple algorithm. This makes it suitable for all imager technologies and all resolutions from QCIF to >10Mpixel, so broad adoption on camera phones is expected in the near term.

Pictures taken on camera phones tend to be very much "spur of the moment" events. The consumer does not want the scene to be posed, and to have to take the time and trouble to place themselves and the camera a suitable distance from the subject. By virtue of its small optics, a conventional camera module is only able to focus on objects a limited range of distance from the camera, typically 60cm to tens of meters. By failing to know and respect this limitation the consumer is frequently disappointed with the resulting pictures. The image enhancement solution for this is "extended field depth." It results in all details in a scene being in focus, provided they are between 10cm and infinity from the camera module. Like the software-enhanced lens zoom solution, this is accomplished through the combination of a special lens providing controlled optical manipulation and a small algorithm. It involves no moving parts and is therefore rugged, reliable, instantaneous, and consumes virtually no power.

In a conventional camera module, the optical train is designed to focus a point source of light, placed a fixed distance from the camera, onto the imager. If the lens is out of focus, or the object is too close to the camera, then the spot is smeared over a diffuse area and the image is blurred. The rule whereby the lens transforms the point source into the blurred spot is described by a mathematical transformation called the point spread function. If the point spread function of a lens is known, the blur can be transformed back to a spot, using digital signal processing. But there is no reliable way of identifying whether a particular area in an image is in or out of focus and therefore whether the transformation should be applied. Software-enhanced optics solves this problem by intentionally de-focusing the entire image in a controlled manner. Effectively the lens creates a uniformly blurred image of a point source located anywhere in the field, from near to far, that can be de-convolved by a straightforward algorithm. The result is a nice, sharp image in which the foreground, middle-distance and background are simultaneously in focus.

One of the principal complaints about camera phones is their low-light performance. This is only a half-truth. Miniaturization of camera modules has certainly resulted in a decline in optical sensitivity compared with digital still cameras, owing to shrinking pixel dimensions. Decreasing the pixel size from 2.2μm in 2007 to 1.75μm in 2008, 1.4μm in 2009 with a road map out to 1.1μm has significant implications for low-light performance and image quality. Simply put, as pixel size shrinks, its sensitivity decreases. From a more technical perspective, the ability of the photo-diode to absorb photons and release electrons (expressed as quantum efficiency) significantly decreases as pixels shrink. Other consequential effects of small pixels include low dynamic range and degraded signal-to-noise ratio. In reality, the perception of poor low-light performance of camera phones is actually more due to the social trend of taking camera phone photographs in low-light environments; typically in the evening and in venues like clubs and restaurants where there can be 5lux or less illumination compared with >350lux outdoors in daylight. As luminance levels decrease, the picture quality from a digital imager deteriorates rapidly, revealing defects like increased noise, loss of detail and color errors.

One of the main reasons for the inadequate low-light performance of camera phones is the inability to alter the F-number of the optical train because this is fixed during manufacture. Most digital still cameras provide the option to increase the aperture size to compensate for the reduced number of photons that actually reach the imager from a dim scene. But a mechanically adjustable aperture is physically quite large, fragile, slow to respond and power hungry. Simply decreasing the F-number of a fixed aperture camera to boost the low-light sensitivity is not an option since a large aperture reduces the field depth, making it difficult to obtain a good quality image where the scene has depth. Typically, standard camera phones use an aperture size from F/2.8 down to F/2.4, mainly to preserve adequate depth of focus under normal luminance conditions. An easy fix for image capture in low light is to increase the exposure time. However, this renders the picture susceptible to motion blur and camera shake, and may not be possible for video capture where exposure time is limited to 67mS by the frame rate.

"Speed" is a convenient shorthand way to describe the ability of an optics system to deliver light to an imager. Operation in good light conditions can be done with a "slow lens." That is, the optics permits the use of a small aperture combined with a slow shutter to get good depth of field. Photography in poor light, or in good light where fast shutter speed is required, such as following certain sports, requires a "fast lens." Thus, the challenge is to sever the normal connection between lighting conditions, depth of focus and shutter speed, and develop a fast lens suitable for low-light scenes.

Software-enhanced optics provides a means for delivering a fully automatic solution for camera phones that enables the consumer to obtain clear images under a wide range of luminance conditions. The basis of the approach is to design the camera module with low F-number optics, typically F/1.75, and restore the depth of field to normal using the extended field depth solution described above. The low F-number optics makes the ultra fast lens solution suitable for both still photography as well as video feeds. The signal processing compensates for loss of contrast and substantially reduces noise in the final image, while preserving edges, fine details and texture. This is possible because the information written to the line buffers necessary to run the algorithms can be reused to provide pixel-averaging data and improve the signal-to-noise ratio of the image by up to 6dB. The effectiveness of this solution can be clearly discerned by comparing the two photographs, taken with identical imagers having 1.75μm pixels, in Figure 9.

Implementation
Software-enhanced optics combines a special lens with a custom algorithm to deliver remarkable quality pictures in a way that is totally transparent to the consumer. However, the camera module designer needs to think ahead to incorporate these image enhancement technologies in a handset, as it cannot be done as an add-on. In principle, all that is required is one custom-designed lens in the optics train that can be manufactured using the existing infrastructure and lens materials. The custom lens can even substitute for an existing lens. Allied with this is the image processing algorithm. The algorithms for these solutions are usually small, taking approximately 100k gates. This is small enough for the algorithm to be embedded in the image pipeline on the CMOS imager, but clearly requires co-operation with the image sensor manufacturer, and the die must then be married to the correct optics.

The alternative placement for the algorithm is as software or firmware running on a dedicated image processor or the phone processor. Again, both of these solutions are very simple from a technical standpoint but require communication outside of the traditional camera module supply chain. Nevertheless, the benefits of these solutions are so compelling that 3Mpixel camera phones with extended field depth are already in production and will be proliferating--together with zoom and ultra fast lens solutions--to higher resolution cameras in 2009.

While software-enhanced optics work to boost the raw performance of highly miniaturized and low-cost camera modules, apart from the zoom solution, they do little to provide features that stoke customer satisfaction with the picture-taking experience. This issue matters little to the camera module designer, but is of very great importance to the original equipment manufacturer. One of the most commonly encountered annoyances from digital camera photographs is red-eye, which explains why red-eye reduction is today implemented on more than 80 percent of digital still cameras. Whether features such as this can be provided on camera phones and how they can be integrated is discussed in Part 4 of this article series.

About the Author
Giles Humpston, Ph.D., serves as Director, Research and Development ofTessera. Dr. Humpston has spent his entire professional career working in the field of semiconductor packaging, initially for military applications and more recently for high volume consumer products. He is a metallurgist by profession and has a doctorate in alloy phase equilibria. Dr. Humpston is a cited inventor on more than 75 patents and has co-authored several text books on metallic joining processes. His work and technical publications have been recognized by five international awards. Dr. Humpston's current interests are packaging of solid state camera modules and product miniaturization through wafer level technologies. He can be reached at: ghumpston@tessera.com

For more on the subject:

Tips and Tricks: The critical nature of cell phone camera packaging
What you need to know about imaging solutions for camera phones
Cameras in handsets evolving from novelty to DSC performance, despite constraints
Image pipeline: Fine-tuning digital camera processing blocks
Get enlightened about camera phone flash units--compare Xenon to high current LEDs