Pixel Magic -- a video enhancement system implemented on embedded platforms -- improves video quality with high efficiency and low cost for a wide variety of portable digital devices.
By Theron Ye, Engineering Director, Pie Del Frate, VP of Marketing, Ian Xie, Founder and CEO, Ipera Technology
Video's presence and importance in the information and media industries has increased dramatically. With consumers no longer satisfied by still pictures, video is becoming a mainstream entertainment component on a variety of portable devices such as mobile phones, PDA's, PMP's etc.
In 2007, 22.4 billion views were logged for User-Generated Videos (UGV), up 70 percent from 2006. Most of these videos were generated on cell phones or low-quality recording equipment, making the viewing experience less than desirable.
Users expect clear, sharp images with vivid colors. Artefacts such as distortion, noise, faded colors, blurred motion and poor lighting are all distractions that minimize the entertainment experience.
Video transmission and processing systems used with digital portable devices today have many inherent problems that cause the delivery of poor-quality video. The main reasons for this are:
Bandwidth of wireless communication is still low with respect to what is required for real time video transmission.
Displays on digital portable devices are relatively small. Enlarging the display size significantly increases the cost, drains batteries, and makes the unit too bulky to carry.
Current technology constraints -- both the computational power and the battery life of embedded systems used for digital portable devices -- restrict many solutions available on the market today.
Distortions arise during all phases of video capture, compression, transmission, decompression and display.
Consider the typical flow for a conventional video processing pipeline depicted in Figure 1. Distortions can be induced throughout the entire procedure. Specifically:
Discontinuities between pixels occur in the CCD when natural scenes are captured and digitized.
Noise is generated during video capture and transmission, both spatially and temporally. In particular, temporal noise is more discernible and annoying.
Blocky and mosquito effects are produced due to loss in some high-frequency components in DCT (Discrete Cosine Transform) domain and motion estimations by sub-pixel interpolations during video encoding. Blocky artifacts form horizontal and vertical pseudo-boundaries are very obvious in playback, especially in case of low bit-rate encoding due to higher quantization factors.
During video encoding, deep quantization on high-frequency components in DCT domain also make video frames/fields lose details. Edges are also blurred for the same reason.
Because only a limited gamut of colors can be received by video capturing system, colors become dim and diverge from those in real world, which causes contrast to be decreased as well.
Next: Video enhancement chips and Ipera Technology's Pixel Magic Video Engine
Video enhancement chips and power
Hardware and software algorithms exist in video display devices that scientifically eliminate the above distortions. However, they are not cost-effective, use too much computational and battery power, and are scene and content dependent.
For instance, algorithms used for DTV's (Digital Television) are implemented in video enhancement chips containing tens of millions of gates or an array of DSP's (Digital Signal Processors) making them too expensive in price and too power consuming to be used in portable devices.
In summary, various distortions degrade video quality prodigiously, which diminish the end users' viewing experience. With the advent of digital portable devices, additional conflicts emerge between improving video quality and compromising video processing speed, efficiency and cost. In other words, what digital portable devices desperately need is an efficient, self-adaptive and effective way to enhance video quality. Simply increasing resolution without better pixel quality doesn't provide the improvement needed -- it's the quality of pixels that matters!
Ipera Technology's Pixel Magic Video Engine
Ipera Technology understands that digital video will dominate the next generation of digital portable products and end users expect the same or better video quality that they are used to seeing on their TVs. After two years of research, Ipera has developed it's Pixel Magic Video Engine which integrates the latest innovations for video enhancements. The Pixel Magic Video Engine helps bridge the gap between the poor-quality video that is delivered today and what the user expects, meeting the ever-increasing demands for video quality in the fast-developing market of portable digital video devices.
Figure 2: Video processing flow with the Pixel Magic Video Engine
The Pixel Magic Video Engine can be integrated into numerous locations in the video decoding and transmission path as shown in Figure 2. Video can be enhanced right after it is captured and before it is encoded to eliminate numerous anomalies created by the video capture device or after it is encoded and before the transmission to enhance the video and reduce the bit-rate required for transmission. The Pixel Magic Video Engine is also designed to be integrated in the portable device or streaming terminal to enhance the video after it is received and decoded but before it is displayed on the screen.
In consideration of the aforementioned distortions that degrade video quality, the Pixel Magic Video Engine has been carefully designed so that:
Codec-related deficits such as blocky, ringing and mosquito effects are eliminated
Both spatial and temporal noise is suppressed
Brightness is adjusted towards a suitable level
Details and edges are restored and become more distinguishable
Contrast is enhanced dynamically
Colors are resumed and naturalized with pleasing saturation
Figure 3: Architecture of Pixel Magic Video Engine
The system has been implemented on several embedded architectures such as TI's DSP and ARM. When running on the TMS320C64x DSP core, it delivers D1 (720x480) resolution video at 30 fps and will soon be able to handle HD content using as little as 900MHz. When running on the ARM926EJ-S core, it can provide enhanced video to quarter VGA resolution (QVGA) video using less than 120 MHz.
In an example where Pixel Magic Video Engine is used to enhance video on the display device, the decoded video data (usually in YUV format, i.e., pixels are represented as a combination of lumas and chromas) is sent into an algorithm core, where it is processed by a variety of proprietary enhancement modules.
Enhanced video data is then transferred into an outputbuffer where the data is converted into a suitable display format. Finally, the data is displayed on the screen.
Input buffer
The Input buffer uses special memory allocation techniques to re-arrange video data into a structure that can be quickly loaded by algorithm modules, so the time for data movement is significantly reduced. It then sends video data to an algorithm core via the bus.
The input buffer also provides an API (Application Program Interface) that is capable of receiving raw video data in several commonly used formats.
Next: Algorithm core, output buffer
Algorithm core
Video data is analyzed, processed and enhanced here. The data then goes through some or all of the video enhancement modules, depending on the results of the analysis process, the various aspects in video quality are improved. In particular, all modules work together seamlessly and dynamically. However, they maintain a level of independence so they are upgradeable and expandable.
The de-blocking module alleviates blocky effects on a block basis without requiring any bit stream information from the decoder. In this way, it differs from either a conventional post-filter approach (as in MPEG2/4) or an in-loop filtering scheme (as in H.264), which makes it more versatile.
The noise reduction module performs 3-D filtering that suppresses spatial and temporal noise simultaneously while retaining the detail. In addition, neither pseudo-outline nor tail effects are induced.
The edge and detail enhancement module identifies edges on different scales and classifies them into different ranks. By determining the locations and importance of edges, the power by which a particular edge shall be strengthened is leveraged. Also, texture richness is examined in some specified region, avoiding over-enhancement.
The dynamic contrast enhancement module enables viewers to perceive details in each video frame/field by tuning contrast automatically based on specified features of the video frame/field. It also prevents enhanced adjacent video frames/fields from flickering or twittering.
The dynamic color enhancement module enhances the colors of video frames/fields in accordance with the HVS (Human Visual System) theory. Therefore, colors become more natural, lively, and saturated.
The automatic brightness adjustment changes the overall brightness of every video frame/field by analyzing its lightening condition, and then making use of the HVS theory adaptively, so a suitable brightness is achieved that is comfortable to viewers' eyes.
Several other proprietary modules are also contained in the algorithm core. They are capable of making video look better by diverse means.
A vital property of the Pixel Magic Video Engine algorithm core lies in the fact that all algorithms have been highly optimized so video can be played in real-time on low-power portable devices -- even when all the different enhancement modules are accessed and functional.
Output buffer
The output buffer obtains enhanced video data via the bus, and transforms it into a proper format that can be used for a specified display device. Depending upon the embedded platform, hardware drivers can be inserted in order to accomplish some functions even more efficiently. The output buffer also provides an API that can output video data in several commonly used formats specified by the customers.
Next: Examples
Examples Figure 4 shows four pairs of comparisons. In each set, the left picture shows the original, unprocessed image and to the right the image after it is processed by the Pixel Magic Video.
Figure 4: Four pairs of comparison
All of the images on the right are more vibrant, have better color balance and saturation and details that are much sharper and more defined. The enhanced examples are even more impressive when viewed as video.
Conclusion
Pixel Magic Video Engine surpasses current video enhancement solutions for portable devices by emphasizing the following features.
Pixel Magic Video Engine performs video enhancement more efficiently. Every algorithm module has been carefully designed and highly optimized so that the maximum computational efficiency has been achieved. Novel design and optimization enables the system to be operated in a real-time fashion and uses an extremely low percentage of resources available in the portable device without compromising performance or quality. The system is therefore highly suitable for digital video portable devices that have very low computational power and limited battery power.
Pixel Magic Video Engine applies state-of-the-art video processing algorithms developed by Ipera Technology. It is able to handle video frames/fields with different sizes and/or formats. The whole system is codec independent making it easy to integrate into almost any video stream.
With this innovative video processing system, Ipera Technology delivers a solution that is unmatched in the industry, enabling the designers and manufacturers of portable digital device such as mobile phones, PDAs and other players with the ability to deliver improved video.
In addition to delivering exceptionally realistic per-pixel video quality at extremely low power consumption for virtually any video codec and any video application, Pixel Magic Video Engine can also be easily adapted to support future formats and other processor platforms. Streaming video providers, mobile OEMs and IC vendors who adopt Pixel Magic technology will find they can improve the video quality of their mobile video content, entertainment and communications devices without increasing their cost when new technologies emerge.
About the authors Theron Ye is an engineering director at Ipera Technology. He's a member of the founding team, and has close to a decade of experience in video and image processing. Theron holds an MS in control engineering from University of Sheffield, UK. Before Ipera, he also worked at other video-related companies.
Pie Del Frate is VP of Marketing at Ipera Technology. He has been managing both product and corporate marketing activities for the company since 2007. Pier has over 20 years of experience with video-related companies including holding executive level positions with National Semiconductors, Mediamatics, Trident, and Sigma design.
Ian Xie is the founder and CEO of Ipera Technology. He has been involved in video processing algorithm and CPU design for over 15 years. Ian is the founder of two cutting-edge video technology start-up companies. He received an MSEE from UT Austin and holds numerous patents.
