Achieving convergence in high volume consumer digital video applications

We're in the midst of a revolution; a technical revolution in theway we access live and recorded media. Digital broadcasting is enablingthis revolution and, to a lesser but growing extent, it is supported bythe market for (home) digital recording. The revolution is HighDefinition (HD) in all its forms; terrestrial and satellite televisionbroadcasting, streaming video and audio over IP, home audio/videorecording, and portable A/V devices.

The justification for calling this a revolution is because itimpacts so many aspects of our lives, requiring new equipment atmultiple points. Arguably, the last technical revolution came with theintroduction of mobile telecommunications, in its analogue form. Whilethis only required change at a single point ” the mobile handset ” itimpacted our lives to such an extent that it caused revolutionarychange in the way we live.

However, since its introduction, everything that has happened sincehas been evolutionary – even the migration to digital cellulartechnology; the revolutionary change happened when we started usingmobile telephones. Even the latest acronym for mobile communications “LTE, which stands for long term evolution ” acknowledges thisfundamental shift.

It can be expected, therefore, that at some point 'HD' will movefrom being revolutionary to evolutionary, but not until it has reacheda level of saturation. Today the market for mobile telephones hasreached that level in the developed world but continues to be buoyed bythe upgrade path and emerging markets. The same may become true for HDdevices, but perhaps not for some time yet.

This highlights an interesting difference between ” and potentiallyconvergent path for ” mobile communications and HD, which is only justbeing addressed; the choice in access devices and the services theyenable.

Mobile broadband is a relatively recent 'evolution' in cellularcommunications, one that is giving rise to a host of new accessdevices, such as 3G dongles or handsets with HSDPA.

Since the introduction of digital broadcasting there have been manyattempts to make it also more portable, by integrating digitalreceivers in to a variety of devices, but with little success. This maybe because the transport media used by handheld or portable devices “over the air terrestrial UHF ” is still relatively unreliable fordigitally encoded video. Other, packet based transport mechanisms, suchas cellular broadcast or even satellite broadcast, is arguably morereliable in this application, particularly in built up areas.

This abundance of choice in transport media, as used already by avariety of devices, is giving rise to demand for convergence; theability to receive digital broadcasts ” including HD ” using a greaterrange of access devices.

In turn, this is fuelling demand for more sophisticated siliconsolutions, which has some surprising consequences. The use of softwaredefined platforms for wireless devices is well established, enabledgreatly by digital signal processors (DSPs). It is these devices thatinvariably enable the access devices used in cellular communicationstoday.

However, it seems HD presents its own unique challenges, that aren'teasily solved using DSPs and software alone. Convergence in digitalbroadcasting is creating demand for a new generation in processingsolutions.

HD Video
One of the leading companies in digital signal processing, indeed thecompany credited with inventing the DSP is, not too surprisingly, alarge player in digital communications. Texas Instruments' DSPs arepresent in the majority of cellular handsets, as well as having astrong presence in basestations. It could, therefore, be seen as one ofthe biggest protagonists for all things software defined. However, thelatest devices from TI show that the real-time processing requirementsfor HD decode are too great for DSPs alone.

Through a combination of integrated processor cores and hardwareacceleration, TI is addressing the need for HD capability in low powerand low cost platforms.

It recently introduced the latest generation of OMAP devices; theOMAP 3. There are currently four devices available, one of whichcombines the ARM Cortex-A8 processor core with one of TI's ownTMS320C6x+ DSP cores and dedicated hardware acceleration blocks forboth HD decode and 2D/3D graphics. The first commercial application forthe OMAP 3 platform was unveiled recently by Archos, which is using theOMAP 3 in its newest portfolio of personal entertainment devices; theInternet Media Tablet.

“TI is committed to helping Archos develop high quality productsthat take advantage of the world's first implementation of thesuperscalar ARM Cortex-A8 processor. Coupled with a TI C64x+ DSP core,the OMAP 3 processors (Figure 1 below) provide the necessaryperformance and multimedia features for enhanced functionality on theArchos devices,” said Suman Narayan, General Manager, CatalogueProcessors and Emerging End Equipment, TI. “As consumers demand HDvideo, media rich data and connectivity anytime and anywhere, Archospushes the boundaries as a leading innovator with its new IMTs.”

Figure1

The ARM Cortex-A8 core delivers over four times the performance ofthe ARM9, the core integrated in previous OMAP platforms and still usedin the latest DaVinci platform. However there is one feature that boththe OMAP 3 platform and latest DaVinci devices share; hardwareaccelerated HD decode, supporting multiple formats. However,architecturally, they are addressing different parts of the HD market;OMAP is aimed at access devices, while DaVinci is described astargeting the 'edge' of the delivery mechanism.

Graphics acceleration
The fact that hardware acceleration for both video and graphics hasbeen added is further evidence that convergence is expected betweenlive HD video and the integration of more sophisticated 2D and 3Dgraphics. Given the power available from processor cores today, itseems somewhat surprising that hardware acceleration is necessary, butindustry experts believe it is, for both live video and renderedgraphics. A recent trend in this field is the development of OpenGL ES2.0, a version of the open API used for graphics acceleration in PCs,aimed at embedded systems, hardware support for which is integrated inthe OMAP 3 platform.

Tom Cooksey is a Software Engineer with Nokia-Qt, and he believes itmarks the beginning of a general shift in the industry: “When Appleintroduced the iPhone it completely changed the way consumers expectedto interact with devices. A big part of that was the graphics andwithout hardware acceleration, devices will have limited graphicscapability.”

Cooksey sees the API as a simple way of programming graphicshardware and although used predominantly for 3D applications, it canalso help accelerate 2D graphics and for adding effects. “This isespecially true for more modern graphics hardware, which is basicallyjust a massively parallel general purpose processor,” explainedCooksey.

“Such hardware is modelled by OpenGL ES 2.0, which allows developersto write custom code called 'shaders', in a C-like language (GLSL).”The driver then compiles the code and runs it on the graphicsprocessor.

The API could, therefore, be used anywhere a user interface isnecessary; from mobile phones to netbooks, gaming consoles or set-topboxes, and even in-car entertainment. “There is also massive potentialin non-consumer applications that have tight graphics performancerequirements, such as industrial, defence and medical computing,”Cooksey added.

With the introduction of hand held devices providing HD playback,there is a definite trend towards integrating video and graphiccapabilities on the same platform, as Cooksey pointed out:”Traditionally, video decoding hardware has used an overlay to putvideo on the screen. From a software point of view, you feed thehardware with the encoded video stream and you never have to worryabout it again ” it will 'magically' appear on the screen.” The sameapproach is used for video input, where the software engineer doesn'teven see the data.

The problem with this approach is that video never passes through GLpipe: “Things like window animations and effects are impossible to do,”explained Cooksey. “One option we have tried is to use a shader to dothe scaling and YUV-RGB conversion.”

As modern graphics hardware is programmable, you could theoreticallyuse it to do the complete decode in a shader, and some vendors haveactually started to do that, according to Cooksey.

“Even just doing the colour-space conversion in a shader works welland allows us to use video frames as textures,” explained Cooksey. “Theproblem we see is power; using the CPU for decode and the 3D core forthe colour-space conversion consumes significantly more power thandedicated video hardware.”

Ideally it would be possible to switch between the two techniques atrun-time, but this isn't made easy by the underlying multimedia APIs.”Hopefully this will change. One thing is for sure, customers wantfancy window animations and HD video playback at the same time. To dothis, hardware vendors will have to provide better integration in thefuture.”

Vector factor
The problem with the current integration of HD decoding hardware andgraphics acceleration is just that; the level of integration betweenthe two needs improving.

The current solution to placing video in to the GL pipeline is torender the video and place it in to video memory, then copy thecontents of that in to the graphics memory used by the GL pipeline;there is currently no direct solution for rendering video in to atextured graphics block.

Why is this important? Well it clearly isn't essential, but usersare likely to demand more sophisticated effects, such as animated albumcovers which pass across the screen at oblique angles, or sweep acrossthe screen as part the selection menu. Part of the solution is toimpose tighter cooperation between the standards bodies responsible forHD video and graphics, which until now there hasn't been a need for.vAnother important feature, according to Tom Cooksey, will be theintroduction of a graphics memory manager to handle the task of movingdata from the HD stream to the graphics pipeline, something he claimsis happening now in Linux.

But sophisticated 3D graphics is still only part of the story,another significant element is vector graphics. Here, again, TheKhronos Group ” the member-funded industry consortium developing openstandards for media authoring and acceleration ” and behind OpenGL ES2.0 ” is working on a solution. OpenVG is also a royalty-freecross-platform API for hardware accelerated vector graphics libraries,also primarily targeting handheld devices.

As OpenGL has no support for text, OpenVG can help. It turnsobjectsin to triangles, so any font ” especially complex ones such as Chinese” can be created, as well as any graphical element. Undoubtedly,leading silicon manufacturers will even now be working on implementinghardware support for OpenVG in to the next generation of mediaprocessing platforms.

Philip Ling is an editor with ESE Magazine.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.