Software makes up a large part of next-generation television receivers. Exactly what software will you find inside and how will it control the hardware and the user experience?
High-definition television (HDTV) involves a significant software component, although nobody is sure just how much
software should be involved, or what it should do. Software clearly has to handle the user interface, and manage the torrent of data running through the MPEG decoder, but what else? This is the type of question engineers and marketers have to consider when they design HDTV software.
This article is an introduction to the software running in an HDTV set—the present reality, and speculation about future evolution. The central issue for today is MPEG processing. That torrent of data drives the hardware
and software architecture. MPEG will be there in the future too. The questions about the future center on how much other software will run on the receiver.
MPEG decoding
A high-definition television (HDTV) built to Advanced Television Systems Committee (ATSC) standards is a digital television, and as digital broadcasting becomes more common, it should work anywhere in the United States. It will probably also work in other places that use NTSC sets today. Other
countries employ different standards, but digital HDTV sets should be similar everywhere. MPEG decoding is the common denominator.
The part of an HDTV set that converts a signal from terrestrial broadcast, cable, satellite, or some other source into analog signals to send toward the display and loudspeakers is called the receiver or, if physically distinct, the set-top box. It may offer a program guide. It might even execute arbitrary software, but the thing it will certainly do—and must do well—is
handle an MPEG stream (
Figure 1
).
Hardware components
The crucial function of HDTV software is to shepherd data from the network or antenna interface to the display and speakers. Feeding a 1,920 x 1,080 interlaced display at 24 bits per pixel, plus 5.1 channels of CD-quality sound, requires a lot of data—as much as 180MB per second.
The DeMultiplexor.
The maximum data rate for an HDTV title is 6MHz. That bandwidth can
deliver about 19Mbits per second for a “terrestrial broadcast,” or about 38Mbits per second from cable. This is a restriction on HDTV broadcast, where channels are separated by frequency. Since a single MPEG stream can contain many titles, a digital network could, in theory, use much higher data rates. It could, for instance, be a 100MB per second network with 10 high-bandwidth HDTV channels, 10 lower-bandwidth HDTV channels, several regular digital television channels, and various computer data
feeds.
The transport DeMultiplexor (or demux) inspects each MPEG packet from the network to determine whether it can be discarded. It routes the remaining packets to the video decoder, to the audio decoder, or to application software.
The video decoder.
An MPEG video decoder is an impressive thing. Several encoding modes have it updating between 50 and 60 megapixels per second. That makes MPEG decompression barely possible on high-end processors and DSPs; rather, specialized MPEG chips are the
tools of choice for consumer electronics.
MPEG compression capitalizes on the fact that typical images don’t change much from frame to frame. Rapid change requires extra bandwidth, or the image may be intentionally degraded to reduce the bandwidth requirement. If you watch an MPEG title that was encoded without much attention, you’ll notice the image getting blocky when it changes suddenly. The solution to that problem is to shift some of the load from that sudden change to earlier slow intervals.
If you watch data flowing into the MPEG decoder, you’ll see the data trickling into it when the picture is fairly static, and roaring into it when there are sudden cuts.
This requires a sizable buffer between the demux and the video decoder, and some interesting work to handle timing that is only loosely connected to input data.
The audio decoder.
Compared to video, an MPEG audio decoder is a simple thing. Its input is no more than half a megabyte per second. It would not even require a
high-end processor to execute in software.
One of the worst things that can happen to an MPEG decoding system is to have the audio and video out of sync. On the surface, this would appear to be easily avoided. The channels need only to ignore the data they’re being fed and pump out decompressed data at a constant rate. If they begin together, they will end together, right? Mostly right. There might be a problem with cumulative errors, given that CD-quality audio uses samples at 48kHz, and video uses
frames at up to 60Hz. However, drifting isn’t generally a problem because decoders can synchronize based on time stamps in MPEG packets.
If packets are corrupted or lost, the decoder has to fill the gap, which isn’t a big problem. The decoder goes silent, or extrapolates, or interpolates, or whatever the designer likes for error concealment. But how long should the fill last? The robust answer is to drive audio and video output from a shared clock, and decode MPEG packets when their time stamp
is called. Although audio and video decoders have separate input streams and output devices, they are joined at the clock.
Mixers.
HDTV sets will certainly need to support video overlays, and probably audio mixing as well. The simplest video overlay is a channel number or a time stamp over the motion video. More complex overlays could support interactive applications, embedding the motion video in a frame of computer-generated graphics, and endless other applications. Audio mixing combines
data from application programs with the output from the MPEG audio decoder, which lets the system respond audibly to its user without shutting off the MPEG sound.
Two reasonable methods exist to mix streams: with software operating on the output from the MPEG decoders, or with hardware, before the decoder output reaches the final output. Software may be able to operate on the video frame buffer during the vertical blanking interval (the interval between frames), and the audio output can be caught and
modified between the audio decoder and the digital to analog converter (DAC) that sends it off to the amplifier.
The easy approach is to use video hardware that supports multiple layers to mix graphics with motion video, and to use an extra set of audio DACs and an analog mixer to mix audio outputs. The extra audio hardware sounds wasteful, but it solves some difficult problems. Program-generated audio data may not have the same sample rate as the output from the decoder—they may use different numbers of
bits per sample, they may need to be scaled to mix them appropriately, or one of them may be delta coded. Handling those combinations in software without generating artifacts of the mixing process is difficult. Generating separate streams and using an analog mixer resolves all of those issues.
Software components
The demux and the MPEG decoder dominate an HDTV receiver or set-top box design, and the software that manages these devices may be the most complex in the
system. The complexity of the MPEG software depends on the hardware used for transport demux and MPEG decoding. The processing for the entire MPEG processing stream from demux through output can be integrated on one chip. This requires substantial memory, but from a software point of view, the chip only needs initialization and a run command for normal operation. It will let you know when the MPEG stream ends.
Toward the other extreme, the MPEG software may need to move data between the demux and the MPEG
decoders, and manage the clock. MPEG devices need large input buffers, so keeping them fed is relatively easy. All the audio and video DACs I’ve seen in this type of system use DMA. The demux is the demanding device. I’ve experienced systems with hard interrupt response time deadlines for the demux of about 25µs and about 40µs. If you don’t meet the deadline, you lose a packet.
HDTV system software is hard to write, but it’s a bounded chore and a lot of it is available off the
shelf. Such packages contain most of the software for basic operation, which is both good and bad. Its availability allows engineers to buy packaged solutions to many knotty problems, but it also starts systems at the same point. These days, HDTV sets are high-end products, and nobody wants to release a product that is “lowest common denominator” in any respect.
Systems can differentiate themselves with special user interface software. Each manufacturer tends to have a characteristic approach to
the user interface, even if it is only placing their logo prominently on the screen. A great deal of work can go into designing and building the user interface. Perhaps the set needs a Web browser. A set of games? Screen savers? E-mail and a calendar? Deciding where to draw the line is difficult. Running the MPEG stream doesn’t require a powerful processor by today’s standards, but it is almost certainly a substantial 32-bit processor. It can do everything I’ve mentioned and more. From a
project deadline point of view, those processors have frightening possibilities.
Application code can receive MPEG packets from the transport demux, send MPEG packets to the decoders, send audio data to the mixers, and graphics to the video output. Applications also handle user input, whether it’s from front panel controls, a touch screen, an IR remote control, or some other mechanism.
The minimum application software required for an HDTV set is a program guide, which reads program guide packets from
the MPEG data stream and offers tools for selecting programs. The data stream for the program guide software is specified by the ATSC. It defines master program guide MPEG packets that carry current programming information, special program guides that may carry information about events beyond the interval covered by the main program guide, and other program-related data streams.
Figure 2
shows an application architecture for digital television. The shell controls the
transport demux’s routing of the MPEG stream. When it finds interesting content, it routes it to one of the three engines. It sends MPEG audio and video to the decoders. It looks at the headers on MPEG data packets and routes the packets to the presentation engine or the execution engine, as appropriate. An application might involve a mixture of all three types of content. If so, the application has to handle interactions on its own. Variations on this model have been used on a variety of platforms. This is
the first model I know of that is likely to be built entirely on industry standards: MPEG, HTML, and Java.
Future systems
The first generation of HDTV systems are on the shelf, or nearly there. I’ve seen a Mitsubishi set available for sale in Des Moines, and the store promises to have several lines shortly. Second-generation sets are being engineered. Some will surely feature lower prices. All of them will look for ways to distinguish themselves.
You lower the
cost of a receiver by using cheaper components, fewer components, or finding an easier way to assemble the system. Some of this cost reduction occurs automatically in the computer field. Chips are always getting cheaper. The fun part comes in picking the least expensive set of chips. Is it better to use a more powerful processor and a weaker MPEG decoder? Can ROM replace some RAM if the ROM is more than eight bits wide? Do we really need an extra set of audio DACs and an analog mixer? Maybe some processor
will be able to decode MPEG in software at a price less than the cost of an ordinary processor plus a decoder chip. These are every-day considerations for engineers in consumer electronics.
This may not be a set-top box
One obvious way to cut the cost of a set-top box is to make it disappear. That box has an enclosure, a power supply, connectors, and probably some controls. It would save a bunch of money to put it in the same box as some nearby electronic
component—like the display.
The set-top box is full of chips, some of them useful only in HDTV decoders. They’re expensive today, but history leads us to believe that in two years they’ll probably drop to one third of their current cost. The display contains a high-end CRT, or LCD, or an exotic display technology like plasma. The price of those ingredients has not been dropping as fast as silicon.
Set-top boxes will not disappear until most television broadcasts are digital. For the moment, a more
effective strategy dominates: leave the box out of the equation. The HDTV set I saw was getting its data from a PC that could play a selection of short data streams. No other way exists yet to get HDTV data in Des Moines, or most other places. Here, and in most other markets, an HDTV set is a television with a wonderful picture and an upgrade path. It makes a lot of sense to sell a display that works now, and promise a future set-top box at prices that can reasonably be expected in several years. A set-top
box is a computer with an interesting set of peripherals, so it will always be cheaper to buy in the future.
The economic reasons for having a separate set-top box will disappear when people are willing to spend the money on a fully ATSC-compliant receiver. But some may still want to buy component sets; perhaps combining company X’s receiver with company Y’s display will yield favorable results. And that will increase flexibility, consumer options, and cost.
Special
hardware vs. general-purpose CPU
An HDTV set is like a large, heavy computer graphics terminal. Sometimes the best designs for such terminals use general-purpose processors to execute graphics operations; sometimes dedicated hardware is a better option.
Specialized hardware can do the job for which it’s designed better or cheaper than general-purpose hardware. That point is difficult to argue with. Consider the particular case of an MPEG decoder. An MPEG decoder contains something like a
general-purpose processor, but it probably uses a Harvard architecture like a DSP, and it doesn’t have memory translation or protection, interrupt processing, exception handling, or floating point. It can probably process arrays of short integers in 64-bit chunks or larger. It is a lot like a DSP, and doesn’t do anything any general-purpose microprocessor can’t do—but it’s about ten times cheaper. The special-purpose chip has the advantage of having hardware designed specifically
to decode MPEG, and it doesn’t need to carry the execution overhead or expense of anything that doesn’t decode MPEG.
The problem is that an MPEG decoder needs a processor to take care of it. Now we have a system that has an MPEG decoder and a processor. Let’s say that the decoder costs $50, and the processor costs $25. You have the cost of buying the two chips, plus the cost of finding a place for them and hooking them together. If a $75 processor could decode MPEG, it would provide for a
less expensive system than the one with a specialized decoder. For an MPEG-decoding processor to be cost-effective, it has only to equal the sum of the cost of a low-end 32-bit processor and the cost of an MPEG decoder. I don’t think that microprocessor exists yet.
A performance issue also factors in. A processor that could decode MPEG and have a few cycles left over for housekeeping could do a lot of interesting things when it isn’t decoding MPEG. The MPEG decoder can’t do anything else.
Replacing the specialized MPEG decoder with a general-purpose DSP leads to a more expensive, but potentially more functional system. The DSP can decode MPEG, and it can do other things like graphics and video conferencing.
The pendulum of today’s market demand seems to have swung toward specialized processors. You can find intelligent I/O cards and graphics processors in anything from high-end workstations down to some game machines. Considering the way the price and performance of embedded processors
is improving, this issue of processor selection needs to be frequently reevaluated.
Parallelism
HTDV sets need to change the state of up to 60 million pixels per second and handle a 6MHz MPEG input stream. Other jobs, like waiting for input from an IR remote and updating the clock, are trivial compared to those tasks.
The processor has what amounts to moments of panic when it has to service interrupts from the hardware processing the MPEG stream, but most of the
time it sits idle. The processor is picked to be fast enough to handle the bursts of MPEG load, and to present a snappy GUI. When the GUI uses a Java virtual machine, you’ll need 50 MIPS or more. For a simple GUI, one or two MIPS would be enough.
Whether the processor is at the bottom of the 32-bit range or in the middle, it is still idle most of the time. I don’t know of anybody who has found an effective use for those cycles. My favorite possibility is for decoding MPEG audio in software (in a
system where the audio decoder isn’t bundled with other hardware, and the MPEG clock is easily accessible). If HDTV sets were critical systems, we’d use all the background cycles running diagnostics. Maybe, in a sense, they are. People have been known to get, shall we say, irritable when their television breaks at the wrong time.
Upgrades
A five-year-old personal computer is about as useful as last week’s newspaper (that is, not completely useless, but
close). A high-end stereo component from the late ’60s, however, is still perfectly satisfactory today. Some of them are even worth a good deal more than they once cost. An HDTV set combines these two sets of expectations. Someone who spends more than $5,000 on a television set would feel completely justified in throwing a fit (if not the HDTV set itself) if it becomes obsolete in five years. Yet the television does contain an expensive computer that will unquestionably be obsolete. What’s the
manufacturer to do?
Hardware upgrades.
Part of the solution is to keep the computer in the set separately upgradable, even after the set-top box migrates into the television. If the computer component of an HDTV set is on a separate board, I’d guess that upgrading a high-end set’s computer hardware would cost much less than trading in the entire thing. However, keeping the computer on a separate board with some type of connector between it and the monitor board will cost more than
putting it all on one board. This cost must be weighed against the advantage of possible upgrades and the convenience of being able to combine computers and monitor components without revising the main board for every combination.
Hardware upgrades run counter to all established practice. The manufacturer wants you to put the old set in the back room and buy a new one. They would be thrilled to convince their customers that a two-year-old set is obsolete because it can’t run the latest software. Generating
fancy software is much cheaper than significantly improving the display.
Software upgrades.
Keeping the software on HDTV sets up to date is in everybody’s best interest. The customer would have bugs fixed as quickly as possible. The manufacturer would avoid service calls and a bad reputation by fixing defects before the customer finds them.
Customers will not want to bring their (heavy) HDTV set into the shop for software upgrades, or to pay to have a serviceman visit their home.
Software has to be downloaded to the device, or installed from some local device (such as a CD player or a card reader).
The ideal upgrade medium is the MPEG stream itself. If the sets were to receive the MPEG stream over a typical interactive network, the set would just keep a path open to its manufacturer and pick up software upgrades seconds after they were made available. If the sets receive all their data from broadcast or cable, though, logistics get a little tricky. I wouldn’t expect broadcasters to
give bandwidth to television manufacturers. Perhaps the manuracturers will buy bandwidth, or maybe they’ll include upgrades in their advertisements. That would be a good reason to watch commercials.
In any case, the set’s system software must be able to handle upgrades. Replacing everything and rebooting is a solution, but it may not be a good one, especially not if the new software uses so much memory that the system can’t operate normally until it processes it. Rebooting to incorporate
the new software would be fine if you could do it at a time you choose. Inserting a flash card in a slot in the television, for instance, and knowing that the set will turn itself off and back on would be fine. But if the television turns itself off for 15 seconds after a particular commercial, that would be bad.
The user would ideally only notice a software upgrade because the set works better afterward.
Java
HDTV opens up a whole new set of opportunities for
standards. It is a new, programmable, digital consumer device with higher bandwidth requirements than we’ve seen before. We need the digital equivalent of the RCA plug to connect digital video components. That connection will likely be IEEE 1394 (Firewire). HDTV sets that run any software beyond the basic program selection shell require a software standard. Every company involved would like the standard to be its proprietary system.
I think the software standard will be either Java or nothing at all. Java
might get an early edge from helping get sets to market quickly, but its processor and memory overhead could force it out of the running when the cost reduction cycle gets going. Java will remain a consideration only if customers want to run software that they get after they buy the set. If Java byte codes start appearing in MPEG content, Java has a lock on becoming the industry standard.
Of course, downloading software into a television set has lots of ramifications. You must be able to prevent
downloaded software from crashing the set. Some classes of software may need local storage, and local software may intensify a desire for two-way communication. One large ramification is a possible identity crisis. An HDTV set is almost a usable personal computer with a nice display and no keyboard. The identity crisis strikes when the HDTV starts running software that isn’t directly related to the multimedia titles it plays.
For example, an intelligent program guide could be considered an agent of sorts
that would search out interesting content, watching the available content constantly for interesting changes. The guide might also watch data packets that contain news, local restaurant menus, weather reports, professional journals, pricing at local stores, and so on. The bandwidth, though a tight squeeze for a movie, can hold tremendous amounts of data, and an agent that filters this data makes it more useful.
Another example might include programs and data in audiovisual content. This path might start with
active hyperlinks appearing in advertisements. Shows might include biographical information on the actors, or statistics on athletes in data packets sent along with the rest of the show. If at least a little bit of up-stream bandwidth is available, even a modem, interactive games of all sorts become an interesting option.
Is this just another PC?
If the HDTV set does run software that isn’t related to content, it begins to resemble a PC; but it probably
won’t go too far in that direction. Users have different expectations for television sets and PCs. One of the requirements for a television is brute reliability, and neither Windows nor the Mac OS are that reliable. Embedded systems run for years without rebooting, partly because they are simple, and partly because people insist on reliability. Imagine what Windows might be like if anyone who experienced a few crashes sent the software back to Microsoft with an angry letter.
Making a hybrid HDTV/PC is
possible, and it could even be good at both jobs, but the result would be a box containing an HDTV and a PC. To keep them from degrading each other, they would need separate processors, separate software, and probably separate memory. They could share the display, the box, and the power supply (see
Figure 3
). Sharing a big, beautiful display sounds good, but even that isn’t as good as it sounds. Most people watch entertainment from more than arm’s length and most people
sit about arm’s length from a computer monitor. Like the system software, they are optimized for different things.
We’re left with the option of combining the systems to save money— and I’m sure it would. You could use one at a time, and you save mostly the cost of a monitor. Come to think of it, I used to plug my Radio Shack color computer into my television—but I bought it a monitor of its own as soon as I could save enough money.
User interface
Today’s televisions all use roughly similar remote controls and a vestigial front panel. Over time the remote control may disappear. Voice commands, gesture-based commands, and user agents may combine to replace switch-based controls.
A normal remote control is probably entirely satisfactory for an ordinary television. The fact that they have remained nearly unchanged for about 20 years is good evidence that people are satisfied with them
Digital television enables a more interactive
experience than conventional television, but a good chance exists that customers and content producers will ignore most of the new potential. If digital sets act like analog sets with a better picture, remote controls will continue to be good enough. If, on the other hand, some aspect of the television experience becomes more interactive, we’ll need a user interface that is as easy as the few buttons on the remote control that most people use, but more powerful.
Adding a track ball or a touch pad to a
remote control might be enough, or maybe the remote will generate a focused beam that lets it work as a long-distance light pen. Maybe the remote will shrink to one button that means “attention,” at which point the set will attend to gestures or spoken commands. From a software point of view, user input is a great opportunity for innovation.
The output side of the user interface is ripe for research too. Television sets aren’t just for technophiles. You should assume that the user will not
even read the manual, much less take a class on operating a digital television. Steve Jobs has referred to the Macintosh as the “computer for the rest of us.” If digital television is going to be something more than a television with a better picture, or a television that can also act a little bit like a PC, it must be the computer for the rest of us.
Peter Dibble is a senior scientist at Microware. He has a PhD in computer science from the University of Rochester. He has written several books
and numerous articles for trade publications. Peter was involved in early work in Microware’s DAVID software for multimedia, and interfacing with partners designing hardware for multimedia systems.
