Virtual and mediated realities: presentation technologies and potential applications
In my previous column on this topic, we considered what we mean when we refer to different types of reality (see A brave new world of virtual, augmented, hybrid, hyper, and diminished realities).
Virtual Reality's New Game! Come hear illustrator and animator Chuck Carter, who helped create Myst and 26 other video games, talk about VR challenges in his keynote presentation, Playing a New Game: VR Challenges and Opportunities, at the Embedded Systems Conference (ESC), September 21-22, 2016 in Minneapolis (Click Here to register).
As we discussed in that article, augmented realities are really only a subset of mediated realities, which also encompass diminished or deletive realities. In this column, we're going to turn our attention to alternative presentation technologies and potential applications.
Virtual and mediated reality presentation technologies
One aspect of all of this that I've been pondering in particular is the way in which mediated (augmented and/or diminished) realities can and will be presented to the user. One way is to view the world through the "window" of a smartphone or tablet computer. A really good example of this is this video showing the real-time translation of signs.
Another example is the GoSkyWatch app that I have loaded on my iPad. When you hold your tablet up to the night sky, you see the same stars and planets on the screen as you do in the heavens, but the ones on the screen are annotated with names and the outlines of constellations and suchlike. Alternatively, you can call up a list of planets and the app will indicate which are presently visible in the sky. When you tap one of these, the app will guide you to its current location in the firmament. (As an aside, I'm not 100% sure how to class this application -- it's not exactly augmented reality and it's not really virtual reality -- I'd like to hear your thoughts on this conundrum.)
The GoSkyWatch app (Source: Max Maxfield / Embedded.com)
Now, I have no problem occasionally using my iPad to view things like the night sky, but I don’t particularly wish to spend all of my days wandering around brandishing a smartphone or tablet computer and beholding the world through its screen.
Something a little less obtrusive was Google Glass, which could have been useful to some extent, but which eventually went the way of the Dodo (having said this, Google Glass does appear to have found a second life in emergency medicine). I never actually tried this little scamp myself, but I don’t think it would have provided me with the immersive experience I desire.
At some stage in the future, it may be that people have virtual/mediated reality systems embedded in their eyes and other parts of their bodies. Before this occurs, I can envisage contact lenses that project virtual and mediated realities directly onto the retinas of their wearers (I have no idea how these would be powered).
Within my lifetime, I can imagine people walking around wearing eyeglass-style goggles that superimpose textual, image, and video data over real-world scenes. To the best of my knowledge, the leader in this space is a company called Magic Leap. Take a look at the video you'll find on the Magic Leap Homepage to give you a taste of what's possible.
(Source: Magic Leap)
But what about the near-term future? Well, I can easily conceive a world in which you see people walking around sporting Oculus Rift-style headsets. The idea is that these headsets would be augmented with two forward-facing cameras giving binocular vision. The default state would be for the images from these cameras to be projected on the displays in front of the wearer's eyes, thereby providing exactly the same view as if the user wasn't wearing the headset at all. Additional information would now be layered over the real-world scene.
An unknown stranger (Source: Max Maxfield)
At this point you might be thinking "Max has lost his mind; there's no way anyone would be seen wandering around in public looking like this." To this I would respond with a series of vignettes as follows:
- Punk rock arrived on the scene in the mid-1970s. I remember visiting friends in London around that time. At some stage during my stay, I found myself sitting on a tube train facing a business man wearing a suit and tie and sporting a briefcase and an umbrella. Sitting one side of this chap was a Goth in full makeup and regalia; on the other side was a Punk rocker wearing an interesting ensemble featuring straps and chains and flaunting a spanking-pink spiky mohawk haircut with the sides of her head shaved. (I myself now boast a "reverse Mohawk" -- it's a funny old world when you come to think about it.) The point is that, having witnessed this scene, I don’t think there's much that would surprise me.
- My mother-in-law likes to ride the crest of the technology wave. She was one of the early adopters of Star Trek-style Bluetooth earpieces. She also used to have her phone set to auto-answer. Once, when she came to visit, I took her to the local supermarket to pick up a few things. Since she has long hair, I had no idea she was wearing an earpiece. Suddenly, while we were ambling around the isles, she started talking animatedly to an unseen person while gesticulating wildly. I honestly thought she'd gone insane. Two things flashed through my mind: (a) How was I going to get her out of the store? and (b) When we got home, my wife (Gina the Gorgeous) was going to say "My God, you were only out with my mother for 30 minutes and you've driven her insane!" These days, of course, you think nothing when you see ladies in supermarkets waving their arms around furiously and talking to themselves (I do occasionally wonder how many of them actually have phones?).
- Way back in the mists of time, during the summer after I graduated from university, just before commencing my first job, some friends and I went to a Greek Island for a couple of weeks. As part of this, we visited a beach where... let's say clothing wasn't encouraged. You might think that this would be somewhat disconcerting but, strange as it might seem, if everyone around you as far as the eye can see is au naturel, then it's more embarrassing to be the one who is garbed.
The point of my meandering musings is that, if you were to see just one person strolling around a public place wearing an Oculus Rift-style headset as illustrated above, then you might think "That's odd" or "He looks like a plonker (but that's a nice Hawaiian shirt he's wearing)." On the other hand, if half of the people around you were sporting these headsets, then it wouldn’t take long before you ceased to notice them at all.
Virtual reality applications
To be totally honest with you, I don’t think any of us really have a clue where virtual and mediated reality technologies may take us, but I have a few ideas and thoughts I'd like to share.
One obvious application domain in the case of virtual realities is games. Until now, I've never considered myself to be a gamer, but I must admit that I love playing around on my Oculus Rift (see It's (Virtual Reality) Life Jim, But Not As We Know It and Minecraft + Oculus Rift Virtual Reality = Mind Goes Boom). However, let's think outside this box...
Another obvious application is education. Let's take the Colosseum (also known as the Flavian Amphitheatre) in Rome, Italy, for example. When I was a kid, we were lucky to get to see black-and-white pictures in books after walking three miles uphill through the snow in our bare feet (Ah, how I miss summertime in England). In later years, schools might show short 8mm educational films. Over time, these were replaced by videos. More recently, the teacher might play a DVD or stream online videos from some educational source. In the not-too-distant future, I can imagine the teacher and students all donning virtual reality headsets and setting off on a guided tour in which all of the participants can see each other's avatars. The ability to look around, focus on areas of interest, whisper comments to your friends in close (virtual) proximity, and even wander off and get lost ("Where's young Max gone now?"), would greatly add to the experience.
Suppose we have someone who handles IT for a large company that has offices and data centers across the globe. Further suppose that people start complaining about slow response times. I don’t know how the IT folks go about tracking down and resolving this sort of problem today... and maybe I've been watching too many science fiction films... but I can envision our IT guru donning her VR headset and using hand gestures to transport herself to a virtual birds-eye view above the facility from whence she can observe virtual network highways entering and leaving the building (think TRON). Maybe most of these highways are a healthy green color, while one is more of an angry red. So we "fly" along the red one across the country, up to a satellite, across an ocean to another satellite, down to a ground-based receiver, and across the land to another building. Now we fly into the building and on to the server room where we observe that one of the server cabinets is looking out of sorts. Then we plunge into the server cabinet where we observe a failing disk drive, and we swap this out for a replacement, all still in our virtual world. Again, I know this may sound like the script for a futuristic movie, but I really can picture (no pun intended) something like this coming our way.
Similar to the above, suppose you are sitting in the control room of a large industrial plant like an oil refinery that occupies square miles of real estate and has hundreds of miles of wiring and pipes and thousands upon thousands of switches and valves and sensors and actuators. Now suppose an alarm sounds indicating that something has gone wrong. Once again, I can envision operators wearing virtual reality headsets "flying around" the facility, homing in on visual indications of troubled mechanisms, looking at the values coming out of virtual representations of real-world sensors, and activating virtual representations of controls and valves where these actions are automatically replicated in the physical world.
What do you think? Am I on the fast train to cuckoo land, or do you think all of the above is a (weak or strong) possibility? Furthermore, while we're here, can you think of any other commercial and/or industrial applications for virtual reality technology?
Mediated reality applications
I'm still trying to wrap my brain around the concept of deleted realities, so -- for the purpose of this column -- let's focus on their augmented counterparts.
In the same way that many people tend to think of virtual reality as being targeted toward games and are surprised to discover there may be more professional applications, many folks tend to think of augmented reality as being targeted towards real-world purposes and are fascinated to realize that this technology may also be applicable in the gaming and educational arenas. As an example, we need go no farther than considering Pokémon Go, which rocketed onto the scene seemingly out of nowhere in July 2016, and had more than 20 million North American users per day after just a few weeks.
In order to give us all something to ruminate on while working our way through the rest of this column, check out this video showing Sesame Street augmented reality dolls and this video showing some very interesting augmented reality implementations.
Returning to serious augmented reality applications, can you imagine the pilot of a small airplane on final approach into an airport wearing an augmented reality headset and seeing a series of glowing rectangles superimposed over the real scene, where said rectangles are floating in the air and receding into the distance? (There was a similar scene in the first Alien movie). All the pilot has to do is fly the plane through the center of each of these rectangles to enjoy the most fuel-efficient landing path.
Similar to the above, the captain of a barge on the Mississippi River could use his augmented reality headset to view a series of virtual port and starboard buoys superimposed on the river accompanied by a dotted line showing the optimum path along which to steer the vessel.
How about when you're driving somewhere unfamiliar? Today's GPS systems are pretty amazing, and it's nice that they talk to you, but you still have to occasionally drag your eyes from the road to check the screen. Wouldn’t it be great to see a dotted line with arrows superimposed on the scene in front of you depicting exactly where to turn? The same thing applies if you find yourself walking around a strange city -- it would be incredibly handy to be able to specify a destination like a local museum and to be presented with a dotted line with arrows guiding you to your destination.
I don’t know about you, but I'm really bad when it comes to remembering names and recognizing faces. This can be a particular problem at conferences where people assume you will recognize them following a chance meeting several years ago. It would be tremendously advantageous to see small "info bubbles" floating above each person's head giving details like their name and company and position, perhaps color-coded to reflect whether or not you've met them before, and so forth.
Still on the topic of interfacing with people, it can sometimes be difficult to "read" someone's mood -- are they happy, sad, confused, homicidal? Wouldn’t it be useful to have this type of information presented via an augmented reality system? (I can imagine entering my house, seeing my wife, warbling "Hello love of my life," and -- when she turns to face me -- having my augmented reality system flash the message "Danger Will Robinson; run as far and as fast as you can!")
In a similar vein, how good are you at telling if someone is lying to you? (Did you see the article about an algorithm that can spot lies in emails and dating sites?) Suppose you were talking to a used car salesman who was singing the praises about one of his offerings and your augmented reality system displayed green tick-marks ("true") or red crosses ("false") in real-time accompaniment to each of his points as he made them. (Note that the system wouldn’t necessarily have to understand the spoken word -- it would be observing and analyzing the salesman's non-verbal cues.)
I could waffle on about this for ages. Instead, may I invite you to watch this video depicting the possible incarnation of an extreme form of augmented reality described as hyper reality (it's worth watching this all the way through).
All of this may seem a bit frantic, but I actually believe that the various capabilities it presents will be with us sooner rather than later.
Hold onto your hat!
The whole virtual and augmented reality scene really has my head buzzing with ideas. In the case of virtual reality, for example, can you imagine a next-generation version of Cluedo ("It was Colonel Mustard in the Library with the Lead Pipe!") taking place in a virtual mansion. Playing the investigator, you could move around the building manipulating objects (e.g. looking under a clock or behind a picture) and looking for clues.
Further imagine that the system was enhanced with speech recognition and generation capabilities, thereby allowing you to interview the characters, asking them questions and trying to detect any lies or omissions in their responses. If you've seen what's already possible with an Oculus Rift, you won’t regard any of this as being too farfetched.
In the case of augmented reality, I don’t think it will be long (relatively speaking) before these systems are enhanced by deep-neural network (DNN)-based embedded vision systems performing real-time object detection, identification, and classification tasks, with the results being presented on top of the scene. I can also imagine everyone's augmented reality systems being networked together, allowing everyone to respond appropriately in the case of an emergency.
In closing, I'd like to reiterate that I really don't think we have clue where virtual and mediated reality technologies may take us in the not-so-distant future. We see new announcements almost every day, such as turning one's arm into a virtual keyboard, using smart glasses to see the world in new ways, or the fact that Google recently decided to acquire an image recognition startup.
Personally, I cannot wait to see all of this unfold. How about you? Are you looking forward to the virtual and mediated future with trepidation or delight? What virtual and/or mediated applications would you most like to see? And what, if anything, scares you about all of this?