In August 2009, I (Joe) took the opportunity to attend the 2009 PGA Championship at Hazeltine National Golf Club in Chaska, Minnesota, with my wife, Julie. The fourth of golf’s majors moves around to a different place every year, but it had also been at Hazeltine, not quite an hour from our home, in 2002. That year, journeyman Rich Beem withstood a furious Sunday charge down the back nine from a red-shirted Tiger Woods (who birdied each of the final four holes) to win his first tournament, and a major at that.
We were there at the 11th green when Beem hit a 5-wood to within 6 feet of the pin, making the putt for the only eagle on that hole for the entire 2002 tournament. But we could not be everywhere down the stretch; with Woods playing in the group ahead of Beem, we had to pick and choose which player to watch from what vantage point and then listen for the roar of the crowd from the other hole to guess what had happened. We did not, however, have the same issue at the 2009 Championship. It was not just that Woods, finishing second once again, played together with eventual winner, Y. E. Yang, in the last group, for we still had the problem of finding a spot from which to see the action amidst the throngs.
The solution to our problem of being there live but not being able to see everything going on? FanVision. Formerly called Kangaroo TV, it is a small but substantive device—about four by eight inches, with a TV screen at its widest point and narrowing down to form a handle with navigation keys, plus an ear jack for listening and shoulder strap for easy carrying—that packs a powerful experiential punch. Whether following the lead group or ensconced in a particularly good spot, we now could keep up with everything else going on in the tournament through the FanVision device. We could watch the live TV feed or just listen to the audio, catch up on the current leaderboard or examine the course layout, and even drill down to the scores and shots by any player we so chose. It was a completely different golf-watching experience; for the first time at any tournament, I did not feel like I was missing out on what was happening elsewhere on the course. I would hear a roar and in moments would know exactly who did what where. Also available for NASCAR, Formula 1, NFL, and other sports events, it’s no wonder Time magazine placed FanVision on its list of best inventions of 2006.1
You can see how the FanVision device, with its associated software and services, effectively enhances sports experiences, augmenting the reality of the live event. It fits squarely into the realm of Augmented Reality, where, as seen in Figure 3.1, the Reality-based experience shifts by one variable, from Matter to No-Matter, from constructing an experience purely physical to one that uses digital substances to enhance the world around us. There is such an explosion of Augmented Reality applications (often abbreviated AR, but we’ll avoid that so as not to confuse it with Alternate Reality) that this chapter cannot help but touch on the possibilities. All the examples we discuss, though, share the essence of this realm: using bits to augment our experience of Reality, overlaying it with digital information constructed to enhance, extend, edit, or amend the way we experience the real world.
The quintessential example of Augmented Reality remains GPS car navigation devices (such as those made by TomTom or Garmin) that help you make your way to your destination. These are special-purpose gadgets, as is the FanVision. So is the Delivery Information Acquisition Device, or DIAD, which underpins the success of the United Parcel Service. UPS produced the first one in 1990 to automate scheduling, time-card reporting, and signature gathering, while today’s fifth-generation device includes GPS tracking of all drivers and a camera for proof-of-delivery—with direct communications between driver and customer not far behind.2
Increasingly, though, people can augment their reality with whatever smartphone they carry. Our favorite example is Amsterdam-based Layar. The company—whose name combines “layer” + “AR”—blends digital data with views of the physical world through what the company refers to as a “Reality Browser.” You view the real world through the built-in camera of your iPhone or Android handset, which then overlays a layer of computer-generated information on top of what you see, in actual time. You can access a layer to find a restaurant for lunch, for example, where as you pan around with your camera restaurants pop up whenever they are in your field of vision, with name, type of food, and the ability to access further information such as the menu. You can similarly look for apartments for rent, the nearest subway station, or the architectural history of buildings in front of you. As cofounder Maarten Lens-FitzGerald told us, Layar “shows you things you can’t see,” providing a “window on the world around you.”
Layar has over 1,500 such content layers—with thousands more coming as developers increasingly use it as the platform for Augmented Reality. Many fit right into an urban lifestyle, while others are particular to a locale. For example, if you direct your mobile phone toward Museo Pablo Casals in San Juan, Puerto Rico, you can access admissions information to the museum or see video clips of Casals, the famous cellist. You can take a 42-stop tour of London dedicated to the Beatles (you never know what you’ll see crossing Abbey Road). In many places you might also encounter a zeppelin or even a UFO in the sky above you, a 3D work of art on the grounds before you, or a new song released just in a particular spot for those “in the know.”
While every Layar layer has to be individually developed, Google wants to bring the entire Web to users of its Android smartphone operating system through Google Goggles. It calls the function of this program “visual search,” enabling people to just take a picture of whatever item on which they would like to know more—such as a building, a painting, or a store—and up pops information found via Google search. One obvious application: comparison shopping in stores, where a quick picture of a product in one store could bring up its price at all the other stores, real or virtual, on the planet. Christine Perey of Perey Research & Consulting says that Augmented Reality holds the promise of turning the world into one huge “interactive catalog.”3 Or how about a dictionary? Word Lens, from San Francisco-based Quest Visual, lets you look at Spanish text through your iPhone camera and replaces it with the word-by-word English translation (or vice versa)—even using the same color and font! Although it’s nowhere near perfect, technology reviewer David Pogue of the New York Times calls it “software magic.”4
All such offerings, known generically as “location-based services,” use the ability of cell phone towers to triangulate your personal location—or the more accurate GPS capabilities increasingly built into smartphones. Sam Altman, CEO of Loopt, whose offering even says who is near you at this very moment that you very well might want to meet, stresses that “while most technology isolates us behind computer screens and virtual worlds, location-based applications help us discover more, experience more, and connect with others in the real world. It’s technology to save us from our technology.”5
We increasingly see location-based experiences as well. “Jewel Collector” lets Layar users find virtual jewels scattered about to earn points and compete with friends. With Foursquare, players “check in” to various locations in their city and around the world, alerting friends (often via Twitter) to their whereabouts, tracking their movements, and uncovering new places to be and recommendations for what to do, eat, drink, or buy there. All the while players earn “check-ins” and, for particular venues at particular times or a set number of visits, badges that display on their Foursquare profile. Check in at a venue more frequently than anyone else, and you can even become “Mayor” of the place! As Nathan Rice, our own social media strategist at Haberman, Inc., in Minneapolis and an avid Foursquarer, told us, “Foursquare helps me both connect with friends and get recommendations at each venue I visit. I have to admit, though, that my real motivation for using it is the gaming. I love building and maintaining my Mayorships, winning badges, and even uncovering the mystery of what some of the badges are.”
Each such location-based experience augments our quotidian existence with not just an informational overlay but by superimposing experience elements onto our daily lives—and into our vacationing lives as well. Recall those old coin-operated viewers at the top of places such as the Empire State Building in New York (“Look! I can see the Statue of Liberty!”) or in tourism locations such as Fisherman’s Wharf in San Francisco (“Look! I can see Alcatraz!”). Well, YDreams of Lisbon offers an “augmented reality scenic viewer” at such places as the Pinhel Castle in central Portugal and the National Pantheon in Lisbon. You can spin it around to view the environs through its camera and then access further information, additional images, and video stories to enhance your learning (“Look! Do you know who that statue represents?”).
Museums have for decades enhanced the visitor learning experience through audio devices that describe objects on display. They democratize the experience by providing a virtual expert for the masses, as if a curator were at your beck and call throughout the time you spend there. Now digital technology enables us to go beyond simple audio tours, with companies offering increasingly sophisticated (and immersive) augmented experiences. The Leeum, Samsung Museum of Art in Seoul rents a “digital guide”—a full-fledged PDA—that allows visitors to access not just audio but also pictures, video, and additional information on its collection. And the Museum of Natural History in Berlin employs a “Jurascope,” developed by ART+COM and WALL AG, through which visitors can see animations of the dinosaur skeletons on exhibition come to life: first come the organs, which are overlaid by the muscles and then the skin, after which the animated dinosaur begins to move around (“Look! I can see what the dinosaur was really like!”). And while the Jurascope currently is limited to showing animations, the Digital Binocular Station from MindSpace Solutions of New Zealand can overlay the real objects themselves and bring them fully to life to interact with the viewer in 3D with immersive audio.
Sports teams also increasingly apply Augmented Reality to their in-stadium experience, going beyond special-purpose devices like the FanVision and connecting to the smartphones patrons already bring with them to the games. Those attending Pittsburgh Penguin games at the Consol Energy Center or Steeler games at Heinz Field, for example, can connect their phones to technology from YinzCam to watch, in realtime, the action from multiple camera angles, access up-to-the-minute statistics, or see instant replays automatically streamed to their seats.
Augmented Reality can also be used for serious business—even though the previously mentioned businesses are all “serious” businesses,6 this realm is fast becoming the basis for a number of sophisticated medical and industrial applications. For example, the VeinViewer, from Christie Medical Innovations of Memphis, takes an image of a patient’s veins via near-infrared light and then superimposes that real-time digital image back onto the patient’s skin so clinicians know exactly what to look for and where to find it. By seeing what they could not otherwise see, the technology takes the guesswork and unnecessary needle pricks out of drawing blood or starting IV drips. And medical device maker Medtronic provides a number of hardware, software, and service solutions to hospitals under the banner of “The Medtronic Navigation Experience” to “allow surgeons to precisely track their surgical instruments in relation to patient anatomy, even as that anatomy is shifting in realtime.”7 For such procedures as spinal fusion, this technology virtually extends the surgeon’s vision into the opaque body itself and, more importantly, increases knowledge, ensures accuracy, and improves outcomes.
Such uses not only augment reality, but they effectively amend it as well. This occurs whenever the digital overlay changes what we do as a result of what we see. This may be particularly important in industrial applications, which can vary from a simple system that illuminates the right part to pick on an assembly line, and indicates the spot where it goes, to a head-mounted display that lets workers completely visualize the task ahead of them as it changes product by product. InterSense of Billerica, Massachusetts, calls its more advanced solution AuRAM, for Augmented Reality Advanced Manufacturing System, which precisely tracks the movement of everything on an assembly line to visually overlay virtual parts on top of physical products.8
The aerospace and automobile industries not only lead the way in applying such systems to their assembly lines but also in providing Augmented Reality to the users of their end products. Military and even commercial pilots have long used heads-up displays (whether via helmets or windshields) to access information about their aircraft and flight, and now the same sort of technology is coming to car drivers as well. Mercedes-Benz uses radar to detect vehicles in a driver’s blind spots, for example, turning on a red light in the respective side mirror if a vehicle is present. Should the driver turn on his signal to indicate a lane change toward such a vehicle, the light flashes while an audible alarm goes off. Ford, meanwhile, places similar technology on the Taurus and adds a Collision Warning with Brake Support system that detects potential collisions ahead, flashing a red warning light directly on the windshield as it provides additional brake support.
Wired magazine parodied—or, rather, predicted—the direction of Augmented Reality in cars with one of its “Artifacts from the Future” features on the last page of its January 2008 issue.9 Produced by Chris Baker, it showed the driver’s eye view of the futuristic (and as yet fictional) “Hitachi 2400 Smart Windshield,” showing not only the speedometer and navigation displayed on the windshield but the current weather, a videoconference call in progress, the turnoff for (naturally) the nearest Starbucks, and—our favorite—the location of a car in the next lane, highlighted in red, whose driver, one “Han, Jennifer // Age 23,” is driving with an expired license and had a DUI conviction in 2012. (We only hope the police departments of the future can afford the same technology.)
And, really, we’re not that far off. While not yet ready for prime drive time, General Motors is developing an “enhanced vision system” for later this decade that uses external cameras to see what is ahead, as well as internal sensors to ascertain the position of the driver’s head and eyes, so it can superimpose visual information on the car windshield. It will highlight speed limit signs, draw the edge of the road under foggy conditions, point out animals near the car, and aid navigation not with a dash-mounted device that takes the driver’s eyes off the road but with superimposed images on top of the physical road itself.10 Can indicators for the turnoff for the nearest Starbucks really be very far behind?
It is important, however, to not make the mistake of limiting this realm of the Multiverse to visual devices; any way of constructing digital technology to enhance reality qualifies, no matter what sense it engages. In the not-too-distant future when your new car is equipped with the sort of “enhanced vision system” described in the previous section, you likely will interact with it via your voice, much as a GPS navigation system already provides audio directions to you via its recorded voice. Certainly the visual will continue to lead, but already we see applications like Shazam, a smartphone app that you can hold up to “listen” to any musical track, which it can then identify for you (and make it easy to purchase).11 Or RjDj, from Reality Jockey Ltd. based in Bizau, Austria, which provides “the soundtrack to your life”—where “YOU are inside the music,” as its iTunes app description attests12—by incorporating whatever sounds happen around you (or by you) into its “reactive music” mix, played into your ears through your iPhone headphones. And then there is LookTel, from Santa Monica-based Ipplex, which lets visually impaired people point their smartphone camera at objects (such as money, medications, or merchandise) and have the app identify the object and state it out loud.
Or consider tactile augmentation. LeapFrog Enterprises builds its entire business around embedding digital technology into education toys to enhance learning via touch. Its latest Tag Reading System lets a child touch words on “Tag-enabled” books with a special digital “reader” to hear the word aloud or get its definition. The technology can read the entire book along with the child, play special sound effects for pictures or characters, or even launch learning games. People of any age can similarly use the Livescribe Echo digital smartpen to remember what they write down (and, with its microphone, hear) for recall later, and workers can apply Sweden-based Anoto Group’s ExpeData Digital Writing Platform to record, transmit, and recall handwritten text, such as for invoice writing or package signing.
Researchers continue to push forward the boundary of such tactile augmentation. On the near horizon, the University of Tokyo’s Ando-Shinoda Laboratory is working on “Touchable Holography,” holograms that you can physically touch—at least that is the sensation created by the “Airborne Ultrasound Tactile Display.” It uses pinpoint ultrasound projectors and hand-tracking sensors to create tactile sensations wherever a digital object is visually projected to be in the physical environment, creating, for example, the ability to “feel” holographic raindrops or the surface of any 3D shape.13 At McGill University in Montreal, researchers “developed floor tiles that can simulate the look, sound, and feel of snow, grass, or pebbles underfoot.”14 And already out of the research lab is GaitAid, from Israeli-based MediGait, which helps those with diminished senses of touch and sight (such as many who suffer from Parkinson’s disease) by superimposing a checkered pattern over the floor or ground via a headset and making a sound with every deliberate step. This additional sensory input helps patients normalize their gait and enables them to walk more effectively where they could not before.
Beyond such tactile enhancements lies kinesthetic augmentation. Consider the work of Hugh Herr, the head of the Biomechatronics group at the MIT Media Lab. After losing both legs in the aftermath of a mountain-climbing accident, Herr dedicated his life to developing new kinds of prosthetic limbs—“smart” prosthetics embedded with digital technology. For example, his microprocessor-controlled knee, commercialized by Reykjavik’s Össur as the Rheo Knee, constantly senses the terrain and how its user walks to make adjustments that make the gait natural. His PowerFoot One, a powered ankle-foot prosthesis that Herr is bringing to market himself via iWalk, based in Cambridge, Massachusetts, has shown much promise in clinical trials: “Early results indicate that it significantly reduces wearers’ metabolic costs while increasing their chosen walking speed. It may even bring the metabolic economy of movement above the levels experienced by people with intact limbs, the world’s first limb prosthesis to do so.”15 As this indicates, Herr desires not just to provide amputees with capabilities of walking or grasping but actually to improve on “normal” for the physically challenged, giving them a “mobility platform” with greater capabilities—more strength, higher jumping ability, a longer reach, increased endurance—than they (and potentially anyone else) ever had before.16 Like baseball pitchers having Tommy John surgery to “repair” uninjured ligaments17 or using Lasik surgery to improve their eyes beyond normal to 20/15 (as Tiger Woods has done, raising some controversy18), one day people with no limb loss may even choose prosthetics to increase their natural capabilities.19
Another research center at the MIT Media Lab, the Fluid Interfaces Group headed by Pattie Maes, developed SixthSense to access the information in the real world we wish we had available at our fingertips—and in this case controlled by our fingertips. Developed by her student Panav Mistry, SixthSense consists of an off-the-shelf Webcam, a smartphone, a pico-projector, and a mirror to shoot images forward onto any surface, which becomes the display; a user wears the assemblage like a pendant around the neck. The device recognizes the movements of the user’s hands via the webcam and color-coded finger-gloves or tape worn on the index finger and thumb of each hand (which Mistry plans on eventually eliminating). This enables a natural “gestural interface,” where, for example, the four fingers forming a classic “frame” gesture make the device snap a photo, or holding up the left hand causes the projector to display a phone keypad on it, which the right hand can then dial. In a bookstore, the device could recognize a book the user selects and project information onto it, such as its Amazon.com rating or its price at competing stores in the area. A newspaper could trigger the device to search for relevant news video clips, and a person in the line of sight might prompt the display to show his contact details.20
Many have likened its gestural interface to the one used by Tom Cruise’s character, John Anderton, in Steven Spielberg’s Minority Report (2002).21 The scientist who advised the Minority Report production, John Underkoffler of MIT’s Tangible Media Group, founded LA-based Oblong Industries to develop the G-Speak Spatial Operating Environment, which he describes as “an interface that has no interface,” adding, “You operate the world as you operate the real world, which is to say, with your hands.”22 G-Speak effectively unites the Space and No-Matter of Augmented Reality. As Keith Kelsen, founder and CEO of 5th Screen, explains, “What Oblong has been doing is essentially teaching the machine about space and its position in it, using what is essentially a new concept for an operating system. In this new world, the machine no longer thinks of the screen as a flat abstract collection of pixels but as a real object, in the real world, that exists at a particular location … and has a relationship to other things in the environment based on that location.”23
The sense of smell is also a part of our environment, but olfactory augmentation may not yet make sense—DigiScents went belly up trying to develop the first Internet-enabled smell machine, the iSmell, and NTT did not make much of a (fragrant) splash when it demonstrated a similar prototype in Japan. At least Procter & Gamble made it into production for a few years before discontinuing its Febreze Scentstories machine, which “played” discs of scents that slowly changed over the course of an hour. But I (Joe) can still remember a decade later the odors of both good and bad wine emanating from their respective smell machines at Vinopolis in London—for no sense sparks memory more effectively than smell. As reality gets more and more enhanced with digital technology, olfactory augmentation will make increasing sense, although actually less so with Augmented Reality—where the real world remains right there in front of us—than with Virtuality, where over time it may become not only desirable but necessary to add aromatic elements to complete the experience (and thereby shift it over to Augmented Virtuality).
Throughout this tour of Augmented Reality we have seen how companies use digital technology to enhance the lives of people, quite often with rather specialized technology—from navigation devices to heads-up displays, from FanVisions to LeapFrogs, from Jurascopes to Vein-Viewers, and even from smart prosthetics to smell machines. Over time the technologies for augmenting reality have grown significantly more advanced (and less material).24 Today companies are working on embedding tiny projectors onto people’s normal glasses, providing information while still being able to see the real world in front of them. Researchers even hope to develop contact lenses that project images seemingly in front of the wearer’s eyes.25 Increasingly, however, dedicated devices are swiftly being replaced by the increased functionality available on smartphones. TomTom and Garmin, for example, face tough times competing against Google Maps Navigation, AroundMe, and other such apps available for free or next to nothing, without any special equipment on the dash.
Whatever the underlying technology, Augmented Reality supplies a flow (either constant or on demand) of sensory information and enhanced experience to the individual. So in a very real sense all of these devices are prosthetics, extending our sight and, increasingly, our other senses—and our mind as well. When viewed through the lens (pun intended) of digital prosthetics, this realm of experience also creates great potential for extending, even transforming, our very selves.
Few people, however, desire to realize Star Trek’s vision of the Borg, a race that assimilates both technology and other species into a Collective in pursuit of constant enhancement, if not perfection.26 Resistance is not futile, for, as with all technologies, we maintain the personal choice to buy or not to buy, to use or not to use, to turn on and to turn off. Many prefer their sporting experiences undisturbed by the vital statistics of players and unmediated by the images of what is happening out of direct sight or current reality, whether forced on them by giant, flashing scoreboards or personally accessed by tiny, handy mobile phones. Some only want to avail themselves of the old possibility to just be.
So before choosing to play in Augmented Reality, you have to figure out how to provide the value many of your guests desire without perturbing the rest. And always keep the needs and wants of your guests—what the humans in the experience truly value—paramount over the technology. As pioneer Robert Rice of Raleigh-based Neogence put it to us, “It isn’t about pages, servers, websites, or everything we have created over the last two decades. Augmented Reality is about WHO you are, WHERE you are, WHAT is around you, WHAT you are doing, and WHO is nearby.”
When you embrace this mindset, you can create great value by understanding the essence of Augmented Reality: overlaying a reality-based experience with digital technology in order to enhance that experience, making it more informational, more effective, more engaging, more memorable. While many, if not most, of the examples cited here in this fast-moving realm will be superseded by the time we publish this physical artifact of a book (or shortly thereafter), these principles will always apply:
∞ More so than with any other realm (outside of Reality its-self-contained-self), use the real world as the background for the Augmented Reality experience. So if you already have a reality-based experience—a store, an event, an office, a room, or any other physical place—consider what information or immaterial experience you might overlay atop the physicality you already have. If you do not have such a place, no worries—the entire world can provide the background if you can figure out new ways of letting people interact with it digitally!
∞ The opposite is also holds true: Use the real world as the foreground for the Augmented Reality experience. Precisely because the real world is so important to this realm, do not let the technology take preeminence over the base reality.
∞ While long requiring special-purpose devices, increasingly you can embrace the smartphone as the platform for augmenting reality. Do you have an app for that?
∞ Whatever it may be, view the augmentation device as a prosthetic extension of the body and/or mind. In what ways, to what directions, for what purposes might you extend your customers’ capabilities?
∞ In so doing, do not view this realm as purely one of informational overlays, but always use digital technology to augment the experience as well.
∞ Moreover, enhance real-world experiences not only visually but also along audio, tactile, and kinesthetic dimensions (with smell and taste best left to Augmented Virtuality, if engaged at all).
∞ Use Augmented Reality for tracking—whether the location of a destination, friends, UPS packages, surgical instruments, assembly line parts, or anything else your customers find of value—and other location-based experiences. (Note that this realm shares tracking as a principle with its cousin Mirrored Virtuality, as both link directly to real time, the one in physical places and the other in virtual places.)
∞ Enhance learning with Augmented Reality, which is appropriate not only for museums and schools but for all forms of lifelong learning, including business skill training. What do your customers need to learn, and especially how might you help them do it on-demand, in real time?
∞ By its very nature, augmenting reality mediates that reality. Recognize that not all your customers will want that experience mediated, nor want to be bothered by those around them who are mediating their personal experience, so tread carefully in augmenting reality for those that want it while not being a nuisance to those that do not.
∞ In many situations you can also use Augmented Reality to amend reality. Beyond information and experience, how might you help your customers change their environments, revise their results, or even transform themselves?
∞ Finally, understand very clearly that Augmented Reality is not about the technology, but about how that technology enhances people’s experiences, lives, relationships, and selves.
So, yes, think of all the fun ways you can apply Augmented Reality, but recognize too its potential for serious fun, even for transformation. If you have a reality-based business, there is no doubt whatsoever that you can augment that reality through digital substances in interesting and innovative ways. What are the right ways for you, and for your customers? For, as MIT scientist Herr declares, “The only limits are physical law and the boundaries of human imagination.”27
