Although virtual technologies have been around in one form or another for a few decades, the hype has mostly exceeded the reality. During the 1990s and beyond, XR experienced one false start after another. Vendors have consistently overpromised and underdelivered for products and features. Graphics were mostly subpar, performance was jittery, and, in the end, the user experience ranked somewhere between mediocre and awful. It usually didn’t take long for the novelty to wear off and a headset or glasses to wind up collecting dust on a shelf or tucked away in a desk drawer.
Virtual technology is finally taking shape, however. As systems and software have advanced and the hardware has shrunk, AR, VR, and MR have begun to filter into daily life. Rachel Ann Sibley, a futurist, consultant, and part of the faculty at Singularity University, believes that almost every corner of the world and every industry will be disrupted by virtual technologies in the years ahead. “There are a lot of reasons why we have not yet seen mass consumer adoption,” she stated at a Trace3 Evolve conference in 2018. Yet, “every leading technology company is vying for leadership in this space.”1
What makes extended reality so disruptive and so potentially valuable? Sibley believes that it is the technology platform of the future. AR, VR, and MR put data, images, and objects into shapes and forms that are natural and understandable. In fact, she refers to augmented and virtual reality as the “connective tissue” of the future. Extended reality bridges the psychological chasm between machines and humans. “Every physical object can be abstracted, demonetized, democratized, digitized,” she explained. So far, most of these digital objects—cameras, calendars, address books, voice recorders, and specialized apps—have resided in our smartphones.
But 2D is not 3D. A flat screen is not immersive, no matter how beautiful or brilliant the display. Simply put, a smartphone can’t create a complete sensory experience. Extended reality takes computing and digital consumption in an entirely different direction. AR, VR, and MR fundamentally change the way we learn, shop, build, interact, and entertain ourselves. These technologies produce new ways of thinking by rewiring sensory processing. They make things possible that once seemed unfathomable: touching a rare fossil, swimming with sea turtles, soaking in the grandeur of Niagara Falls, and experiencing the magnificence of the Taj Mahal, Eiffel Tower, or Lincoln Memorial.
Let’s take a brief tour of how extended reality will likely play out in the world and how it will shape and reshape several industries and disciplines.
The internet has already changed education in remarkable ways. It hasn’t replaced physical or virtual books, which remain a powerful tool for critical thinking. But it has enabled distance learning, spawned new types of online interactions, and introduced entirely different ways to view and explore data and information. At some point, however, human interaction is essential. Although online spaces are ideal for allowing users to sift through materials at any time and from almost any place—while progressing at their own pace—the downside is the lack of intuitive interaction that takes place in a classroom. Online learning doesn’t deliver a particularly dynamic experience.
It’s clear that virtual technologies will fundamentally change the way we learn, shop, build, interact, and entertain ourselves. They will open up new vistas of thinking—and sensory processing. They will also make things possible that once seemed unfathomable: touching a rare fossil, swimming with sea turtles, exploring a spaceship, and experiencing a famous edifice such as the Taj Mahal, Eiffel Tower, or Lincoln Memorial.
AR and VR take direct aim at this challenge. Suddenly, it’s possible to experience the familiarity of a real-world school without traveling to a campus and setting foot in an actual classroom. Using avatars, virtual representations of physical objects and places, and graphical interfaces and models, participants can step past 2D tools like messaging, chat, video, and whiteboards and experience a lecture, discussion, or virtual field trip in a lifelike way. They can view things—from molecules and mathematical formulas to interstellar objects—in shapes and forms that make sense. Ultimately, the technology allows people to interact in a more natural and seamless way.
Academic studies support the notion that AR, VR, and MR are effective tools for enabling learning. Extended reality also makes learning more appealing and enjoyable. For example, research conducted at the University of Saskatchewan in Canada found that the use of VR improved learning accuracy by about 20 percent for medical students studying spatial relationships.2 When the research team tested participants five to nine days after undergoing instruction, those who learned through virtual reality scored higher than those who used textbooks. “A week later it seemed like I was able to go back into my mind and bring back the experience,” said one participant in the study.3 Likewise, a 2018 study conducted by researchers at the University of Maryland showed an 8.8 percent improvement in overall recall accuracy.4
Extended reality introduces opportunities to expand learning from kindergarten through college—and beyond. Students might visit a zoo, step onto the surface of the moon, view the signing of the Declaration of Independence, stroll through graphical representations of molecules in their chemistry lab, or watch a volcano erupt in Iceland. They might join virtual lectures and engage in virtual discussions. Already, a handful of schools are experimenting with AR, VR, and MR technologies. For instance, the Washington Leadership Academy, an open-enrollment public charter high school in Washington, DC, is reinventing learning through digital technology, including extended reality.5 The initiative was partially funded by Laurene Powell Jobs, the wife of former Apple CEO Steve Jobs.
An enormous challenge for businesses of all shapes and sizes is training employees. In the United States alone, companies spend upward of $70 billion annually.6 It’s no secret that skills change rapidly. Critical knowledge a few years ago is now out-of-date. What’s more, as digital technologies become more pervasive in all industries and lines of work, there’s a growing need to learn new skills. Although online training has revolutionized the way many workers update their skills and knowledge, it has never delivered the realism of assembling a jet engine, mastering a sales technique, or installing a flow control valve at a nuclear power plant.
Augmented reality presents opportunities to enhance training by putting manuals, technical specs, and other data on glasses or lenses as a person learns or performs work. Virtual reality reduces, if not eliminates, the risks associated with many types of training. It’s possible to be prepared to walk into a structure that’s on fire or deal with terrorists holding a hostage before stepping into a real-world situation. Moreover, individuals gain important skills before working with valuable equipment or supplies. The fact that VR removes the need for physical classrooms and transporting people to a specific spot at a specific time is no less appealing.
One company that understands the value of XR is British aircraft engine manufacturer Rolls-Royce Holdings. It has turned to virtual reality for far better results in manufacturing.7 Rolls-Royce uses the technology to teach workers how to assemble critical components in jet engines used on aircraft. A worker dons a VR headset and then steps through the entire process of placing specific components and parts in a gearbox. If the technician makes a mistake, the system issues a warning and forces the technician to start over. Once a worker learns the right sequence and placement for parts, it’s time to move on to the actual assembly station and work with actual engines.
There’s also the ability to hold conferences and symposiums in virtual worlds and mixed-reality spaces. Accenture is developing a teleportation system that allows participants to view a virtual representation of a speaker, view avatars for other participants as they ask questions, and virtually walk through spaces and view booths and exhibits that are part of the physical world. “As bandwidth has improved, video quality has gotten better, and digital technologies have connected to one another, we have advanced online meetings from simply connecting to other people to more advanced telepresence and now virtual and augmented realities,” explains Emmanuel Viale, a managing director at Accenture Labs.
In fact, Viale and a team of researchers are designing the conference room of the future from a lab in the south of France. They have combined scanned and mapped 3D models of physical things—walls, spaces, and objects—with immersive virtual spaces so that 10, 20, or more people can congregate to view presentations, watch videos, listen to lectures, ask questions, and interact. Everything—video, audio, avatars, and movements in the space—are synced in real time. There is also an element of augmented reality within the virtual-reality environment. It’s possible to view names, titles, and information about avatars that represent participants and also view data about objects visible in the space. This includes everything from videos to people and things.
Accenture researchers are also exploring other ideas. One of them is virtual autographs. In the VR environment, participants would capture data about another person, with his or her permission, and the system would automatically recognize and reproduce the avatar or representation in the future. Another idea is creating a framework to support people in the physical world interacting with people in the virtual world, without both groups being together in the same space at the same time. In order to make all of this possible, some participants would wear headsets and venture into an immersive virtual environment, while others would wear smart glasses or lenses—or use a smartphone—to interact. The technology blends all elements and groups into the same digital space.
Other mixed-reality environments are emerging as well. Microsoft’s SharePoint Spaces allows participants to share documents, images, videos, and 3D graphics within a 360-degree shared virtual space that’s visible on a web browser. The application, which doesn’t require a special headset or glasses, lets participants interact in a more lifelike way on their PCs. The platform might help new employees explore an office or a campus, or a business team view data in more visual and understandable ways. Microsoft offers the environment within its popular Office 365 platform.
In 1998, the National Football League (NFL) introduced augmented reality to telecasts. The initial use of the technology was to help viewers see first down lines that are not visible on a physical field. The “First and Ten” system, which features an augmented yellow line projected onto the field, has become a standard feature for televised NFL games. In recent years, television networks have added virtual scoreboards along with graphics that show fans the anatomy of plays, replays, stats, and other information in formats that make a telecast more compelling and enjoyable.
Other sports leagues are also transforming telecasts through AR features. The National Basketball Association (NBA) introduced Virtual 3 in 2016. It highlights the three-point line when a player takes a shot outside the arc. Sports network ESPN has also experimented with AR for major league baseball (MLB) games. This includes a “K-Zone that shows balls and strikes. The objective, in all these cases, is for fans to have a better idea of what’s going on in a game, particularly when it’s something that’s difficult to spot with the human eye. Not surprisingly, many of these AR features borrow techniques that first appeared in video games.
Virtual reality is also scoring points in the sports world. VR creates opportunities to insert fans into the action—and view plays from various angles and perspectives. It introduces ways to view graphics and information that aren’t possible on a TV screen or at a ballgame. Consequently, professional sports leagues, including MLB, the NBA, and the NFL, have begun broadcasting select games and highlights in virtual reality. In addition, at the 2016 Summer Olympics in Rio de Janeiro, Brazil, NBC and the BBC streamed some events in virtual reality. At the 2018 FIFA World Cup, all 64 games were available to viewers in VR via an app from the television network Telemundo.
VR is still a work in progress in the sports world, however. The biggest problem, for now, is that the video resolution in virtual-reality environments doesn’t match the most advanced high-definition television sets that display images at 4K. Many viewers have described the VR experience as incredible but the graphics as underwhelming or disappointing. Televising games in VR also requires substantial human and technical resources. For instance, a virtual-reality NBA game broadcast in 2017 required a crew of 30, a TV production truck, banks of cameras, and three announcers.8
Analysts have speculated that in the future VR could be used for college recruiting—allowing a recruit to visit schools, view facilities, and even check out a weight room or arena, without actually setting foot at the school. It could also become part of an athlete’s training regimen. Golden State Warriors guard Stephen Curry, a league MVP and NBA champion, announced in June 2018 that his personal trainer was working on a three-minute pregame drill that uses VR. “We always look for new things, just to keep him stimulated and to keep pushing his workouts forward,” his personal trainer, Brandon Payne, stated.9
Architects, engineers, and designers were among the early adopters of augmented and virtual reality. They now use XR and 3D modeling tools for visualizing spaces as well as prototyping things as diverse as buildings and automobiles. This allows them to see what something will actually look like before it’s built—and spot flaws that could lead to cost overruns, design errors, or safety hazards. AR and mixed-reality tools also allow a person to step through a virtual space—say an office or museum—and view technical specs and information about electrical systems, plumbing, and other elements. Wearing a pair of AR glasses during the construction phase, data and graphics are superimposed over an actual physical view of the room or object.
The technology boosts productivity and cut costs. It allows organizations to operate in a more agile and flexible way. “There are an enormous number of industrial uses for extended reality,” points out Mary Hamilton, managing director of Accenture Labs in San Francisco. “These technologies introduce capabilities that simply didn’t exist in the past. They fundamentally change and improve processes.”
Aerospace giant Boeing has experimented with Google Glass Enterprise Edition to streamline assembly processes in its factories. Typically, aircraft require assembly of tens of thousands of individual components, including wires that come in myriad shapes and sizes. In recent years, engineers and assemblers have used PDF files to view assembly instructions. However, using a keyboard while engaged in the assembly process is tedious and slow. Navigating a keyboard also increases the risk of error. Instead, the AR system delivers specific contextual information about how to find specific wires, cut them, and install them.
Although Boeing has experimented with heads-up displays in the past—the earliest pilot projects at the company extend back to 1995—Google Glass helped the initiative get off the ground. Technicians authenticate themselves by simply putting on the glasses and scanning a QR code. They immediately view detailed wiring instructions and other crucial data in a corner of the eyewear. The app that drives the system, Skylight, incorporates voice and touch gestures. This makes it possible to glide through steps seamlessly and ensure that the right instructions and assembly phase are visible at any given moment. Glass and Skylight helped workers decrease the assembly time by 25 percent while significantly reducing error rates, according to a Boeing executive who oversaw the pilot project.10
Ford Motor Co is also using VR to build automobiles. It has created an immersive vehicle laboratory. In 2013, the auto manufacturer examined 135,000 items on 193 vehicle prototypes without building a physical model.11 In the future, AR and VR will be used across industries for assembly instructions, maintenance, technical support, quality assurance, and much more. One device, DAQRI Smart Glasses,12 essentially brings a control panel to a person’s eyes. This means a worker can roam through a factory and handle tasks and functions as the need arises. It’s possible to video chat, view critical specs, review 3D models, and manage digital controls hands free.
Extended reality is already reshaping the way consumers view videos and movies and participate in games. In 2016, the augmented-reality game Pokémon Go captured the hearts and minds of millions of people around the globe. Suddenly, participants combed backyards, street corners, parks, and other public places in search of virtual creatures with names like Bulbasaur, Charmander, and Squirtle. When all was said and done, the AR treasure hunt—which resembled nothing before it—resulted in upward of 800 million downloads worldwide.13 More important, it set the stage for a tsunami of AR games that followed along with more widespread acceptance of augmented reality.
Immersive virtual-reality gaming is also advancing. In Catan VR, for example, the popular board game becomes a rich interactive 3D experience. Objects, characters, and avatars emerge from the virtual board, scurry about, and engage in a variety of actions and activities. In Ark Park, gamers enter a spectacular futuristic fantasy world where they ride and hunt dinosaurs. Star Trek: Bridge Crew assembles participants from all over the world. People take on different tasks—navigating, operating the engines, managing the weapons—as the USS Aegis explores uncharted sectors of space in the pursuit of finding Vulcans a new home. Along the way, there are dangerous objects and attacks from Klingons.
Gaming and entertainment are also taking on new and different forms. In 2016, for instance, a movie theater in Amsterdam became the world’s first permanent VR movie cinema,14 in which theatergoers plunked down €12.50, plopped themselves into a rotating chair, and strapped on Samsung VR headsets along with a pair of headphones. They watched a specially made 35-minute film and, by rotating their seat or turning their head, they could view the virtual reality action in 360 degrees. Although the theater shuttered its doors in 2018 the concept lives on at pop up VR cinemas across Europe. In Japan, cinemas also show films in virtual-reality formats.15 Mainstream studios such as Paramount and theater formats such as IMAX have experimented with VR as well.16
The capabilities of virtual reality are clearly expanding. For example, VR “theme park” The VOID thrusts visitors into a Star Wars battle on a molten planet and recover “intelligence” vital to the rebellion’s survival. Participants, wearing helmets, gloves, and special vests, see, feel, and hear everything around them as if they were actually in a physical space. Specialized hardware and software—including haptic gloves, motion tracking systems, and built-in special effects—make it possible to move around in the holodeck. Without goggles, the rooms are essentially bare walls and stages. With the headset, they become completely immersive spaces. A door becomes an elevator; a hallway becomes a bridge to a spaceship. There are controls, enemy storm troopers, and monsters at every turn.
Consumers are also exploring virtual reality in arcades and from the comfort of their sofas. With an HMD such as the Oculus Go, PlayStation VR, or HTC Vive, it’s possible to ride an extreme roller coaster, battle an army of homicidal robots, race cars through a rocky desert landscape, fly spaceships, embark on a treasure hunt for hidden gems, and even play popular games. What’s more, visual programming tools such as Microsoft’s AltSpaceVR, help users design and build their own virtual worlds—and share them with others.17 This essentially adapts the concept of the popular online space Second Life to the 3D virtual world.
In the coming years, as graphics and displays improve, haptic gloves advance, body suits and exoskeletons emerge, and more advanced motion tracking takes shape, ultrarealistic immersive gaming will become the new normal. People will also attend virtual plays and music festivals that, in some cases, take on novel forms.
The travel industry is also likely to change significantly as a result of VR and AR. By the end of 2018, worldwide online travel sales had reached about $700 billion.18 And while websites and apps provide a wealth of information about destinations and getaways—they allow consumers to book flights, cruises, resorts, hotels, and tours—it can be difficult to know what a place is really like before physically setting foot there. There’s also the fact that a lot of people prefer to travel vicariously, or, because of physical limitations, cannot travel to faraway places.
In a virtual space, it’s possible to explore the glaciers of Greenland, the Buddhist temples of Bhutan, the emerald cliffs of Kauai, or the diverse wildlife of the Galapagos Islands. It’s also possible to stroll through resorts and hotels and have a much better sense of what swimming pools, beaches, and rooms look like before reserving a room. A virtual-reality brochure or journey also allows a person to explore a region or destination in a more personalized and intuitive way. A click of a pointer on a map sends the visitor to that virtual place. Once there, selecting menu items, say the birds of the Galapagos or the chants of Bhutanese monks, makes the experience come alive.
Virtual reality has already unleashed change in the travel industry. For example, in 2015, Marriott Hotels introduced an in-room VR travel experience called VRoom Service that displays VR Postcards—essentially immersive travel experiences—using Samsung Gear virtual reality. This made it possible to experience an array of exotic places.19 In 2016, German airline Lufthansa introduced VR kiosks at Berlin’s Schönefeld Airport. Passengers passing through the terminal and awaiting flights used an Oculus headset to embark on a virtual tour of Miami or the Great Barrier Reef in Australia.20 Others, such as National Geographic and Smithsonian, have introduced immersive travel experiences that span the globe.
The Smithsonian and the Great Courses, for example, have introduced an interactive journey to Venice, Italy, that includes a gondola ride and visits to the Grand Canal, the Piazza San Marco, Marco Polo’s house, and the Basilica di Santa Maria della Salute.21 A virtual adventurer sits in a small boat with a professor of Italian history, Kenneth R. Bartlett, who delivers a private tour for the virtual journey. The boat pitches and sways while other gondolas and boats pass by. A turn of the head reveals a 360-degree view of the scene—including the spectacular structures surrounding the canal and the gondolier at the back of the boat.
Others are stretching the technology further—and even creating virtual museums that make art, culture, and history more accessible. In 2018, a firm called Timescope introduced the idea of self-service virtual-reality kiosks that virtually transport users to places all over the world via a telescope-like system equipped with a 360-degree rotation mechanism, directional speakers, and 4K screen resolution. A visitor controls the system with a multilingual touchscreen that transports a person to an immersive environment such as the Monument des Fraternisations in France. There, it’s possible to view the trenches of World War I during a cease-fire between Allied and German forces.
AR also changes the way people travel and interact with others. Smartphone apps such as Google Translate already deliver instant translations for signs, menus, brochures, and other printed items. Apps such as Looksee use AR to display places and things, as well as distances for major sights, over live views on a smartphone screen. It includes cities such as Barcelona, Paris, Los Angeles, and Orlando. Meanwhile, at Gatwick Airport in the UK, an award-winning app projects a path—displayed as a series of green arrows—to guide a passenger to the correct gate;22 a user simply holds his or her phone up and the path is projected onto the screen with the live physical space behind it.
Few industries are likely to undergo as big a change as the news media in the coming years. For better or for worse, there’s a growing emphasis on news as a “spectacle” and form of entertainment than a source of information about the world. In March 2018, CNN announced that it was introducing a virtual-reality news app for the Oculus platform.23 The app allows viewers to experience events in an immersive way—and at far higher resolution than ever before offered in the VR world. A person can view what it’s like to be on an aircraft carrier when a jet takes off, in a helicopter during a rescue or in the middle of a battle, on a camel in the middle of the Sahara Desert, or at the center of a festival or sports event. The CNNVR app introduced 360-degree 4K video digital content with a news ticker at the bottom of the screen. It also included integration with social media, such as Twitter.
Other news outlets, including Huffington Post and the New York Times, have also begun to produce AR, VR, and MR content. Some of these experiences put a viewer at a sports event or rock concert; others involve stepping into refugee camps and crime scenes. While societal standards have broadened about what’s acceptable to show in a video, the issue takes on new meaning when the scene is ultrarealistic and immersive. Some, like Robert Hernandez, an associate professor at the University of Southern California’s School for Communication and Journalism, have stated that VR could invoke bad memories and even unleash trauma for some people. The role of journalists and the news media may require reexamination. “As a journalist I have to ask what’s my job? Is my job to hurt you? It’s to inform you, and sometimes it’s a little in the middle,” he stated.24
One of the first uses of AR in marketing occurred in 2008. A firm in Germany created a printed magazine ad of a model BMW Mini that, when held up to a camera on a computer, appeared on screen in 3D.25 A virtual model connected to markers on the physical ad made it possible for a user to control the paper version of the car and move it around to view different 3D angles on screen. This real-time digital interaction demonstrated the potential of augmented reality. Other brands, from National Geographic to Disney, have since embraced the technology to depict things as diverse as environmental problems and cartoon characters interacting with people on the street.26
Other companies are expanding the concept of XR in other ways. IKEA’s augmented-reality app that allows users to preview a piece of furniture has garnered press attention and consumer interest. Sherwin-William’s paint app that lets a buyer view the color of paint in a virtual representation of their bedroom or office has proven popular. Others are now joining the XR party. For instance, cosmetic company Sephora has turned to AR technology to allow customers to see what various products look like—from lipstick to eyeliner—on their digital face. A user simply snaps a photo in the app or submits it to a website, selects a product, and instantly views the results.27 Sephora Virtual Artist also offers tips that coincide with the products.
In the future, an HMD will allow individuals to venture online and view items more authentically, walk through a shopping space in a more realistic way, and, through the use of haptics and feedback technology, feel the texture of a silk blouse or a sofa. In fact, virtual reality could deliver online shopping experiences that resemble a physical store. Instead of viewing clothing or power lawnmowers on a 2D screen, a person might travel down a virtual aisle and gaze at items—while pulling up specs and information by reaching out and touching them or directing a virtual laser pointer at them. AR and VR will also likely revolutionize real estate by allowing people to walk through property from across town or from the other side of the world. One firm, Matterport, already offers hardware and software that creates 3D virtual home tours.28
The concept of virtual shopping hasn’t escaped retailing giant Walmart, which filed a patent in August 2018 for a virtual-reality showroom that lets users view images of shelves and products and select them virtually. Other retailers are also dialing into the concept.29 In 2017, home improvement giant Lowe’s introduced a virtual-reality shopping experience called Holoroom.30 Customers could view VR tutorials of home improvement projects and gain hands-on experience. During a trial run of the initiative, Lowe’s reported that customers demonstrated 36 percent better recall of how to complete a do-it-yourself project. The company also gained valuable feedback about where customers typically become confused and frustrated.
Auto manufacturers such as Audi and BMW are also steering toward VR. An app from Audi, for example, allows a prospective buyer to view and explore the interior of a vehicle in 3D—using a VR headset at home. In 2016, Audi introduced an app that allows customers to view different colors and options using an Oculus Rift or HTC Vive HMD at a dealer. Now it’s expanding its virtual-reality offerings to users at home. Using an app developed by a firm named ZeroLight, potential buyers can enter a virtual cabin and see what the dashboard, seats, and space look like in any of the manufacturer’s 52 models. Shoppers can also view a vehicle’s engine and, using headphones, hear what the car sounds like.
Augmented reality could dramatically change policing. Although law enforcement officials in China already use facial recognition glasses to identify suspected criminals, the technology—in conjunction with the Internet of Things (IoT), has other applications, including license plate recognition, identifying chemicals in a bomb or other explosive device, displaying vitals when first responders encounter injured individuals, enhanced night vision, and capturing video with GPS stamps to record and authenticate events. Virtual reality is equally attractive for police. In Morristown New Jersey, for instance, officers train in virtual-reality simulators. This helps them learn when and where to shoot—and when to refrain.31 VR could also aid in recruiting and allow the public to experience what it’s like to be a police officer on a beat.
There’s also the issue of AR and VR used in the courtroom. These technologies could lead to new types of evidence and allow juries to see and experience events in entirely different ways. Instead of viewing scratchy and blurry CCTV images, jurors could be transported to a virtual crime scene. Caroline Sturdy Colls, an associate professor in forensic archaeology and genocide investigation at Staffordshire University in the UK, has stated: “Traditional means of documenting, sketching and photographing crime scenes can be laborious and they don’t provide data suitable for presentation in court to non-experts. A number of novel, digital, non-invasive methods used in archaeology, computing and games design present the opportunity to increase search efficiency and accuracy and provide more effective means of presenting evidence in court.”32
In 2016, Colls received a £140,000 research grant from a European Union commission to study the use of virtual-reality head-mounted displays for the criminal system. Using an HMD, a juror could walk around a crime scene and view it from different angles and in different ways. In 2018, she received another grant to study how officers can better analyze buried and concealed evidence at crime scenes. She and other researchers are collaborating with UK police forces to take policing and courtroom trials into the digital age.
Virtual reality is also introducing new ways to treat physical and psychological conditions. At Cedars-Sinai Hospital in Los Angeles, psychiatrists have experimented with VR to treat opioid addiction. Results show the approach is promising.33 At the University of Maryland, doctors use AR to view ultrasounds while examining a patient.34 This makes it possible for a technician to avoid looking away from a patient while conducting the exam. It also puts essential information in view of the technician at all times. Another use for the technology comes from a company called AccuVein, which has developed a device that scans veins and projects an AR visualization onto the skin area a technician scans.35 Using the handheld unit, it’s possible to view veins inside the body. The process has led to 45 percent fewer escalations, the company claims.
Virtual reality is also changing medicine. At Cedars-Sinai Hospital, psychiatrists have experimented with VR to treat opioid addiction. At the University of Maryland, doctors use AR to view ultrasounds while examining a patient. This makes it possible for a technician to avoid looking away from a patient while conducting the exam.
Another firm, Surgical Theater, has introduced a VR platform that allows patients to view their anatomy and doctors to view tumors, blood vessels, and other structures in the brain or another part of the body.36 The system displays a patient’s unique issue and helps doctors plan the operation. Physicians at George Washington University Hospital in Washington, DC, already use the technology, which it calls Precision Virtual Reality.37 Apart from reviewing information and planning the procedure with much greater insight, they can use the personalized 3D virtual images to demonstrate to patients what will take place.
Meanwhile, a San Diego, California, organization called the Virtual Reality Medical Center has turned to VR to aid people with phobias like fear of flying, public speaking, agoraphobia, and claustrophobia. VRMC uses 3D virtual-reality exposure therapy in combination with biofeedback and cognitive behavioral therapy to treat phobias, anxiety, stress, and chronic pain.38
Pharmaceutical and biotech companies are also tapping virtual technologies. Jonas Boström, a drug designer in the Department of Medicinal Chemistry at AstraZeneca in Sweden, has developed a molecular visualization tool called Molecular Rift, which runs on Oculus goggles. It offers an environment where a person can use gestures to interact with molecules and examine how they behave in different environments and situations. Boström believes the tool is “the next generation of molecular visualization.”39 Other pharma and biotech firms rely on the CAVE and other VR tools to engage in advanced drug discovery. This includes virtual mock-up studies, design reviews, safety studies, ergonomic studies, failure mode effects analysis, training, machine redesign assessments, computer-aided engineering, and air flow visualization.
Virtual reality will also aid in detecting and diagnosing dementia and other problems, including PTSD.40 At Rush University Medical Center in Chicago researchers have developed modules to help medical and pharmacy students recognize signs and symptoms of dementia.41 At the University of Southern California, Skip Rizzo, who has worked on the BRAVEMIND project, is exploring ways to treat veterans and others suffering from post-traumatic stress disorder. He has built simulated environments where people “relive” traumatic events in a virtual space that resembles a computer game. The technique builds on a widely accepted treatment approach called “prolonged exposure” therapy. The initial versions of the VR program, called Virtual Iraq and Virtual Afghanistan, were adapted from the 2004 Xbox video game Full Spectrum Warrior. He later developed the more advanced virtual-space BRAVEMIND as a tool for helping PTSD patients confront and process difficult emotional memories.
These VR environments feature a variety of combat situations, conditions, and wound levels. At any time, a psychologist can trigger a reenactment of the original traumatic event. This approach clearly shows promise. A study conducted by the Office of Naval Research in the United States found that 16 of 20 participants displayed significant reductions in PTSD symptoms as a result of the VR treatment.42 One soldier indicated that reliving his traumatic experiences in a virtual environment substituted the need to think about the trauma when he was at home with his family. Researchers have also used virtual environments to aid victims of sexual trauma and other types of abuse.
Rizzo believes that virtual reality will eventually emerge as a mainstream tool for addressing both physiological and psychological issues. Today, “physical rehabilitation for a stroke, traumatic brain injury or spinal cord problem requires a person or robotic device to deliver resistance,” he explains. “The next frontier is introducing VR simulations that can gauge a person’s physical capacity and apply pressure and resistance exactly how it is needed to rehab the person.” Instead of actually touching or picking up physical objects, for example, a system will send signals and auditory cues that result in a believable experience. “You reach for a balloon and it pops when you squeeze it. You hear the sound, you feel the air and you experience the vibration.” Further out, he says, systems may include exoskeletons that simulate walking and allow patients to engage in activities that aid in rehab or allow a person with a disability to experience the sensation of walking, running, or swimming.
It’s shouldn’t be the least bit surprising that the military and organizations like the US Defense Advanced Research Projects Agency (DARPA) are among the biggest adopters of AR and VR tools. The quest for superiority on the battlefield has led to massive research and development efforts in the XR space. Much of the effort has revolved around training and simulations. Teaching pilots how to fly or soldiers how to react to dangerous situations is a lot less expensive—and far safer—in a virtual world. A simulator that costs $800,000, such as the US Army’s Stryker,43 is more practical—and effective—than dropping tens or hundreds of millions of dollars into actual war games and training exercises.
Virtual-reality simulators also collect valuable data about how soldiers and others act and react in different situations. Using artificial intelligence and analytics tools, it’s possible to spot patterns and trends that lead to more effective combat methods. Yet, the use of XR extends beyond simulators. For example, the US Army has developed a head-mounted display that projects relevant data, information, and graphics into a soldier’s view—while providing a view of the physical battlefield.44
In the future, these systems could also include bio-detection capabilities and they could integrate with sensors that might be worn in clothing or carried on a person. Across the Atlantic Ocean, the British Army has introduced a virtual-reality recruiting tool that allows potential recruits to experience training exercises. It witnessed a 66 percent increase in signups after offering the immersive experience at various sites across the UK.45
Extended reality is filtering into other parts of life as well. A pastor in Reading, Pennsylvania, is working to create a virtual church, complete with avatars, music, and sermons.46 New York’s Museum of Modern Art has introduced augmented reality to a Jackson Pollock exhibit. With a smartphone in hand and an app called MoMAR Gallery, a visitor sees the art radically remade along with illustrations that completely revamp the space.47 Even rock bands and fashion designers are dabbling in the space. They’re introducing new ways to experience their designs, music, and other creations. While VR might prove valuable for viewing traditional paintings and sculpture, it will likely usher in completely new forms of art—including remarkably different installation art—in the years ahead.
Of course, as these various forms of extended reality take shape—and reshape the way people go about their daily lives—a multitude of questions, concerns, and potential problems arise. The psychological and sociological changes extended reality unleashes intersect with ethics, morality, legality, and behavior. While some of these issues are merely intriguing, others veer into the category of somewhat to extraordinarily disturbing.