The technologies that collectively comprise AR, VR, and MR are evolving at a rapid rate. As microchips become more powerful, components shrink in size, and researchers develop far more sophisticated software, extended reality is taking shape. Yet, there’s no single path toward augmented and virtual technologies. Different industries and use cases demand entirely different tools and components. A fully immersive gaming experience is markedly different than a system—or the headgear—a military aircraft pilot would use while flying a jet or an engineer would need to view a virtual building. An app on a smartphone is considerably different than augmented-reality eyeglasses a firefighter might rely on.
System design and engineering are critical to usability. Yet, the technical and practical challenges associated with creating a realistic virtual environment are formidable. This is particularly true for complex virtual-reality and mixed-reality environments. While it’s apparent that a person entering a virtual space will require a realistic visual experience—through pictures, video or graphics—it’s also necessary to tie in sensations, sounds, and other stimuli, including possibly scent, in order to convince the human brain that it’s all real. Connecting and coordinating digital technologies are at the center of producing a convincing and meaningful virtual experience.
Consider: a person, say an architect, decides to walk through a virtual building and see what rooms and spaces look like before finalizing a set of designs and blueprints. This may include viewing walls, ceiling, flooring, architectural and design elements, and even layers of hidden infrastructure such as wiring, lighting, or plumbing. The problem is that walking through an actual room with a head-mounted display could mean tripping over physical objects, stumbling and falling, or crashing into walls or girders. Sitting down and bending into spaces introduces other challenges. All of this points to a need for a convincing virtual environment that’s coordinated with the existing physical environment.
Similarly, augmented reality requires an understanding of physical objects and spaces, including the relationships between and among them. This might include words on a sign or menu, or the dimensions or characteristics of a room. In order to obtain this information, the AR system may require a camera or other type of sensor to measure a room’s size or shape or discern the color or texture of the walls. If this system does not work correctly—say the AR app and the underlying algorithm can’t determine the physical parameters and translate all the data points into physical representations—a piece of furniture will appear out of perspective or the color of the room won’t look right.
The result is a poorly performing app or a useless experience. Grossly distorted rooms or furniture floating in the air will likely lead, at the very least, to a person abandoning the app. The consequences can be far worse, possibly even dangerous. An AR system that renders images or data incorrectly could result in an injury or death—particularly if the application is used in dangerous work environments, for policing, or in combat situations. What makes XR so challenging from a design and engineering perspective is that even the tiniest glitch or flaw in hardware, software, user interface, or network performance can torpedo the entire experience. Suddenly, the environment morphs from convincing to implausible.
What makes augmented reality so enticing is that it allows humans to step outside the physical world—while remaining in the physical world. The visual piece of the AR puzzle is particularly important. Superimposing graphics, images, and text onto a smartphone screen or a pair of glasses is a complex multistep process. This varies, of course, depending on what the creator of an AR app or feature is attempting to do.
A starting point for developing an augmented-reality app is for a digital artist or graphic designer to create a 3D model, or for an app developer to incorporate a model from an existing 3D library from a private company or an open-source group. Specialized software such as 3DS Max, Blender, Cinema 4D, Maya, Revit, or SketchUp deliver tools that designers use to create visuals within an AR app. A platform like Sketchfab lets designers discover, share, and purchase content and components to create apps. An artist typically begins with a rough sketch or wireframe and, using these tools, refines the drawing until a 3D model takes shape.
Once a concept drawing exists, the artist transforms the drawing into an actual model. This animation process may include adding colors, shapes, textures, human features, or the behavior or physics of how an object or device operates in the physical world. In some cases, the goal may be to create a character that appears to have lifelike features. If the goal is to display an actual object—say David Bowie’s “Ashes to Ashes” costume from 1980 (which appears in the New York Times app mentioned in chapter 2)—there’s a need to capture an image or video of an item and convert it through the use of 3D projection mapping software. This transforms the 2D structure into a 3D representation.
The next step is to create the actual augmented-reality experience. After software transforms the mathematical construct into polygons—the fundamental geometry that underlies a 3D visual model—it’s possible to add movement or give the person, creature, or object real-world features within the AR app. Yet, there’s more to this task than simply creating an object of interest. When a person uses an app or AR glasses, the object must be visible from different angles and perspectives. Too few polygons and a virtual object won’t look or feel real. Too high of a polygon count and the AR object may not render correctly.
The final step is to finalize the AR features using a software developer kit, such as Apple’s ARKit, Google’s ARCore, or an independent platform like Vuforia. These tools aid in building an app and embedding AR features. At this stage of the development process, app creators add things like motion tracking and depth perception—as well as area-learning features that integrate the virtual object with the actual physical environment. This helps the smartphone camera or eyeglasses capture and display the object correctly. The seamless blending of the physical and virtual objects is what makes an AR scene appear convincing and real. Without using these tools, the display device will not orient and coordinate all the elements. Motion may be off register and objects may appear distorted.
In fact, the ability to orient and coordinate physical and virtual objects—and manipulate the latter—is at the center of a successful AR app. Some augmented-reality SDKs, such as the Vuforia platform, include features that automatically recognize objects and compare them to existing 3D models built into the software framework. By estimating camera position and the overall orientation of objects it’s possible to introduce a more realistic experience. Motion tracking software that operates in the app—simultaneous localization and mapping (SLAM)—creates the necessary depth tracking and pattern tracking.
SLAM, which is also used for autonomous driving and robotics systems, relies on algorithms and sensor data from the device to generate a 3D map of the space. It determines where and how to display the AR object on the smartphone or AR glasses. SLAM also uses global lighting models to render images with the right colors and tones. All of this information about registered objects is stored in a database that ultimately determines the visual outcome in the VR or AR environment. Then, using a rendering engine such as Unity or Unreal Engine, the phone or glasses generate the visual images. The iPhone X, for example, incorporates a TrueDepth sensor that projects 30,000 invisible infrared dots on an object. This allows sensors in the device to map a physical space and generate an accurate representation of a virtual object or an environment.
The final effect is the ability to render an object or scene in a way that appears photorealistic. A larger object typically means that the object is closer, and a smaller pattern implies that it is further away. The distance between objects helps determine the overall perspective of the scene—as well as how objects rotate and move in the AR space. The embedded algorithm—and the device’s ability to use microelectromechanical systems (MEMS) such as accelerometers, proximity sensors, and gyroscopes—determines how the final scene appears on the phone or on a pair of glasses.
Although modern augmented-reality features and apps began to appear on smartphones as early as 2010, advances in electronics have transformed the space in recent years. For one thing, cameras in smartphones have improved to the point where they capture images in high definition. For another, today’s OLED screens deliver high contrast ratios at an ultrahigh resolution. All of this allows colors and objects—including computer-generated images—to appear more real. SDKs and rendering platforms such as Unity and Unreal Engine have also continued to evolve.
Of course, form factors matter. A smartphone display offers the advantage of portability. The device is easy to tuck into a pocket or purse and it’s convenient to use in a wide array of situations. The screen is available instantly and it is able to adapt and adjust to lighting conditions by dimming and brightening automatically. Yet a phone screen isn’t as convenient as a pair of glasses or goggles, which reduce or eliminate the need for hands-on interaction with the device—particularly while using physical objects. For technicians, engineers, emergency responders, and many others, including consumers attempting to fix things—say replacing a light switch or sprinkler head—holding a phone to view written instructions or a video can prove challenging, if not impossible.
Indeed, AR glasses and goggles are all about going hands free. Google Glass is perhaps the highest-profile example of AR optics, but a growing number of firms are developing products for consumers and businesses. The list includes Epson, Sony, Microsoft, Nokia, and Apple. Prototypes and commercially available systems use several different technologies to project AR imagery onto lenses and connect to a smartphone or the internet. Form factors range from traditional eyeglasses to head-mounted displays, virtual retina displays (VRDs) that use low-powered LED light and special optics to project an image directly into the human eye without the use of a screen, and the heads-up displays used in vehicles and aircraft. HUDs use a laser, or a mirror that projects or reflects light onto the glass surface.
A key consideration in AR is field of view (FOV). Many of today’s systems offer 50-degree or lower FOVs. This produces a scene that is best described as adequate but hardly compelling. Essentially, limited FOV restricts the field of vision and forces users to view objects within a narrower scope, which isn’t natural. In order to achieve a superior view, a system requires an FOV of 180 to 220 degrees. A limited view also has potential repercussions for technicians and others who require a broader view of the space they are in. If the AR display obstructs their view they are more prone to make mistakes or cause an accident.
Display technology continues to advance. Researchers at the University of Arizona have begun to experiment with holographic optics.1 This could produce a larger eye box—the display system that creates the image—and thus improve the clarity and viewing angles for data displayed on a screen or windshield. At the same time, researchers are looking for new and innovative ways to improve these systems. For instance, a Honda Motor Company patent granted in 2015 proposed to improve safety through an augmented-reality HUD that maps a forward view of a pedestrian along with a crosswalk path.2 The system could flash a warning or apply the brakes automatically if it detects a potential hazard.
Glasses, goggles, or any other device designed to project images onto the human field of vision operate on a basic principle: they redirect light into the eye. This includes the natural light generated from the physical world as well as the artificial digital images created by a computer. The latter typically takes the form of LEDs and OLEDs. The optical device must combine natural and artificial light in order to create augmented reality. The system uses what is referred to as a “combiner” to meld these light sources into a single field of vision. The combiner achieves this result by behaving like a partial mirror. It selectively filters light from each source to produce an optimal augmented image.
Wearable AR optical systems come in two basic forms: HMD optical combiners and waveguide combiners.3 Each offers advantages and disadvantages involving shape, feel, aesthetics, weight, image quality, and resolution. Optical combiners rely on two different methods to generate images. The first is polarized beam combiner systems such as Google Glass and smart glasses from Epson. These glasses are typically lightweight and affordable, but they are not capable of generating the best possible image. The split-beam approach they use typically results in some blurriness. The second approach, the off-axis, semi-spherical combiner, includes devices like the Meta 2, which look like a futuristic pair of goggles from Star Wars.4 The $1,495 device delivers relies on a single OLED flat panel built into the lenses to create an impressive 2560 × 1440 pixel display.
Another experimental method, wavelength combiners, revolves around a technology called waveguide grating, also known as a waveguide hologram, to produce optics. The technique taps a method called total internal reflection (TIR) to progressively extract collimated, or parallel, images within a wavelength pipe. The wavelength pipe is made from a thin sheet of glass or plastic that allows light to pass through. The diffraction that occurs in this process produces the high-quality holographic image—along with a potentially greater field of vision. For now, however, the technical challenges related to manufacturing these glasses in volume are formidable. They remain outside the realm of a commercial product. VRDs also fit into the “future” category, although Intel has announced its intention to sell smart glasses that use retinal projection.5
AR technology isn’t just about presenting interesting, useful, or fun images. These devices increasingly tie into other smartphone or AR glass features, including cameras, microphones, and speech recognition. It’s critical for glasses to coordinate various functions—in much the same way a smartphone handles the task. This requires an operating system (OS) and application programming interfaces (APIs) to connect software programs, apps, and tools. Finally, these devices require processing power along with memory, storage, and a battery. One of the persistent and ongoing problems with all portable tools—from smartphones to smart watches—is the ability to use a device for an extended period of time.
For now, no clear standard for AR glasses or HMDs exists. Researchers in labs and at technology firms continue to experiment with a variety of approaches and form factors. They’re also exploring new ways to input data. This might include virtual cursive or character-based handwriting technology or gestures that use fingers, hands, head or eye movements, or an entire body. A system might also rely on a specialized stylus or entirely different types of keyboards. For example, a company named MyScript has developed an AR management technology called Interactive Ink, which supports augmented reality and other digital technologies.6
Technical obstacles aside, the use of AR on phones and glasses will likely soar in the years ahead. As the price of microchips and electronic components continues to drop and software and systems become more sophisticated, the technology will filter into apps used for entertainment, marketing, training, and technical support. AR will deliver information on the road, at the beach, in the classroom, and on the factory floor. A 2017 survey conducted by Evans Data Corporation found that 74 percent of mobile developers are either incorporating or evaluating the use of AR in apps.7
Investment and financial services firm Goldman Sachs predicts that the market for augmented and virtual reality will reach $80 billion by 2025.8 This would put AR technologies on par with today’s desktop PC market. Heather Bellini, business unit leader for Telecommunications, Media and Technology at Goldman Sachs, believes that augmented reality has the “potential to transform how we interact with almost every industry today, and we think it will be equally transformative both from a consumer and an enterprise perspective.”9
The enormous appeal of virtual reality isn’t difficult to grasp. The technology transports a person into an immersive and seemingly real digital world. Designing, engineering, and assembling the technology required to produce a convincing VR experience, however, is no simple task. It’s one thing to stream images to a head-mounted display or create a crude sense of feel using haptic gloves that vibrate; it’s entirely another to coordinate and orchestrate computer-generated processes into an experience that convinces the mind the scene is real. This requires an understanding of how a person moves, reacts, and thinks in both the physical and VR worlds. As a result, there’s a need to think of VR as a framework that extends beyond the human body.
HMDs are at the core of virtual reality. They provide the visuals and other sensory cues that allow a person to feel immersed. Unlike AR glasses or goggles, they must close off the outside world. Today, HMDs typically use liquid crystal display (LCD) technology or liquid crystal on silicon (LCoS) to generate images. In the past, HMDs relied on cathode ray tube (CRT) technology. Future displays will likely use OLED technology, which delivers clearer and brighter images. The transition from CRT to LCD and OLED has allowed engineers to build HMDs that are less bulky and more comfortable. Smaller and lighter units are also important because they allow a user to move around more freely.
An HMD requires several core components to produce the visual experience. Lenses built into the display focus a user’s eyes on objects in the VR space and achieve the correct depth of field. In fact, a person’s eyes don’t focus on the lenses; they focus into the virtual distance. The display itself sits just beyond the lenses. It produces two slightly different images that when calibrated between two eyes produce depth and dimensionality. This stereoscopic effect contributes to the brain believing it’s navigating within a 3D world. Aiding in this process are earbuds, headphones, or built-in speakers that generate spatially accurate sound. This makes the bark of a dog or the sound of a car engine seem realistic as it moves closer or further away, or across the horizontal plane.
Other technical requirements also define a VR experience. A minimum acceptable refresh rate for virtual reality is generally pegged at 60 frames per second (fps). Anything less than this refresh rate causes video stuttering that often leads to motion sickness. While high-end gaming consoles run at 60 fps, televisions operate at about 30 fps. The Oculus Go, released in early 2018, has redefined consumer VR. It runs at 72 fps and at a resolution of 2560 × 1440. This generates more realistic, smoother, and more seamless graphics. The Go also uses a technique called foveated rendering, which detects where a person is looking and renders these areas in full detail while rendering other areas in less detail. This allows developers to use computing resources more efficiently.10
The field of view for HMDs typically ranges from 90 to 110 degrees, though a few systems deliver an FOV as high as 150 or above. Like AR eyeglasses and goggles, the FOV for a system determines how authentic scenes appear to a person. Too low an FOV and a user experiences a “snorkel mask” effect that hinders the overall experience. A wider view creates a greater sense of realism and immersion. An FOV of 180 or above generally extends to the edges of human peripheral vision. Currently, the challenge for engineers is that higher FOVs result in reduced graphics resolution and sharpness.
Engineers are attempting to tackle this challenge. As the GPU chips that power these displays become more powerful and better adapted to virtual reality, FOVs will increase and, in conjunction with foveated rendering, the visual quality of VR systems will continue to improve. Newer chipsets from the likes of Nvidia and Qualcomm are advancing VR further. These chips are far more adept at handling motion, pixel shading, and other technical and practical elements that can make or break a VR experience.
Tracking and navigation are also critical in VR. The Oculus Rift, for example, offers a video-game style controller called Touch that a person wears over the wrists. This provides the mechanism required to move around inside the virtual world. Newer designs, such as the Oculus Go, offer a more compact single controller with a touch surface as well as dedicated buttons. Within the VR space, the Go controller depicts a laser pointer that allows a user to navigate the interface. The HTC Vive, by contrast, offers a VR controller that not only allows a user to navigate through the VR world but also brings real-world objects into the virtual world. It also offers wrist strap controllers and special rackets for sports.
Of course, a VR system must also detect how a person moves and responds, ideally within 30 milliseconds but generally at a maximum latency of 50 milliseconds. This may include hand, body, eye, and head movements. The latter is particularly critical because the display must adjust as a person turns or tilts his or her head—including looking up, down, or backward. Some systems, such as Sony’s PlayStation VR (PSVR), use a series of LED lights to track movement with external cameras. The Oculus Rift relies on a tripod system equipped with IR LEDs to track a user’s movements. Software translates all this data into the scene a person experiences inside the VR environment. Other systems tap sensors—usually some combination of gyroscopes, accelerometers, and magnetometers—to track and measure movements.
The most advanced HMDs incorporate eye tracking, which delivers feedback about what a person is looking at and how he or she is responding. Foveated rendering uses a sensor in the HMD to follow eye movements. The system prioritizes what a person is specifically looking at and taps digital image processing to produce higher-resolution graphics. This also allows developers to dial down non-essential graphics and other events without a user noticing it. Using this technique, engineers and developers deliver maximum performance from the GPUs and CPUs that produce the VR graphics and audio.
Until recently, HMDs required cables to power the device and receive signals from a smartphone or personal computer. VR systems, however, are rapidly evolving into self-contained devices. Cables and cords are being replaced by on-board processing, memory, and storage, along with rechargeable batteries that last upward of 30 hours. Newer devices, such as the Oculus Go and Lenovo Daydream, operate entirely independent of other computing devices. The Go also has built in audio and the platform eliminates the need for external tracking devices. Its built-in sensors track motion and capture the position and movements of a person’s head.
Researchers have also explored the idea of direct brain interfaces to control VR spaces.11 One startup, Neurable, is working to perfect an HMD that uses electrodes and a technique called electroencephalography (EEG) to capture brain activity. Analytics software deciphers the signals and transforms them into actions.12 In the future, this noninvasive brain–computer interface (BCI) could revolutionize the way people use extended reality. It might also allow the blind to achieve at least some sense of sight.13
Haptic gloves are also advancing rapidly. Researchers are redefining touch technology and the user experience. The problem with many of today’s VR gloves is that they provide only basic haptic feedback that’s designed to mimic pressure, texture, and vibration. A slight buzz, rumble, tingle, or tug on the fingers, however, isn’t like the corresponding real-world event. “These systems don’t deliver a realistic analogue of what your brain expects to feel,” explains HaptX founder and CEO Jake Rubin. “Smooth objects don’t feel smooth. Bumpy objects don’t feel bumpy. An orange doesn’t feel like an orange and a handful of dirt doesn’t feel like dirt. Every time an experience doesn’t match the expectations of the brain the virtual-reality experience is diminished to some degree.”
The HaptX system takes aim at these challenges. The gloves rely on microfluidics within a smart textile to deliver a higher density and greater displacement of feedback points. The system uses air channels with micropneumatic actuators built into a thin, flexible sheet of material to deform skin in a way that mimics touching an object. In other words, a person wearing HaptX Gloves in the virtual environment feels a virtual object as if it’s a physical object. Software that manages the valves and sensors in the HaptX system translates data about pressures, textures, and feelings into a mathematical model that simulates how the human musculoskeletal system works. “The goal is to build a system that actually matches how your brain expects to feel the things you interact with in a virtual world,” Rubin says.
Others are also exploring haptics technology. A Microsoft Research Team has developed a multifunctional haptic device called the CLAW that allows a user to grab virtual objects and touch virtual surfaces and receive tactile feedback as if they were real.14 The device adapts haptic rendering by sensing differences in the user’s grasp and the situational context of the virtual scene. “As a user holds a thumb against the tip of the finger, the device simulates a grasping operation: the closing of the fingers around a virtual object is met with a resistive force, generating a sense that the object lies between the index finger and the thumb,” a Microsoft blog noted. The company is also experimenting with a haptic wheel and other devices that could be used in VR and AR apps.
Oculus has experimented with haptic gloves that include internal “tendons” that tense and relax. The system simulates a person’s sense of touch within a VR environment. By creating the feeling of resistance—much like stretching a rubber band—a person gains a more accurate sense of touch using the haptic gloves. According to a US patent filed by Oculus: “The haptic feedback facilitates an illusion that a user is interacting with a real object when in fact the object is a virtual object. The mechanism resists movement by one or more portions of a user’s body.”15 Another firm, Kaya Tech, has developed a full-body VR tracking and haptics system called the HoloSuit. The most advanced version of the product includes 36 total sensors, nine haptic feedback devices, and six firing buttons.16
The software frameworks that support virtual reality are also advancing rapidly. SDKs are expanding virtual horizons and virtual-reality operating systems are emerging. For example, Google debuted the virtual-reality OS Daydream in 2016.17 It allows Android smartphone users to use a wide range of VR apps from content providers such as YouTube and Netflix—as well as Google’s own offerings, including Google Earth. Meanwhile, Microsoft has developed Holographic OS for VR headsets, and others are entering the VR OS space.
The number of virtual-reality apps is also growing. The Oculus platform offers thousands of app titles—from car racing and mountain climbing apps to space missions, historical events, and movies. It also offers travel apps, such as Flight over Ancient Rome and Google Earth VR with Street View. This allows a user to fly over the Manhattan Skyline, across the Andes Mountains of Ecuador, along the Nile River, and through the Grand Canyon—all with the photorealism and 3D sensation of actually being there. While a computer may generate stunning graphics, the virtual-reality space actually makes it seem real.
The Holy Grail of virtual reality is a concept called 6 DOF (six degrees of freedom tracking). It revolves around the idea that a person in a virtual-reality environment is able to move freely. This includes forward and backward, up and down, and sideways. Of course, walking blindly around a physical room—compete with walls, furniture, and other hazards—isn’t a good idea for someone wearing an HMD. That’s why, for virtual reality to gain even greater realism, there’s a need for omnidirectional treadmills. As the name implies, the mechanical device allows a person to move in any direction at any moment. When the VR system delivers contextual cueing, walking or running on the treadmill can seem entirely natural.
Omnidirectional treadmills can be used to simulate a wide range of terrains, from hilly to flat, and they operate with or without a harness. The resulting whole-body interaction can transform the way a person perceives various stimuli and events. So far, researchers for the US Army have explored the idea, and a handful of companies has attempted to develop a treadmill that could be used commercially. One system, called Infinadeck, lets a user step in any direction and change directions in real-time. Television network CBS has reportedly explored the idea of using the omnidirectional treadmill for green-screen sets, where an actor could move around on camera in a virtual framework.18
Another omnidirectional system is called Virtusphere.19 Individuals enter a fully immersive environment inside a 10-foot hollow sphere, which is placed on a special platform with a “ball” that rotates freely in all directions. When someone takes a step or changes direction the system adapts in real time. The sphere, wheel platform, and rolling bar support unfettered motion. A person in the Virtusphere wears a head-mounted display that integrates the motion through software. So far, the device has been used for training military personnel, police, and workers at nuclear power plants and other high-risk facilities that require highly specialized skills. It also has been used by museums, by architects, and for home gaming.
Another movement technology is a virtual holodeck. It allows people to move around freely in a virtual space by generating a 3D simulation of the physical space, with added elements. For example, The VOID has introduced a VR theme attraction that recreates a Star Wars film, Secrets of the Empire, or the movie Ghostbusters. The virtual space, dubbed “hyper-reality,” allows participants to move around as they would in the physical world.20 It features sights, sounds, feeling, and smell. The VOID is available at several sites, including Downtown Disney in Anaheim, California, the Venetian and Palazzo Hotels in Las Vegas, Nevada, and at Hub Zero, City Walk in Dubai, UAE. In Hong Kong, Korea, and elsewhere, other arcades are taking shape.21 They have introduced holodecks, omnidirectional treadmills, and haptics systems for multiplayer gaming and other scenarios.
Researchers are also exploring ways to improve the CAVE. One commonly used approach involves “redirected walking,” which is deployed in a CAVE space that’s 10 square feet or smaller.22 It tricks a visitor into thinking that he or she is walking along a straight line while the person is actually traveling along a curved path.23 The VR system rotates and adapts imagery slowly and imperceptibly so that a user feels slightly off balance and adjusts his or her gait accordingly. This technology makes it possible to use the CAVE space far more efficiently.
As the definition of reality is reframed and reinvented to match the technologies of the digital age, an array of experiences becomes possible. Mixed reality, sometimes referred to as hybrid reality, combines virtual and physical realities to create yet another experience in the reality-virtuality spectrum. Within this computer-generated space, actual objects coexist with virtual objects and things. At the most basic level, anything that falls between a complete virtual environment and a completely physical environment falls into the category of mixed reality, though augmented and virtual realities use mixed reality in different ways.24 MR can be useful in a wide range of applications, including simulations, training environments, interactive product content management, and manufacturing environments.
For example, a VR environment that drops a person into a virtual conference space could also include a live person making a presentation or actual booths that are virtualized for the environment and fed into the environment via a live video stream. The conference space could also include a virtual environment with augmented overlays. This might allow a shopper to step into a virtual showroom and view different cars or appliances but also tap on them to view features and specifications. At the other end of the spectrum, a person might use AR glasses to view a scene—say, a sporting event—but then watch a virtual-reality instant replay, where he or she is inserted into the scene.
Not surprisingly, MR introduces other opportunities and challenges. Most important, all the technologies and systems used for virtual and augmented reality must integrate seamlessly. The right images, text, video, and animations must appear at the right time and they must be contextually accurate. Yet, it’s also vital that the visual, haptic, and auditory experience in a mixed-reality space is presented in an easy to navigate way. As with any computing device, interface and usability are critical elements. The challenges are magnified when live video is inserted into a virtual environment or a virtual environment is embedded into a live scenario. The underlying software and algorithms must interconnect components seamlessly.
Virtual-reality systems orchestrate all these tasks and technologies through three types of integrated computer processing: input, which controls the devices that deliver data to the computer or smartphone—including a controller, keyboard, 3D tracker, or voice recognition system; simulation, which transfers user behavior and actions into virtual results within the environment; and rendering, which produces the sights, sounds, and sensations that the user experiences in a virtual environment. The latter includes haptics and auditory generation systems.
When designers, engineers, and systems developers assemble the right mix of XR technologies and optimize the way systems work, the result is a continuous feedback loop that delivers images, sounds, and haptic cues at precisely the right instant. The resulting stream of images, graphics, video, and audio puts the person at the center of the action—or displays the right data or images on AR glasses. A user feels as though events are taking place as they would in the physical world. Ultimately, the AR, VR, or MR becomes more than a new way to view objects and things. It’s a framework for an entirely new way to navigate the digital and physical worlds.
The result is a new and different type of reality that extends beyond the limitations of human perception. In some cases, it may lead to synesthesia, a process where the right combination of electronically generated signals tricks the mind into believing parts of the body have been touched or stimulated—or some realistic seeming action or event has taken place. To be sure, augmented, virtual, and mixed reality are more than a tangle of technologies. There’s a need to understand human behavior, social interactions, and how XR affects the human body and the mind. This requires a deeper understanding of the mind–body connection—and the influence of technology on everything from thinking to proprioception.
When designers, engineers, and systems developers assemble the right mix of XR technologies and optimize the way systems work, the result is a continuous feedback loop that delivers the right images, sounds, and haptic cues at any given instant. The resulting stream of images, graphics, video, and audio puts the person at the center of the action.