Introduction

If you’ve ever watched any of the Iron Man movies starring Robert Downey Jr., you’re probably familiar with Jarvis. Jarvis, short for “Just a Rather Very Intelligent System,” is a highly advanced computerized AI system. Recently, Jarvis inspired Mark Zuckerberg to build his own AI-powered home-automation system (https://youtu.be/ZGLPxEv_EWo).

Incredibly, you can build your own wearable Jarvis-like system by combining the thrilling power of novel holographic devices like HoloLens with Microsoft Cognitive Services and Microsoft Mixed Reality. Such systems can quickly process visual content and describe it via spoken words or text displayed on screen. With just a few voice commands or air gestures, you can ask your Mixed Reality “Jarvis” system to look up what you see on the web and tell you more about it. You could use this system with AI to, say, develop a device to support visually impaired persons (https://youtu.be/R2mC-NUAmMk).

In this book you will learn how to develop AI-powered Mixed Reality apps using various strategies and technologies, including Universal Windows Platform (UWP), Microsoft Cognitive Services, Unity, Vuforia, and Xamarin UrhoSharp. First, you will learn how to set up the development environment, install the necessary tools, and use the HoloLens emulator and Mixed Reality simulators. Then you will start writing UWP two-dimensional apps that run across every Windows 10 device (including HoloLens and immersive headsets) and adjust their views and functionalities to specific devices. Subsequently, you will learn how to transfer images from the world-facing camera of the headset to the machine learning modules of the computer vision service to obtain image descriptions that are spoken by the device or displayed in text form. Afterward, you will use sensor readings from the mobile device to control content displayed on the Windows Mixed Reality headset.

After learning about 2D app development, you will move on to building 3D apps. You’ll learn how to do this from the ground up. First, you’ll explore the fundamental concepts of 3D graphics. You will then learn how to use Unity Editor to build 3D Mixed Reality apps, including setting up scenes, adding built-in and custom 3D objects, and formatting these objects with materials to create holograms. Afterward, you will learn how to make your holograms behave like real objects with various physics simulations and to interact with these holograms through air gestures and voice commands. Next, you will learn how to attach augmented reality to real objects with Vuforia. Finally, you will build a Mixed Reality app with UrhoSharp—a library for writing cross-platform 3D apps. This will enable you to transfer your skills to other platforms, such as iOS or Android.

Audience and Expected Skills

This book is devoted to developers, students, engineers, enthusiasts, designers, scientists, and researchers who would like to use their existing programming skills to develop software for Windows Mixed Reality (HoloLens and immersive headsets) with Unity, Vuforia, and UrhoSharp. I assume the reader knows fundamental aspects of C# programming and is experienced in Windows programming. I do not, however, assume any previous knowledge of Unity, Vuforia, or UrhoSharp.

Tools and Required Hardware

To implement all examples presented in this book, you will need a system that runs Windows 10 (Creators Update or later) and uses Visual Studio 2017 Community as the development environment.

Organization of This Book

The book is divided into three parts:

Conventions

The following conventions are used in this book:

About the Companion Content

This book includes companion code to enrich your learning experience. The companion code for this book can be downloaded from the following page:

https://github.com/dawidborycki/MixedReality-Samples.

As shown in Figure I-1, the code is partitioned into subfolders, which correspond to particular chapters. To improve book readability, in many places I refer to the companion code rather than showing the full listings, so it is good to have the solution open while reading this book.

You can also find the companion files at https://aka.ms/ProgMixedReality/downloads.

A screenshot showing the Solution Explorer with companion code folders displayed.
FIGURE I-1 The structure of the companion code.

Acknowledgments

The publication of this book would not have been possible without Loretta Yates and Laura Norman, who accepted my book proposal and provided initial feedback. I am grateful to John Ray for thoroughly checking every project presented in this book and providing useful, positive comments on the content. Many thanks, too, to Kate Shoup for copyediting the book. Finally, I appreciate ongoing support from my family: my wife, Agnieszka; my daughter, Zuzanna; and my son, Ksawery. I would achieve nothing without them.

Errata and Book Support

We have made every effort to ensure the accuracy of this book and its companion content. Any errors that have been reported since this book was published are listed on our Microsoft Press site at:

https://aka.ms/ProgMixedReality/errata

If you find an error that is not already listed, you can report it to us through the same page.

If you need additional support, email Microsoft Press Book Support at mspinput@microsoft.com.

Please note that product support for Microsoft software is not offered through the addresses above.

Stay in Touch

Let’s keep the conversation going! We’re on Twitter: @MicrosoftPress.