© Wallace Wang 2018

Wallace WangBeginning ARKit for iPhone and iPadhttps://doi.org/10.1007/978-1-4842-4102-8_2

2. Getting to Know ARKit

Wallace Wang¹ 

(1)

San Diego, CA, USA

 

Augmented reality works by tracking the real world through a camera. By identifying solid objects in the real world such as floors, tables, and walls, augmented reality can then accurately place virtual objects on the scene that create the illusion of actually being there. Even if the virtual object is nothing more than a cartoon Pokémon character, augmented reality must overlay that virtual object so the virtual object doesn’t get cut in half by furniture, walls, or tables.

Since creating algorithms for detecting objects in the real world can be difficult even for experienced programmers, Apple created a software framework called ARKit, which provides much of the basic needs of any augmented reality app. By using ARKit, you can create augmented reality apps by focusing on the unique features of your app rather than on the details of detecting, displaying, and tracking virtual objects in the real world.

ARKit acts as a platform for you to develop your own augmented reality apps. To help you get familiar using ARKit, Xcode provides a simple augmented reality project that you can compile and run on any compatible iOS device physically connected to your Macintosh through its USB cable. To create this ARKit sample app, follow these steps:

1. 1.

   Start Xcode. (Make sure you’re using Xcode 10 or greater.)

    
2. 2.

   Choose File ➤ New ➤ Project. Xcode asks you to choose a template, as shown in Figure 2-1.

    

Figure 2-1

Choosing an Xcode project template

1. 3.

   Click the Augmented Reality App icon and click the Next button. Xcode asks for a product name, organization name, organization identifiers, and content technology, as shown in Figure 2-2.

    

Figure 2-2

Defining the options for an augmented reality project

1. 4.

   Click in the Product Name text field and type a descriptive name for your project, such as ARExample. (The exact name does not matter.)

    

Note

The organization name and identifier can be any text such as your name or company name. The organization identifier is typically the website address of your company spelled backwards, such as com.microsoft.

1. 5.

   Make sure the Content Technology popup menu displays SceneKit. SpriteKit and Metal give you more versatility at the expense of more complexity. For the purposes of this book, all augmented reality apps will rely on SceneKit.

    
2. 6.

   Make sure the Include Unit Tests and Include UI Tests check boxes are not checked since we won’t be using these features in any of the apps created in this book.

    
3. 7.

   Click the Next button. Xcode asks where you want to store your project.

    
4. 8.

   Choose a folder and click the Create button. Xcode creates an augmented reality project that’s ready to run.

    
5. 9.

   Connect your iPhone or iPad to your Macintosh using its USB cable.

    
6. 10.

   Click on the popup menu near the top of Xcode window that displays the device to run your project on, as shown in Figure 2-3.

    

Figure 2-3

Defining the target to run the project on

1. 11.

   Choose your iOS device, such as an iPhone or iPad.

    
2. 12.

   Click the Run button or choose Product ➤ Run. A dialog appears, asking if you want to allow your project to access the iOS device’s camera, as shown in Figure 2-4.

    

Figure 2-4

Apps must always ask for permission to access an iOS device’s camera

Note

Xcode may ask for your password to allow your app to access additional libraries. To grant access, type your password and click the Allow button.

1. 13.

   Your project appears on your iOS device. Notice that a cartoon airplane appears, as shown in Figure 2-5. Each time you run this project, point your iOS device in a different direction. Whatever direction your iOS device’s camera points at is where the cartoon airplane will appear. Move your iOS device around and you’ll be able to see the cartoon airplane from different angles.

    

Figure 2-5

Viewing the virtual airplane through an iPhone screen

1. 14.

   Click the Stop button in Xcode or choose Product ➤ Stop.

    

Understanding the Swift Source Code

By creating a project using the Augmented Reality App template, you can create a working augmented reality app without writing or modifying a single line of code. To better understand how to use ARKit, let’s dissect the Swift code so you can understand exactly what’s happening.

Your entire augmented reality project consists of several files, but the ViewController.swift file contains all the Swift code you need to add augmented reality to any project. First, notice that the ViewController.swift file imports three software frameworks: UIKit (defines the user interface), SceneKit (defines the 3D animation used to create virtual images), and ARKit (links to the ARKit augmented reality library).

import UIKit

import SceneKit

import ARKit

Note

SceneKit is Apple’s framework for creating 3D animation, but two other options are SpriteKit and Metal. If you choose either of these options, your project would need to import the SpriteKit or Metal framework instead of SceneKit.

Next, you must define the ViewController class as the ARSCNViewDelegate :

class ViewController: UIViewController, ARSCNViewDelegate {

Now you need to create a scene for displaying a virtual image. To do this, you need to create an IBOutlet. The name of this IBOutlet can be anything you wish, but the Augmented Reality App template names this IBOutlet sceneView and it represents an ARSCNView object:

@IBOutlet var sceneView: ARSCNView!

Within the viewDidLoad method , you need to define four items. First, you must set the ViewController class to its own ARSCNViewDelegate:

// Set the view's delegate

sceneView.delegate = self

Second, you can display statistics on the screen that let you know information such as frames per second (fps):

// Show statistics such as fps and timing information

sceneView.showsStatistics = true

Third, you need to define the actual image to display on the augmented reality scene. Remember, the scene is defined by the IBOutlet named sceneView :

// Create a new scene

let scene = SCNScene(named: "art.scnassets/ship.scn")!

If you click on the art.scnassets folder in Xcode, you can see two graphic files called ship.scn and texture.png , as shown in Figure 2-6.

Figure 2-6

The contents of the art.scnassets folder

The ship.scn file represents a SceneKit image (notice the .scn file extension). Another type of graphic image you can use with ARKit is the COLLADA (COLLAborative Design Activity) file, which has the .dae file extension. Nearly all 3D authoring tools, such as SketchUp, can export files to the .dae format, which is an open standard for storing 3D images.

The ship.scn file defines the 3D shape of the plane. The texture.png graphic file defines the image that gets applied on the ship.scn image to display different colors or patterns. In most cases, you’ll need both a 3D image (a .scn or .dae file) and a texture (a .png file) that wraps around the 3D image and provides the “skin” or outside graphics for that 3D image. If you click on the texture.png file, you can see what it looks like, as shown in Figure 2-7.

Figure 2-7

The texture.png file defines the “skin” of a 3D image

Fourth, after defining the 3D image with a variable name (scene), you need to put this 3D image into the actual scene view:

// Set the scene to the view

sceneView.scene = scene

In the viewWillAppear method , you need two additional lines of Swift code. The first line turns on the iOS device’s tracking to measure the location and angle you aim the iOS device’s camera:

// Create a session configuration

let configuration = ARWorldTrackingConfiguration()

The second line runs the actual augmented reality session:

// Run the view's session

sceneView.session.run(configuration)

Understanding the User Interface

The user interface for the Augmented Reality App Template consists of a single view in a storyboard. On that view is an ARSCNView object that fills the entire view, as shown in Figure 2-8. This ARKit SceneKit View allows SceneKit 3D images to appear on the user interface.

Figure 2-8

The ARKit SceneKit View defines where to display the 3D image on the user interface

Every augmented reality app must access the iOS device’s camera. However, every app must first ask for permission to use the camera. To ask for permission to access the camera, your app must define a privacy setting for the camera. You can view this camera privacy setting in the Info.plist file , as shown in Figure 2-9.

Figure 2-9

The Info.plist file defines the privacy setting for the iOS device camera

The text that appears in the Value column of the camera privacy setting will appear in the dialog that asks the user’s permission for your app to access the iOS device’s camera. This text is simply an explanation for why your app needs access to the camera, such as “This application will use the camera for Augmented Reality.” You can always change this text to something else if you wish.

The second line in the Info.plist file that every augmented reality app needs is the Required Device Capabilities setting. It must be set to arkit, as shown in Figure 2-10.

Figure 2-10

The Info.plist file defines the hardware requirements for an iOS device to run augmented reality apps

This setting in the Info.plist file makes sure your app will only attempt to run on an iOS device with the proper hardware capable of running ARKit such as an iPhone 6s or higher, or an iPad Pro or higher.

Creating Augmented Reality with the Single View App Template

The Augmented Reality App template provides the basic Swift code needed to display augmented reality. Rather than use the Augmented Reality App template, you can create an augmented reality app using the simple Single View App template instead. By creating an augmented reality app through the Single View App template, you can get a better idea what code you need to write and what user interface elements you need to give any app augmented reality capabilities.

1. 1.

   Start Xcode. (Make sure you’re using Xcode 10 or greater.)

    
2. 2.

   Choose File ➤ New ➤ Project. Xcode asks you to choose a template (see Figure 2-1).

    
3. 3.

   Click the Single View App icon and click the Next button. Xcode asks for a product name, organization name, organization identifiers, and content technology (see Figure 2-2). Make sure all check boxes are clear for additional options such as Use Core Data or Include Unit Tests.

    
4. 4.

   Click in the Product Name text field and type a descriptive name for your project, such as ARExample2. (The exact name does not matter, but make sure it’s different from the project you created using the Augmented Reality App template.)

    
5. 5.

   Click the Next button. Xcode asks where you want to store your project.

    
6. 6.

   Choose a folder and click the Create button. Xcode creates a basic iOS project that’s ready to run.

    

At this point, you have a basic iOS app with no augmented reality features. To add augmented reality to an app, we need to write Swift code, modify the user interface, and define settings in the Info.plist file to allow access to the camera and run only on ARKit-compatible iOS devices such as the iPad Pro or iPhone 6s and higher.

First, we need to make sure our iOS app can access the ARKit framework and use the camera. To do this, we need to modify the Info.plist file .

1. 1.

   Click the Info.plist file in the Navigator pane. Xcode displays a list of keys, types, and values.

    
2. 2.

   Click the disclosure triangle to expand the Required Device Capabilities category to display Item 0.

    
3. 3.

   Move the mouse pointer over Item 0 to display a plus (+) icon.

    
4. 4.

   Click this plus (+) icon to display a blank Item 1.

    
5. 5.

   Type arkit under the Value category in the Item 1 row (see Figure 2-10).

    
6. 6.

   Move the mouse pointer over the last row to display a plus (+) icon.

    
7. 7.

   Click on the plus (+) icon to create a new row. A popup menu appears.

    
8. 8.

   Choose Privacy – Camera Usage Description, as shown in Figure 2-11.

    

Figure 2-11

The Privacy – Camera Usage Description line lets your app access an iOS device’s camera

1. 9.

   Type AR needs to use the camera under the Value category in the Privacy – Camera Usage Description row.

    

With “arkit” and “Privacy – Camera Usage Description” defined in the Info.plist file, our app can now access ARKit and use an iOS device’s camera. The next step is to modify the ViewController.swift file and write Swift code to display augmented reality.

1. 1.

   Click on the ViewController.swift file in the Navigator pane of Xcode.

    
2. 2.
   Add the following two lines under the Import UIKit line, as follows:

   import SceneKit

   import ARKit
    
3. 3.
   Modify the class ViewController line to add the ARSCNViewDelegate as follows:

   class ViewController: UIViewController, ARSCNViewDelegate {
    
4. 4.
   At the bottom of the ViewController.swift file, add the viewWillAppear and viewWillDisappear functions as follows:

       override func viewWillAppear(_ animated: Bool) {

           super.viewWillAppear(animated)

           let configuration = ARWorldTrackingConfiguration()

           sceneView.session.run(configuration)

       }

       override func viewWillDisappear(_ animated: Bool) {

           super.viewWillDisappear(animated)

           sceneView.session.pause()

       }
    

In this code, we’re referencing sceneView, which hasn’t been defined yet. This sceneView variable name will represent the user interface view that displays augmented reality in our app. The user interface object that displays augmented reality is called ARKit Scene View.

To add the ARKit Scene View object, we need to drag it to the Main.storyboard file and create an IBOutlet for it in the ViewController.swift file. To do this, follow these steps:

1. 1.

   Click on the Main.storyboard file in the Navigator pane of Xcode. Xcode displays an iOS device on the storyboard screen that you can change by clicking View As at the bottom of the storyboard screen, as shown in Figure 2-12.

    

Figure 2-12

The View As option lets you choose different iOS devices for a storyboard

1. 2.

   Click the Object Library icon to display the Object Library window , as shown in Figure 2-13.

    

Figure 2-13

The Object Library icon opens the Object Library window

1. 3.

   Click in the search field at the top of the Object Library window and type ARKit. The Object Library window displays all ARKit objects available, as shown in Figure 2-14.

    

Figure 2-14

Displaying ARKit objects in the Object Library window

1. 4.

   Drag the ARKit SceneKit View from the Object Library on to the storyboard.

    
2. 5.

   Resize the ARKit SceneKit View on the storyboard, as shown in Figure 2-15. The exact size and position of the ARKit SceneKit View isn’t important but make it large enough because the size of the ARKit SceneKitView defines how large the image will appear when viewed through the iOS device’s camera.

    

Figure 2-15

Resizing the ARKit SceneKit View

1. 6.

   Click on the ARKit SceneKit View to select it and then choose Editor ➤ Resolve AutoLayout Issues ➤ Reset to Suggested Constraints. Xcode adds constraints to keep your ARKit SceneKit View properly aligned no matter which size or orientation the user holds the iOS device.

    
2. 7.

   Click the Show Assistant Editor icon, as shown in Figure 2-16, or choose View ➤ Assistant Editor ➤ Use Assistant Editor. Xcode displays the ViewController.swift file side by side with the storyboard.

    

Figure 2-16

The Show Assistant Editor icon lets you view a storyboard and Swift controller file at the same time

1. 8.

   Move the mouse over the ARKit SceneKIt View, hold down the Control key, and drag the mouse underneath the class ViewController line, as shown in Figure 2-17.

    

Figure 2-17

Control-dragging from the ARKit SceneKit View to the ViewController.swift file

1. 9.

   Release the Control key and the mouse. Xcode displays a popup menu to define a name for the IBOutlet, as shown in Figure 2-18.

    

Figure 2-18

Defining a name for an IBOutlet

1. 10.

   Click in the Name field, type sceneView, and press Return. Xcode creates an IBOutlet in the ViewController.swift file as follows:

    

@IBOutlet var sceneView: ARSCNView!

1. 11.

   Click the Use Standard Editor icon or choose View ➤ Standard Editor ➤ Use Standard Editor.

    
2. 12.

   Click the ViewController.swift file in the Navigator pane. Xcode displays the Swift code stored in the ViewController.swift file.

    
3. 13.
   Edit the viewDidLoad function as follows:

       override func viewDidLoad() {

           super.viewDidLoad()

           sceneView.delegate = self

           sceneView.showsStatistics = true

           let scene = SCNScene(named: "")!

           sceneView.scene = scene

       }
    

These code changes essentially duplicate the ARKit iOS template. However, we still need an object to place in our augmented reality view. When we created an augmented reality app from the augmented reality template, that template included a ship.scn file, where the .scn file extension stands for SceneKit.

What we need initially are files stored in the .dae COLLADA file format, which stands for COLLAborative Design Activity. This file format is used as a standard file format for sharing graphic designs for three-dimensional programs.

To find .dae COLLADA files, visit your favorite search engine and look for “.dae public domain” files that you can download. (For the artistically-inclined, you can create your own three-dimensional objects using graphics editors, such as the free Blender program available at www.blender.org .) Most COLLADA files consist of a .dae file that defines the shape of the object and a texture file that defines the outer design of that shape. Two sites that offer free (and paid) COLLADA files include Free3D ( https://free3d.com ) and TurboSquid ( www.turbosquid.com ).

Once you’ve downloaded a .dae COLLADA file along with any accompanying texture files, you must create a special scnassets folder to store those images. To create an scnassets folder, follow these steps:

1. 1.

   Choose File ➤ New ➤ File. Xcode displays different file templates.

    
2. 2.

   Click iOS at the top of the template window and scroll down to click on the SceneKit Catalog icon under the Resource category, as shown in Figure 2-19.

    

Figure 2-19

Choosing the SceneKit Catalog icon

1. 3.

   Click the Next button. Xcode asks where you want to store this folder.

    
2. 4.

   Click the Create button. Xcode creates a SceneKit Asset Catalog.scnassets folder in the Navigator pane.

    
3. 5.

   Click on the SceneKit Asset Catalog.scnassets folder and press Return. Xcode highlights the entire folder name, as shown in Figure 2-20.

    

Figure 2-20

Changing the name of the SceneKit Assets Catalog folder

1. 6.

   Change the name of the folder to art.scnassets and press Return.

    

Now that we’ve written the bulk of the Swift code needed in the ViewController.swift file and designed the user interface to display augmented reality through the ARKit SceneKit View, the last step is to import a .dae file and its texture file into the .scnassets folder you created in the Xcode Navigator pane.

To do add 3D images to Xcode, follow these steps:

1. 1.

   Drag and drop the .dae and accompanying texture file image from the Finder window to the scnassets folder, as shown in Figure 2-21.

    

Figure 2-21

Drag and drop a .dae and texture file from the Finder window to the scnassets folder in Xcode

1. 2.

   Click on the .dae file in the scnassets folder to select it.

    
2. 3.

   Choose Editor ➤ Convert to SceneKit scene file format (.scn). A dialog appears, asking you to verify you want to convert the .dae file to a .scn file, as shown in Figure 2-22.

    

Figure 2-22

Xcode asks for confirmation to convert the .dae file to an .scn file

1. 4.

   Click the Convert button. Xcode converts your .dae file to an .scn file.

    
2. 5.

   (Optional) Click on the .scn file and press Return to edit the filename to something simple and descriptive.

    
3. 6.
   Edit the following line in the viewDidLoad function to include the name of your .scn file. If your .scn file was named satellite.scn, the code would look like this:

   let scene = SCNScene(named: "art.scnassets/satellite.scn")!
    

This Swift code will load the .dae file (converted to an .scn file) into your augmented reality view. However, there’s still one last step. With most .dae files, there’s an accompanying texture file that defines the outer appearance or “skin” of the three-dimensional object. The final step is to apply this texture or “skin” to the .scn file. To do this, follow these steps:

1. 1.

   Click on the .scn file in your scnassets folder displayed in the Navigator pane. Xcode displays your image as a general shape but with no outer appearance.

    
2. 2.

   Click the Show Scene Graph View icon near the bottom of the Xcode window, as shown in Figure 2-23. Xcode displays the Scene Graph View.

    

Figure 2-23

The Show Scene Graph View icon

1. 3.

   Click on each item displayed in the Scene Graph View pane and then click on the Show the Material Inspector icon, as shown in Figure 2-24. Or choose View ➤ inspectors ➤ Show Material Inspector.

    

Figure 2-24

The Show Material Inspector icon

1. 4.

   Click on the Diffuse popup menu and choose the name of your texture file, such as texture.jpg, as shown in Figure 2-25.

    

Figure 2-25

The Diffuse popup menu lets you choose the texture image

If your original .dae file came with two or more texture files, you may need to include those multiple texture files in the scnassets folder and use the Diffuse popup menu to select each appropriate texture file for different parts of your three-dimensional object.

Now attach an iOS device to your Macintosh through its USB cable and click the Run icon or choose Product ➤ Run. You should now see your .scn file displayed over the image captured by your iOS device’s camera.

Summary

While it’s possible to create augmented reality apps on your own, it’s far simpler to rely on Apple’s ARKit framework. ARKit takes care of the details of managing a camera and real-world objects around you to combine reality with virtual images.

The simplest way to create an augmented reality app is to start with the Augmented Reality template when creating a new iOS project. However, you can also add augmented reality features to an existing app. First, you must import the ARKit framework along with a graphics framework such as SceneKit. Next you must create an ARKit SceneKit View on your app’s user interface to view the actual augmented reality image. Finally, you must import a three-dimensional image into Xcode and convert it to an .scn SceneKit file.

When you want an app focused on augmented reality, it’s best to create a new project using the Augmented Reality project template. When you want to add augmented reality features to an existing app, you can easily do so at any time as well.

Now that you have a basic idea how to create an augmented reality app and the various steps you need to follow to create augmented reality, it’s time to go into more detail about the specific parts of the different augmented reality features available through ARKit.