Setting up ARKit

ARKit requires the usage of macOS High Sierra, because of the XCode 9 and above requirement for compiling and making changes to the API itself. So, I would highly suggest having a macOS from late 2011 or newer. I am utilizing a Mac Mini 2011 model with 8 GB of RAM, although the standard 4 GB should be plenty. Unity3D does utilize OpenCL/OpenGL extensively, which requires a GFX card capable of utilizing Metal. 2011 and earlier macOSs do not have this ability natively; this could be circumvented by having an external GPU (Radeon RX 480 is currently the only GPU supported officially for this).

With that out of the way, we can begin with installing and configuring ARKit for Unity3D on our macOS.

There are a couple of ways you can install ARKit:

  1. We can navigate to the plugin page on the Asset Store (https://www.assetstore.unity3d.com/en/#!/content/92515):
  1. Or we can download the plugin directly from the Bitbucket repository (https://bitbucket.org/Unity-Technologies/unity-arkit-plugin/overview):

  1. If we go the first route and install from the Asset Store, we don't have to worry about copying the files into our project ourselves, but either way it is simple enough to do, so take your pick on the method you want to go with and create a new project called ARKitTutorial:

Next up, we have quite a bit to unpack regarding what is actually in this package:

  • /Assets/Plugins/iOs/UnityARKit/NativeInterface/ARsessionNative.mm – This is the Objective-C code that is the actual interfaces with the ARKit SDK.
  • /Assets/Plugins/iOS/UnityARKit/NativeInterface/UnityARSessionNativeInterface.cs – This is the scripting API that glues the native code to ARKit.
  • /Assets/Plugins/iOS/UnityARKit/NativeInterface/AR*.cs – These are the equivalents of the data structures exposed by ARKit.
  • /Assets/Plugins/iOS/UnityARKit/Utility/UnityARAnchorManager.cs – This is a utility script that tracks the anchor updates from ARKit and can create the proper corresponding GameObjects in Unity for them.
  • /Assets/Plugins/iOS/UnityARKit/Editor/UnityARBuildPostprocessor.cs – This is an editor script that runs at build time on iOS.
  • /Assets/Plugins/iOS/UnityARKit/UnityARCameraManager.cs – This is the component that should be placed on a GameObject in the scene that references the camera that you want to control. It will position and rotate the camera as well as provide the correct projection matrix based on updates from ARKit. This component also initializes as ARKit Session.
  • /Assets/Plugins/iOS/UnityARKit/UnityARVideo.cs – This is the component that should be placed on the camera and grabs the textures needed for rendering the video. It sets the material needed for blitting to the backbuffer and sets up the command buffer for blitting.
  • /Assets/Plugins/iOS/UnityARKit/UnityARUserAnchorComponent.cs – This is the component that adds and removes Anchors from ARKit based on the life cycle of the GameObject it is added to.

Before we build our own "Hello World" example, we should build the UnityARKitScene.unity to iOS to get a taste of what ARKit is capable of, as it demonstrates all of the basic functionality of ARKit in that scene.

UnityARKitScene is included in the plugin as well as a couple of other example projects. We will compile the UnityARKitScene as our Hello World application.

However, before we do that, we need to talk about the file structure, because those who are not well-versed with compiling to iOS will have some serious issues compiling without further clarification. You may have noticed quite a few items that we have not mentioned that are in the plugin, so let's go ahead and discuss what all of them do before moving on.

\UnityARKitPlugin main directory files:

  • ARKitRemote – Allows you to send remote commands from your device to the Unity3D editor
  • Examples – This directory houses example scripts and scenes to showcase various things you can do with ARKit and this plugin
  • Plugins – Houses the directories required to run ARKit
  • Resources – Houses the resource files required for ARKit

Plugins\iOS\UnityARKit\NativeInterface cs files:

  • ARAnchor – Anchors an object to a location in the world from the camera feed.
  • ARCamera – Tracks the camera's position.
  • ARErrorCode – Error codes.
  • ARFaceAnchor – Face tracking anchor.
  • ARFrame – Returns data about the camera, anchors, and light estimates.
  • ARHitTestResult –Returns any collision results.
  • ARHitTestResultType – Enumeration for the hit test types available.
  • ARLightEstimate – Calculates how much luminosity is in the image or video.
  • ARPlaneAnchor – Anchors a plane to a specific 4x4 matrix.
  • ARPlaneAnchorAlignment – Aligns the anchor horizontally with respect to gravity.
  • ARPoint – A point struct for x and y values as a double.
  • ARRect – A struct that takes ARPoint as the origin and ARSize as the size.
  • ARSessionNative – Native plugin used to specify framework dependencies and the platforms the plugin should work for.
  • ARSize –A struct that takes a width and height value as a double.
  • ARTextureHandles – A native Metal texture handler for the camera buffer which takes an IntPtr  (int pointer) for both textureY and textureCbCr values.
  • ARTrackingQuality – Enumeration for tracking qualities available.
  • ARTrackingState –Enumeration for tracking states. Limited, Normal, and NoAvailable are the options.
  • ARTrackingStateReason – Enumeration for the state reasons. Options are Excessive Motion, Insufficient Features, and Initializing.
  • ARUserAnchor – Defines this anchor's transformation matrix for rotation, translation, and scale in world coordinates.
  • UnityARSessionNativeInterface – Native plugin wrapper code.

\Plugins\iOS\UnityARKit\Helpers cs files:

  • AR3DOFCameraManager – A helper class for 3D objects with the AR Camera
  • ARPlaneAnchorGameObject – A class that attaches a GameObject with the ARPlaneAnchor
  • DontDestroyOnLoad – Makes sure the GameObject doesn't get destroyed on load
  • PointCloudParticleExample – Creates a point cloud particle system
  • UnityARAmbient – A helper function for ambient lighting
  • UnityARAnchorManager – A manager for ARPlaneAnchorGameObjects
  • UnityARCameraManager – A helper class for the ARCamera
  • UnityARCameraNearFar – Sets the Near Far distance of the camera appropriately
  • UnityARGeneratePlane – Creates an ARPlaneAnchorGameObject
  • UnityARHitTestExample – Tests collisions with various amounts of planes, from few to infinite
  • UnityARKitControl – A helper class designed for creating a test ARSession
  • UnityARKitLightManager – A helper class for managing the various lighting possibilities
  • UnityARMatrixOps – A class for converting a 4x4 matrix to Euclidean space for quaternion rotation
  • UnityARUserAnchorComponent – A helper class for creating Anchor added and removed events
  • UnityARUtility – A helper class to do coordinate conversions from ARKit to Unity
  • UnityARVideo – A helper function to render the video texture to the scene
  • UnityPointCloudExamples – A helper function to draw a point cloud using particle effects

\Plugins\iOS\UnityARKit\Shaders shader files:

  • YUVShader – A gamma Unlit Shader for rendering Textures
  • YUVShaderLinear – A linear Unlit Shader for rendering Textures

\UnityARKitPlugin\Resources file:

  • UnityARKitPluginSettings.cs – Is a scriptable object that toggles whether ARKit is required for the app and toggles Facetracking for the iPhone X.

UnityARKitPlugin\ARKitRemote files:

  • ARKitRemote.txt – A text file that explains how to set up and use ARKitRemote
  • EditorTestScene.unity – Test scene that should run when running ARKitRemote
  • UnityARKitRemote.unity – Scene that should be compiled and installed on an applicable device
  • ARKitRemoteConnection.cs – Used to send data from the device to the UnityEditor
  • ConnectionMessageIds – GUIDs for the Editor Session Message
  • ConnectToEditor.cs – Creates a network connection between the editor and the device
  • EditorHitTest – Returns collision data from device to editor
  • ObjectSerializationExtension.cs – An extension to convert an object to a byte array
  • SerializableObjects.cs – Serializes Vector 4 data and a 4x4 matrix
  • UnityRemoteVideo.cs – Connects to the editor and transfers a video feed from the device to the editor

UnityARKitPlugin\Examples files:

  • AddRemoveAnchorExample – An example program to add and remove anchors
  • Common – Has common materials, models, prefabs, shaders, and textures that are used in various projects
  • FaceTracking – Face tracking example application
  • FocusSquare – An example scene where it finds a specific anchor
  • UnityARBallz – An example scene where you play a game with balls
  • UnityARKitScene – A basic scene with minimal scripts attached to test if ARKit works appropriately
  • UnityAROcclusion – An example project that showcases various lighting conditions
  • UnityARShadows – An example project that handles low lighting conditions
  • UnityParticlePainter – An example project that lets you paint with particles

Now that we have a fundamental understanding of everything inside this package, let's build our Hello World with ARKit.