Now, we can finally start working in Unity in this section. This is the easier part, where we just need to create our wrapper and our MonoBehaviour script to attach to an object.
Navigate to the dll file that we created. This should be in the x64 | Debug folder of the source project:
Create two folder called Plugins and Scripts in Unity, just as we did in Chapter3.
Now, we will create two scripts. One for our Wrapper class, and the other for our MonoBehaviour. The Wrapper class will be called OpenCVWrapper, and the MonoBehaviour class will be called OpenCVFaceDetection.
Open the OpenCVWrapper class in Visual Studio. It is time to write some more code.
We only need to use the InteropServices namespace for this class:
using System.Runtime.InteropServices;
We will create an internal static class this time around:
internal static class OpenCVWrapper
{
We will import the Init function that we created in the last step, and we need to make sure that we reference the parameters. The ref keyword is very similar to the & keyword in C++:
[DllImport("UnityOpenCVSample")]
internal static extern int Init(ref int outCameraWidth, ref int outCameraHeight);
We will import the Close function, which closes the connection and will avoid memory leaks when we use the functions that we've created:
[DllImport("UnityOpenCVSample")]
internal static extern int Close();
We will import the SetScale function we created, along with keeping the parameters that we required in C++:
[DllImport("UnityOpenCVSample")]
internal static extern int SetScale(int downscale);
We will import the Detect function, and this one is a bit different, as we are actually using a pointer; this will be very important very soon, as this deals with unsafe code in C# and Unity. If you aren't familiar, the * keyword denotes a pointer, which is the address of the object in memory:
[DllImport("UnityOpenCVSample")]
internal unsafe static extern void Detect(CvCircle* outFaces, int maxOutFacesCount, ref int outDetectedFacesCount);
}
Lastly, we will create a structure that needs to be sequential and with the correct byte size (3 ints = 4 bytes * 3 = 12 bytes) for CvCircle:
[StructLayout(LayoutKind.Sequential, Size = 12)]
public struct CvCircle
{
public int X, Y, Radius;
}
This takes care of the wrapper class, and we can now move over to our MonoBehaviour class.
We need a few namespaces, as they will be fully utilized in this script:
using UnityEngine;
using System.Collections;
using System.Collections.Generic;
We have our class named the same as the file in the Unity Editor and inherit from MonoBehaviour:
public class OpenCVFaceDetection : MonoBehaviour
{
The main thing to notice here is that I have a reference to the camera and a WebCamTexture. This is because we will feed the data from the webcam to the camera:
public Camera camera;
public static List<Vector2> NormalizedFacePositions { get; private set; }
public static Vector2 CameraResolution;
private const int DetectionDownScale = 1;
private bool _ready;
private int _maxFaceDetectCount = 5;
private CvCircle[] _faces;
private Quaternion baseRotation;
private WebCamTexture webCamTexture;
In this Start method, we get everything set up and running. We also check to make sure that the cascades.xml file is able to be found (more on that in the next section):
void Start()
{
int camWidth = 0, camHeight = 0;
webCamTexture = new WebCamTexture();
Renderer renderer = GetComponent<Renderer>();
renderer.material.mainTexture = webCamTexture;
baseRotation = transform.rotation;
webCamTexture.Play();
camWidth = webCamTexture.width;
camHeight = webCamTexture.height;
int result = OpenCVWrapper.Init(ref camWidth, ref camHeight);
if (result < 0)
{
if (result == -1)
{
Debug.LogWarningFormat("[{0}] Failed to find cascades definition.", GetType());
}
else if (result == -2)
{
Debug.LogWarningFormat("[{0}] Failed to open camera stream.", GetType());
}
return;
}
CameraResolution = new Vector2(camWidth, camHeight);
_faces = new CvCircle[_maxFaceDetectCount];
NormalizedFacePositions = new List<Vector2>();
OpenCVWrapper.SetScale(DetectionDownScale);
_ready = true;
}
This method will make sure that the connections are closed to the webcam. This will free up the resources and make sure that we don't leak any memory:
void OnApplicationQuit()
{
if (_ready)
{
OpenCVWrapper.Close();
}
}
This Update method makes sure that the orientation of the webcam is corrected, checks whether the camera is read or not, and actively tracks for face detection:
void Update()
{
if (!_ready)
{
return;
}
transform.rotation = baseRotation * Quaternion.AngleAxis(webCamTexture.videoRotationAngle, Vector3.up);
int detectedFaceCount = 0;
unsafe
{
fixed (CvCircle* outFaces = _faces)
{
OpenCVWrapper.Detect(outFaces, _maxFaceDetectCount, ref detectedFaceCount);
}
}
NormalizedFacePositions.Clear();
for (int i = 0; i < detectedFaceCount; i++)
{
NormalizedFacePositions.Add(new Vector2((_faces[i].X * DetectionDownScale) / CameraResolution.x, 1f - ((_faces[i].Y * DetectionDownScale) / CameraResolution.y)));
}
}
}
Save the script and go back to the Unity Editor. You will immediately notice that Unity will show an error along the lines of unsafe code needs to be allowed. Let's go ahead and enable this feature. To do this, go to your Player Settings, which is located inside the Build Settings.
Inside the Player Settings, look down at the configuration inside Other Settings, and there is a checkbox called Allow 'unsafe' Code. Make sure that it is checked:
In the Scripts folder, you need one more file to be added; in my example file that you can download, I have quite a few more .xml files than what I am going to tell you to add. The reason for this is to allow you to play around with the different .xml files to see their results. You will have to update the C++ plugin to account for the proper .xml file you want to use; alternatively, you can update the Init function to take a string parameter to be able to change the .xml file in the Unity Editor.
In your OpenCV folder, navigate to OpenCV\opencv\build\etc\lbpcascades:
You want to copy lbpcascade_frontalface.xml into the scripts folder in Unity. (My project has everything in an XML folder, as I have many more .xml files to play with.)
Finally, we just need to create a plane to face the camera.
The last step is to attach the OpenCVFaceDetection script to the plane.
Now, the project will compile and run appropriately (if you get a dll import error, make sure you have the dll set to x86-x64 and that the project is built for Windows):
