Adding identification to our smart-house application

As a part of our smart-house application, we want the application to recognize who we are. Doing so opens up the opportunity to get responses and actions from the application, tailored to you.

Creating our smart-house application

Create a new project for the smart-house application, based on the MVVM template we created earlier.

With the new project created, add the Microsoft.ProjectOxford.Face NuGet package.

As we will be building this application throughout this book, we will start small. In the MainView.xaml file, add a TabControl property containing two items. The two items should be two user controls, one called the AdministrationView.xaml file and the other called the HomeView.xaml file.

The administration control will be where we administer different parts of the application. The home control will be the starting point and the main control to use.

Add corresponding ViewModel instances to the Views. Make sure they are declared and created in MainViewModel.cs, as we have seen throughout this chapter. Make sure that the application compiles and runs before moving on.

Adding people to be identified

Before we can go on to identify a person, we need to have something to identify them from. To identify a person, we need a PersonGroup property. This is a group that contains several Persons properties.

In the administration control, we will execute several operations in this regard. The UI should contain two textbox elements, two list box elements, and six buttons. The two textbox elements will allow us to input a name for the person group and a name for the person. One list box will list all person groups that we have available. The other will list all the persons in any given group.

We have buttons for each of the operations that we want to execute, which are as follows:

The View model should have two ObservableCollection properties: one of a PersonGroup type and the other of a Person type. We should also add three string properties. One will be for our person group name, the other for our person name. The last will hold some status text. We also want a PersonGroup property for the selected person group. Finally, we want a Person property holding the selected person.

In our View model, we want to add a private variable for the FaceServiceClient method, as shown in the following code:

    private FaceServiceClient _faceServiceClient;

This should be assigned in the constructor, which should accept a parameter of a FaceServiceClient type. It should also call an initialization function, which will initialize six ICommand properties. These maps to the buttons, created earlier. The initialization function should call the GetPersonGroups function to list all person groups available, as shown in the following code:

    private async void GetPersonGroups() {
        try {
            PersonGroup[] personGroups = await
            _faceServiceClient.ListPersonGroupsAsync();

The ListPersonGroupsAsync function does not take any parameters, and returns a PersonGroup array if successfully executed, as shown in the following code:

            if(personGroups == null || personGroups.Length == 0)
            {
                StatusText = "No person groups found.";
                return;
            }

            PersonGroups.Clear();

            foreach (PersonGrouppersonGroup in personGroups)
            {
                PersonGroups.Add(personGroup); 
            }
        }

We then check to see whether the array contains any elements. If it does, we clear out the existing PersonGroups list. Then we loop through each item of the PersonGroup array and add them to the PersonGroups list.

If no person groups exist, we can add a new one by filling in a name. The name you fill in here will also be used as a person group ID. This means that it can include numbers and English lowercase letters, the "-" character (hyphen), and the "_" character (underscore). The maximum length is 64 characters. When it is filled in, we can add a person group.

First, we call the DoesPersonGroupExistAsync function, specifying PersonGroupName as a parameter, as shown in the following code. If this is true, then the name we have given already exists, and as such, we are not allowed to add it. Note how we call the ToLower function on the name. This is so we are sure that the ID is in lowercase:

    private async void AddPersonGroup(object obj) {
        try {
            if(await DoesPersonGroupExistAsync(PersonGroupName.ToLower())) {
                StatusText = $"Person group {PersonGroupName} already exist";
                return;
            }

If the person group does not exist, we call the CreatePersonGroupAsync function, as shown in the following code. Again, we specify the PersonGroupName as lowercase in the first parameter. This represents the ID of the group. The second parameter indicates the name we want. We end the function by calling the GetPersonGroups function again, so we get the newly added group in our list:

            await _faceServiceClient.CreatePersonGroupAsync (PersonGroupName.ToLower(), PersonGroupName);
            StatusText = $"Person group {PersonGroupName} added";
            GetPersonGroups();
        }

The DoesPersonGroupExistAsync function makes one API call. It tries to call the GetPersonGroupAsync function, with the person group ID specified as a parameter. If the resultant PersonGroup list is anything but null, we return true.

To delete a person group, a group must be selected as follows:

    private async void DeletePersonGroup(object obj)
    {
        try
        {
            await _faceServiceClient.DeletePersonGroupAsync (SelectedPersonGroup.PersonGroupId);
            StatusText = $"Deleted person group {SelectedPersonGroup.Name}";

            GetPersonGroups();
        }

The API call to the DeletePersonGroupAsync function requires a person group ID as a parameter. We get this from the selected person group. If no exception is caught, then the call has completed successfully, and we call the GetPersonGroups function to update our list.

When a person group is selected from the list, we make sure that we call the GetPersons function. This will update the list of persons, as follows:

    private async void GetPersons()
    {
        if (SelectedPersonGroup == null)
            return;

        Persons.Clear();

        try
        {
            Person[] persons = await _faceServiceClient.GetPersonsAsync(SelectedPersonGroup.PersonGroupId);

We make sure the selected person group is not null. If it is not, we clear our persons list. The API call to the GetPersonsAsync function requires a person group ID as a parameter. A successful call will result in a Person array.

If the resultant array contains any elements, we loop through it. Each Person object is added to our persons list, as shown in the following code:

            if (persons == null || persons.Length == 0)
            {
                StatusText = $"No persons found in {SelectedPersonGroup.Name}.";
                return;
            }

            foreach (Person person in persons)
            {
                Persons.Add(person);
            }
        }

If no persons exist, we can add new ones. To add a new one, a person group must be selected, and a name of the person must be filled in. With this in place, we can click on the Add button:

    private async void AddPerson(object obj)
    {
        try
        {
            CreatePersonResultpersonId = await _faceServiceClient.CreatePersonAsync(SelectedPersonGroup.PersonGroupId, PersonName);
            StatusText = $"Added person {PersonName} got ID: {personId.PersonId.ToString()}";
               
            GetPersons();
        }

The API call to the CreatePersonAsync function requires a person group ID as the first parameter. The next parameter is the name of the person. Optionally, we can add user data as a third parameter. In this case, it should be a string. When a new person has been created, we update the persons list by calling the GetPersons function again.

If we have selected a person group and a person, then we will be able to delete that person, as shown in the following code:

    private async void DeletePerson(object obj)
    {
        try
        {
            await _faceServiceClient.DeletePersonAsync (SelectedPersonGroup.PersonGroupId, SelectedPerson.PersonId);

            StatusText = $"Deleted {SelectedPerson.Name} from {SelectedPersonGroup.Name}";

            GetPersons();
        }

To delete a person, we make a call to the DeletePersonAsync function. This requires the person group ID of the person group the person lives in. It also requires the ID of the person we want to delete. If no exceptions are caught, then the call succeeded, and we call the GetPersons function to update our person list.

Our administration control now looks similar to the following screenshot:

Adding new persons

Before we can identify a person, we need to associate faces with that person. With a given person group and person selected, we can add faces. To do so, we open a file dialog. When we have an image file, we can add the face to the person, as follows:

        using (StreamimageFile = File.OpenRead(filePath))
        {
            AddPersistedFaceResultaddFaceResult = await _faceServiceClient.AddPersonFaceAsync(
            SelectedPersonGroup.PersonGroupId,
            SelectedPerson.PersonId, imageFile);

            if (addFaceResult != null)
            {
                StatusText = $"Face added for {SelectedPerson.Name}. Remember to train the person group!";
            }
        }

We open the image file as a Stream. This file is passed on as the third parameter in our call to the AddPersonFaceAsync function. Instead of a stream, we could have passed a URL to an image.

The first parameter in the call is the person group ID of the group in which the person lives. The next parameter is the person ID.

Some optional parameters to include are user data in the form of a string and a FaceRectangle parameter for the image. The FaceRectangle parameter is required if there is more than one face in the image.

A successful call will result in an AddPersistedFaceResult object. This contains the persisted face ID for the person.

Each person can have a maximum of 248 faces associated with it. The more faces you can add, the more likely it is that you will receive a solid identification later. The faces that you add should from slightly different angles.

With enough faces associated with the persons, we need to train the person group. This is a task that is required after any change to a person or person group.

We can train a person group when one has been selected, as shown in the following code:

    private async void TrainPersonGroup(object obj)
    {
        try
        {
            await _faceServiceClient.TrainPersonGroupAsync(
SelectedPersonGroup.PersonGroupId);

The call to the TrainPersonGroupAsync function takes a person group ID as a parameter, as shown in the following code. It does not return anything, and it may take a while to execute:

            while(true)
            {
                TrainingStatustrainingStatus = await _faceServiceClient.GetPersonGroupTrainingStatusAsync (SelectedPersonGroup.PersonGroupId);

We want to ensure that the training completed successfully. To do so, we call the GetPersonGroupTrainingStatusAsync function inside a while loop. This call requires a person group ID, and a successful call results in a TrainingStatus object, as shown in the following code:

                if(trainingStatus.Status != Status.Running)
                {
                    StatusText = $"Person group finished with status: {trainingStatus.Status}";
                    break;
                }

                StatusText = "Training person group...";
                await Task.Delay(1000);
            }
        }

We check the status and we show the result if it is not running. If the training is still running, we wait for one second and run the check again.

When the training has succeeded, we are ready to identify people.

There are a few API calls that we have not looked at, which will be mentioned briefly in the following bullet list:

Identifying a person

To identify a person, we are first going to upload an image. Open the HomeView.xaml file and add a ListBox element to the UI. This will contain the person groups to choose from when identifying a person. We will need to add a button element to find an image, upload it, and identify the person. A TextBox element is added to show the working response. For our own convenience, we also add an image element to show the image we are using.

In the View model, add an ObservableCollection property of a PersonGroup type. We need to add a property for the selected PersonGroup type. Also, add a BitmapImage property for our image, and a string property for the response. We will also need an ICommand property for our button.

Add a private variable for the FaceServiceClient type, as follows:

    private FaceServiceClient _faceServiceClient;

This will be assigned in our constructor, which should accept a parameter of a FaceServiceClient type. From the constructor, call on the Initialize function to initialize everything, as shown in the following code:

    private void Initialize()
    {
        GetPersonGroups();
        UploadOwnerImageCommand = new DelegateCommand(UploadOwnerImage,CanUploadOwnerImage);
    }

First, we call the GetPersonGroups function to retrieve all the person groups. This function makes a call to the ListPersonGroupsAsync API, which we saw earlier. The result is added to our PersonGroup list's ObservableCollection parameter.

Next, we create our ICommand object. The CanUploadOwnerImage function will return true if we have selected an item from the PersonGroup list. If we have not, it will return false, and we will not be able to identify anyone.

In the UploadOwnerImage function, we first browse to an image and then load it. With an image loaded and a file path available, we can start to identify the person in the image, as shown in the following code:

    using (StreamimageFile = File.OpenRead(filePath))
    {
        Face[] faces = await _faceServiceClient.DetectAsync(imageFile);
        Guid[] faceIds = faces.Select(face =>face.FaceId).ToArray();

We open the image as a Stream type, as shown in the following code. Using this, we detect faces in the image. From the detected faces, we get all the face IDs in an array:

        IdentifyResult[] personsIdentified = await _faceServiceClient.IdentifyAsync (SelectedPersonGroup.PersonGroupId,
faceIds, 1);

The array of face IDs will be sent as the second parameter to the IdentifyAsync API call. Remember that when we detect a face, it is stored for 24 hours. Proceeding to use the corresponding face ID will make sure that the service knows which face to use for identification.

The first parameter used is the ID of the person group we have selected. The last parameter in the call is the number of candidates returned. As we do not want to identify more than one person at a time, we specify one. Because of this, we should ensure that there is only one face in the image we upload.

A successful API call will result in an array of the IdentifyResult parameter, as shown in the following code. Each item in this array will contain candidates:

    foreach(IdentifyResultpersonIdentified in personsIdentified) { 
        if(personIdentified.Candidates.Length == 0) {
            SystemResponse = "Failed to identify you.";
            break;
        }
        GuidpersonId = personIdentified.Candidates[0].PersonId;

We loop through the array of results, as shown in the following code. If we do not have any candidates, we just break out of the loop. If, however, we do have candidates, we get the PersonId parameter of the first candidate (we asked for only one candidate earlier, so this is okay):

        Person person = await faceServiceClient.GetPersonAsync(
SelectedPersonGroup.PersonGroupId, personId);

        if(person != null) {
            SystemResponse = $"Welcome home, {person.Name}";
            break;
        }
    }
}

With the personId parameter, we get a single Person object, using the API to call the GetPersonAsync function. If the call is successful, we print a welcome message to the correct person (as shown in the following screenshot) and break out of the loop:

Identifying a person