Chapter 4. Controlling a Phone App with Your Suave Gestures

 

You've got all the moves.

 
 --Lani Hall, Never Say Never Again (1983)>

He raises an eyebrow; he lowers his chin; he twists the corners of his mouth; he folds one arm into the crook of the other as he points his pistol at the ceiling. It all looks very impressive, but is he simply dithering while trying to remember people's names?

Agent 007 has a few old friends with normal names, such as Bill Tanner and Felix Leiter. Almost every other name is a number, a single letter, a mash-up of multiple languages, or a blindingly obvious double entendre. After a few vodka martinis and tranquilizer darts, any man would start to wonder whether his memory for names was playing tricks on him.

To put such doubts to rest, we will develop an Android app that determines a person's name based on a series of yes/no questions. To allow a secret agent to use it discretely, the app will rely on gesture controls and audio output, which can go to a Bluetooth headset so that others cannot hear.

The app's logic is like the parlor game, Twenty Questions. First, the app asks a question by playing an audio clip. Then, the user responds with a nod or a shake of the head. Each question is more specific than the last, until the app is ready to guess a name or give up. Recognizing the two possible head gestures—a nod or a shake—is our computer vision task for this chapter.

The app's codename is Goldgesture.

Note

The completed project for this chapter can be downloaded from my website at http://nummist.com/opencv/7376_04.zip.

Planning the Goldgesture app

Goldgesture is a GUI app built with the Android SDK and OpenCV's Java bindings for Android. It has just a single view, as shown in the screenshot on the next page. The app has the following flow of execution:

  1. The app constantly displays a live video feed from the front-facing (self-portrait) camera.
  2. It performs human face detection using OpenCV's CascadeClassifier class.
  3. When a human face is detected:
    1. It draws a blue rectangle around the face.
    2. It detects features of the face (points that should be easy to track in subsequent frames despite movement) using OpenCV's goodFeaturesToTrack function. It then draws green circles around these features.
  4. As the face moves, the app tracks the features in every frame using OpenCV's calcOpticalFlowPyrLK function. This function can continuously track the features even though CascadeClassifier is unlikely to continuously detect a face.
  5. When the features' center point moves up and down by a certain amount and a certain number of times, the app deems that a nod has occurred.
  6. When the features' center point moves left and right by a certain amount and a certain number of times, the app deems that a shake of the head has occurred.
  7. The app plays a sequence of audio clips. At each juncture, it chooses the next clip depending (in part) on whether a nod or shake of the head has occurred.
  8. The app resets the tracking when its reliability deteriorates to a certain extent or when the user's head appears to be nodding and shaking at the same time.
    Planning the Goldgesture app

The face detection functionality in Goldgesture should already be familiar to you from last chapter's project, Angora Blue. However, feature tracking, and specifically optical flow, is a new topic for us. Let's talk about the concepts a little before proceeding to set up our project.