James Bond lives in a nightmarish world where laws are written at the point of a gun | ||
| --Yuri Zhukov, Pravda, 30 September 1965 | ||
Bond: Just a moment. Three measures of Gordon's, one of vodka, half a measure of Kina Lillet. Shake it very well until it's ice-cold, then add a large thin slice of lemon peel. Got it? | ||
| --Casino Royale, Chapter 7: Rouge et Noir (1953) | ||
James Bond is a precise man. Like a physicist, he seems to see order in a world where others see chaos. Another mission, another romance, another shaken drink, another crashing car or helicopter or skier, and another gunshot do not change the way the world works or the way the Cold War worked. He seems to take comfort in this consistency.
A psychologist might say that Bond has a habit of re-enacting an unhappy childhood, which the novels reveal to us in brief glimpses. The boy lacked a permanent home. His father was an international arms dealer for the Vickers company, so the family moved often for work. When James was 11, his parents died in a mountain climbing accident—the first of many dramatic, untimely deaths in the Bond saga. An aunt in Kent, England took in the orphaned James, but the next year he was sent to board at Eton College. There, the lonesome boy became infatuated with a maid, got into trouble over it, and was expelled—the first of his many short-lived and fraught romances. Next, he was sent even further from family, to Fettes College in Scotland. The pattern of displacement and trouble was set. By 16, he was trying to live the life of a playboy in Paris. By 20, he was a dropout from the University of Geneva, and he was off to join the Royal Navy at the height of the Second World War.
Amid all that upheaval, Bond did manage to learn a thing or two. He is clever, not just with his eyebrow-raising witty remarks, but also with his fast solutions to puzzles that involve mechanics, kinematics, or physics. He is never caught flat-footed (though he is sometimes caught in other ways).
The moral of the story is that a secret agent must practice his physics even under the most trying of circumstances. An app can help with that.
When I think about problems of geometry or physics, I like to draw them with pen and paper. However, I also like to see animations. Our app, Rollingball, will allow us to combine these two media. It will use computer vision to detect simple geometric shapes that the user can draw on paper. Then, based on the detected shapes, the app will create a physics simulation that the user can watch. The user can also influence the simulation by tilting the device to alter the simulated direction of gravity. The experience is like designing and playing one's own version of a ball-in-a-maze puzzle, a fine toy for aspiring secret agents.
Building games is fun—but it is not all fun and games! We have a new list of skills to master in this chapter:
- Detecting linear and circular edges using the Hough transform
- Detecting solid circles as blobs
- Using OpenCV in the Unity game engine
- Building a Unity game for Android
- Converting coordinates from OpenCV's space to Unity's space, and creating 3D objects in Unity, based on our detection results in OpenCV
- Customizing the appearance and physics behavior of 3D objects in Unity using shaders, materials, and physics materials
- Drawing lines and rectangles explicitly using OpenGL calls from Unity
With these goals in mind, let's get ready to play ball!
Note
The completed project for this chapter can be downloaded from my website at http://nummist.com/opencv/7376_07.zip. The download does not include the OpenCV for Unity plugin, which must be licensed and added to the project as described in the Setting up OpenCV for Unity section.
Rollingball will be a mobile app. We will develop it in the Unity game engine using a third-party plugin called OpenCV for Unity. The app will be compatible with both Android and iOS. Our build instructions will focus on Android, but we will also provide a few notes for readers who are experienced with the iOS build process (on Mac).
Note
For instructions on setting up Unity and finding relevant documentation and tutorials, please refer back to the Unity section in Chapter 1, Preparing for the Mission. At the time of writing, Unity's supported development environments are Windows and Mac, but sadly not Linux.
Using the mobile device's camera, Rollingball will scan two types of primitive shapes: circles and lines. The user will start by drawing any combination of these primitive shapes. For example, refer to the following photograph:
Here we have three circles and three lines drawn on the facing side of the paper. The reverse side also has some shapes drawn or printed on it, but these shapes are only faintly visible, so our detector should ignore them. The edges of the paper and some elements of the background can be considered as clearly visible lines, so our detector should detect them.
Rollingball is a simple app with only one Android activity or one iOS view controller. A live video feed fills most of the background. When circles or lines are detected, they are highlighted in red, as seen in the following screenshot:
Note that some lines, especially the drawn lines, are detected multiple times because they are not perfectly straight nor are they perfectly continuous.
The user may press a button to start the physics simulation. During the simulation, the video pauses, the detector stops running, and the red-highlighted areas are replaced with cyan balls and lines. The lines are stationary but the balls fall freely and may bounce off each other and off the lines. Real-world gravity, as measured by the mobile device's gravity sensor, is used to control the simulated direction of gravity. However, the simulation is two-dimensional, and gravity is flattened such that it points toward an edge of the screen. The following screenshot shows the simulated balls after they have fallen partway down the page:
The user may press the button again to clear all the simulated objects and resume the live video and detection. The cycle can continue indefinitely with the user choosing to simulate different drawings or different views of the same drawing.
Now, let's consider the techniques to detect circles and lines.