Chapter 1. Preparing for the Mission

Q: I've been saying for years, sir, that our special equipment is obsolete. And now, computer analysis reveals an entirely new approach: miniaturization.

On Her Majesty's Secret Service (1969)

James Bond is not a pedestrian. He cruises in a submarine car, he straps on a rocket belt, and oh, how he skis, how he skis! He always has the latest stuff and he is never afraid to put a dent in it, much to the dismay of Q, the engineer.

As software developers in the 2010s, we are witnessing an explosion in the adoption of new platforms. Under one family's roof, we might find a mix of Windows, Mac, iOS, and Android devices. Mom and Dad's workplaces provide different platforms. The kids have three game consoles or five if you count the mobile versions. The toddler has a LeapFrog learning tablet. Smart glasses are becoming more affordable.

We must not be afraid to try new platforms and consider new ways to combine them. After all, most users do.

This book embraces multi-platform development. It presents weird and wonderful applications that we can deploy in unexpected places. It uses several of the computer's senses, but especially uses computer vision to breathe new life into the humdrum, heterogeneous clutter of devices that surround us.

Before Agent 007 runs amok with the gadgets, he is obligated to listen to Q's briefing. This chapter performs Q's role. This is the setup chapter.

By the end of this chapter, you will obtain all the tools to develop OpenCV applications in C++ or Python for Windows, Mac, or Linux, and in C++ or Java for Android. You will also be the proud new user of a Raspberry Pi single-board computer (this additional hardware is optional). You will even know a bit about Unity, a game engine into which we can integrate OpenCV.

If you find yourself a bit daunted by the extent of this setup chapter, be reassured that not all of the tools are required and no single project uses all of them in combination. Although Q and I live for the big event of setting up multiple technologies at once, you could just skim this chapter and refer back to it later when the tools become useful, one by one, in our projects.

Note

Where basic OpenCV setup and reference materials are concerned, this chapter includes excerpts from my introductory books, OpenCV Computer Vision with Python and Android Application Programming with OpenCV, published by Packt Publishing. All contents are retested, updated, and expanded to cover newer OpenCV versions and additional operating systems. Also, there are all new sections on the optional hardware and game engine used in this book.

Setting up a development machine

We can develop our OpenCV applications on a desktop, a notebook, or even the humble Raspberry Pi (covered later in the Setting up a Raspberry Pi section). Most of our apps have a memory footprint of less than 128 MB, so they can still run (albeit slowly) on old or low-powered machines. To save time, develop on your fastest machine first and test on slower machines later.

This book assumes that you have one of the following operating systems on your development machine:

Windows XP or a later version
Mac OS 10.6 or a later version
Debian Wheezy or a derivative such as the following:
- Raspbian
- Ubuntu 12.04 or a later version
- Linux Mint 13 or a later version
Fedora 18 or a later version, or a derivative such as the following:
- Red Hat Enterprise Linux (RHEL) 7 or a later version
- CentOS 7 or a later version
openSUSE 13.1 or a later version, or a derivative

Other Unix-like systems can also work but they are not covered in this book.

You should have a USB webcam and any necessary drivers. Most webcams come with instructions for installing drivers on Windows and Mac. Linux distributions typically include the USB Video Class (UVC) Linux driver, which supports many webcams, listed at http://www.ideasonboard.org/uvc/#devices.

We are going to set up the following components:

A C++ development environment. On Windows, we will use Visual Studio 2010 or a later version. Alternatively, Windows users can follow Kevin Hughes' helpful tutorial on setting up OpenCV with MinGW and the Code::Blocks IDE at http://kevinhughes.ca/tutorials/opencv-install-on-windows-with-codeblocks-and-mingw/. On Mac, we will use Xcode. On Linux, we will use GCC, which comes as standard.
On Mac, we will use a third-party package manager to help us install libraries and their dependencies. We will use either MacPorts or Homebrew.
A Python 2.7 development environment. At the time of writing, the best option is to use version 2.7 as it is the most recent Python version supported by OpenCV's stable branch. (Python 2.6 is also supported by the stable branch.)
Popular libraries such as NumPy (for numeric functions), SciPy (for numeric and scientific functions), Requests (for web requests), and wxPython (for cross-platform GUIs).
PyInstaller, a cross-platform tool used for bundling Python scripts, libraries, and data as redistributable apps, such that users machines do not require installations of Python, OpenCV, and other libraries. For this book's purposes, building redistributables of Python projects is an optional topic. We will cover the basics in Chapter 2, Searching for Luxury Accommodations Worldwide, but you might need to do your own testing and debugging as PyInstaller (like other Python bundling tools) does not show entirely consistent behavior across operating systems, Python versions, and library versions. It is not well supported on Raspberry Pi or other ARM systems.
A build of OpenCV with C++ and Python support plus optimizations for certain desktop hardware. At the time of writing, OpenCV 2.4.x is the stable branch and our instructions are tailored for this branch.
Another build of OpenCV with C++ and Java support plus optimizations for certain Android hardware. Specifically, we will use the OpenCV build that comes with Tegra Android Development Pack (TADP). At the time of writing, TADP 3.0r4 is the most recent release.
An Android development environment, including Eclipse, ADT, Android SDK, and Android NDK. TADP includes these too.
On Windows or Mac, a 3D game engine called Unity.
Tip
Eclipse has a big memory footprint. Even if you want to use Raspberry Pi for developing desktop and Pi apps, use something with more RAM for developing Android apps.

Let's break this setup down into three sets of platform-dependent steps, plus a set of platform-independent steps for TADP, and another set of platform-independent steps for Unity.

Windows

On Windows, we have the option of setting up a 32-bit development environment (to make apps that are compatible with both 32-bit and 64-bit Windows) or a 64-bit development environment (to make optimized apps that are compatible with 64-bit Windows only). Recent versions of OpenCV are available in 32-bit and 64-bit versions.

We also have a choice of either using binary installers or compiling OpenCV from source. For our Windows apps in this book, the binary installers provide everything we need. However, we will also discuss the option of compiling from source because it enables us to configure additional features, such as support for Kinect and Asus depth cameras, which might be relevant to your future work or to our projects in other books.

Note

For an OpenCV project that uses a depth camera, refer to my book OpenCV Computer Vision with Python, published by Packt Publishing.

Regardless of our approach to obtain OpenCV, we need a general-purpose C++ development environment and a general-purpose Python 2.7 development environment. We will set up these environments using binary installers.

As our C++ development environment, we will use Visual Studio 2010 or a later version. Use any installation media you might have purchased, or go to the downloads page at http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs.aspx. Download and run the installer for one of the following:

Visual C++ 2010 Express, which is free
Visual Studio Express 2013 for Windows desktop, which is free
Any of the paid versions, which have 90-day free trials

If the installer lists optional C++ components, we should opt to install them all. After the installer runs till completion, reboot.

Tip

If we plan to compile OpenCV from source (as described in the OpenCV on Windows with CMake and Compilers section), I recommend you use Visual Studio 2010 (and not any later version). At the time of writing, OpenCV and some of its optional dependencies do not compile easily with Visual Studio 2012 or Visual Studio 2013.

Installers for Python 2.7 are available at http://www.python.org/getit/. Download and run the latest revision of Python 2.7 in either the 32-bit variant or the 64-bit variant.

To make Python scripts run using our new Python 2.7 installation by default, let's edit the system's Path variable and append ;C:\Python2.7 (assuming Python 2.7 is installed in the default location). Remove any previous Python paths, such as ;C:\Python2.6. Log out and log back in (or reboot).

Let's assume that we also want to use binary installers for NumPy, SciPy, and wxPython. Download and run the installers for the latest stable library versions that target Python 2.7. We can find these installers at the following locations:

NumPy: The official installers are 32-bit only and are located at http://sourceforge.net/projects/numpy/files/NumPy/. Unofficial 64-bit installers are available at http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy.
SciPy: The official installers are 32-bit only and are located at http://sourceforge.net/projects/scipy/files/scipy/. Unofficial 64-bit installers are available at http://www.lfd.uci.edu/~gohlke/pythonlibs/#scipy.
wxPython: This can be downloaded from http://www.wxpython.org/download.php. The apps in this book are successfully tested with wxPython 2.8, 2.9, and 3.0. If in doubt, choose version 3.0. However, if you choose version 2.8, get its Unicode variant.

Requests does not have a binary installer but we can download the latest source bundle from https://github.com/kennethreitz/requests/archive/master.zip. Unzip it to any destination, which we will refer to as <unzip_destination>. Open Command Prompt and run the following commands:

> cd <unzip_destination>
> python setup.py install

Next, we can put PyInstaller in any convenient location, since it is treated as a set of tools rather than a library. Let's download the latest release version from http://www.pyinstaller.org/ and unzip it to C:\PyInstaller or any another location of your choice.

Now, we are ready to set up OpenCV and, optionally, other computer vision libraries.

> cmake -D CMAKE_BUILD_TYPE=RELEASE -D WITH_OPENNI=ON -D OPENNI_LIB_DIR="<openni_install_destination>\Lib" -D OPENNI_INCLUDE_DIR="<openni_install_destination>\Include" -D OPENNI_PRIME_SENSOR_MODULE_BIN_DIR="<sensorkinect_install_destination>\Bin" -D WITH_TBB=ON -D TBB_LIB_DIR="<tbb_unzip_destination>\lib\ia32\vc10" -D TBB_INCLUDE_DIR="<tbb_unzip_destination>\include" -G "Visual Studio 10" "<opencv_unzip_destination>\opencv\sources"

Alternatively, to create a 64-bit project for Visual Studio 2010, run the following command (but replace the angle brackets and their contents with the actual paths):

> cmake -D CMAKE_BUILD_TYPE=RELEASE -D WITH_OPENNI=ON -D OPENNI_LIB_DIR="<openni_install_destination>\Lib" -D OPENNI_INCLUDE_DIR="<openni_install_destination>\Include" -D OPENNI_PRIME_SENSOR_MODULE_BIN_DIR="<sensorkinect_install_destination>\Bin" -D WITH_TBB=ON -D TBB_LIB_DIR="<tbb_unzip_destination>\lib\intel64\vc10" -D TBB_INCLUDE_DIR="<tbb_unzip_destination>\include" -G "Visual Studio 10 Win64" "<opencv_unzip_destination>\opencv\sources"

If OpenNI is not installed, omit -D WITH_OPENNI=ON -D OPENNI_LIB_DIR="<openni_install_destination>\Lib" -D OPENNI_INCLUDE_DIR="<openni_install_destination>\Include" -D OPENNI_PRIME_SENSOR_MODULE_BIN_DIR="<sensorkinect_install_destination>\Bin". (In this case, depth cameras will not be supported.)

If OpenNI is installed but SensorKinect is not, omit -D OPENNI_PRIME_SENSOR_MODULE_BIN_DIR="<sensorkinect_install_destination>\Bin". (In this case, Kinect will not be supported.)

If TBB is not installed, omit -D WITH_TBB=ON -D TBB_LIB_DIR="<tbb_unzip_destination>\lib\ia32\vc10" -D TBB_INCLUDE_DIR="<tbb_unzip_destination>\include" (32-bit) or -D WITH_TBB=ON -D TBB_LIB_DIR="<tbb_unzip_destination>\lib\intel64\vc10" -D TBB_INCLUDE_DIR="<tbb_unzip_destination>\include" (64-bit). (In this case, Intel multiprocessing will not be supported.)

CMake will produce a report on the dependencies that it did or did not find. OpenCV has many optional dependencies, so do not panic (yet) about missing dependencies. However, if the build does not finish successfully, try installing missing dependencies (many are available as prebuilt binaries). Then, repeat this step.

Now that our build system is configured, we can compile OpenCV. Open <build_folder>\OpenCV.sln in Visual Studio. Select Release configuration and build the project (you might get errors if you select another build configuration besides Release.)
Copy <build_folder>\lib\RELEASE\cv2.pyd to C:\Python2.7\Lib\site-packages (assuming that Python 2.7 is installed in the default location). Now, the Python installation can find part of OpenCV.
Finally, we need to make sure that Python and other processes can find the rest of OpenCV and its dependencies. Edit the system's Path variable and append ;<build_folder>\bin\RELEASE. If we are using TBB, also append ;<tbb_unzip_destination>\lib\ia32\vc10 (32-bit) or ;<tbb_unzip_destination>\lib\intel64\vc10 (64-bit). Log out and log back in (or reboot).

You might want to look at the code samples in <unzip_destination>/opencv/samples.

Mac

Let's begin by setting up Xcode and the Xcode Command Line Tools, which give us a complete C++ development environment:

Download and install Xcode from the Mac App Store or http://connect.apple.com/. If the installer provides an option to install Command Line Tools, select it.
Open Xcode. If a license agreement is presented, accept it.
If the Xcode Command Line Tools were not already installed, we must install them now. Go to Xcode | Preferences | Downloads and click on the Install button next to Command Line Tools. Wait for the installation to finish. Then, quit Xcode. Alternatively, if you do not find an option to install the Command Line Tools from inside Xcode, open Terminal and run the following command:
```
$ xcode-select install
```

Next, we need a Python 2.7 development environment. Recent versions of Mac come with Python 2.7 preinstalled. However, the preinstalled Python is customized by Apple for the system's internal needs. Normally, we should not install any libraries atop Apple's Python. If we do, our libraries might break during system updates or worse, might conflict with preinstalled libraries that the system requires. Instead, we should install standard Python 2.7 and then install our libraries atop it.

For Mac, there are several possible approaches to obtain standard Python 2.7 and Python-compatible libraries such as OpenCV. All approaches ultimately require OpenCV to be compiled from source using Xcode Command Line Tools. However, depending on the approach, this task is automated for us by third-party tools in various ways. We will look at approaches using MacPorts or Homebrew. These two tools are package managers, which help us resolve dependencies and separate our development libraries from the system libraries.

Tip

I recommend MacPorts. Compared to Homebrew, MacPorts offers more patches and configuration options for OpenCV. Also, I maintain a MacPorts repository to ensure that you can get continue to get an OpenCV build that is compatible with all of my books. Particularly, my version includes support for depth cameras such as Kinect, which were used in OpenCV Computer Vision with Python.

Normally, MacPorts and Homebrew should not be installed on the same machine.

Regardless of the approach to set up our Python environment, we can put PyInstaller in any convenient location, since it is treated as a set of tools rather than a library. Let's download the latest release version from http://www.pyinstaller.org/ and unzip it to ~/PyInstaller or another location of your choice.

Note

Our installation methods for Mac do not give us the OpenCV sample projects. To get these, download the latest source code archive from http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/ and unzip it to any location. Find the samples in <opencv_unzip_destination>/samples.

Now, depending on your preference, let's proceed to either the Mac with MacPorts section or the Mac with Homebrew section.

Mac with MacPorts

MacPorts provides Terminal commands that automate the process of downloading, compiling, and installing various pieces of open source software (OSS). MacPorts also installs dependencies as needed. For each piece of software, the dependencies and build recipe are defined in a configuration file called a Portfile. A MacPorts repository is a collection of Portfiles.

Starting from a system where Xcode and its Command Line Tools are already set up, the following steps will give us an OpenCV installation via MacPorts:

Download and install MacPorts from http://www.macports.org/install.php.
If we want an
OpenCV build that is fully compatible with all of my books, we need to inform MacPorts where to download some custom Portfiles that I have written. To do so, edit /opt/local/etc/macports/sources.conf (assuming MacPorts is installed in the default location). Just above the line rsync://rsync.macports.org/ release/ports/ [default], add the following line:
```
http://nummist.com/opencv/ports.tar.gz
```
Tip
Downloading the example code
You can download the example code fies from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the fies e-mailed directly to you. The latest and updated example code for this book is also available from the author's website at http://nummist.com/opencv/.
Save the file. Now, MacPorts knows to search for Portfiles in my online repository first and then the default online repository.
Open Terminal and run the following command to update MacPorts:
```
$ sudo port selfupdate
```
When prompted, enter your password.
Now (if you are using my repository), run the following command to install OpenCV
with Python 2.7 bindings, plus extras such as support for Intel TBB multiprocessing and support for depth cameras including Kinect:
```
$ sudo port install opencv +python27 +tbb +openni_sensorkinect
```
Alternatively (with or without my repository), run the following command to install OpenCV with Python 2.7 bindings, plus extras such as support for Intel TBB multiprocessing and support for depth cameras excluding Kinect:
```
$ sudo port install opencv +python27 +tbb +openni
```
Dependencies, including Python 2.7, NumPy, OpenNI, and (in the first example) SensorKinect, are automatically installed as well.
By adding +python27 to the command, we are specifying that we want the OpenCV variant (build configuration) with Python 2.7 bindings. Similarly, +tbb specifies the variant with support for Intel TBB multiprocessing, which can greatly improve the performance on compatible hardware. The +openni_sensorkinect tag specifies the variant with the broadest possible support for depth cameras via OpenNI and SensorKinect. You can omit +openni_sensorkinect if you do not intend to use depth cameras or you can replace it with +openni if you do intend to use OpenNI-compatible depth cameras but just not Kinect. To see the full list of available variants before installing, we can enter:
```
$ port variants opencv
```
Depending on our customization needs, we can add other variants to the install command.

Run the following commands to install SciPy, Requests, and wxPython:

$ sudo port install py27-scipy
$ sudo port install py27-requests
$ sudo port install py27-wxpython-3.0

The Python installation's executable is named python2.7. If we want to link the default python executable to python2.7, let's also run:
```
$ sudo port install python_select
$ sudo port select python python27
```

Now we have everything we need to develop OpenCV applications for Mac. To also develop the same for Android, we need to set up TADP as described in the section Tegra Android Development Pack, covered later in this chapter.

Mac with Homebrew

Like MacPorts, Homebrew is a package manager that provides Terminal commands to automate the process of downloading, compiling, and installing various pieces of open source software.

Starting from a system where Xcode and its Command Line Tools are already set up, the following steps will give us an OpenCV installation via Homebrew:

Open Terminal and run the following command to install Homebrew:
```
$ ruby -e "$(curl -fsSkLraw.github.com/mxcl/homebrew/go)"
```
Unlike MacPorts, Homebrew does not automatically put its executables in PATH. To do so, create or edit the file ~/.profile and add this line at the top:
```
export PATH=/usr/local/bin:/usr/local/sbin:$PATH
```
Save the file and run this command to refresh PATH:
```
$ source ~/.profile
```
Note that executables installed by Homebrew now take precedence over executables installed by the system.
For Homebrew's self-diagnostic report, run:
```
$ brew doctor
```
Follow any troubleshooting advice it gives.
Now, update Homebrew:
```
$ brew update
```
Run the following command to install Python 2.7:
```
$ brew install python
```
Now, we can install NumPy. Homebrew's selection of Python library packages
is limited, so we will use a separate package management tool called pip, which comes with Homebrew's Python:
```
$ pip install numpy
```
SciPy contains some Fortran code, so we need an appropriate compiler. We can use Homebrew to install the GFortran compiler:
```
$ brew install gfortran
```
Now, we can install SciPy:
```
$ pip install scipy
```
Similarly, we can install Requests and wxPython:
```
$ pip install requests wxpython
```
We need to install OpenCV from an optional repository called homebrew/science. Run the following commands:
```
$ brew tap homebrew/science
$ brew install opencv
```
Lastly, we
must tell Python where to find Homebrew's installation of OpenCV. Edit your ~/.profile file to add the following line:
```
export PYTHONPATH=/usr/local/Cellar/opencv/2.4.7/lib/python2.7/site-packages:$PYTHONPATH
```

Debian Wheezy and its derivatives, including Raspbian, Ubuntu, and Linux Mint

Note

For information on setting up the Raspbian operating system, refer to the section Setting up a Raspberry Pi covered later in this chapter.

On Raspbian, Ubuntu 13.04 and its later versions, and Ubuntu derivatives such as Linux Mint 15 and its later versions, a recent version of OpenCV is available in the standard repository. This OpenCV package includes support for many video codes and for Intel TBB multiprocessing, but it does not include support for depth cameras (as used in my other book, OpenCV Computer Vision with Python). To install OpenCV and SciPy, open Terminal and run the following commands:

$ sudo apt-get install python-opencv
$ sudo apt-get install python-scipy

On other systems, the standard repository contains OpenCV 2.3.1 but this version is old (from 2011) and lacks some of the functionality used in this book. Thus, we want to compile a newer version of OpenCV from source. Compiling from source is likewise a requirement if we want support for Asus and Kinect depth cameras. Because the dependencies are complex, I have written a script that downloads, configures, and builds OpenCV and related libraries. Here are the steps to obtain and use the script, and then install a few additional elements of our Python environment:

Download the script from http://nummist.com/opencv/install_opencv_debian_wheezy.sh and put it in any destination, say <script_folder>.
If you prefer Qt over GTK (for example, if you intend to build apps for KDE), then edit <script_folder>/install_debian_wheezy.sh and replace WITH_QT=0 with WITH_QT=1. Save and close the file.
Open Terminal. Change the directory to <script_folder>:
```
$ cd <script_folder>
```
Set the script's permissions so that it is executable:
```
$ chmod +x install_opencv_debian_wheezy.sh
```
Execute the script:
```
$ ./install_opencv_debian_wheezy.sh
```
When prompted, enter your password.
The script creates a folder named <script_folder>/opencv that contains downloads and built files that are temporarily used in the installation process. Once the script terminates, <script_folder>/opencv can safely be deleted; although first, you might want to look at the code samples in <script_folder>/opencv/samples.
SciPy is already installed by the script. Run the following commands to install Requests and wxPython:
```
$ sudo apt-get install python-requests
$ sudo apt-get install python-wxgtk2.8
```
Finally, we can put PyInstaller in any convenient location since it is treated as a set of tools rather than a library. Let's download the latest release version from https://github.com/pyinstaller/pyinstaller/wiki and unzip it to ~/PyInstaller or another location of your choice.

If the script ran successfully, we now have a recent version of OpenCV configured to support C++, Python 2.7, and (on compatible systems) several extras such as video codecs, support for Intel TBB multiprocessing, and support for Kinect and Asus depth cameras. Not all the extras are relevant to our current projects but they might be useful in your future work.

Note

For an OpenCV project that uses a depth camera, refer to my book OpenCV Computer Vision with Python.

Now we have everything we need to develop OpenCV applications for Debian Wheezy or a derivative. To also develop the same for Android, we need to set up TADP as described in the section Tegra Android Development Pack, later in this chapter.

Fedora and its derivatives, including RHEL and CentOS

Recent versions of OpenCV, SciPy, Requests, and wxPython are in the standard repository. To install them, open Terminal and run the following commands:

$ sudo yum install opencv-python
$ sudo yum install scipy
$ sudo yum install python-requests
$ sudo yum install wxPython

Download the latest release version of PyInstaller from http://www.pyinstaller.org/ and unzip it to ~/PyInstaller or another location of your choice.

As an alternative to using the packaged build of OpenCV, the following official tutorial provides instructions to install OpenCV's dependencies and compile it from source: http://docs.opencv.org/trunk/doc/py_tutorials/py_setup/py_setup_in_fedora/py_setup_in_fedora.html.

For optional dependencies and additional compilation options, including depth camera support, refer to my Debian-compatible build script as a starting point: http://nummist.com/opencv/install_opencv_debian_wheezy.sh.

Now we have everything we need to develop OpenCV applications for Fedora or a derivative. To also develop the same for Android, we need to set up TADP as described in the section Tegra Android Development Pack, later in the chapter.

openSUSE and its derivatives

Recent versions of OpenCV, SciPy, Requests, and wxPython are in the standard repository. To install them, open Terminal and run the following commands:

$ sudo yum install python-opencv
$ sudo yum install python-scipy
$ sudo yum install python-requests
$ sudo yum install python-wxWidgets

Download the latest release version of PyInstaller from http://www.pyinstaller.org/ and unzip it to ~/PyInstaller or another location of your choice.

As an alternative to using the packaged build of OpenCV, the steps to install OpenCV from source should be similar to the steps on Fedora, though dependencies' package names might differ.

Next, we need to follow the cross-platform steps to set up an Android development environment.

Tegra Android Development Pack

Tegra Android Development Pack (TADP) contains a complete, preconfigured development environment for Android, OpenCV, and some other libraries. TADP builds apps that are optimized for NVIDIA's Tegra processors. Despite being optimized for Tegra, the apps are compatible with other hardware too.

Note

TADP's OpenCV is a third-party build of OpenCV. OpenCV's standard Android build does not offer the optimizations that are present in TADP. Moreover, TADP includes all components of an Android development environment and its setup process is simple. For these reasons, I recommend TADP.

TADP also contains some extras that we do not require for this book. For a complete list of TADP's contents, refer to the official description at https://developer.nvidia.com/tegra-android-development-pack.

To set up TADP, we just need to download and install it from a secure section of NVIDIA's website. Here are the required steps:

Join the NVIDIA Registered Developer Program at https://developer.nvidia.com/registered-developer-programs. (It is free.)
Log in at https://developer.nvidia.com/user/login.
Complete your user profile at https://developer.nvidia.com/user/me/profile/rdp_profile.
Apply to join the Tegra Registered Developer Program at https://developer.nvidia.com/rdp/applications/tegra-registered-developer-program (it is free, too!). Wait for NVIDIA to send you an acceptance e-mail. Normally, you might receive it a few minutes after applying.
Go to https://developer.nvidia.com/tegra-resources and find the download link for TADP's latest version. At the time of writing, the latest version of TADP is 3.0r4. There are installers for Windows (32-bit or 64-bit), Mac, and Ubuntu (32-bit or 64-bit). Download and run the appropriate installer. On Linux, run the installer from a sudoer account because the installer will prompt you to sudo at some stages.
Note
TADP's Ubuntu installer depends on a more recent version of libc (the GNU C Library) than the version available in standard Debian Wheezy. However, on a Debian Wheezy system, we can upgrade libc to a newer version available in Debian Sid (Debian's unstable version) without upgrading the whole system to Debian Sid. Then, our Debian system will be compatible with the TADP Ubuntu installer.
Upgrading libc worked for me but it has the potential to break a lot of things. You do it at your own risk. The steps are as follows:
With root privileges, edit /etc/apt/sources.list. Add this line to the file:
```
deb http://ftp.debian.org/debian sid main
```
Save the file.
Run the following commands in Terminal:
```
$ sudo apt-get update
$ sudo apt-get -t sid install libc6 libc6-dev libc6-dbg
```
Edit /etc/apt/sources.list again to remove the line that we added. This way, we cannot accidentally upgrade anything else to Debian Sid. Save the file.
Now, proceed with running the TADP installer.
When the installer presents the Installation Directory step, we can enter any destination, which we will refer to as <tadp>. By default, <tadp> is C:\NVPACK (in Windows) or ~/NVPACK ( in Mac and Linux).
When the installer presents the Installation Options step, we can select any option: Complete, Express, or Custom. Compared to an Express installation, a Complete or Custom installation might include additional versions of Android SDK and binary images of Tegra Android OS, which is NVIDIA's customization of Android. If in doubt, choose Express.
When the installer presents the Proxy Configuration step, we can leave all fields blank unless we are using a proxy server.
After finishing all of the installer's configuration steps, wait for TADP's content to be downloaded and installed.

That's all! Before proceeding, let's just take a note of the locations where TADP has installed certain components. For TADP 3.0r4 (the latest version at the time of writing), the locations are as follows:

Android SDK is located at <tadp>/android-sdk-windows (in Windows), <tadp>/android-sdk-macosx (in Mac), or <tadp>/android-sdk-linux (in Linux). We will refer to this location as <android_sdk>.
Android NDK is located at <tadp>/android-ndk-r10c. We will refer to this location as <android_ndk>.
OpenCV is located at <tadp>/OpenCV-2.4.8.2-Tegra-sdk. We will refer to this location as <opencv>.
Eclipse is located at <tadp>/eclipse. We will refer to this location as <eclipse>.

The TADP installer automatically edits the system's PATH to include <android_sdk>/platform-tools and <android_sdk>/ tools. Also, it creates an environment variable called NDKROOT, whose value is <android_ndk>.

Building OpenCV Android sample projects with Eclipse

By building and running a few sample applications, we can test our OpenCV installation. At the same time, we can practice using Eclipse.

Let's start by launching Eclipse. The Eclipse launcher should be located at <eclipse>/eclipse.exe (in Windows), <eclipse>/Eclipse.app (in Mac), or <eclipse>/eclipse (in Linux). Run it.

As shown in the following screenshot, we should see a window called Workspace Launcher, which asks us to select a workspace:

Building OpenCV Android sample projects with Eclipse

A workspace is the root directory for a set of related Eclipse projects. Enter <tadp>/nvsample_workspace, which is a workspace where the OpenCV library, samples, and tutorials are already set up as projects.

Tip

We can return to Workspace Launcher anytime via the menu: File | Switch Workspace | Other….

If the Welcome to Eclipse screen appears, click on the Workbench button.

Now, we should see a window with several panels, including Package Explorer. The OpenCV library, samples, and tutorials should be listed in Package Explorer. We might need to fix some configuration issues in these projects. Our development environment can have different paths and different versions of the Android SDK, than the ones in the sample's default configuration. Any resulting errors are reported in the Problems tab as shown in the following screenshot:

Tip

We should start by resolving any errors in the OpenCV Library project as the samples and tutorials depend on the library.

The following are some of the common configuration problems and their symptoms and solutions:

The target Android version might not be properly specified. The symptoms of these are that the imports from the java and android packages fail, and there are error messages such as The project was not built since its build path is incomplete. The solution is to right-click on the project in Package Explorer, select Properties from the context menu, select the Android section, and checkmark one of the available Android versions. These steps should be repeated for all projects. At compile time, OpenCV and its samples must target Android 3.0 (API level 11) or greater, though at runtime they also support Android 2.2 (API level 8) or greater.
If we are working on Mac or Linux, the C++ samples might be misconfigured to use the Windows build executable. The symptom of this problem is an error message such as Program "{ndk}/ndk-build.cmd" not found in PATH. The solution is to right-click on the project in Package Explorer, select Properties from the context menu, select the C/C++ Build section, and edit the Build command: field to remove the .cmd extension. These steps should be repeated for all the native (C++) projects, which include OpenCV Sample - face-detection and OpenCV Tutorial 2 - Mixed Processing as shown in the following screenshot:
Tip
We only need to troubleshoot the projects that have names starting with OpenCV. For this book's purposes, the other TADP samples are not relevant.

Once the OpenCV projects no longer show any errors, we can prepare to test them on an Android device. Remember that the device must have Android 2.2 (Froyo) or a later version, and a camera. For Eclipse to communicate with the device, we must enable the device's USB debugging option with the help of the following steps:

Open the Settings app.
On Android 4.2 or a later version, go to the About phone or About tablet section and tap Build number seven times. This step enables the Developer options section.
Go to the Developer options section (on Android 4.0 or a later version) or the Applications | Development section (on Android 3.2 or an earlier version). Enable the USB debugging option. Now, open the Play Store app and find and install the OpenCV Manager app. OpenCV Manager takes care of checking for any OpenCV library updates when we run any OpenCV applications.
Tip
If you do not have the Play Store app on your device, then you need to install OpenCV Manager and certain OpenCV libraries via USB as per the instructions at http://docs.opencv.org/android/service/doc/UseCases.html.

Now, we must prepare our main computer for communication with the Android device. The required steps vary depending on our operating system:

On Windows, we need to install the proper USB drivers for the Android device. Different vendors and devices have different drivers. The official Android documentation provides links to the various vendors' driver download sites at http://developer.android.com/tools/extras/oem-usb.html#Drivers.
On Linux, before connecting an Android device via USB, we must specify the device's vendor in a permissions file. Each vendor has a unique ID number, as listed in the official Android documentation at http://developer.android.com/tools/device.html#VendorIds. We will refer to this ID number as <vendor_id>. To create the permissions file, open a command prompt application (such as Terminal) and run the following commands:
```
$ cd /etc/udev/rules.d/
$ sudo touch 51-android.rules
$ sudo chmod a+r 51-android-rules
```
Note that the permissions file needs to have root ownership, so we will use sudo while creating or modifying it. Now, open the file in an editor such as gedit:
```
$ sudo gedit 51-android-rules
```
For each vendor, append a new line to the file. Each of these lines should have the following format:
```
SUBSYSTEM=="usb", ATTR{idVendor}=="<vendor_id>", MODE="0666",
GROUP="plugdev"
```
Save the permissions file and quit the editor.
On Mac, no special drivers or permissions are required.

Plug the Android device into your computer's USB port. In Eclipse, select one of the OpenCV sample projects in Package Explorer. Then, from the menu system, navigate to Run | Run As... | Android Application:

An Android Device Chooser window should appear. Your Android device should be listed under Choose a running Android device. (If the device is not listed, try unplugging it and plugging it back in. If that does not work, also try disabling and re-enabling the device's USB debugging option, as described earlier.)

Select the device and click on OK.

If the Auto Monitor Logcat window appears, select the Yes radio button and the verbose drop-down option and click on OK. This option ensures that all the log output from the application will be visible in Eclipse.

On the Android device, you might get a message, OpenCV library package was not found! Try to install it?. Make sure that the device is connected to the Internet and then click on the Yes button on your device. The Play Store will open to show an OpenCV package. Install the package and then press the physical back button to return to the sample application, which should be ready for use.

For OpenCV 2.4.8.2, the samples and tutorials have the following functionality:

Sample – 15 puzzle: This splits up a camera feed to make a sliding block puzzle. The user can swipe blocks to move them.
Sample – camera-calibration: This estimates the projection and distortion characteristics of the camera. The user prints a test pattern located at https://raw.githubusercontent.com/Itseez/opencv/2.4/doc/acircles_pattern.png and then takes pictures of it from various angles by tapping the screen. After taking several pictures, the user must press the … menu and the Calibrate button to run the calibration algorithm.
Sample – color-blob-detection: This detects color regions in a camera feed. The user can touch anywhere to see the outline of a color region.
Sample – face-detection: This draws green rectangles around faces in a camera feed.
Sample – image-manipulations: This applies filters to a camera feed. The user can press the Android menu button to select from a list of filters.
Sample – native-activity: This displays a camera feed using native (C++) code.
Tutorial 1 – Camera Preview: This displays a camera feed. The user can press the … menu to select a different camera feed implementation (Java or native C++).
Tutorial 2 – Mixed Processing: This applies filters to a camera feed using native (C++) code. The user can press the … menu to select from a list of filters. One of the filters draws red circles around interest points or features in a camera feed. Generally speaking, interest points or features lie along the high-contrast edges in an image. They are potentially useful in image recognition and tracking applications.
Tutorial 3 – Camera Control: This applies filters to a camera feed, which has a customizable resolution. The user can press the … menu to select from a list of filters and a list of resolutions.

Try these applications on your Android device! While an application is running, its log output should appear in the LogCat tab in Eclipse as shown in the following screenshot:

Feel free to browse the project's source code via Package Explorer to see how it was made. Alternatively, you might want to return to the official samples and tutorials later, after we have built our own projects over the course of this book.

Unity

Unity is a 3D game engine that supports development on Windows or Mac and deployment to many platforms, including Windows, Mac, Linux, iOS, Android, a web browser plugin, and several game consoles. For one of our projects, we will use an OpenCV plugin for Unity.

Unity comes in two editions, Standard and Pro, which both support the plugin that we want to use. The Standard edition is free. The Pro edition has a free 30-day trial; otherwise, its price starts from $75 a month or a one-time payment of $1,500. If you do not already have a Unity Pro license, wait until you are ready to start working on our Unity project (which comes late in the book) in case you want to try out some Pro-only functionality at the same time. Once you are ready, download and install the trial from http://unity3d.com/unity/download.

Even before installing Unity, we can get inspiration from the playable demos at https://unity3d.com/gallery/demos/live-demos. Most of these demos are web-based. You will be prompted to download and install the free Unity Web Player when you navigate to one of the web-based games.

After installing Unity, we can learn from other demo projects that include complete source code and art assets. They can be downloaded and opened from https://unity3d.com/gallery/demos/demo-projects. Also check out the tutorials, videos, and documentation at https://unity3d.com/learn.

As you can see, there are a lot of official resources for Unity beginners, so I will let you explore these on your own for now.