Back in Chapter 2, Handling Files, Cameras, and GUIs, we discussed the concept that a computer can have multiple video capture devices and each device can have multiple channels. Suppose a given device is a stereo camera. Each channel might correspond to a different lens and sensor. Also, each channel might correspond to a different kind of data, such as a normal color image versus a depth map. When working with OpenCV's VideoCapture class or our wrapper CaptureManager, we can choose a device on initialization and we can read one or more channels from each frame of that device. Each device and channel is identified by an integer. Unfortunately, the numbering of devices and channels is unintuitive. The C++ version of OpenCV defines some constants for the identifiers of certain devices and channels. However, these constants are not defined in the Python version. To remedy this situation, let's add the following definitions in depth.py:
# Devices. CV_CAP_OPENNI = 900 # OpenNI (for Microsoft Kinect) CV_CAP_OPENNI_ASUS = 910 # OpenNI (for Asus Xtion) # Channels of an OpenNI-compatible depth generator. CV_CAP_OPENNI_DEPTH_MAP = 0 # Depth values in mm (CV_16UC1) CV_CAP_OPENNI_POINT_CLOUD_MAP = 1 # XYZ in meters (CV_32FC3) CV_CAP_OPENNI_DISPARITY_MAP = 2 # Disparity in pixels (CV_8UC1) CV_CAP_OPENNI_DISPARITY_MAP_32F = 3 # Disparity in pixels (CV_32FC1) CV_CAP_OPENNI_VALID_DEPTH_MASK = 4 # CV_8UC1 # Channels of an OpenNI-compatible RGB image generator. CV_CAP_OPENNI_BGR_IMAGE = 5 CV_CAP_OPENNI_GRAY_IMAGE = 6
The depth-related channels require some explanation, as given in the following list:
- A depth map is a grayscale image in which each pixel value is the estimated distance from the camera to a surface. Specifically, an image from the
CV_CAP_OPENNI_DEPTH_MAPchannel gives the distance as a floating-point number of millimeters. - A point cloud map is a color image in which each color corresponds to a spatial dimension (x, y, or z). Specifically, the
CV_CAP_OPENNI_POINT_CLOUD_MAPchannel yields a BGR image where B is x (blue is right), G is y (green is up), and R is z (red is deep), from the camera's perspective. The values are in meters. - A disparity map is a grayscale image in which each pixel value is the stereo disparity of a surface. To conceptualize stereo disparity, let's suppose we overlay two images of a scene, shot from different viewpoints. The result would be like seeing double images. For points on any pair of twin objects in the scene, we can measure the distance in pixels. This measurement is the stereo disparity. Nearby objects exhibit greater stereo disparity than far-off objects. Thus, nearby objects appear brighter in a disparity map.
- A valid depth mask shows whether the depth information at a given pixel is believed to be valid (shown by a non-zero value) or invalid (shown by a value of zero). For example, if the depth camera depends on an infrared illuminator (an infrared flash), then depth information is invalid in regions that are occluded (shadowed) from this light.
The following screenshot shows a point-cloud map of a man sitting behind a sculpture of a cat:
The following screenshot has a disparity map of a man sitting behind a sculpture of a cat:
A valid depth mask of a man sitting behind a sculpture of a cat is shown in the following screenshot:
