Learning OpenCV 3 · Computer Vision in C++ With the OpenCV Library by Kaehler, Adrian -- Read -- Imperial Library of Trantor

Index

Preface

Purpose of This Book

Who This Book Is For What This Book Is Not

About the Programs in This Book Prerequisites How This Book Is Best Used Conventions Used in This Book Using Code Examples O’Reilly Safari We’d Like to Hear from You Acknowledgments

Thanks for Help on OpenCV Thanks for Help on This Book Adrian Adds... Gary Adds...

1. Overview

What Is OpenCV? Who Uses OpenCV? What Is Computer Vision? The Origin of OpenCV

OpenCV Block Diagram Speeding Up OpenCV with IPP Who Owns OpenCV?

Downloading and Installing OpenCV

Installation

Windows Linux Mac OS X

Getting the Latest OpenCV via Git More OpenCV Documentation

Supplied Documentation Online Documentation and the Wiki

OpenCV Contribution Repository

Downloading and Building Contributed Modules

Portability Summary Exercises

2. Introduction to OpenCV

Include Files

Resources

First Program—Display a Picture Second Program—Video Moving Around A Simple Transformation A Not-So-Simple Transformation Input from a Camera Writing to an AVI File Summary Exercises

3. Getting to Know OpenCV Data Types

The Basics OpenCV Data Types

Overview of the Basic Types Basic Types: Getting Down to Details

The point classes The cv::Scalar class The size classes The cv::Rect class The cv::RotatedRect class The fixed matrix classes The fixed vector classes The complex number classes

Helper Objects

The cv::TermCriteria class The cv::Range class The cv::Ptr template and Garbage Collection 101 The cv::Exception class and exception handling The cv::DataType<> template The cv::InputArray and cv::OutputArray classes

Utility Functions

cv::alignPtr() cv::alignSize() cv::allocate() cv::deallocate() cv::fastAtan2() cvCeil() cv::cubeRoot() cv::CV_Assert() and CV_DbgAssert() cv::CV_Error() and CV_Error_() cv::error() cv::fastFree() cv::fastMalloc() cvFloor() cv::format() cv::getCPUTickCount() cv::getNumThreads() cv::getOptimalDFTSize() cv::getThreadNum() cv::getTickCount() cv::getTickFrequency() cvIsInf() cvIsNaN() cvRound() cv::setNumThreads() cv::setUseOptimized() cv::useOptimized()

The Template Structures

Summary Exercises

4. Images and Large Array Types

Dynamic and Variable Storage

The cv::Mat Class: N-Dimensional Dense Arrays Creating an Array Accessing Array Elements Individually The N-ary Array Iterator: NAryMatIterator Accessing Array Elements by Block Matrix Expressions: Algebra and cv::Mat Saturation Casting More Things an Array Can Do The cv::SparseMat Class: Sparse Arrays Accessing Sparse Array Elements Functions Unique to Sparse Arrays Template Structures for Large Array Types

Summary Exercises

5. Array Operations

More Things You Can Do with Arrays

cv::abs() cv::absdiff() cv::add() cv::addWeighted() cv::bitwise_and() cv::bitwise_not() cv::bitwise_or() cv::bitwise_xor() cv::calcCovarMatrix() cv::cartToPolar() cv::checkRange() cv::compare() cv::completeSymm() cv::convertScaleAbs() cv::countNonZero() cv::cvarrToMat() cv::dct() cv::dft() cv::cvtColor() cv::determinant() cv::divide() cv::eigen() cv::exp() cv::extractImageCOI() cv::flip() cv::gemm() cv::getConvertElem() and cv::getConvertScaleElem() cv::idct() cv::idft() cv::inRange() cv::insertImageCOI() cv::invert() cv::log() cv::LUT() cv::magnitude() cv::Mahalanobis() cv::max() cv::mean() cv::meanStdDev() cv::merge() cv::min() cv::minMaxIdx() cv::minMaxLoc() cv::mixChannels() cv::mulSpectrums() cv::multiply() cv::mulTransposed() cv::norm() cv::normalize() cv::perspectiveTransform() cv::phase() cv::polarToCart() cv::pow() cv::randu() cv::randn() cv::randShuffle() cv::reduce() cv::repeat() cv::scaleAdd() cv::setIdentity() cv::solve() cv::solveCubic() cv::solvePoly() cv::sort() cv::sortIdx() cv::split() cv::sqrt() cv::subtract() cv::sum() cv::trace() cv::transform() cv::transpose()

Summary Exercises

6. Drawing and Annotating

Drawing Things

Line Art and Filled Polygons

cv::circle() cv::clipLine() cv::ellipse() cv::ellipse2Poly() cv::fillConvexPoly() cv::fillPoly() cv::line() cv::rectangle() cv::polyLines() cv::LineIterator

Fonts and Text

cv::putText() cv::getTextSize()

Summary Exercises

7. Functors in OpenCV

Objects That “Do Stuff”

Principal Component Analysis (cv::PCA)

cv::PCA::PCA() cv::PCA::operator()() cv::PCA::project() cv::PCA::backProject()

Singular Value Decomposition (cv::SVD)

cv::SVD() cv::SVD::operator()() cv::SVD::compute() cv::SVD::solveZ() cv::SVD::backSubst()

Random Number Generator (cv::RNG)

cv::theRNG() cv::RNG() cv::RNG::operator T(), where T is your favorite type cv::RNG::operator() cv::RNG::uniform() cv::RNG::gaussian() cv::RNG::fill()

Summary Exercises

8. Image, Video, and Data Files

HighGUI: Portable Graphics Toolkit Working with Image Files

Loading and Saving Images

Reading files with cv::imread() Writing files with cv::imwrite()

A Note About Codecs Compression and Decompression

Compressing files with cv::imencode() Uncompressing files with cv::imdecode()

Working with Video

Reading Video with the cv::VideoCapture Object

Reading frames with cv::VideoCapture::read() Reading frames with cv::VideoCapture::operator>>() Reading frames with cv::VideoCapture::grab() and cv::VideoCapture::retrieve() Camera properties: cv::VideoCapture::get() and cv::VideoCapture::set()

Writing Video with the cv::VideoWriter Object

Writing frames with cv::VideoWriter::write() Writing frames with cv::VideoWriter::operator<<()

Data Persistence

Writing to a cv::FileStorage Reading from a cv::FileStorage cv::FileNode

Summary Exercises

9. Cross-Platform and Native Windows

Working with Windows

HighGUI Native Graphical User Interface

Creating a window with cv::namedWindow() Drawing an image with cv::imshow() Updating a window and cv::waitKey() An example displaying an image Mouse events Sliders, trackbars, and switches Surviving without buttons

Working with the Qt Backend

Getting started The actions menu The text overlay Writing your own text into the status bar The properties window Trackbars revisited Creating buttons with cv::createButton() Text and fonts Setting and getting window properties Saving and recovering window state Interacting with OpenGL

Integrating OpenCV with Full GUI Toolkits

An example of OpenCV and Qt An example of OpenCV and wxWidgets An example of OpenCV and the Windows Template Library

Summary Exercises

10. Filters and Convolution

Overview Before We Begin

Filters, Kernels, and Convolution

Anchor points

Border Extrapolation and Boundary Conditions

Making borders yourself Manual extrapolation

Threshold Operations

Otsu’s Algorithm Adaptive Threshold

Smoothing

Simple Blur and the Box Filter Median Filter Gaussian Filter Bilateral Filter

Derivatives and Gradients

The Sobel Derivative Scharr Filter The Laplacian

Image Morphology

Dilation and Erosion The General Morphology Function Opening and Closing Morphological Gradient Top Hat and Black Hat Making Your Own Kernel

Convolution with an Arbitrary Linear Filter

Applying a General Filter with cv::filter2D() Applying a General Separable Filter with cv::sepFilter2D Kernel Builders

cv::getDerivKernel() cv::getGaussianKernel()

Summary Exercises

11. General Image Transforms

Overview Stretch, Shrink, Warp, and Rotate

Uniform Resize

cv::resize()

Image Pyramids

cv::pyrDown() cv::buildPyramid() cv::pyrUp() The Laplacian pyramid

Nonuniform Mappings Affine Transformation

cv::warpAffine(): Dense affine transformations cv::getAffineTransform(): Computing an affine map matrix cv::transform(): Sparse affine transformations cv::invertAffineTransform(): Inverting an affine transformation

Perspective Transformation

cv::warpPerspective(): Dense perspective transform cv::getPerspectiveTransform(): Computing the perspective map matrix cv::perspectiveTransform(): Sparse perspective transformations

General Remappings

Polar Mappings

cv::cartToPolar(): Converting from Cartesian to polar coordinates cv::polarToCart(): Converting from polar to Cartesian coordinates

LogPolar

cv::logPolar()

Arbitrary Mappings

cv::remap(): General image remapping

Image Repair

Inpainting Denoising

Basic FNLMD with cv::fastNlMeansDenoising() FNLMD on color images with cv::fastNlMeansDenoisingColor() FNLMD on video with cv::fastNlMeansDenoisingMulti() and cv::fastNlMeansDenoisingColorMulti()

Histogram Equalization

cv::equalizeHist(): Contrast equalization

Summary Exercises

12. Image Analysis

Overview Discrete Fourier Transform

cv::dft(): The Discrete Fourier Transform cv::idft(): The Inverse Discrete Fourier Transform cv::mulSpectrums(): Spectrum Multiplication Convolution Using Discrete Fourier Transforms cv::dct(): The Discrete Cosine Transform cv::idct(): The Inverse Discrete Cosine Transform

Integral Images

cv::integral() for Standard Summation Integral cv::integral() for Squared Summation Integral cv::integral() for Tilted Summation Integral

The Canny Edge Detector

cv::Canny()

Hough Transforms

Hough Line Transform

cv::HoughLines(): The standard and multiscale Hough transforms cv::HoughLinesP(): The progressive probabilistic Hough transform

Hough Circle Transform

cv::HoughCircles(): the Hough circle transform

Distance Transformation

cv::distanceTransform() for Unlabeled Distance Transform cv::distanceTransform() for Labeled Distance Transform

Segmentation

Flood Fill Watershed Algorithm Grabcuts Mean-Shift Segmentation

Summary Exercises

13. Histograms and Templates

Histogram Representation in OpenCV

cv::calcHist(): Creating a Histogram from Data

Basic Manipulations with Histograms

Histogram Normalization Histogram Threshold Finding the Most Populated Bin Comparing Two Histograms

Correlation method (cv::COMP_CORREL) Chi-square method (cv::COMP_CHISQR_ALT) Intersection method (cv::COMP_INTERSECT) Bhattacharyya distance method (cv::COMP_BHATTACHARYYA)

Histogram Usage Examples

Some More Sophisticated Histograms Methods

Earth Mover’s Distance Back Projection

Basic back projection: cv::calcBackProject()

Template Matching

Square Difference Matching Method (cv::TM_SQDIFF) Normalized Square Difference Matching Method (cv::TM_SQDIFF_NORMED) Correlation Matching Methods (cv::TM_CCORR) Normalized Cross-Correlation Matching Method (cv::TM_CCORR_NORMED) Correlation Coefficient Matching Methods (cv::TM_CCOEFF) Normalized Correlation Coefficient Matching Method (cv::TM_CCOEFF_NORMED)

Summary Exercises

14. Contours

Contour Finding

Contour Hierarchies

Finding contours with cv::findContours()

Drawing Contours A Contour Example Another Contour Example Fast Connected Component Analysis

More to Do with Contours

Polygon Approximations

Polygon approximation with cv::approxPolyDP() The Douglas-Peucker algorithm explained

Geometry and Summary Characteristics

Length using cv::arcLength() Upright bounding box with cv::boundingRect() A minimum area rectangle with cv::minAreaRect() A minimal enclosing circle using cv::minEnclosingCircle() Fitting an ellipse with cv::fitEllipse() Finding the best line fit to your contour with cv::fitLine() Finding the convex hull of a contour using cv::convexHull()

Geometrical Tests

Testing if a point is inside a polygon with cv::pointPolygonTest() Testing whether a contour is convex with cv::isContourConvex()

Matching Contours and Images

Moments

Computing moments with cv::moments()

More About Moments

Central moments are invariant under translation Normalized central moments are also invariant under scaling Hu invariant moments are invariant under rotation Computing Hu invariant moments with cv::HuMoments()

Matching and Hu Moments Using Shape Context to Compare Shapes

Structure of the shape module The shape context distance extractor Hausdorff distance extractor

Summary Exercises

15. Background Subtraction

Overview of Background Subtraction Weaknesses of Background Subtraction Scene Modeling

A Slice of Pixels Frame Differencing

Averaging Background Method

Accumulating Means, Variances, and Covariances

Computing the mean with cv::Mat::operator+=() Computing the mean with cv::accumulate() Variation: Computing the mean with cv::accumulateWeighted() Finding the variance with the help of cv::accumulateSquare() Finding the covariance with cv::accumulateWeighted() A brief note on model testing and cv::Mahalanobis()

A More Advanced Background Subtraction Method

Structures Learning the Background Learning with Moving Foreground Objects Background Differencing: Finding Foreground Objects Using the Codebook Background Model A Few More Thoughts on Codebook Models

Connected Components for Foreground Cleanup

A Quick Test

Comparing Two Background Methods OpenCV Background Subtraction Encapsulation

The cv::BackgroundSubtractor Base Class KaewTraKuPong and Bowden Method

cv::BackgroundSubtractorMOG

Zivkovic Method

cv::BackgroundSubtractorMOG2

Summary Exercises

16. Keypoints and Descriptors

Keypoints and the Basics of Tracking

Corner Finding

Finding corners using cv::goodFeaturesToTrack() Subpixel corners

Introduction to Optical Flow Lucas-Kanade Method for Sparse Optical Flow

How Lucas-Kanade works Pyramid Lucas-Kanade code: cv::calcOpticalFlowPyrLK() A worked example

Generalized Keypoints and Descriptors

Optical Flow, Tracking, and Recognition How OpenCV Handles Keypoints and Descriptors, the General Case

The cv::KeyPoint object The (abstract) class that finds keypoints and/or computes descriptors for them: cv::Feature2D The cv::DMatch object The (abstract) keypoint matching class: cv::DescriptorMatcher

Core Keypoint Detection Methods

The Harris-Shi-Tomasi feature detector and cv::GoodFeaturesToTrackDetector

Keypoint finder Additional functions

A brief look under the hood The simple blob detector and cv::SimpleBlobDetector

Keypoint finder

The FAST feature detector and cv::FastFeatureDetector

Keypoint finder

The SIFT feature detector and cv::xfeatures2d::SIFT

Keypoint finder and feature extractor

The SURF feature detector and cv::xfeatures2d::SURF

Keypoint finder and feature extractor Additional functions provided by cv::xfeatures2d::SURF

The Star/CenSurE feature detector and cv::xfeatures2d::StarDetector

Keypoint finder

The BRIEF descriptor extractor and cv::BriefDescriptorExtractor

Feature extractor

The BRISK algorithm

Keypoint finder and feature extractor Additional functions provided by cv::BRISK

The ORB feature detector and cv::ORB

Keypoint finder and feature extractor Additional functions provided by cv::ORB

The FREAK descriptor extractor and cv::xfeatures2d::FREAK

Feature extractor

Dense feature grids and the cv::DenseFeatureDetector class

Keypoint finder

Keypoint Filtering

The cv::KeyPointsFilter class

Matching Methods

Brute force matching with cv::BFMatcher Fast approximate nearest neighbors and cv::FlannBasedMatcher

Linear indexing with cv::flann::LinearIndexParams KD-tree indexing with cv::flann::KDTreeIndexParams Hierarchical k-means tree indexing with cv::flann::KMeansIndexParams Combining KD-trees and k-means with cv::flann::CompositeIndexParams Locality-sensitive hash (LSH) indexing with cv::flann::LshIndexParams Automatic index selection with cv::flann::AutotunedIndexParams FLANN search parameters and cv::flann::SearchParams

Displaying Results

Displaying keypoints with cv::drawKeypoints Displaying keypoint matches with cv::drawMatches

Summary Exercises

17. Tracking

Concepts in Tracking Dense Optical Flow

The Farnebäck Polynomial Expansion Algorithm

Computing dense optical flow with cv::calcOpticalFlowFarneback

The Dual TV-L1 Algorithm

Computing dense optical flow with cv::createOptFlow_DualTVL1

The Simple Flow Algorithm

Computing Simple Flow with cv::optflow::calcOpticalFlowSF()

Mean-Shift and Camshift Tracking

Mean-Shift Camshift

Motion Templates Estimators

The Kalman Filter

What goes in and what comes out Assumptions required by the Kalman filter Information fusion Systems with dynamics Kalman equations Tracking in OpenCV with cv::KalmanFilter Kalman filter example code

A Brief Note on the Extended Kalman Filter

Summary Exercises

18. Camera Models and Calibration

Camera Model

The Basics of Projective Geometry Rodrigues Transform Lens Distortions

Calibration

Rotation Matrix and Translation Vector Calibration Boards

Finding chessboard corners with cv::findChessboardCorners() Subpixel corners on chessboards and cv::cornerSubPix() Drawing chessboard corners with cv::drawChessboardCorners() Circle-grids and cv::findCirclesGrid()

Homography Camera Calibration

How many chess corners for how many parameters? What’s under the hood? Calibration function Computing extrinsics only with cv::solvePnP() Computing extrinsics only with cv::solvePnPRansac()

Undistortion

Undistortion Maps Converting Undistortion Maps Between Representations with cv::convertMaps() Computing Undistortion Maps with cv::initUndistortRectifyMap() Undistorting an Image with cv::remap() Undistortion with cv::undistort() Sparse Undistortion with cv::undistortPoints()

Putting Calibration All Together Summary Exercises

19. Projection and Three-Dimensional Vision

Projections Affine and Perspective Transformations

Bird’s-Eye-View Transform Example

Three-Dimensional Pose Estimation

Pose Estimation from a Single Camera

Computing the pose of a known object with cv::solvePnP()

Stereo Imaging

Triangulation Epipolar Geometry The Essential and Fundamental Matrices

Essential matrix math Fundamental matrix math How OpenCV handles all of this

Computing Epipolar Lines Stereo Calibration Stereo Rectification

Uncalibrated stereo rectification: Hartley’s algorithm Calibrated stereo rectification: Bouguet’s algorithm Rectification map

Stereo Correspondence

The stereo matching classes: cv::StereoBM and cv::StereoSGBM Block matching Computing stereo depths with cv::StereoBM Semi-global block matching Computing stereo depths with cv::StereoSGBM

Stereo Calibration, Rectification, and Correspondence Code Example Depth Maps from Three-Dimensional Reprojection

Structure from Motion Fitting Lines in Two and Three Dimensions Summary Exercises

20. The Basics of Machine Learning in OpenCV

What Is Machine Learning?

Training and Test Sets Supervised and Unsupervised Learning Generative and Discriminative Models OpenCV ML Algorithms Using Machine Learning in Vision Variable Importance Diagnosing Machine Learning Problems

Cross-validation, bootstrapping, ROC curves, and confusion matrices

Cost of misclassification Mismatched feature variance

Legacy Routines in the ML Library

K-Means

Problems and solutions K-means code

Mahalanobis Distance

Using the Mahalanobis distance to condition input data Using the Mahalanobis distance for classification

Summary Exercises

21. StatModel: The Standard Model for Learning in OpenCV

Common Routines in the ML Library

Training and the cv::ml::TrainData Structure

Constructing cv::ml::TrainData Constructing cv::ml::TrainData from stored data Secret sauce and cv::ml::TrainDataImpl Splitting training data Accessing cv::ml::TrainData

Prediction

Machine Learning Algorithms Using cv::StatModel

Naïve/Normal Bayes Classifier

The naïve/normal Bayes classifier and cv::ml::NormalBayesClassifier

Binary Decision Trees

Regression impurity Classification impurity OpenCV implementation Decision tree usage Decision tree results

Boosting

AdaBoost Boosting code

Random Trees

Random trees code Using random trees

Expectation Maximization

Expectation maximization with cv::EM()

K-Nearest Neighbors

Using K-nearest neighbors with cv::ml::KNearest()

Multilayer Perceptron

Back propagation The Rprop algorithm Using artificial neural networks and back propagation with cv::ml::ANN_MLP Parameters for training

Support Vector Machine

About kernels Handling outliers Multiclass extension of SVM One-class SVM Support vector regression Using support vector machines and cv::ml::SVM() Additional members of cv::ml::SVM

Summary Exercises

22. Object Detection

Tree-Based Object Detection Techniques

Cascade Classifiers

Haar-like features Local binary pattern features Training and pretrained detectors

Supervised Learning and Boosting Theory

Boosting in the Haar cascade

Rejection cascades

Viola-Jones classifier summary The cv::CascadeClassifer object Searching an image with detectMultiScale() Face detection example

Learning New Objects

Detailed arguments to createsamples Detailed arguments to traincascade

Object Detection Using Support Vector Machines

Latent SVM for Object Detection

Object detection with cv::dpm::DPMDetector Other methods of cv::dpm::DPMDetector Where to get models for cv::dpm::DPMDetector

The Bag of Words Algorithm and Semantic Categorization

Training with cv::BOWTrainer K-means and cv::BOWKMeansTrainer Categorization with cv::BOWImgDescriptorExtractor Putting it together using a support vector machine

Summary Exercises

23. Future of OpenCV

Past and Present

OpenCV 3.x

How Well Did Our Predictions Go Last Time? Future Functions

Current GSoC Work Community Contributions OpenCV.org

Some AI Speculation Afterword

A. Planar Subdivisions

Delaunay Triangulation, Voronoi Tesselation

Creating a Delaunay or Voronoi Subdivision Navigating Delaunay Subdivisions

Points from edges Locating a point within a subdivision Orbiting around a vertex Rotating an edge Identifying the bounding triangle Identifying the bounding triangle or edges on the convex hull and walking the hull

Usage Examples

Exercises

B. opencv_contrib

An Overview of the opencv_contrib Modules

Contents of opencv_contrib

C. Calibration Patterns

Calibration Patterns Used by OpenCV

Bibliography Index

← Prev
Back
Next →

← Prev
Back
Next →