Detecting points using the Harris corner detector

Corner detection is a technique used to detect points of interest in an image. These interest points are also called feature points, or simply features, in computer vision terminology. A corner is basically an intersection of two edges. An interest point is basically something that can be uniquely detected in an image. A corner is a particular case of an interest point. These interest points help us characterize an image. These points are used extensively in applications such as object tracking, image classification, and visual search. Since we know that the corners are interesting, let's see how can detect them.

In computer vision, there is a popular corner detection technique called the Harris corner detector. We basically construct a 2 x 2 matrix based on partial derivatives of the grayscale image, and then analyze the eigenvalues. What does that even mean? Well, let's dissect it so that we can understand it better. Let's consider a small patch in the image. Our goal is to identify whether this patch has a corner in it. So, we consider all the neighboring patches and compute the intensity difference between our patch and all those neighboring patches. If the difference is high in all directions, then we know that our patch has a corner in it. This is an oversimplification of the actual algorithm, but it covers the gist. If you want to understand the underlying mathematical details, you can check out the original paper by Harris and Stephens at http://www.bmva.org/bmvc/1988/avc-88-023.pdf. A corner is a point with strong intensity differences along two directions.

If we run the Harris corner detector, it will look like this:

As we can see, the green circles on the TV remote are the detected corners. This will change based on the parameters you choose for the detector. If you modify the parameters, you can see that more points might get detected. If you make it strict, you might not be able to detect soft corners. Let's look at the code to detect Harris corners:

int main(int argc, char* argv[])
{
// Variable declaration and initialization

// Iterate until the user presses the Esc key
while(true)
{
    // Capture the current frame
    cap >> frame;

    // Resize the frame
    resize(frame, frame, Size(), scalingFactor, scalingFactor, INTER_AREA);

    dst = Mat::zeros(frame.size(), CV_32FC1);

    // Convert to grayscale
    cvtColor(frame, frameGray, COLOR_BGR2GRAY );

    // Detecting corners
    cornerHarris(frameGray, dst, blockSize, apertureSize, k, BORDER_DEFAULT);

    // Normalizing
    normalize(dst, dst_norm, 0, 255, NORM_MINMAX, CV_32FC1, Mat());
    convertScaleAbs(dst_norm, dst_norm_scaled);

We converted the image to grayscale and detected corners using our parameters. You can find the full code in the .cpp files. These parameters play an important role in the number of points that will be detected. You can check out the OpenCV documentation of cornerHarris() at https://docs.opencv.org/master/dd/d1a/group__imgproc__feature.html#gac1fc3598018010880e370e2f709b4345.

We now have all the information we need. Let's go ahead and draw circles around our corners to display the results:

        // Drawing a circle around each corner
        for(int j = 0; j < dst_norm.rows ; j++)
        {
            for(int i = 0; i < dst_norm.cols; i++)
            {
                if((int)dst_norm.at<float>(j,i) > thresh)
                {
                    circle(frame, Point(i, j), 8, Scalar(0,255,0), 2, 8, 0);
                }
            }
        }

        // Showing the result
        imshow(windowName, frame);

        // Get the keyboard input and check if it's 'Esc'
        // 27 -> ASCII value of 'Esc' key
        ch = waitKey(10);
        if (ch == 27) {
            break;
        }
    }

    // Release the video capture object
    cap.release();

    // Close all windows
    destroyAllWindows();

    return 1;
}

As we can see, this code takes an input argument: blockSize. Depending on the size you choose, the performance will vary. Start with a value of four and play around with it to see what happens.