Feature extraction

The next thing we need to do is extract the features for each object. To understand the feature vector concept, we are going to extract very simple features in our example, as this is enough to get good results. In other solutions, we can get more complex features such as texture descriptors, contour descriptors, and so on. In our example, we only have nuts, rings, and screws in different positions and orientations in the image. The same object can be in any position of image and orientation, for example, the screw or the nut. We can see different orientations in the following image:

We are going to explore some features or characteristics that could improve the accuracy of our machine learning algorithm. These possible characteristics of our different objects (nuts, screws, and rings) are as follows:

These characteristics can describe our objects very well, and if we use all of them, the classification error will be very small. However, in our implemented example, we are only going to use the first two characteristics, area and aspect ratio, for learning purposes, because we can plot these characteristics in a 2D graphic and show that these values correctly describe our objects. We can also show that we can visually differentiate between one kind of object and another in the graphic plot. To extract these features, we are going to use the black/white ROI image as input, where only one object appears in white with a black background. This input is the segmentation result of Chapter 5, Automated Optical Inspection, Object Segmentation, and Detection. We are going to use the findCountours algorithm for segmenting objects and create the ExtractFeatures function for this purpose, as we can see in the following code:

vector< vector<float> > ExtractFeatures(Mat img, vector<int>* left=NULL, vector<int>* top=NULL) 
{ 
  vector< vector<float> > output; 
  vector<vector<Point> > contours; 
  Mat input= img.clone(); 
   
  vector<Vec4i> hierarchy; 
  findContours(input, contours, hierarchy, RETR_CCOMP, CHAIN_APPROX_SIMPLE); 
  // Check the number of objects detected 
  if(contours.size() == 0){ 
    return output; 
  } 
  RNG rng(0xFFFFFFFF); 
  for(auto i=0; i<contours.size(); i++){ 
     
    Mat mask= Mat::zeros(img.rows, img.cols, CV_8UC1); 
    drawContours(mask, contours, i, Scalar(1), FILLED, LINE_8, hierarchy, 1); 
    Scalar area_s= sum(mask); 
    float area= area_s[0]; 
 
    if(area>500){ //if the area is greater than min. 
       
      RotatedRect r= minAreaRect(contours[i]); 
      float width= r.size.width; 
      float height= r.size.height; 
      float ar=(width<height)?height/width:width/height; 
 
      vector<float> row; 
      row.push_back(area); 
      row.push_back(ar); 
      output.push_back(row); 
      if(left!=NULL){ 
          left->push_back((int)r.center.x); 
      } 
      if(top!=NULL){ 
          top->push_back((int)r.center.y); 
      } 
      
      // Add image to the multiple image window class, See the class on full github code   
      miw->addImage("Extract Features", mask*255); 
      miw->render(); 
      waitKey(10); 
    } 
  } 
  return output; 
}

Let's explain the code that we use to extract features. We are going to create a function that has one image as input and return two vectors of the left and top position for each object detected in the image as a parameter. This data will be used for drawing the corresponding label over each object. The output of a function is a vector of vectors of floats. In other words, it is a matrix where each row contains the features of each object that's detected.

First, we have to create the output vector variable and the contours variable that are going to be used in our find contours algorithm segmentation. We also have to create a copy of our input image, because the findCoutours OpenCV functions modify the input image:

  vector< vector<float> > output; 
  vector<vector<Point> > contours; 
  Mat input= img.clone(); 
  vector<Vec4i> hierarchy; 
  findContours(input, contours, hierarchy, RETR_CCOMP, CHAIN_APPROX_SIMPLE);

Now, we can use the findContours function to retrieve each object in an image. If we don't detect any contour, we return an empty output matrix, as we can see in the following snippet:

if(contours.size() == 0){ 
    return output; 
  }

If objects are detected, for each contour we are going to draw the object in white on a black image (zero values). This will be done using 1 values, like a mask image. The following piece of code generates the mask image:

for(auto i=0; i<contours.size(); i++){ 
    Mat mask= Mat::zeros(img.rows, img.cols, CV_8UC1); 
    drawContours(mask, contours, i, Scalar(1), FILLED, LINE_8, hierarchy, 1);

It's important to use the value of 1 to draw inside the shape because we can calculate the area by summing all of the values inside the contour, as shown in the following code:

    Scalar area_s= sum(mask); 
    float area= area_s[0];

This area is our first feature. We are going to use this value as a filter to remove all possible small objects that we have to avoid. All objects with an area less than the minimum threshold area that we considered will be discarded. After passing the filter, we create the second feature and the aspect ratio of the object. This refers to the maximum of the width or height, divided by the minimum of the width or height. This feature can tell the difference between the screw and other objects easily. The following code describes how to calculate the aspect ratio:

if(area>MIN_AREA){ //if the area is greater than min. 
      RotatedRect r= minAreaRect(contours[i]); 
      float width= r.size.width; 
      float height= r.size.height; 
      float ar=(width<height)?height/width:width/height;

Now we have the features, we only have to add them to the output vector. To do this, we will create a row vector of floats and add the values, followed by adding this row to the output vector, as shown in the following code:

vector<float> row; 
row.push_back(area); 
row.push_back(ar); 
output.push_back(row);

If the left and top parameters are passed, then add the top-left values to output the parameters:

  if(left!=NULL){ 
      left->push_back((int)r.center.x); 
  }
  if(top!=NULL){ 
      top->push_back((int)r.center.y); 
  }

Finally, we are going to show the detected objects in a window for user feedback. When we finish processing all of the objects in the image, we are going to return the output feature vector, as described in the following code snippet:

      miw->addImage("Extract Features", mask*255); 
      miw->render(); 
      waitKey(10); 
    } 
  } 
  return output;

Now that we have extracted the features of each input image, we can continue with the next step.