Segmentation

Segmentation is the process of dividing an image into multiple segments. This process is to simplify the image for analysis and make feature extraction easier.

One important feature of plate segmentation is the high number of vertical edges in a license plate, assuming that the image was taken frontally and the plate is not rotated and without perspective distortion. This feature can be exploited during the first segmentation step to eliminate regions that don't have any vertical edges.

Before finding vertical edges, we need to convert the color image to a grayscale image (because color can't help us in this task) and remove possible noise generated from the camera or other ambient noise. We will apply a 5x5 gaussian blur and remove noise. If we don't apply a noise removal method, we can get a lot of vertical edges that produce fail detection:

//convert image to gray 
Mat img_gray; 
cvtColor(input, img_gray, CV_BGR2GRAY); 
blur(img_gray, img_gray, Size(5,5));

To find the vertical edges, we will use a Sobel filter and find the first horizontal derivate. The derivate is a mathematical function that allows us to find vertical edges on an image. The definition of the Sobel function in OpenCV is as follows:

void Sobel(InputArray src, OutputArray dst, int ddepth, int xorder, int yorder, int ksize=3, double scale=1, double delta=0, int borderType=BORDER_DEFAULT )

Here, ddepth is the destination image depth; xorder is the order of the derivate by x; yorder is the order of the derivate by y; ksize is the kernel size of one, three, five, or seven; scale is an optional factor for computed derivative values; delta is an optional value added to the result; and borderType is the pixel interpolation method.

Then, for our case, we can use xorder=1, yorder=0, and ksize=3:

//Find vertical lines. Car plates have high density of vertical 
lines 
Mat img_sobel; 
Sobel(img_gray, img_sobel, CV_8U, 1, 0, 3, 1, 0);

After applying a Sobel filter, we will apply a threshold filter to obtain a binary image with a threshold value obtained through Otsu's method. Otsu's algorithm needs an 8-bit input image, and Otsu's method automatically determines the optimal threshold value:

//threshold image 
Mat img_threshold; 
threshold(img_sobel, img_threshold, 0, 255, CV_THRESH_OTSU+CV_THRESH_BINARY);

To define Otsu's method in the threshold function, we will combine the type parameter with the CV_THRESH_OTSU value, and the threshold value parameter is ignored.

When the CV_THRESH_OTSU value is defined, the threshold function returns the optimal threshold value obtained by Otsu's algorithm.

By applying a close morphological operation, we can remove blank spaces between each vertical edge line and connect all regions that have a high number of edges. In this step, we have possible regions that can contain plates.

First, we will define our structural element to use in our morphological operation. We will use the getStructuringElement function to define a structural rectangular element with a 17 by 3 dimension size in our case; this may be different in other image sizes:

Mat element = getStructuringElement(MORPH_RECT, Size(17, 3));

Then, we will use this structural element in a close morphological operation using the morphologyEx function:

morphologyEx(img_threshold, img_threshold, CV_MOP_CLOSE, element);

After applying these functions, we have regions in the image that could contain a plate; however, most of the regions do not contain license plates. These regions can be split by means of connected component analysis, or by using the findContours function. This last function retrieves the contours of a binary image with different methods and results. We only need to get the external contours with any hierarchical relationship and any polygonal approximation results:

//Find contours of possibles plates 
 vector< vector< Point>> contours; 
 findContours(img_threshold, 
    contours, // a vector of contours 
    CV_RETR_EXTERNAL, // retrieve the external contours 
    CV_CHAIN_APPROX_NONE); // all pixels of each contours

For each contour detected, extract the bounding rectangle of minimal area. OpenCV brings up the minAreaRect function for this task. This function returns a rotated RotatedRect rectangle class. Then, using a vector iterator over each contour, we can get the rotated rectangle and make some preliminary validations before we classify each region:

//Start to iterate to each contour founded 
 vector<vector<Point>>::iterator itc= contours.begin(); 
 vector<RotatedRect> rects; 

 //Remove patch that has no inside limits of aspect ratio and 
 area. 
 while (itc!=contours.end()) { 
     //Create bounding rect of object 
     RotatedRect mr= minAreaRect(Mat(*itc)); 
     if(!verifySizes(mr)){ 
         itc= contours.erase(itc); 
     }else{ 
         ++itc; 
         rects.push_back(mr); 
     } 
 }

We make basic validations for the regions detected based on their area and aspect ratio. We will consider that a region can be a plate if the aspect ratio is approximately 520/110 = 4.727272 (plate width divided by plate height), with an error margin of 40% and an area based on a minimum of 15 pixels and a maximum of 125 pixels for the height of the plate. These values are calculated depending on the image size and camera position:

bool DetectRegions::verifySizes(RotatedRect candidate ){ 
    float error=0.4; 
    //Spain car plate size: 52x11 aspect 4,7272 
    const float aspect=4.7272; 
    //Set a min and max area. All other patchs are discarded 
    int min= 15*aspect*15; // minimum area 
    int max= 125*aspect*125; // maximum area 
    //Get only patches that match to a respect ratio. 
    float rmin= aspect-aspect*error; 
    float rmax= aspect+aspect*error; 

    int area= candidate.size.height * candidate.size.width; 
    float r= (float)candidate.size.width 
    /(float)candidate.size.height; 
    if(r<1) 
        r= 1/r; 

    if(( area < min || area > max ) || ( r < rmin || r > rmax )){ 
        return false; 
    }else{ 
        return true; 
    } 
}

We can make even more improvements using the license plate's white background property. All plates have the same background color, and we can use a flood fill algorithm to retrieve the rotated rectangle for precise cropping.

The first step to crop the license plate is to get several seeds near the last rotated rectangle center. Then, we will get the minimum size of the plate between the width and height, and use it to generate random seeds near the patch center.

We want to select the white region, and we need several seeds to touch at least one white pixel. Then, for each seed, we use a floodFill function to draw a new mask image to store the new closest cropping region:

for(int i=0; i< rects.size(); i++){ 
 //For better rect cropping for each possible box 
 //Make floodFill algorithm because the plate has white background 
 //And then we can retrieve more clearly the contour box 
 circle(result, rects[i].center, 3, Scalar(0,255,0), -1); 
 //get the min size between width and height 
 float minSize=(rects[i].size.width < rects[i].size.height)? 
 rects[i].size.width:rects[i].size.height; 
 minSize=minSize-minSize*0.5; 
 //initialize rand and get 5 points around center for floodFill algorithm 
 srand ( time(NULL) ); 
 //Initialize floodFill parameters and variables 
 Mat mask; 
 mask.create(input.rows + 2, input.cols + 2, CV_8UC1); 
 mask= Scalar::all(0); 
 int loDiff = 30; 
 int upDiff = 30; 
 int connectivity = 4; 
 int newMaskVal = 255; 
 int NumSeeds = 10; 
 Rect ccomp; 
 int flags = connectivity + (newMaskVal << 8 ) + CV_FLOODFILL_FIXED_RANGE + CV_FLOODFILL_MASK_ONLY; 
 for(int j=0; j<NumSeeds; j++){ 
     Point seed; 
     seed.x=rects[i].center.x+rand()%(int)minSize-(minSize/2); 
     seed.y=rects[i].center.y+rand()%(int)minSize-(minSize/2); 
     circle(result, seed, 1, Scalar(0,255,255), -1); 
     int area = floodFill(input, mask, seed, Scalar(255,0,0), &ccomp, Scalar(loDiff, loDiff, loDiff), Scalar(upDiff, upDiff, upDiff), flags);

The floodFill function fills a connected component with a color into a mask image starting from a point seed, and sets the maximum lower and upper brightness/color difference between the pixel to fill and the pixel's neighbors or pixel seed:

int floodFill(InputOutputArray image, InputOutputArray mask, Point seed, Scalar newVal, Rect* rect=0, Scalar loDiff=Scalar(), Scalar upDiff=Scalar(), int flags=4 )

The newval parameter is the new color we want to incorporate into the image when filling. The loDiff and upDiff parameters are the maximum lower and maximum upper brightness/color difference between the pixel to fill and the pixel neighbors or pixel seed.

The parameter flag is a combination of the following bits:

CV_FLOODFILL_FIXED_RANGE sets the difference between the current pixel and the seed pixel. CV_FLOODFILL_MASK_ONLY will only fill the image mask and not change the image itself.

Once we have a crop mask, we will get a minimal area rectangle from the image mask points and check the validity size again. For each mask, a white pixel gets the position and uses the minAreaRect function to retrieve the closest crop region:

//Check new floodFill mask match for a correct patch. 
 //Get all points detected for get Minimal rotated Rect 
 vector<Point> pointsInterest; 
 Mat_<uchar>::iterator itMask= mask.begin<uchar>(); 
 Mat_<uchar>::iterator end= mask.end<uchar>(); 
 for( ; itMask!=end; ++itMask) 
     if(*itMask==255) 
         pointsInterest.push_back(itMask.pos()); 
     RotatedRect minRect = minAreaRect(pointsInterest); 
     if(verifySizes(minRect)){

The segmentation process is finished, and we have valid regions. Now, we can crop each detected region, remove possible rotation, crop the image region, resize the image, and equalize the light of the cropped image regions.

First, we need to generate the transform matrix with getRotationMatrix2D to remove possible rotations in the detected region. We need to pay attention to height, because RotatedRect can be returned and rotated at 90 degrees. So, we have to check the rectangle aspect and, if it is less than 1, we need to rotate it by 90 degrees:

//Get rotation matrix 
float r= (float)minRect.size.width / (float)minRect.size.height; 
float angle=minRect.angle; 
if(r<1) 
    angle=90+angle; 
Mat rotmat= getRotationMatrix2D(minRect.center, angle,1);

With the transform matrix, we now can rotate the input image by an affine transformation (an affine transformation preserves parallel lines) with the warpAffine function, where we set the input and destination images, the transform matrix, the output size (same as the input in our case), and the interpolation method to use. We can define the border method and border value if required:

//Create and rotate image 
Mat img_rotated; 
warpAffine(input, img_rotated, rotmat, input.size(), 
CV_INTER_CUBIC);

After we rotate the image, we will crop the image with getRectSubPix, which crops and copies an image portion of width and height centered on a point. If the image is rotated, we need to change the width and height sizes with the C++ swap function:

//Crop image 
Size rect_size=minRect.size; 
if(r < 1) 
    swap(rect_size.width, rect_size.height); 
Mat img_crop; 
getRectSubPix(img_rotated, rect_size, minRect.center, img_crop);

Cropped images are not good for use in training and classification since they do not have the same size. Also, each image contains different light conditions, accentuating the differences between them. To resolve this, we resize all the images to the same width and height, and apply a light histogram equalization:

Mat resultResized; 
resultResized.create(33,144, CV_8UC3); 
resize(img_crop, resultResized, resultResized.size(), 0, 0, INTER_CUBIC); 
//Equalize croped image 
Mat grayResult; 
cvtColor(resultResized, grayResult, CV_BGR2GRAY); 
blur(grayResult, grayResult, Size(3,3)); 
equalizeHist(grayResult, grayResult);

For each detected region, we store the cropped image and its position in a vector:

output.push_back(Plate(grayResult,minRect.boundingRect()));

Now that we have possible detected regions, we have to classify whether each possible region is a plate or not. In the next section, we are going to learn how to create a classification based on SVM.