Overlaying a face mask in a live video

OpenCV provides a nice face detection framework. We just need to load the cascade file and use it to detect the faces in an image. When we capture a video stream from the webcam, we can overlay funny masks on our faces. It will look something like this:

Let's look at the main parts of the code to see how to overlay this mask on the face in the input video stream. The full code is available in the downloadable code bundle provided along with this book:

#include "opencv2/core/utility.hpp"
#include "opencv2/objdetect/objdetect.hpp"
#include "opencv2/imgproc.hpp"
#include "opencv2/highgui.hpp"

using namespace cv;
using namespace std;

...

int main(int argc, char* argv[]) 
{ 
    string faceCascadeName = argv[1]; 
     
    // Variable declaration and initialization 
    ...
    // Iterate until the user presses the Esc key 
    while(true) 
    { 
        // Capture the current frame 
        cap >> frame; 
         
        // Resize the frame 
        resize(frame, frame, Size(), scalingFactor, scalingFactor, INTER_AREA); 
         
        // Convert to grayscale 
        cvtColor(frame, frameGray, COLOR_BGR2GRAY); 
         
        // Equalize the histogram 
        equalizeHist(frameGray, frameGray); 
         
        // Detect faces 
        faceCascade.detectMultiScale(frameGray, faces, 1.1, 2, 0|HAAR_SCALE_IMAGE, Size(30, 30) ); 

Let's take a quick stop to see what happened here. We start reading input frames from the webcam and resize it to our size of choice. The captured frame is a color image and face detection works on grayscale images. So, we convert it to grayscale and equalize the histogram. Why do we need to equalize the histogram? We need to do this to compensate for any issues, such as lighting or saturation. If the image is too bright or too dark, the detection will be poor. So, we need to equalize the histogram to ensure that our image has a healthy range of pixel values:

        // Draw green rectangle around the face 
        for(auto& face:faces) 
        { 
            Rect faceRect(face.x, face.y, face.width, face.height); 
             
            // Custom parameters to make the mask fit your face. You may have to play around with them to make sure it works. 
            int x = face.x - int(0.1*face.width); 
            int y = face.y - int(0.0*face.height); 
            int w = int(1.1 * face.width); 
            int h = int(1.3 * face.height); 
             
            // Extract region of interest (ROI) covering your face 
            frameROI = frame(Rect(x,y,w,h));

At this point, we know where the face is. So we extract the region of interest to overlay the mask in the right position:

            // Resize the face mask image based on the dimensions of the above ROI 
            resize(faceMask, faceMaskSmall, Size(w,h)); 
             
            // Convert the previous image to grayscale 
            cvtColor(faceMaskSmall, grayMaskSmall, COLOR_BGR2GRAY); 
             
            // Threshold the previous image to isolate the pixels associated only with the face mask 
            threshold(grayMaskSmall, grayMaskSmallThresh, 230, 255, THRESH_BINARY_INV); 

We isolate the pixels associated with the face mask. We want to overlay the mask in such a way that it doesn't look like a rectangle. We want the exact boundaries of the overlaid object so that it looks natural. Let's go ahead and overlay the mask now:

            // Create mask by inverting the previous image (because we don't want the background to affect the overlay) 
            bitwise_not(grayMaskSmallThresh, grayMaskSmallThreshInv); 
             
            // Use bitwise "AND" operator to extract precise boundary of face mask 
            bitwise_and(faceMaskSmall, faceMaskSmall, maskedFace, grayMaskSmallThresh); 
             
            // Use bitwise "AND" operator to overlay face mask 
            bitwise_and(frameROI, frameROI, maskedFrame, grayMaskSmallThreshInv); 
             
            // Add the previously masked images and place it in the original frame ROI to create the final image 
            add(maskedFace, maskedFrame, frame(Rect(x,y,w,h))); 
        } 
      
    // code dealing with memory release and GUI 
 
    return 1; 
}