Image capturing from web camera

We are starting our introduction to advanced image processing with capturing the content from a video camera.

Capturing frame from the camera is be done in multiple steps:

1. Identifying the camera
2. Capturing the frame
3. Converting the frame to Julia images
4. Previewing the result

We will also find the most efficient way of getting the results to Julia images; that is, running the conversion process from C++ on a Julia side or saving and reloading the image.

First, we start by initializing libraries and configuration parameters:

ENV["PKG_CONFIG_PATH"] = "/Users/dc/anaconda/envs/python35/lib/pkgconfig"

using OpenCV
using Images
using Cxx

Next, we proceed with defining the function to convert Open CV images to Julia images:

function opencv_to_image(img_opencv)

    converted_image = zeros(Float16, (3, rows(img_opencv), 
     cols(img_opencv)));

    for i = 1:size(converted_image, 2)
        for j = 1:size(converted_image, 3)
            pixel_value = @cxx at_v3b(img_opencv, i, j)
            converted_image[:, i, j] = map(x -> Int(at(pixel_value, 
             x)), [2, 1, 0]) ./ 255
        end
    end

    return converted_image
end

Julia code is ready, and we need to proceed with defining C++ code. C++ code is defined within the cxx""" <<C++ code>> """ tag. As the code is put inside """, we should first come up with all code and then execute it in REPL.

We start with initializing C++ libraries and namespaces. Next, we define the first function that will connect to the web camera. Connection to the web camera is done by initializing the VideoCapture class from Open CV namespace and reading the first frame:

#include "opencv2/videoio.hpp"
#include "opencv2/imgproc.hpp"

using namespace std;
using namespace cv;

/// Retrieve video device by index
cv::VideoCapture get_video_device(int device_index) {

    cv::VideoCapture capture(device_index);
    cv::Mat frame;

    capture.read(frame);

    return capture;
}

When we know how to connect to the web camera, we need to define a function to capture and return the frame. It is done by calling the read function of the VideoCapture object. This is shown in the following code:

/// Capture frame
cv::Mat capture_frame(cv::VideoCapture capture) {
   
    cv::Mat frame;
    bool Success = capture.read(frame);
    return frame;
}

We also add another function that will capture the frame, but instead of returning it to Julia, we save the image to disk. The implementation is very similar to capture_frame, but we also use the success status and the imwrite function to save the file:

/// Capture and save frame
void capture_save_frame(cv::VideoCapture capture, String dest) {
   
    cv::Mat frame;
    bool success = capture.read(frame);

    if (success) {
        cv::imwrite(dest, frame);
    }
}

The last C++ function we define is responsible for releasing the camera resources:

/// Release an active camera
void release_camera(cv::VideoCapture capture) {
    capture.release();
}

Now, let's proceed to calling the the functions and getting the results. We start with creating a video_device object keeping a reference to our web camera. CAP_ANY determines the camera; in this case, it is equal to zero and returns a built-in web camera:

video_device = @cxx get_video_device(CAP_ANY);

Next, we test two different scenarios. First, we try to read the frame and convert the OpenCV Mat object to a Julia image. We also use the @time macro to have a rough estimate of the execution time:

@time current_frame = @cxx capture_frame(video_device);
# Main> 0.011622 seconds (7 allocations: 400 bytes)

@time current_frame_image = opencv_to_image(current_frame);
# Main> 6.573967 seconds (9.12 M allocations: 439.680 MiB, 2.15% gc time)

imshow(colorview(RGB, current_frame_image));

You can see that the code is very inefficient. Let's try running capture_save_frame and reloading the image from Julia:

filename = joinpath(pwd(), "camera-frame.jpg");

@time @cxx capture_save_frame(video_device, pointer(filename));
# Main> 0.080999 seconds (7 allocations: 304 bytes)

@time current_frame_image_2 = load(filename);
# Main> 0.034937 seconds (265 allocations: 7.046 MiB)

imshow(current_frame_image_2)

This option is way better. It does not allocate that much memory and is over 50 times better. I suspect that the timing can be decreased by resizing the image when it is captured and operating with a smaller resolution.

Both results will output a photoshoot from your web camera. You can see the view from my place in the following photo:

As the code contains a mix of C++ and Julia code, I urge you to review the GitHub version to ensure that you implement it correctly.