Processing a neighborhood of pixels with convolution filters

For a convolution filter, the channel values at each output pixel are a weighted average of the corresponding channel values in a neighborhood of input pixels. We can put the weights in a matrix, called a convolution matrix or kernel . For example, consider the following kernel:

{{ 0, -1,  0},
 {-1,  4, -1},
 { 0, -1,  0}}

The central element is the weight for the source pixel that has the same indices as of the destination pixel. Other elements represent weights for the rest of the neighborhood of input pixels. Here, we are considering a 3 x 3 neighborhood. However, OpenCV supports kernels with any square and odd-numbered dimensions. This particular kernel is a type of edge-finding filter called a Laplacian filter. For a neighborhood of flat (same) color, it yields a black output pixel. For a neighborhood of high contrast, it yields a bright output pixel.

Let's consider another kernel where the central element is greater by 1:

{{ 0, -1,  0},
 {-1,  5, -1},
 { 0, -1,  0}}

This is equivalent to taking the result of a Laplacian filter and then adding it to the original image. Instead of edge-finding, we get edge-sharpening. That is, edge regions get brighter while the rest of the image remains unchanged.

Tip

Beware big kernels

The bigger the kernel, the more expensive the computation. Kernels larger than 5 x 5 (that is, 25 input pixels per output pixel) are probably not practical for live video processing on typical Android devices today.

OpenCV provides many static methods for convolution filters that use certain popular kernels. The following are some examples:

• Imgproc.blur(Mat src, Mat dst, Size ksize): It blurs the image by taking a simple average of a neighborhood of size ksize. For example, if ksize is new Size(5, 5), then the kernel is the following:

  {{0.04, 0.04, 0.04, 0.04, 0.04},
   {0.04, 0.04, 0.04, 0.04, 0.04},
   {0.04, 0.04, 0.04, 0.04, 0.04},
   {0.04, 0.04, 0.04, 0.04, 0.04},
   {0.04, 0.04, 0.04, 0.04, 0.04}}

• Laplacian(Mat src, Mat dst, int ddepth, int ksize, double scale, double delta): It is a Laplacian edge-finding filter, as described previously. Results are multiplied by a constant (the scale argument) and added to another constant (the delta argument).

Moreover, OpenCV provides a static method, Imgproc.filter2D(Mat src, Mat dst, int ddepth, Mat kernel), which enables us to specify our own kernels. For learning purposes, we will take this approach. The ddepth argument determines the numeric type of the destination's data. This argument may be any of the following:

• -1: It means the same numeric type as in the source.
• CvType.CV_16S: It means 16-bit signed integers.
• CvType.CV_32F: It means 32-bit floats.
• CvType.CV_64F: It means 64-bit floats.

Let's use a convolution filter as part of a more complex filter that draws heavy, black lines atop edge regions in the image. To achieve this effect, we also rely on two more static methods from OpenCV:

• Core.bitwise_not(Mat src, Mat dst): This method inverts the image's brightness and colors, such that white becomes black, red becomes cyan, and so on. It is useful to us because our convolution filter will produce white edges on a black field, whereas we want the opposite: black edges on a white field.
• Core.multiply(Mat src1, Mat src2, Mat dst, double scale): This method blends a pair of images by multiplying their values together. The resulting values are scaled by a constant (the scale argument). For example, scale can be used to normalize the product to the [0, 255] range. For our purposes, Core.multiply can serve to superimpose the black edges on the original image.

The following is the implementation of the blackened edge effect in StrokeEdgesFilter:

public class StrokeEdgesFilter implements Filter {
  private final Mat mKernel = new MatOfInt(
    0, 0,   1, 0, 0,
    0, 1,   2, 1, 0,
    1, 2, -16, 2, 1,
    0, 1,   2, 1, 0,
    0, 0,   1, 0, 0
  );
  private final Mat mEdges = new Mat();
  @Override
  public void apply(final Mat src, final Mat dst) {
    Imgproc.filter2D(src, mEdges, -1, mKernel);
    Core.bitwise_not(mEdges, mEdges);
    Core.multiply(src, mEdges, dst, 1.0/255.0);
  }
}

We will look at some other complex uses of convolution filters in subsequent chapters.

Next, let's add a user interface for enabling and disabling all our filters.