Mixing color channels

As we saw in Chapter 2, Working with Camera Frames, OpenCV stores image data in a matrix of type Mat, which is like a two-dimensional array. The columns and rows (specified by the first and second indices, respectively) correspond to the y and x pixel coordinates in the image. The elements are the pixel values. A pixel value may be represented by one number (in the case of a grayscale image) or multiple numbers (in the case of a color image). Each of these numbers is said to belong to a channel. A grayscale image may have just one channel, value (brightness), which is abbreviated as V. A color image may have as many as four channels—for example, red, green, blue, and alpha (transparency), which constitute the RGBA format. Other useful formats for color images include RGB (red, green, blue), HSV (hue, saturation, value), and L*a*b (luminosity, green-versus-magenta, yellow-versus-blue). In this book, we focus on RGB and RGBA images, but OpenCV supports other formats too. As we saw in the previous chapter, we can convert between color formats with the Imgproc.cvtColor static method.

If we separated the channels of an RGB image matrix, we could make three different grayscale image matrices, each having one channel. We could then apply some matrix arithmetic to these single-channel matrices, and merge the results to get another RGB image matrix. The resulting RGB image would look as if it were mixed from a different color palette than the original image was. This technique is called channel mixing. For an RGB image, we may define channel mixing in pseudocodeas follows:

dst.b = funcB(src.b, src.g, src.r)
dst.g = funcG(src.b, src.g, src.r)
dst.r = funcR(src.b, src.g, src.r)

That is to say, each channel in the destination image is mapped from a function of any or all channels in the source image. We will not restrict our definition to any particular kind of function. However, let's note the visual effects of the following operations, which I find useful when working with RGB images:

With these effects in mind, let's look at the OpenCV functionality that we would use to produce them. OpenCV's Core class provides all the relevant functionality as static methods. The Core.split(Mat m, List<Mat> mv) method is responsible for channel splitting. As arguments, it takes a source matrix and a list of destination matrices. Each channel from the source is copied into a single-channel matrix in the destination list. If necessary, the destination list is populated with new matrices.

After using the Core.split method, we can apply matrix operations to the individual channels. The Core.addWeighted(Mat src1, double alpha, Mat src2, double beta, double gamma, Mat dst) method can be used to take a weighted average of two channels. The first four arguments are weights and source matrices. The fifth argument is a constant that is added to the result. The last argument is the destination matrix. In pseudocode, dst = alpha * src1 + beta * src2 + gamma.

The Core.min(Mat src1, Mat src2, Mat dst) and Core.max(Mat src1, Mat src2, Mat dst) methods each take a pair of source matrices and a destination matrix. These methods perform a per-element min or max.

Finally, the converse of Core.split is Core.merge(List<Mat> mv, Mat m). We can use it to recreate a multichannel image from the split channels.

To do a practical example of channel mixing, let's open RecolorRCFilter.java and write the following implementation of the class:

public class RecolorRCFilter implements Filter {
  private final ArrayList<Mat> mChannels = new ArrayList<Mat>(4);
  @Override
  public void apply(final Mat src, final Mat dst) {
    Core.split(src, mChannels);
    final Mat g = mChannels.get(1);
    final Mat b = mChannels.get(2);
    // dst.g = 0.5 * src.g + 0.5 * src.b
    Core.addWeighted(g, 0.5, b,  0.5, 0.0, g);
    // dst.b = dst.g
    mChannels.set(2, g);
    Core.merge(mChannels, dst);
  }
}

The effect of this filter is to turn greens and blues to cyan, leaving a limited color palette of red and cyan. It resembles the color palette of certain old movies and old computer games.

As a member variable, RecolorRCFilter has a list of four matrices. Whenever the apply() method is called, this list is populated with the four channels of the source matrix. (We assume that the source and destination matrices each have four channels, in RGBA order.) We get the green and blue channels (at indices 1 and 2 in the list), take their average, and assign the result back to the same channels. Last, we merge the four channels into the destination matrix, which may be the same as the source matrix.

The code for our other two channel mixing filters is similar, so, for brevity, we will omit most of it. Just note that RecolorRGVFilter relies on the following operations:

// dst.b = min(dst.r, dst.g, dst.b)
Core.min(b, r, b);
Core.min(b, g, b);

The effect of this filter is to desaturate blues, leaving a limited color palette of red, green, and white. It, too, resembles the color palette of certain old movies and old computer games.

Similarly, RecolorCMVFilter relies on the following operations:

// dst.b = max(dst.r, dst.g, dst.b)
Core.max(b, r, b);
Core.max(b, g, b);

The effect of this filter is to desaturate yellows, leaving a limited color palette of cyan, magenta, and white. Nobody ever made a movie in this color palette (yet!), but it will be a familiar sight to gamers of the 1980s.

Arbitrary channel mixing functions, in RGB, tend to produce effects that are bold and stylized, not subtle. This is true of our examples here. Next, let's look at a family of filters that are easier to parameterize for subtle, natural-looking results.