Curves are another technique for remapping colors. Channel mixing and curves are similar insofar as the color at a destination pixel is a function of the color at the corresponding source pixel (only). However, in the specifics, channel mixing and curves are dissimilar approaches. With curves, a channel's value at a destination pixel is a function of (only) the same channel's value at the source pixel. Moreover, we do not define the functions directly; instead, for each function, we define a set of control points from which the function is interpolated. In pseudocode, for a BGR image:
dst.b = funcB(src.b) where funcB interpolates pointsB dst.g = funcG(src.g) where funcG interpolates pointsG dst.r = funcR(src.r) where funcR interpolates pointsR
The type of interpolation may vary between implementations, though it should avoid discontinuous slopes at control points and, instead, produce curves. We will use cubic spline interpolation whenever the number of control points is sufficient.
Our first step toward curve-based filters is to convert control points to a function. Most of this work is done for us by a SciPy function called interp1d(), which takes two arrays (x and y coordinates) and returns a function that interpolates the points. As an optional argument to interp1d(), we may specify a kind of interpolation, which, in principle, may be linear, nearest, zero, slinear (spherical linear), quadratic, or cubic, though not all options are implemented in the current version of SciPy. Another optional argument, bounds_error, may be set to False to permit extrapolation as well as interpolation.
Let's edit utils.py and add a function that wraps interp1d() with a slightly simpler interface:
def createCurveFunc(points):
"""Return a function derived from control points."""
if points is None:
return None
numPoints = len(points)
if numPoints < 2:
return None
xs, ys = zip(*points)
if numPoints < 4:
kind = 'linear'
# 'quadratic' is not implemented.
else:
kind = 'cubic'
return scipy.interpolate.interp1d(xs, ys, kind,
bounds_error = False)Rather than two separate arrays of coordinates, our function takes an array of (x, y) pairs, which is probably a more readable way of specifying control points. The array must be ordered such that x increases from one index to the next. Typically, for natural-looking effects, the y values should increase too, and the first and last control points should be (0, 0) and (255, 255) in order to preserve black and white. Note that we will treat x as a channel's input value and y as the corresponding output value. For example, (128, 160) would brighten a channel's midtones.
Note that cubic interpolation requires at least four control points. If there are only two or three control points, we fall back to linear interpolation but, for natural-looking effects, this case should be avoided.
Now we can get the function of a curve that interpolates arbitrary control points. However, this function might be expensive. We do not want to run it once per channel, per pixel (for example, 921,600 times per frame if applied to three channels of 640 x 480 video). Fortunately, we are typically dealing with just 256 possible input values (in 8 bits per channel) and we can cheaply precompute and store that many output values. Then, our per-channel, per-pixel cost is just a lookup of the cached output value.
Let's edit utils.py and add functions to create a lookup array for a given function and to apply the lookup array to another array (for example, an image):
def createLookupArray(func, length = 256):
"""Return a lookup for whole-number inputs to a function.
The lookup values are clamped to [0, length - 1].
"""
if func is None:
return None
lookupArray = numpy.empty(length)
i = 0
while i < length:
func_i = func(i)
lookupArray[i] = min(max(0, func_i), length - 1)
i += 1
return lookupArray
def applyLookupArray(lookupArray, src, dst):
"""Map a source to a destination using a lookup."""
if lookupArray is None:
return
dst[:] = lookupArray[src]Note that the approach in createLookupArray() is limited to whole-number input values, as the input value is used as an index into an array. The applyLookupArray() function works by using a source array's values as indices into the lookup array. Python's slice notation ([:]) is used to copy the looked-up values into a destination array.
Let's consider another optimization. What if we always want to apply two or more curves in succession? Performing multiple lookups is inefficient and may cause loss of precision. We can avoid this problem by combining two curve functions into one function before creating a lookup array. Let's edit utils.py again and add the following function that returns a composite of two given functions:
def createCompositeFunc(func0, func1):
"""Return a composite of two functions."""
if func0 is None:
return func1
if func1 is None:
return func0
return lambda x: func0(func1(x))The approach in createCompositeFunc() is limited to input functions that each take a single argument. The arguments must be of compatible types. Note the use of Python's lambda keyword to create an anonymous function.
Here is a final optimization issue. What if we want to apply the same curve to all channels of an image? Splitting and remerging channels is wasteful, in this case, because we do not need to distinguish between channels. We just need one-dimensional indexing, as used by applyLookupArray(). Let's edit utils.py to add a function that returns a one-dimensional interface to a preexisting, given array that may be multidimensional:
def createFlatView(array):
"""Return a 1D view of an array of any dimensionality."""
flatView = array.view()
flatView.shape = array.size
return flatViewThe return type is numpy.view, which has much the same interface as numpy.array, but numpy.view only owns a reference to the data, not a copy.
The approach in createFlatView() works for images with any number of channels. Thus, it allows us to abstract the difference between grayscale and color images in cases when we wish to treat all channels the same.
Since we cache a lookup array for each curve, our curve-based filters have data associated with them. Thus, they need to be classes, not just functions. Let's make a pair of curve filter classes, along with corresponding higher-level classes that can apply any function, not just a curve function:
VFuncFilter: This is a class that is instantiated with a function, which it can later apply to an image using apply(). The function is applied to the V (value) channel of a grayscale image or to all channels of a color image.VcurveFilter: This is a subclass of VFuncFilter. Instead of being instantiated with a function, it is instantiated with a set of control points, which it uses internally to create a curve function.BGRFuncFilter: This is a class that is instantiated with up to four functions, which it can later apply to a BGR image using apply(). One of the functions is applied to all channels and the other three functions are each applied to a single channel. The overall function is applied first and then the per-channel functions.BGRCurveFilter: this is a subclass of BGRFuncFilter. Instead of being instantiated with four functions, it is instantiated with four sets of control points, which it uses internally to create curve functions.Additionally, all these classes accept a constructor argument that is a numeric type, such as numpy.uint8 for 8 bits per channel. This type is used to determine how many entries should be in the lookup array.
Let's first look at the implementations of VFuncFilter and VcurveFilter, which may both be added to filters.py:
class VFuncFilter(object):
"""A filter that applies a function to V (or all of BGR)."""
def __init__(self, vFunc = None, dtype = numpy.uint8):
length = numpy.iinfo(dtype).max + 1
self._vLookupArray = utils.createLookupArray(vFunc, length)
def apply(self, src, dst):
"""Apply the filter with a BGR or gray source/destination."""
srcFlatView = utils.flatView(src)
dstFlatView = utils.flatView(dst)
utils.applyLookupArray(self._vLookupArray, srcFlatView,
dstFlatView)
class VCurveFilter(VFuncFilter):
"""A filter that applies a curve to V (or all of BGR)."""
def __init__(self, vPoints, dtype = numpy.uint8):
VFuncFilter.__init__(self, utils.createCurveFunc(vPoints),
dtype)Here, we are internalizing the use of several of our previous functions: createCurveFunc(), createLookupArray(), flatView(), and applyLookupArray(). We are also using numpy.iinfo() to determine the relevant range of lookup values, based on the given numeric type.
Now, let's look at the implementations of BGRFuncFilter and BGRCurveFilter, which may both be added to filters.py as well:
class BGRFuncFilter(object):
"""A filter that applies different functions to each of BGR."""
def __init__(self, vFunc = None, bFunc = None, gFunc = None,
rFunc = None, dtype = numpy.uint8):
length = numpy.iinfo(dtype).max + 1
self._bLookupArray = utils.createLookupArray(
utils.createCompositeFunc(bFunc, vFunc), length)
self._gLookupArray = utils.createLookupArray(
utils.createCompositeFunc(gFunc, vFunc), length)
self._rLookupArray = utils.createLookupArray(
utils.createCompositeFunc(rFunc, vFunc), length)
def apply(self, src, dst):
"""Apply the filter with a BGR source/destination."""
b, g, r = cv2.split(src)
utils.applyLookupArray(self._bLookupArray, b, b)
utils.applyLookupArray(self._gLookupArray, g, g)
utils.applyLookupArray(self._rLookupArray, r, r)
cv2.merge([b, g, r], dst)
class BGRCurveFilter(BGRFuncFilter):
"""A filter that applies different curves to each of BGR."""
def __init__(self, vPoints = None, bPoints = None,
gPoints = None, rPoints = None, dtype = numpy.uint8):
BGRFuncFilter.__init__(self,
utils.createCurveFunc(vPoints),
utils.createCurveFunc(bPoints),
utils.createCurveFunc(gPoints),
utils.createCurveFunc(rPoints), dtype)Again, we are internalizing the use of several of our previous functions: createCurveFunc(), createCompositeFunc(), createLookupArray(), and applyLookupArray(). We are also using iinfo(), split(), and merge().
These four classes can be used as is, with custom functions or control points being passed as arguments at instantiation. Alternatively, we can make further subclasses that hard-code certain functions or control points. Such subclasses could be instantiated without any arguments.
A common use of curves is to emulate the palettes that were common in pre-digital photography. Every type of photo film has its own, unique rendition of color (or grays) but we can generalize about some of the differences from digital sensors. Film tends to suffer loss of detail and saturation in shadows, whereas digital tends to suffer these failings in highlights. Also, film tends to have uneven saturation across different parts of the spectrum. So each film has certain colors that pop or jump out.
Thus, when we think of good-looking film photos, we may think of scenes (or renditions) that are bright and that have certain dominant colors. At the other extreme, we may remember the murky look of underexposed film that could not be improved much by the efforts of the lab technician.
We are going to create four different film-like filters using curves. They are inspired by three kinds of film and a processing technique:
Each film emulation effect is a very simple subclass of BGRCurveFilter. We just override the constructor to specify a set of control points for each channel. The choice of control points is based on recommendations by photographer Petteri Sulonen. See his article on film-like curves at http://www.prime-junta.net/pont/How_to/100_Curves_and_Films/_Curves_and_films.html.
The Portra, Provia, and Velvia effects should produce normal-looking images. The effect should not be obvious except in before-and-after comparisons.
Portra has a broad highlight range that tends toward warm (amber) colors, while shadows are cooler (more blue). As a portrait film, it tends to make people's complexions fairer. Also, it exaggerates certain common clothing colors, such as milky white (for example, a wedding dress) and dark blue (for example, a suit or jeans). Let's add this implementation of a Portra filter to filters.py:
class BGRPortraCurveFilter(BGRCurveFilter):
"""A filter that applies Portra-like curves to BGR."""
def __init__(self, dtype = numpy.uint8):
BGRCurveFilter.__init__(
self,
vPoints = [(0,0),(23,20),(157,173),(255,255)],
bPoints = [(0,0),(41,46),(231,228),(255,255)],
gPoints = [(0,0),(52,47),(189,196),(255,255)],
rPoints = [(0,0),(69,69),(213,218),(255,255)],
dtype = dtype)Provia has strong contrast and is slightly cool (blue) throughout most tones. Sky, water, and shade are enhanced more than sun. Let's add this implementation of a Provia filter to filters.py:
class BGRProviaCurveFilter(BGRCurveFilter):
"""A filter that applies Provia-like curves to BGR."""
def __init__(self, dtype = numpy.uint8):
BGRCurveFilter.__init__(
self,
bPoints = [(0,0),(35,25),(205,227),(255,255)],
gPoints = [(0,0),(27,21),(196,207),(255,255)],
rPoints = [(0,0),(59,54),(202,210),(255,255)],
dtype = dtype)Velvia has deep shadows and vivid colors. It can often produce azure skies in daytime and crimson clouds at sunset. The effect is difficult to emulate but here is an attempt that we can add to filters.py:
class BGRVelviaCurveFilter(BGRCurveFilter):
"""A filter that applies Velvia-like curves to BGR."""
def __init__(self, dtype = numpy.uint8):
BGRCurveFilter.__init__(
self,
vPoints = [(0,0),(128,118),(221,215),(255,255)],
bPoints = [(0,0),(25,21),(122,153),(165,206),(255,255)],
gPoints = [(0,0),(25,21),(95,102),(181,208),(255,255)],
rPoints = [(0,0),(41,28),(183,209),(255,255)],
dtype = dtype)Cross-processing produces a strong, blue or greenish-blue tint in shadows and a strong, yellow or greenish-yellow in highlights. Black and white are not necessarily preserved. Also, contrast is very high. Cross-processed photos take on a sickly appearance. People look jaundiced, while inanimate objects look stained. Let's edit filters.py to add the following implementation of a cross-processing filter:
class BGRCrossProcessCurveFilter(BGRCurveFilter):
"""A filter that applies cross-process-like curves to BGR."""
def __init__(self, dtype = numpy.uint8):
BGRCurveFilter.__init__(
self,
bPoints = [(0,20),(255,235)],
gPoints = [(0,0),(56,39),(208,226),(255,255)],
rPoints = [(0,0),(56,22),(211,255),(255,255)],
dtype = dtype)