The main object of computer vision is the image. So before diving into image processing and analysis, we should learn how to work with OpenCV-related images freely. We will consider images of classes ofxCvColorImage
, ofxCvGrayscaleImage
, ofxCvFloatImage
, and ofxCvShortImage
. These class names have the prefix ofxCv
, so we will call them ofxCv images. To be precise, we will assume that we have these image objects:
ofxCvColorImage image, image2; //Color images ofxCvGrayscaleImage grayImage, grayImage2; //Grayscale images ofxCvFloatImage floatImage; //Float-type image ofxCvShortImage shortImage; //Image with "unsigned short" pixels
It is convenient to group functions and operations into several groups.
You always need to initialize an image before using it for the first time. Let us look at a few functions used to initialize images:
allocate( w, h )
function initializes the image with width w
and height h
pixels; for example, image.allocate( 320, 240 )
creates an image with size 320 × 240 pixels.=
operator copies images of equal or different types and performs the necessary pixel conversions; for example:image2 = image;
copies image
to image2
. Note that there is no need to initialize image2
using image2.allocate( w, h )
because =
initializes it automatically.grayImage = image;
converts a color image to a grayscale imagefloatImage = grayScale;
converts a grayscale image to a float imageThe important thing here is initialization. If destination image was not initialized, the =
operator performs the required initialization automatically. However, if the image was initialized, it must have a size equal to the size of the source image. In opposite cases, you should clear the image using the clear()
function, or set a prerequisite size using the resize()
function.
During image type conversion, the range of pixel values is transforming correspondingly with the range of the image class. The range for ofxCvColorImage
and ofxCvGrayscaleImage
is from 0 to 255, while that for ofxCvFloatImage
is segment [0, 1], and the range for ofxCvShortImage
is from 0 to 65535.
For example, during the floatImage = grayImage
operation, the assigned pixel values of floatImage
are equal to the pixel values of grayImage
image multiplied by 1.0 / 255.0
. So, the pixel values of floatImage
will lie in [0, 1]. Similarly, during the grayImage = shortImage
operation, the assigned pixel values of grayImage
are equal to the pixel values of the shortImage
image multiplied by 255.0 / 65535.0
.
You can change the range of ofxCvFloatImage
to any value using the setNativeScale( vMin, vMax )
function; for example, floatImage.setNativeScale( 0.0, 255.0 )
sets the range to [0, 255].
setFromPixels( data, w, h )
function sets image dimensions to w
× h
pixels and sets its pixel values to the values from an unsigned char
array data
. Note that the size of the data array should be equal to w
× h
× 3
for ofxCvColorImage
and w
× h
for other types. There is an alternative form, setFromPixels( pixels )
, with pixels
having the type ofPixels
; for example, if you are getting video frames from a camera using the object ofVideoGrabber grabber
(see the Processing a live video from camera section in Chapter 5, Working with Videos), you can initialize an image in the following way:image.setFromPixels( grabber.getPixelsRef() );
setFromPixels( dataFloat, w, h )
where dataFloat
is the array of floats.bAllocated
member, which has the type bool
as shown in the following code:if ( image.bAllocated ) { //... }
There are two functions that are close to the initialization stage:
set( value )
function sets all the image pixels to the value
value. For color images, there is also a function set( valueRed, valueGreen, valueBlue )
, that sets each pixel of red, green, and blue color components to valueRed
, valueGreen
, and valueBlue
respectively.clear()
function clears all the memory allocated for an image. Usually, you don't need to use this function because it is called automatically by the image's destructor.Algebraic operations apply mathematical operations such as addition and subtraction to each pixel of the images. The following are a few algebraic functions:
+=
, -=
, and *=
operations with an operand of the ofxCV image type are operations that are applicable for images of an equal size and type. The operations do the corresponding operation on the corresponding pixels of the both images; for example, image += image2
adds image2
to image
.+=
, -=
, *=
, and /=
operations with the float operand argument value
perform addition, subtraction, multiplication, and division respectively on all pixel values in the image with the value specified in the value
variable; for example, image += 1
adds 1
to the image's pixel values. The *=
and /=
operations are currently only available for float images of the class ofxCvFloatImage
.Currently, the *=
operation truncates negative pixel values to zero. So do not use this operation when you need to work with negative pixel values during intermediate operations. So, instead of floatImage *= value
, call the multiplyByScalar( floatImage, value )
function, where the function's code is as follows:
void multiplyByScalar( ofxCvFloatImage &floatImage, float value ){ int w = floatImage.width; int h = floatImage.height; float *floatPixels = floatImage.getPixelsAsFloats(); for (int y=0; y<h; y++) { for (int x=0; x<w; x++) { //Change pixels values floatPixels[ x + w * y ] *= value; } } //Notify openFrameworks that //the image was changed floatImage.flagImageChanged(); }
grayImage.absDiff( grayImage2 )
function calculates the absolute difference value between the corresponding pixels of grayImage
and grayImage2
and writes the result to grayImage
. There is an overloaded grayImage.absDiff( grayImageA, grayImageB )
function, which puts the result of the absolute difference between grayImageA
and grayImageB
to grayImage
. Though this is not formally an algebraic operation, it is obviously related to them. The function is useful for marking the regions where the two given images differ. Note, the absDiff
function is currently available for grayscale images only.All the image types except ofxCvFloatImage
have a limited range of pixel values (see the discussion of ranges in the preceding information box for the operator =
). In the case when the result of any operation goes beyond the range, the saturation arithmetic is applied, that is, the value is truncated to the range. For example, for an image of type ofxCvGrayscaleImage
, the values 300 and -10 will be truncated to 255 and 0 respectively. Hence, if you perform advanced mathematical calculations with images, it is a good idea to convert input images to ofxCvFloatImage
first, then perform calculations, and finally convert the final result to the required type.
The drawing functions are similar to the corresponding functions of the ofImage
class for openFrameworks' images, discussed in Chapter 4, Images and Textures. The following are a few drawing functions:
draw( x, y, w, h )
function draws the image to the screen. Note, for images of classes ofxCvFloatImage
and ofxCvShortImage
, the pixel values are mapped from the corresponding ranges [0, 1] and 0 to 65,535 to the range 0 to 255 for screen output. The range for a float image can be changed using the setNativeScale( vMin, vMax )
function. There are overloaded versions: draw( x, y )
, draw( point )
, and draw( rect )
with point
of type ofPoint
and rect
of type ofRectangle
.setAnchorPercent( xPct, yPct )
, setAnchorPoint( x, y )
, and resetAnchor()
functions let us control the origin of the output image, just like in ofImage
.setUseTexture( use )
function with use
of the type bool
enables or disables using texture for the image. This texture is automatically recalculated only before the image drawing. If the image is used only for internal calculations and will never be shown on the screen, call setUseTexture( false )
for saving the video memory.getTextureReference()
function returns the reference on the ofTexture
object of the image. If you change the image and need its texture, you need to call updateTexture()
before getting the texture reference.For implementing custom image processing functions or using OpenCV functions not implemented in the ofxOpenCv addon, you need to have access to the pixel values and OpenCV image inside the ofxCv image. There are number of functions for dealing with it:
width
and height
values can be used for getting the current size of the image.getPixels()
function returns an array of unsigned char
values, corresponding with the image's values. For ofxCvFloatImage
and ofxCvShortImage
images, the pixel's values are mapped to 0 to 255 range, as in the draw()
function described earlier.getPixelsRef()
function returns a reference to a pixel array of the current frame represented by a class ofPixels
.getPixelsAsFloats()
and getFloatPixelsRef()
functions respectively return an array of floats and reference to ofFloatPixels
for images of class ofxCvFloatImage
.getShortPixelsRef()
function returns a reference to ofShortPixels
for images of class ofxCvShortImage
.getCvImage()
function returns a pointer to an object of type IplImage
. This is an OpenCV type used for holding an image. The function is used for applying any OpenCV operations to the images directly. Normally, you will use this function for those OpenCV capabilities that are not implemented in the ofxOpenCv addon.IplImage
object, you need to call flagImageChanged()
to notify the ofxOpenCv addon that the image was changed. If you need a texture reference to the image, you should call updateTexture()
. Note, when calling image.draw()
, the texture updates automatically, if needed.There are a number of functions for manipulating color planes. They are as follows:
image.setFromGrayscalePlanarImages( planeR, planeG, planeB )
function creates a color image with color planes from three ofxCvGrayscaleImage
images, planeR
, planeG
, and planeB
. These images planeR
, planeG
, and planeB
should be allocated before calling setFromGrayscalePlanarImages()
.image.convertToGrayscalePlanarImages( planeR, planeG, planeB )
function does the opposite. It splits an image into its color planes, and writes them to planeR
, planeG
, and planeB
. Note, currently image
should be allocated before calling convertToGrayscalePlanarImages()
.image.convertToGrayscalePlanarImage( grayImage, index )
function extracts the color plane number index from an image and writes it into grayImage
. Here index
= 0, 1, and 2 corresponds to red, green and blue color components respectively, and grayImage
does not need to be allocated before calling convertToGrayscalePlanarImage()
.The class ofxCvColorImage
has two functions for converting between RGB (Red, Green, Blue) and HSV (Hue, Saturation, Value) color spaces: convertRgbToHsv()
and convertHsvToRgb()
.
Now we will consider an example of using ofxCv images for a simple motion detector.
Let's consider a live video from the camera or a video from a movie, and consider the absolute difference between its two successive frames, which is computed using the function grayImage.absDiff( grayImage2 )
, considered in the Algebraic operations with images section. The regions in this difference image, corresponding to the moving objects, will have higher values than the static regions. So, for getting pixels with high values, it is possible to detect the regions of motion in the video. This information can be used for controlling the behavior of your application; for example, if you have a particle system, the motion areas can be used as places of particles' emitting or as areas of particles' attraction. Then, people walking in front of your camera will see how their silhouette controls the particles on the screen.
This method of using difference image is simple and has been successfully used in a number of interactive projects for more than thirty years now. However, if you consider two successive difference images, they will most likely have very few common pixels with high values. The reason is that difference image emphasizes the changed pixels of successive frames, or in another words, "motion border", which changes each frame. So difference image is not stable in time. To regularize it, it is a good idea to accumulate the differences in an image buffer that slowly decreases its values.
The following example illustrates the calculation of absolute differences and accumulation of them in the buffer using ofxCv images of the ofxOpenCv addon.
Use the Project Generator for creating an empty project with the linked ofxOpenCv addon (see the Using ofxOpenCv section). Then, copy the handsTrees.mov
movies into the bin/data
folder of the project.
Include the addon's header into the testApp.h
file, just after the #include "ofMain.h"
line:
#include "ofMain.h"
#include "ofxOpenCv.h"
Also, add the following lines in the testApp
class declaration:
ofVideoPlayer video; //Declare the video player object ofxCvColorImage image; //The current video frame //The current and the previous video frames as grayscale images ofxCvGrayscaleImage grayImage, grayImagePrev; ofxCvGrayscaleImage diff; //Absolute difference of the frames ofxCvFloatImage diffFloat; //Amplified difference images ofxCvFloatImage bufferFloat; //Buffer image
Now let's assemble testApp.cpp
. The testApp::setup()
function loads and starts the video as follows:
void testApp::setup(){ video.loadMovie( "handsTrees.mov" ); //Load the video file video.play(); //Start the video to play }
The testApp::update()
function reads video frames and processes them by calculating the absolute difference diff
, amplifying it for better visibility in diffFloat
, and then updating the accumulate buffer bufferFloat
. Note, we check the fact that an image is initialized using the bAllocated
value:
void testApp::update(){ video.update(); //Decode the new frame if needed //Do computing only if the new frame was obtained if ( video.isFrameNew() ) { //Store the previous frame, if it exists till now if ( grayImage.bAllocated ) { grayImagePrev = grayImage; } //Getting a new frame image.setFromPixels( video.getPixelsRef() ); grayImage = image; //Convert to grayscale image //Do processing if grayImagePrev is inited if ( grayImagePrev.bAllocated ) { //Get absolute difference diff.absDiff( grayImage, grayImagePrev ); //We want to amplify the difference to obtain //better visibility of motion //We do it by multiplication. But to do it, we //need to convert diff to float image first diffFloat = diff; //Convert to float image diffFloat *= 5.0; //Amplify the pixel values //Update the accumulation buffer if ( !bufferFloat.bAllocated ) { //If the buffer is not initialized, then //just set it equal to diffFloat bufferFloat = diffFloat; } else { //Slow damping the buffer to zero bufferFloat *= 0.85; //Add current difference image to the buffer bufferFloat += diffFloat; } } } }
Finally, the testApp::draw()
function draws four images (from left to right and from top to bottom). These four images are as follows:
grayImage
.diffFloat
image, which is the absolute difference of the current and the previous frames amplified for better visibility.bufferFloat
.draw()
function. The pixel is regarded as a motion pixel if its value in bufferFloat
exceeds the threshold value 0.9
.Let's take a look at the code:
void testApp::draw(){
ofBackground( 255, 255, 255 ); //Set the background color
//Draw only if diffFloat image is ready.
//It happens when the second frame from the video is obtained
if ( diffFloat.bAllocated ) {
//Get image dimensions
int w = grayImage.width;
int h = grayImage.height;
//Set color for images drawing
ofSetColor( 255, 255, 255 );
//Draw images grayImage, diffFloat, bufferFloat
grayImage.draw( 0, 0, w/2, h/2 );
diffFloat.draw( w/2 + 10, 0, w/2, h/2 );
bufferFloat.draw( 0, h/2 + 10, w/2, h/2 );
//Draw the image motion areas
//Shift and scale the coordinate system
ofPushMatrix();
ofTranslate( w/2+10, h/2+10 );
ofScale( 0.5, 0.5 );
//Draw bounding rectangle
ofSetColor(0, 0, 0);
ofNoFill();
ofRect( -1, -1, w+2, h+2 );
//Get bufferFloat pixels
float *pixels = bufferFloat.getPixelsAsFloats();
//Scan all pixels
for (int y=0; y<h; y++) {
for (int x=0; x<w; x++) {
//Get the pixel value
float value = pixels[ x + w * y ];
//If value exceed threshold, then draw pixel
if ( value >= 0.9 ) {
ofRect( x, y, 1, 1 );
//Rectangle with size 1x1 means pixel
//Note, this is slow function,
//we use it here just for simplicity
}
}
}
ofPopMatrix(); //Restore the coordinate system
}
}
Run the example. You will see an animation consisting of four images on the screen: the current movie frame in grayscale, absolute difference of the current and the previous frame, the accumulated buffer, and finally, the motion areas shown in black color:
You can observe that all the operations in the example can be done using pixel-by-pixel processing, described in Chapter 5, Working with Videos. So, why use complicated stuff like OpenCV? The answer is that although the example is very simple, when you do more complicated image processing and then use pixel-by-pixel programming, it makes the code cumbersome. When using OpenCV, you can do most of the image operations with a single line of code. Also, OpenCV uses various optimizations, so using it usually improves the performance of your project.
Now we will consider the control parameters of the motion detection algorithm.
There are three parameters of the motion detection algorithm:
5.0
is in the following line of the update()
function:diffFloat *= 5.0; //Amplify pixel values
Change the parameter to 1.0
and 10.0
to see the decrease and increase in the brightness of the second image. (Note that the parameter affects the third and fourth image as well).
0.85
is in the following line of the update()
function:bufferFloat *= 0.85;
Increase or decrease the parameter for slower or faster damping correspondingly; for example, change the parameter value to 0.8
and 0.95
to see results on the third image. (Note, the parameter affects the fourth image too).
0.9
for motion area detection is in the following line of the draw()
function:if ( value >= 0.9 ) {
Increasing or decreasing the value leads to a corresponding decrease or increase in the sensitivity of the algorithm, that is, the detected area becomes larger or smaller correspondingly. Try to change the parameter to 0.3
and 2.0
and see the result on the fourth image.
The second and third images on the screen are the result of drawing float-valued images diffFloat
and bufferFloat
. As we discussed earlier, pixel values of these images are mapped from the range [0, 1] to 0 to 255 while drawing. So, all the pixel values greater than 1.0 are rendered in white color. In our case, pixels of the bufferFloat
image can have values greater than 1.0, so its image on the screen is clamped in the sense of color representation. To reduce the color clamping on the screen, decrease the amplification parameter.
It is easy to change the previous example to search motion areas in a live video from a camera. That is, find the following line in testApp.h
:
ofVideoPlayer video; //Declare the video player object
Replace the preceding line with the following:
ofVideoGrabber video; //Video grabber from the camera
Now, find the following lines in testApp.cpp
in the testApp::setup()
function:
video.loadMovie( "handsTrees.mov" ); //Load the video file video.play(); //Start the video to play
Replace the preceding lines with the following:
video.initGrabber( 640, 480 ); //Start the video grabber
For details on capturing images from cameras, see the Processing a live video from the camera section in Chapter 5, Working with Videos.
When you run the example with a live video, you will possibly find that the motion has not tracked so well as in the prerecorded video handsTrees.mov
. In this case, you need to adjust the algorithm's parameters, see the Discussing the algorithm's parameters section.
We have considered the algorithm of motion detection, which is great for learning the basics of computer vision and making simple interactive installations. Though, it has a drawback in that you need to adjust its parameters depending on the light conditions. Also, the result of detection depends on the colors of the objects.
There are several ways to avoid this problem:
c/bgfg_segm.cpp
(you can find the example in OpenCV's Github repository). Currently, this algorithm is not integrated in openFrameworks, so you need to adopt the code in your project on your own.Now we will consider the methods of image filtering, which include smoothing and other essential image processing operations.