OpenCV now implements the ECC algorithm, which is only available as of Version 3.0. This method estimates the geometric transformation (warp) between the input and template frames and returns the warped input frame, which must be close to the first template. The estimated transformation is the one that maximizes the correlation coefficient between the template and the warped input frame. In the OpenCV examples ([opencv_source_code]/samples/cpp/image_alignment.cpp), a related program can be found.
Note
The ECC algorithm is based on the ECC criterion of the paper Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization. You can find this at http://xanthippi.ceid.upatras.gr/people/evangelidis/george_files/PAMI_2008.pdf.
We are now going to see an example (findCameraMovement) that uses this ECC technique using the findTransformECC() function:
#include <opencv2/opencv.hpp>
using namespace cv;
using namespace std;
int findCameraMovement()
{
//1-Set the parameters and initializations
bool finish = false;
Mat frame;
Mat initial_frame;
Mat warp_matrix;
Mat warped_frame;
int warp_mode = MOTION_HOMOGRAPHY;
TermCriteria criteria(TermCriteria::COUNT | TermCriteria::EPS, 50, 0.001);
VideoCapture capture(0);
Rect rec(100,50,350,350); //Initial rectangle
Mat aux_initial_frame;
bool follow = false;
//Check if video camera is opened
if(!capture.isOpened()) return 1;
//2-Initial capture
cout << "
Press 'c' key to continue..." << endl;
while(!follow)
{
if(!capture.read(initial_frame)) return 1;
cvtColor(initial_frame ,initial_frame, COLOR_BGR2GRAY);
aux_initial_frame = initial_frame.clone();
rectangle(aux_initial_frame, rec, Scalar(255,255,255),3);
imshow("Initial frame", aux_initial_frame);
if (waitKey(1) == 99) follow = true;
}
Mat template_frame(rec.width,rec.height,CV_32F);
template_frame = initial_frame.colRange(rec.x, rec.x + rec.width).rowRange(rec.y, rec.y + rec.height);
imshow("Template image", template_frame);
while(!finish)
{
cout << "
Press a key to continue..." << endl;
waitKey();
warp_matrix = Mat::eye(3, 3, CV_32F);
//3-Read each frame, if possible
if(!capture.read(frame)) return 1;
//Convert to gray image
cvtColor(frame ,frame, COLOR_BGR2GRAY);
try
{
//4-Use findTransformECC function
findTransformECC(template_frame, frame, warp_matrix, warp_mode, criteria);
//5-Obtain the new perspective
warped_frame = Mat(template_frame.rows, template_frame.cols, CV_32F);
warpPerspective (frame, warped_frame, warp_matrix, warped_frame.size(), WARP_INVERSE_MAP + WARP_FILL_OUTLIERS);
}
catch(Exception e) { cout << "Exception: " << e.err << endl;}
imshow ("Frame", frame);
imshow ("Warped frame", warped_frame);
//Press Esc to finish
if(waitKey(1) == 27) finish = true;
}
capture.release();
return 0;
}
main()
{
findCameraMovement();
}Some screen captures are shown in the following screenshot. The left-column frames represent the initial and template frames. The upper-right image is the current frame and the lower-right image is the warped frame.
Output of the findCameraMovement example.
The code example shows you four windows: the initial template, the initial frame, the current frame, and the warped frame. The first step is to set the initial parameters as warp_mode (MOTION_HOMOGRAPHY). The second step is to check whether the video camera is opened and to obtain a template, which will be calculated using a centered rectangle. When the C key is pressed, this area will be captured as the template. The third step is to read the next frame and convert it to a gray frame. The findTransformECC() function is applied to calculate warp_matrix with this matrix, and using warpPerspective(), the camera movement can be corrected using warped_frame.