Now that you have seen both the methods, it is time to try and reconstruct the first frame of the pair, using the second one and the optical flow field. We will try to accomplish that by a straightforward method called interpolation. This method actually tries to estimate unknown data, given a known set of data. In our case, the known set of data comprises the pixel values in the second frame of the pair and the motion vectors. We will repeat the same procedure for both the Horn-Schunck and Lucas-Kanade methods.
- We start by loading our video:
>> clear all; >> videoObj = VideoReader('atrium.avi'), % Open video - Now, we'll create two system objects, one for each optical flow method:
>> ofHS = vision.OpticalFlow('ReferenceFrameDelay', 1,... 'Method','Horn-Schunck', 'OutputValue', 'Horizontal and vertical components in complex form'), % Horn Schunck method >> ofLK = vision.OpticalFlow('ReferenceFrameDelay', 1,... 'Method','Lucas-Kanade', 'OutputValue', 'Horizontal and vertical components in complex form'), % Lucas-Kanade method - Perform the optical flow estimation with both the methods:
>> for i = 89:90 % For frames 89 and 90 frame = read(videoObj,i); % Load one frame at a time temp = rgb2gray(frame); % Convert frame to grayscale im(:,:,i-88) = single(temp); % Convert frame to single (for calculations) hs(:,:,i-88) = step(ofHS, im(:,:,i-88)); %Estimate HS optical flow lk(:,:,i-88) = step(ofLK, im(:,:,i-88)); %Estimate LK optical flow end
- To make our jobs easier, we will create the x and y motion matrices from the optical flow results (the second one estimated):
>> xHS = real(hs(:,:,2)); >> yHS = imag(hs(:,:,2)); >> xLK = real(lk(:,:,2)); >> yLK = imag(lk(:,:,2));
- We will also create the absolute motion matrices:
>> absHS = abs(hs(:,:,2)); >> absLK = abs(lk(:,:,2));
- Now, it is time to perform the interpolation for both the results. First, we will make the grid for the coordinates of the pixels, using
meshgrid:>> [x,y]=meshgrid(1:videoObj.Width ,1:videoObj.Height);
- This is the time to perform our warping, that is, the interpolation that will re-create the first frame from our pair. We will do this twice, once for every optical flow method:
>> warpHS=interp2(x,y,im(:,:,2),x+xHS,y+yHS); % Warped H-S >> warpLK=interp2(x,y,im(:,:,2),x+xLK,y+yLK); % Warped L-K
- Finally, we will display our results from the warping process next to the original first frame that we are trying to re-create. First, we convert every result back to type
uint8:>> im = uint8(im); >> warpHS = uint8(warpHS); >> warpLK = uint8(warpLK); >> subplot(2,2,1), imshow(warpHS), title('Horn-Schunck Warp') >> subplot(2,2,3), imshow(im(:,:,2)), title('Original frame') >> subplot(2,2,2), imshow(warpLK), title('Lucas-Kanade Warp')
- It is worth displaying the result in the areas around the two moving persons, so that we get a better idea of the power of each algorithm:
>> figure,subplot(2,3,1), imshow(warpHS(181:274,40:114)), title('Horn-Schunck Warp') >> subplot(2,3,2), imshow(im(181:274,40:114,2)), title('Original frame') >> subplot(2,3,3), imshow(warpLK(181:274,40:114)), title('Lucas-Kanade Warp') >> subplot(2,3,4), imshow(warpHS(91:170,383:424)), title('Horn-Schunck Warp') >> subplot(2,3,5), imshow(im(91:170,383:424,2)), title('Original frame') >> subplot(2,3,6), imshow(warpLK(91:170,383:424)), title('Lucas-Kanade Warp')
This example has demonstrated the quality of the two optical flow algorithms included in MATLAB, in terms of frame reconstruction. This time, we repeated the process presented in the previous example, for both the methods. After acquiring the results and isolating the vertical, horizontal, and absolute motion values, we performed warping, using two-dimensional interpolation. The results were then demonstrated, proving the superiority of the Lucas-Kanade optical flow algorithm over the Horn-Schunck one, at least for the example examined here and for the default settings for the optical flow implementations. However, even though the Lucas-Kanade method achieved better results, it as far from ideal. The ghosting effect as more apparent, especially in the case of the faster moving person.
It is now your turn to experiment some more. First, you should thoroughly read the help page of the vision.OpticalFlow system object, either inside MATLAB, or at http://www.mathworks.com/help/vision/ref/vision.opticalflowclass.html.
Next, you should try to implement different versions of the Horn-Schunck and the Lucas-Kanade algorithms by tweaking the settings. Some settings that you should pay attention to are the smoothness setting, the iteration count for the Horn-Schunck algorithm, and the temporal gradient filter for the Lucas-Kanade algorithm.