For this example, we will use the shaky_car.avi video file included in the Computer Vision System Toolbox of MATLAB. In the first frame, we will manually choose an area in which we will be looking for feature points. Then, we will try to track these feature points in the rest of the frames in the video. Finally, the coordinate differences between the points we track will be used for motion compensation. Let's start:
- The first step will be, as always, to load our video:
>> videoObj = VideoReader('shaky_car.avi'), - Now, we will read a frame:
>> frame = read(videoObj,1);
- Next, we choose a rectangular area:
>> imshow(frame); >> ROI=round(getPosition(imrect))
- We can also show our selected area of interest (we will use the preceding car):
>> im = insertShape(frame, 'Rectangle', ROI, 'Color', 'red'), >> figure; imshow(im);
- It is the time to detect and show the feature points in the area we chose. We will use the
FASTcorner detector:>> points = detectFASTFeatures(rgb2gray(frame), 'ROI', ROI); >> im = insertMarker(im, points.Location, '+', 'Color', 'white'), >> figure, imshow(im);
- This is the point where we initialize the
trackerfunction for the feature points and theMarkerInserterfunction for drawing the points we track:>> tracker = vision.PointTracker('MaxBidirectionalError', 1); >> initialize(tracker, points.Location, frame); >> markerInserter = vision.MarkerInserter('Shape','Plus','BorderColor','White'), - Finally, we must loop through all the frames in our video and track the features inside our region of interest (
ROI). The result will be saved in a new matrix, initialized right before theforloop:>> vOut = zeros(videoObj.Height,videoObj.Width,3,videoObj.NumberOfFrames); >> for i = 1:videoObj.NumberOfFrames % For all frames frame = read(videoObj,i); % Load a frame [points, validity] = step(tracker, frame); % Track points out = insertMarker(frame, points(validity, :), '+'), % Draw points end >> vOut = uint8(vOut); % Convert vOut to uint8 for displaying
- Let's see if our method proved to be robust by displaying a montage of thefirst 18 frames:
>> montage(uint8(vOut(:,:,:,1:18)),'Size',[3 6])
- Since we are sure that our feature tracking is successful and robust, we can use the coordinates of the feature points to stabilize the video. All we have to do is to shift the horizontal and vertical dimensions of the second frame of each pair, by the mean amount of pixels our features have moved to each direction (relative to the first frame):
>> stab = zeros(videoObj.Height,videoObj.Width,3,videoObj.NumberOfFrames); >> for i = 1:videoObj.NumberOfFrames % For all frames frame = read(videoObj,i); % Load a frame if i == 1 [points, validity] = step(tracker, frame); % Track points pFirst = points(validity,:); % Save coordinates stab(:,:,:,i) = frame; % Store 1st frame else [points, validity] = step(tracker, frame); % Track points % Calculate shifting values sh = round(mean(points(validity,:)-pFirst)); stab(:,:,:,i) = circshift(frame,[sh(1) sh(2) 0 0]); end >> stab = uint8(stab); % Convert stab to uint8 for displaying
- At this point, we can check to see if the result is acceptable, by playing back the video matrix
stab:>> implay(stab)
Note
Our results are only a little bit better. This is because our method of shifting by the mean value is very simplistic and does not really ensure the stabilization of the video. A more sophisticated, yet beyond the scope of this book, way to accomplish this is included in MATLAB demos and can also be found at http://www.mathworks.com/help/vision/ug/feature-detection-extraction-and-matching.html#btj3w6s.
You have just been introduced to basic methods of feature point tracking and a first attempt of its usage in video stabilization. The first two steps involved the usual opening of a video file. In steps 3 and 4, we chose and displayed a region of interest for our feature tracking process. Step 5 detected the features residing in the ROI in the first frame and then we use step 6 to initialize a tracker object. In step 7, a loop through all the frames of the video, performed tracking of the feature points and drew them in each frame. All the annotated frames are saved in a new matrix called
vOut. The first 18 frames of this matrix are displayed in a montage in step 8. Finally, in step 9, we attempted a very simplistic stabilization method by shifting the dimensions of each frame, by the mean number of pixels our feature points have moved towards each direction compared to their positions in the first frame. The result of this method was displayed in step 10, to come to a realization that the simplicity of the method had a very negative effect on its efficiency.
As we mentioned before, MATLAB comes with a much more complicated, yet also much more efficient stabilization method. For the sake of challenging yourself, why don't you try to go to the link provided in the previous example and alter the feature detection process of the method to use alternative methods? The original demo uses the detectFASTFeatures function also, but on the entire image. It also introduces various complicated concepts, such as geometric transformation, but you shouldn't mess with that at this point. Your goal will be to make a function using the code you'll find in the URL, which will be able to work with other videos as well and also use the other feature detection methods (according to the choice the user provides as input):
detectHarrisFeaturesdetectMinEigenFeaturesdetectMSERFeaturesdetectSURFFeatures