The methods and techniques we have presented in the previous chapters introduced you to the very basics of video processing. We did not get into the mathematical details involved in such processing and preferred to demonstrate how the theory comes into action using hands-on, practical exercises that hopefully helped you comprehend the essence of the algorithms. In this final chapter, we will try to spice things up a little, going into more complex methods that require some more mathematical insight. We will avoid going into the most difficult details, but nevertheless, this chapter may seem a little more challenging than the others, as it will demonstrate basic motion detection and estimation techniques, combine them with the notion of image registration and use them for video stabilization. Furthermore, we will indulge in the field of three-dimensional image and video processing and create our own 3-D videos.
In this chapter, we will cover:
- How we can detect motion in videos
- How motion can be estimated using optical flow
- Ways to compensate camera movement in videos using registration
- How stereoscopic videos can be easily created
So, let's dive right in!
So far, the only time we had to handle inter-frame processing was for the filtering methods for de-interlacing and smoothing of videos demonstrated in the previous chapter. These methods were not very sophisticated, as they did not aim at extracting useful information, for example, detecting and estimating motion from the video frame sequences.
Motion detection can be loosely defined as the process of detecting changes in the relative position of an object to its background (however, this background is defined) in the consecutive frames of a video. Intuitively, this process could be based on a simple subtraction of these consecutive frames, so that the values of the pixels that change appear brighter than those that remain constant. For various reasons, this simplistic approach is not always successful. In this chapter, we will discuss why.
On the other hand, motion estimation is a more challenging problem; it aims at estimating a set of vectors that describes the motions of pixels in consecutive frames. This means that ideally, a motion estimation process can perfectly describe the changes of all pixel positions from one frame of a video to the next. This description has several flaws, since a video captures the three-dimensional real world images and transforms them to fit the two-dimensional plane of a screen. You can understand that mapping all the pixels from one frame to all the pixels in the other frame is impossible, since real world scenes captured in video suffer from occlusions, lighting changes, camera movements, and so on.
Let's start our discussion on the issues of motion detection and estimation with some examples.
The first problem in motion detection is to identify which pixels change their values in consecutive frames. Usually, this is an indication that the particular pixels represent parts of moving objects, since one of the most popular constraints used in motion estimation, as we will see later in the chapter, is the brightness constancy constraint.