In this chapter, we are going to learn how to stitch multiple images of the same scene together to create a panoramic image.
By the end of this chapter, you will know:
In the last chapter, we learned how to extract keypoints using various methods. The reason that we extract keypoints is because we can use them for image matching. Let's consider the following image:
As you can see, it's the picture of a school bus. Now, let's take a look at the following image:
The preceding image is a part of the school bus image and it's been rotated anticlockwise by 90 degrees. We could easily recognize this because our brain is invariant to scaling and rotation. Our goal here is to find the matching points between these two images. If you do that, it would look something like this:
Following is the code to do this:
import sys import cv2 import numpy as np def draw_matches(img1, keypoints1, img2, keypoints2, matches): rows1, cols1 = img1.shape[:2] rows2, cols2 = img2.shape[:2] # Create a new output image that concatenates the two images together output_img = np.zeros((max([rows1,rows2]), cols1+cols2, 3), dtype='uint8') output_img[:rows1, :cols1, :] = np.dstack([img1, img1, img1]) output_img[:rows2, cols1:cols1+cols2, :] = np.dstack([img2, img2, img2]) # Draw connecting lines between matching keypoints for match in matches: # Get the matching keypoints for each of the images img1_idx = match.queryIdx img2_idx = match.trainIdx (x1, y1) = keypoints1[img1_idx].pt (x2, y2) = keypoints2[img2_idx].pt # Draw a small circle at both co-ordinates and then draw a line radius = 4 colour = (0,255,0) # green thickness = 1 cv2.circle(output_img, (int(x1),int(y1)), radius, colour, thickness) cv2.circle(output_img, (int(x2)+cols1,int(y2)), radius, colour, thickness) cv2.line(output_img, (int(x1),int(y1)), (int(x2)+cols1,int(y2)), colour, thickness) return output_img if __name__=='__main__': img1 = cv2.imread(sys.argv[1], 0) # query image (rotated subregion) img2 = cv2.imread(sys.argv[2], 0) # train image (full image) # Initialize ORB detector orb = cv2.ORB() # Extract keypoints and descriptors keypoints1, descriptors1 = orb.detectAndCompute(img1, None) keypoints2, descriptors2 = orb.detectAndCompute(img2, None) # Create Brute Force matcher object bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True) # Match descriptors matches = bf.match(descriptors1, descriptors2) # Sort them in the order of their distance matches = sorted(matches, key = lambda x:x.distance) # Draw first 'n' matches img3 = draw_matches(img1, keypoints1, img2, keypoints2, matches[:30]) cv2.imshow('Matched keypoints', img3) cv2.waitKey()
In the preceding code, we used the ORB detector to extract the keypoints. Once we extracted the keypoints, we used the Brute Force matcher to match the descriptors. Brute Force matching is pretty straightforward! For every descriptor in the first image, we match it with every descriptor in the second image and take the closest one. To compute the closest descriptor, we use the Hamming distance as the metric, as shown in the following line:
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
You can read more about the Hamming distance at https://en.wikipedia.org/wiki/Hamming_distance. The second argument in the preceding line is a Boolean variable. If this is true, then the matcher returns only those keypoints that are closest to each other in both directions. This means that if we get (i, j) as a match, then we can be sure that the i-th descriptor in the first image has the j-th descriptor in the second image as its closest match and vice versa. This increases the consistency and robustness of descriptor matching.
Let's consider the following line again:
matches = bf.match(descriptors1, descriptors2)
Here, the variable matches is a list of DMatch objects. You can read more about it in the OpenCV documentation. We just need to quickly understand what it means because it will become increasingly relevant in the upcoming chapters. If we are iterating over this list of DMatch objects, then each item will have the following attributes:
Now that we know how to access different attributes of the matcher object, let's see how we can use them to draw the matching keypoints. OpenCV 3.0 provides a direct function to draw the matching keypoints, but we will not be using that. It's better to take a peek inside to see what's happening underneath.
We need to create a big output image that can fit both the images side by side. So, we do that in the following line:
output_img = np.zeros((max([rows1,rows2]), cols1+cols2, 3), dtype='uint8')
As we can see here, the number of rows is set to the bigger of the two values and the number of columns is simply the sum of both the values. For each item in the list of matches, we extract the locations of the matching keypoints, as we can see in the following lines:
(x1, y1) = keypoints1[img1_idx].pt (x2, y2) = keypoints2[img2_idx].pt
Once we do that, we just draw circles on those points to indicate their locations and then draw a line connecting the two points.