In this chapter, we are going to learn how to stitch multiple images of the same scene together to create a panoramic image.
By the end of this chapter, you will know:
- How to match keypoint descriptors between multiple images
- How to find overlapping regions between images
- How to warp images based on the matching keypoints
- How to stitch multiple images to create a panoramic image
In the last chapter, we learned how to extract keypoints using various methods. The reason that we extract keypoints is because we can use them for image matching. Let's consider the following image:
As you can see, it's the picture of a school bus. Now, let's take a look at the following image:
The preceding image is a part of the school bus image and it's been rotated anticlockwise by 90 degrees. We could easily recognize this because our brain is invariant to scaling and rotation. Our goal here is to find the matching points between these two images. If you do that, it would look something like this:
Following is the code to do this:
import sys
import cv2
import numpy as np
def draw_matches(img1, keypoints1, img2, keypoints2, matches):
rows1, cols1 = img1.shape[:2]
rows2, cols2 = img2.shape[:2]
# Create a new output image that concatenates the two images together
output_img = np.zeros((max([rows1,rows2]), cols1+cols2, 3), dtype='uint8')
output_img[:rows1, :cols1, :] = np.dstack([img1, img1, img1])
output_img[:rows2, cols1:cols1+cols2, :] = np.dstack([img2, img2, img2])
# Draw connecting lines between matching keypoints
for match in matches:
# Get the matching keypoints for each of the images
img1_idx = match.queryIdx
img2_idx = match.trainIdx
(x1, y1) = keypoints1[img1_idx].pt
(x2, y2) = keypoints2[img2_idx].pt
# Draw a small circle at both co-ordinates and then draw a line
radius = 4
colour = (0,255,0) # green
thickness = 1
cv2.circle(output_img, (int(x1),int(y1)), radius, colour, thickness)
cv2.circle(output_img, (int(x2)+cols1,int(y2)), radius, colour, thickness)
cv2.line(output_img, (int(x1),int(y1)), (int(x2)+cols1,int(y2)), colour, thickness)
return output_img
if __name__=='__main__':
img1 = cv2.imread(sys.argv[1], 0) # query image (rotated subregion)
img2 = cv2.imread(sys.argv[2], 0) # train image (full image)
# Initialize ORB detector
orb = cv2.ORB()
# Extract keypoints and descriptors
keypoints1, descriptors1 = orb.detectAndCompute(img1, None)
keypoints2, descriptors2 = orb.detectAndCompute(img2, None)
# Create Brute Force matcher object
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
# Match descriptors
matches = bf.match(descriptors1, descriptors2)
# Sort them in the order of their distance
matches = sorted(matches, key = lambda x:x.distance)
# Draw first 'n' matches
img3 = draw_matches(img1, keypoints1, img2, keypoints2, matches[:30])
cv2.imshow('Matched keypoints', img3)
cv2.waitKey()In the preceding code, we used the ORB detector to extract the keypoints. Once we extracted the keypoints, we used the Brute Force matcher to match the descriptors. Brute Force matching is pretty straightforward! For every descriptor in the first image, we match it with every descriptor in the second image and take the closest one. To compute the closest descriptor, we use the Hamming distance as the metric, as shown in the following line:
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
You can read more about the Hamming distance at https://en.wikipedia.org/wiki/Hamming_distance. The second argument in the preceding line is a Boolean variable. If this is true, then the matcher returns only those keypoints that are closest to each other in both directions. This means that if we get (i, j) as a match, then we can be sure that the i-th descriptor in the first image has the j-th descriptor in the second image as its closest match and vice versa. This increases the consistency and robustness of descriptor matching.
Let's consider the following line again:
matches = bf.match(descriptors1, descriptors2)
Here, the variable matches is a list of DMatch objects. You can read more about it in the OpenCV documentation. We just need to quickly understand what it means because it will become increasingly relevant in the upcoming chapters. If we are iterating over this list of DMatch objects, then each item will have the following attributes:
- item.distance: This attribute gives us the distance between the descriptors. A lower distance indicates a better match.
- item.trainIdx: This attribute gives us the index of the descriptor in the list of train descriptors (in our case, it's the list of descriptors in the full image).
- item.queryIdx: This attribute gives us the index of the descriptor in the list of query descriptors (in our case, it's the list of descriptors in the rotated subimage).
- item.imgIdx: This attribute gives us the index of the train image.
Now that we know how to access different attributes of the matcher object, let's see how we can use them to draw the matching keypoints. OpenCV 3.0 provides a direct function to draw the matching keypoints, but we will not be using that. It's better to take a peek inside to see what's happening underneath.
We need to create a big output image that can fit both the images side by side. So, we do that in the following line:
output_img = np.zeros((max([rows1,rows2]), cols1+cols2, 3), dtype='uint8')
As we can see here, the number of rows is set to the bigger of the two values and the number of columns is simply the sum of both the values. For each item in the list of matches, we extract the locations of the matching keypoints, as we can see in the following lines:
(x1, y1) = keypoints1[img1_idx].pt (x2, y2) = keypoints2[img2_idx].pt
Once we do that, we just draw circles on those points to indicate their locations and then draw a line connecting the two points.