Project 4: Stitching Photo Mosaics

Part A

Homographies and Warping

To create the mosaic that we want in the end, we would need to have the sequences of images transformed to their corresponding position. And for that, we need to compute the Homography matrix for transformation.

The homography transformation is given by

where

Then we can solve for $H$ by solving a least square solution to a list of coordinates:

Once we get $H$, we can created an targeted images using inverse warp.

Image Rectification: Naive result

Before we dive into the image mosaics by blending multiple images in a sequence. It is perhaps better to test if my homographies are correct. Here’s just a random photo I took to see if this warping would work.

Mosaic Result

For mosaic, I decided to calculated the average position of the coordinates in the sequence as the target, and calculate homography for each image and warp them to target. Of course, before combining them together, the size of a larger canvas is calculated based on the shift of alignment points for each image.

Here are the two images I took in Moffit 4th floor.

Here are two of a sequence of images I took at Crater lake, Oregon. To make the visual result better, I used about 12 coordinates on both mountain and trees.

In the naive version of mosaic function, it simply shift the warped images to the right position to align the target coordinates. But this isn’t good enough. To make the visual effect better, I added a mask feature to the mosaic function: for the overlap area, take the weighted average from each warped image, where the weights are calculated based on the distance to the center of warped images.

The blended mosaic looks much smoother on the edge between images.

Lets try another pair of images.

Part B

Corner Detection and Adaptive Non-Maximal Suppression

In this project, I pretty much use the provided Harris Corner Detection function. The only small modification I make is that instead of discard a fixed amount of pixels, I decided to discard 10% of pixels from all side, this is to mitigate the effect of some distortion around the edge, since some of the photos are shot with wider angle from camera lens. The issue with Harris corners is that they are all jammed together and isn’t really useful for us. So have to use ANMS as described in the MOPS paper where it calculate $r_i = \min_{j} |x_i - x_j| \quad \text{s.t.} \quad f(x_i) < c_{\text{robust}} f(x_j), \quad x_j \in I$ for all Harris corners, and pick the top K amount of coordinates as sparse interests point for future usage.

Feature Matching

Once we get the ANMS points, we want to find a way to find the most matching pairs so we could use them to calculate the transformation homography matrix $H$. At each interest point, we extract “feature” as we select a 8x8 sub-image from the original image with a 40x40 window around each interest point, using a step size of 5. Here we followed the advice from the paper, thresholding the ratio of 1st and 2nd nearest neighbors’ distance, and get the feature matching points.

RANSAC

Of course, not all feature matching points are correct matching. We have to select a few best points to compute the best homography. We implement RANSAC described in class: every time we randomly select 4 matching points, compute the homography $H$, and calculate the error of this $H$ on all available matching point. If error is acceptable, we recompute this $H$ with all the inliers. We iterate over this random selection scheme for a good number of times until we find the best $H$ with the largest amount of inliers (and we keep those inliers).

Here we can see all the RANSAC inliers are good matches in two images. Since this is a randomized algorithm, there is possibility that this doesn’t work well and return a bad homogrpahy. Just run them again and it will be good! Once we get a good homogrpahy, we can do the warp as in previous part.

More Auto Mosaic

lets try a new sequence of images I took at Colorado