Project 4: IMAGE WARPING and MOSAICING

Overview

The goal of this project is to create a seamless mosaic image by stitching together multiple photographs taken from slightly different angles. This involves understanding and implementing key concepts in image processing, including homography estimation, image warping, and blending techniques. By completing this project, you will develop skills in computer vision and gain practical experience in manipulating and transforming images.

1. Shooting the Pictures

To initiate my project, I captured multiple photographs while keeping the center of projection fixed. This involved rotating my camera to maintain perspective. I avoided fisheye or lenses with significant barrel distortion, as they can warp straight lines, and opted for a wide-angle lens to create more engaging mosaics.

When taking the pictures, I made sure to shoot them closely in time to minimize changes in lighting and subject movement. I ensured there was a significant overlap—between 40% and 70%—between adjacent images to facilitate easier stitching. It was acceptable to adjust the zoom between shots, and I focused on capturing scenes rich in detail for better results.

2. Recovering Homographies

After capturing the images, the next step was to compute the homography that defines the transformation between pairs of images. I gathered corresponding points from the two images and wrote a function, computeH(im1_pts, im2_pts), to calculate the 3x3 homography matrix. To achieve a stable and accurate transformation, I ensured to gather more than four corresponding points, forming an overdetermined system of equations that I solved using least squares.

3. Warping the Images

With the homography matrix calculated, I proceeded to warp the images. I created a function called warpImage(im, H) that applied this matrix to warp each image into alignment with the reference image. I chose to utilize inverse warping to prevent aliasing during the resampling process, employing techniques like scipy.ndimage.map_coordinates to handle this efficiently, focusing on vectorized implementations without explicit loops.

4. Image Rectification

After validating the warping process, I tackled image rectification. This involved taking a photograph of a known rectangular object, such as a painting, and computing the homography that transforms it into a perfectly rectangular shape. This step was crucial for testing and ensuring that my homography and warping methods were functioning correctly.

5. Creating a Mosaic Image

With the images aligned, I moved on to create a mosaic. I opted to warp all images into a new projection, gradually adding them one by one. I calculated the final mosaic size beforehand and blended the images using techniques like weighted averaging or Laplacian pyramid blending to produce a seamless composite image. This approach helped minimize any artifacts that arose from direct image overlaps. Initially, I added a mask to the valid parts of each image, setting the pixel values to the original in the center and to 0 at the edges. However, no matter how I choose the mask, the output image is always very bright or very dark in some areas. Below are the masks and the generated blended image.

Then I used Laplacian pyramid blending for the images and found that the results were significantly better than before.

6. Bells & Whistles

To further enhance the effect, I projected the original coordinates onto cylindrical coordinates. To achieve this, I only needed to use the scipy.ndimage.map_coordinates function to project the transformed images onto the cylinder, resulting in the following:

Overview

The project aims to develop a system that can automatically stitch images into a mosaic. The process involves several key steps, including harris corners detection, descriptor extraction, feature matching with RANSAC algorithm, and the final mosaic creation.

Harris Corners Detection

The process of selecting effective Harris corners is divided into two steps. First, implement the function get_harris_corners to detect Harris corners in a given black-and-white image. From all detected Harris corners, select the most representative points by calculating their relative significance, which is based on the Harris response values and the distances between them. When making these calculations, consider both the distances and intensities of the corners to ensure that the returned corners are distributed more evenly across the image, making them suitable for subsequent feature matching and image processing tasks.

Feature Descriptor Extraction

We will implement a function that takes the original image and the list of detected corners. For each corner, extract a 20x20 patch, sample an 8x8 sub-patch, and normalize the resulting descriptor using bias and gain normalization techniques to ensure consistency.

Homography Estimation using RANSAC

Use the RANSAC algorithm to robustly compute the homography matrix from the matched feature points. This step involves selecting random sets of four points from the matches and estimating the homography, then checking how many points fit this model (inliers). Repeat this process multiple times to ensure a reliable estimate, filtering out outliers to enhance the stitching accuracy. Develop a function that finds matches between two sets of feature descriptors. For each descriptor in the first set, calculate distances to all descriptors in the second set, apply Lowe’s ratio test to filter out weak matches, and return a list of matched index pairs.

Results

Combine the transformed images using the computed homography to create a seamless mosaic. This involves warping the images according to the homography and blending them together. Compared to the previous results, the mosaic created using the Harris corners automatic alignment method has clearer overlaps in the images, and it does not exhibit highlights or strange wedge-like artifacts in other areas.