Image Stitching

Chun-Wei Liu Yi-Hsin Liu

National Taiwan University

Artifact

Overview

In this project, we synthesis a single image by combining multiple images with their overlapping field of view. Typically, there are two ways of image stitching technique. One is called direct method and the other is called feature-base method. Because of the benefits in speed and information preserving, we chose feature-base method in this project.

Implementation

Data Acquisition

Our images are captured by NIKON D80 camera with 18mm lens, and the size of image are at least 1936 x 1296 resolution.

Feature Extraction

We use Multi-Scale Oriented Patches (MOPS) base on Brown et al. in [1]. However, we make some simplification and extension in our implementation, and they will be briefly described as follows.

Feature detection. The idea of interesting point detection is based on Harris Conner detector applying on blurring images. In [1], they used image pyramid in different scales to extract more feature points. In our implementation, a single layer perform empirically well, and thus we simplify the computation for better performance.

In the other hand, the crowded interesting points are not useful for image stitching. Therefore, we apply the Adaptive Non-Maximal Suppression (ANMS) to gain better distributed interesting points. This step is the bottleneck of performance. A non-uniform grid is used to speed up in our implementation.

However, the ANMS is using, some feature points with large conner response would be ignored. Some feature points with smaller conner response were selected, but they can only provide limited information. Figure 1 (Middle) show the problem: The percentage of those feature points on the turf is arose because the ANMS has limited some feature points on the dome of the baseball field. It makes the matching more difficult. A Region Select method is used here to help the detector away from those noise feature points.

Detection without ANMS Detection with ANMS Detection with region selected

Figure 1. MOPS detected feature points without ANMS (Left), with ANMS (Middle) and with Region Selected (Right).

Feature description. The feature description, as described in [1], is a 40 x 40 image frame around each feature point which is sampled into a 8 x 8 gray scale image and oriented, and normalized as the paper's suggestion. Finally, a reshaped patch with 64-dimensional vector are used as the feature description.

Feature matching. We apply two-norm as distance metrics with the threshold ( E_1NN / E_2NN ) ≤ 0.65, where E_knn is the matching distance of K-nearest neighbor. Figure 2 shows the result of matching.

Matching result

Figure 2. The matching result of image side by side.

Image Stitching

In this part we referred Brown's method [2] as follows:

Image warping. First, we warp image into spherical coordinate, and distort image with widely used formula by κ1 and κ2.

Image alignment. Our implementation can perform auto stitch, namely judging all pairs of images whether connecting. We use RANSEC procedure to estimate an affine matrix for all possible image pairs. And we compute the global alignment matrix for each input image.

Image blending. We stitch the image series by matrix alignment, blending, smoothing, and removing artifacts. The blending method in here is linear interpolation.

Bonus Part

We implement two functions to improve our system.

Recognizing panorama: For different images, our program automatically discriminated different image series.


Figure 3. Input images can be auto detected by our program.

Multi-band blending. We refer Burt in [3]. The Gaussian and Laplacian pyramids are generated for each image. We apply blending for each level of image.

Results

All results here are automatically generated. There are different black boundaries side by picture, depending on the focus of scene. There are two kind of data of our results, one by test data set, the other is our own gallery. Please click the picture and see the panorama view!

Test Data

Graphics and Imaging Laboratory of University of Washington, USA

National Taiwan University CSIE Building, Taipei, Taiwan

Gallery

Taichung Intercontinental Baseball Stadium, Taichung, Taiwan

Shin Kong Mitsukoshi (SKM) Department Store, Taipei, Taiwan

References

  1. “Multi-image matching using multi-scale oriented patches.” [Link]
    Matthew Brown, Richard Szeliski and Simon Winder.
    CVPR 2005
  2. “Recognising Panorama.” [Link]
    Matthew Brown and David Lowe.
    ICCV 2003
  3. “A Multiresolution Spline With Application to Image Mosaics.” [Link]
    Peter J. Burt and Edward H. Adelson.
    ACM Transactions on Graphics, 1983