Multimodal Image Co-Registration Using Gabor-Jet Feature Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image co-registration techniques for multi-modal images, such as SAR and optical images, lack accuracy and efficiency due to their focus on either spatial or temporal domains, failing to exploit heterogeneous features and leading to significant false matches.
Innovation Solution
A deep-learning based method using a multi-input-output encoder-decoder network, specifically a Siamese U-Net architecture, combined with a Gabor-Jet model for feature extraction and similarity matching, to determine similarity scores between multi-modal images, enabling accurate co-registration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If conventional template matching techniques are used for multi-modal image co-registration, then the process can be automated, but the matching accuracy deteriorates due to failure in exploiting heterogeneous features
Solution Approach 1:
The patent transforms multi-modal images into a unified frequency domain representation using Gabor wavelets, changing the parameter space from spatial domain heterogeneous features to frequency domain homogeneous features. This allows automated template matching to work effectively across different modalities (SAR and optical) by converting them into comparable Gabor jet descriptors.
Solution Approach 2:
The patent introduces Gabor wavelets as an intermediary transformation that bridges the gap between heterogeneous SAR and optical images. By converting both modalities into Gabor jet features in the frequency domain, it creates a common intermediate representation that enables accurate automated matching without requiring manual feature design.
2Productivity
If FFT algorithm is used for fast image matching, then the processing speed is improved, but the matching accuracy deteriorates due to consideration of global features only
Solution Approach 1:
The patent segments the image matching process into local Gabor jet feature extraction followed by frequency domain correlation using FFT. By dividing the image into local patches and extracting Gabor jet descriptors for each patch, it maintains both speed (through FFT) and accuracy (through local feature representation).
Solution Approach 2:
The patent transitions from spatial domain matching to frequency domain matching using Gabor wavelets and FFT. By transforming the problem into the frequency domain, it achieves fast correlation computation while preserving local structural information through the Gabor jet representation, thus resolving the speed-accuracy tradeoff.
3Adaptability or versatility
If deep learning-based dot product matching is used, then homogeneous feature maps are obtained, but false matches increase due to intensive computation and reliance on local spatial features only
Solution Approach 1:
The patent replaces the intensive dot product computation with frequency domain correlation using FFT and Gabor wavelets. This substitution reduces computational complexity from O(n^2) to O(n log n) while improving reliability by capturing both local and global structural information through the frequency domain representation.
Solution Approach 2:
The patent moves from spatial domain dot product matching to frequency domain correlation matching. By transforming images into the frequency domain using Gabor wavelets and FFT, it achieves both computational efficiency and improved match reliability by capturing multi-scale structural information that is lost in local spatial feature matching.
4Difficulty of detecting and measuring
If SIFT feature extraction is used, then points of interest are identified, but matching accuracy deteriorates due to use of homogeneous image features for heterogeneous modalities
Solution Approach 1:
The patent changes the feature extraction parameter space from spatial domain gradient information (SIFT) to frequency domain Gabor wavelet coefficients. This parameter transformation enables effective feature extraction for both SAR and optical images by capturing texture and structural information in the frequency domain, where heterogeneous modalities can be represented homogeneously.
Solution Approach 2:
The patent introduces Gabor wavelets as an intermediary that bridges heterogeneous SAR and optical images. Instead of relying on SIFT's gradient-based features that work well only for homogeneous images, the Gabor wavelet transformation provides a common feature representation that captures essential structural information across different modalities, enabling accurate matching.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
The disclosure generally relates to methods and systems for image co-registration of multi-modal temporal sensing. Conventional techniques for image co-registration of multi-modality images focus on either the spatial or temporal domain and thus are not of high accuracy and do not preserve both global and local characteristics for the matching. The present disclosure solves the technical problems in the art for image co-registration of multi-modal temporal sensing using a deep-learning based multi-input-output encoder-decoder network with a Gabor Jet Model. The deep-learning based multi-input-output encoder-decoder network is utilized for the feature extraction. A distinctive Gabor-jet layer of the Gabor Jet Model is utilized for the similarity matching. The Gabor-jet layer generates a Gabor jet graph which provides sparse feature points for matching between matching images and reference images.