Optical-SAR Image Matching with Self-Supervised Global-Local Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image matching methods for synthetic aperture radar (SAR) and optical images face challenges such as poor performance in capturing deep and complex feature representations, sensitivity to nonlinear intensity differences, and reliance on shallow features, leading to noise interference and high computational demands.
Innovation Solution
A multimodal image matching method using self-supervised feature extraction to identify repetitive feature points, followed by a dual-branch network for global and local feature extraction, and a two-stage training process to enhance matching precision, including a lite-transformer and CSP modules for robust feature detection and description.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If feature-based methods (SIFT, SAR-SIFT, RIFT) are used for image matching, then matching can be performed with moderate computational requirements, but the methods rely on shallow features and cannot capture deep and complex feature representations
Solution Approach 1:
The patent segments the feature extraction process into multiple hierarchical levels using a pyramid structure. The image is divided into multiple scales (pyramid levels), and at each level, features are extracted and matched. This segmentation allows the system to capture both local detailed features and global contextual information, thereby achieving deep feature representation without overwhelming computational complexity at any single level.
Solution Approach 2:
The patent introduces a temporal dimension by utilizing video sequences instead of static images. By tracking features across multiple frames, the system extracts deep temporal features that capture motion patterns and temporal consistency. This dimensional transformation from spatial-only to spatio-temporal feature space enables deeper feature representation while maintaining computational feasibility through temporal aggregation.
2Measurement precision
If deep neural networks are used for feature extraction, then deep and complex feature representations can be captured, but computational demands increase significantly
Solution Approach 1:
The patent applies partial action by selectively processing only certain regions or features at different pyramid levels rather than exhaustively processing the entire image at maximum depth. The system performs feature extraction at multiple scales but focuses computational resources on promising regions identified at coarser levels, thereby achieving deep feature representation with reduced overall computational demand compared to full exhaustive deep processing.
Solution Approach 2:
The patent performs preliminary feature extraction and filtering at coarser pyramid levels before proceeding to finer levels. By pre-identifying candidate regions and filtering out non-promising areas at lower resolutions, the system reduces the computational burden on subsequent deeper processing stages, enabling deep feature extraction with optimized computational resource utilization.
3Measurement precision
If repetitive feature points are extracted through self-supervised learning, then matching precision improves, but the training process requires additional computational resources
Solution Approach 1:
The patent employs periodic action by training the self-supervised learning model on periodically sampled video frames rather than continuously processing all frames. The system selects representative frames at regular intervals for training, which reduces the total training time and computational resources required while still capturing sufficient temporal variations to learn robust repetitive feature points that generalize across the video sequence.
4Measurement precision
If global and local features are extracted using a dual-branch network, then feature description capability is enhanced, but device complexity increases
Solution Approach 1:
The patent merges the global and local feature extraction branches at multiple levels of the pyramid structure. Rather than maintaining completely separate processing paths throughout, the system combines features from both branches at each pyramid level, allowing the merged representation to capture both broad contextual information and fine-grained details. This merging strategy enhances feature description capability while controlling overall system complexity through structured integration.
Data Source
AI summary
Provided are a multimodal image matching method and system, terminal device, and storage medium. The method includes: performing self-supervised feature extraction on an optical image and a synthetic aperture radar (SAR) image to obtain a repetitive feature point between the optical image and the SAR image; segmenting the optical image and the SAR image into a first image block sequence based on the repetitive feature point, and performing feature extraction on the first image block sequence through a dual-branch network to obtain feature description vectors of the optical image and the SAR image respectively, where the dual-branch network includes a first branch network for extracting a global feature and a second branch network for extracting a local feature; and performing feature matching on the optical image and the SAR image based on the feature description vectors to obtain a matching point pair between the optical image and the SAR image.