Multimodal Remote Sensing Fusion with Dedicated-Shared Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing remote sensing data interpretation methods are limited to fixed combinations of modal data, failing to efficiently interpret data from any combination of modalities, which reduces the efficiency of Earth observation.
Innovation Solution
A fusion method for multimodal remote sensing data using dedicated and shared architecture, involving data correction processing with dedicated networks to align different modal types, confidence value determination with shared feature association networks, and replacement processing to generate fused remote sensing data with improved accuracy and richness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing remote sensing data interpretation methods are used, then fixed combinations of modal data can be interpreted, but the system fails to handle any combination of modalities and reduces Earth observation efficiency
Solution Approach 1:
The patent implements a universal fusion network architecture that can process any combination of remote sensing modalities (optical, SAR, hyperspectral, etc.) through a unified framework. The network uses modality-agnostic feature extraction layers followed by configurable fusion modules, allowing the same system to handle diverse modal combinations without requiring separate specialized models for each modality type, thereby achieving both versatility and efficiency
2Measurement precision
If data correction processing is performed using dedicated networks for each modal type, then spatial and channel dimension alignment is achieved, but the processing complexity increases
Solution Approach 1:
The patent divides the data correction process into distinct dedicated network modules for each remote sensing modality type. Each modality-specific network performs targeted correction operations (spatial alignment, channel normalization) optimized for its particular data characteristics. This segmentation allows precise correction for each modality while maintaining modular architecture that manages complexity through clear separation of concerns
Solution Approach 2:
The patent applies modality-specific correction strategies tailored to the unique characteristics of each remote sensing type. Optical data receives corrections optimized for spectral properties, SAR data gets corrections for radar-specific artifacts, and hyperspectral data undergoes corrections suited to its high-dimensional spectral space. This local quality approach ensures each modality is corrected with appropriate precision while the overall system handles multiple modalities through specialized local processing
3Reliability
If confidence value determination is performed using shared feature association networks, then the representation of ground feature information is enhanced, but the computational load increases
Solution Approach 1:
The patent merges the confidence value determination functionality into a shared feature association network that is jointly trained with the main fusion task. Instead of separate confidence estimation models, the system uses a unified neural network that performs both feature association and confidence prediction in an integrated manner, sharing computational resources and parameters between tasks to reduce overall computational energy consumption while maintaining reliable confidence values
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
The present disclosure relates to the field of remote sensing data processing, and in particular to a fusion method for multimodal data based on dedicated and shared architecture. The method includes: obtaining multiple pieces of initial remote sensing data of a region of interest, performing data correction processing using a dedicated network to generate intermediate remote sensing data, obtaining a confidence value using a shared feature association network, performing replacement processing based on the confidence value to generate sub fused remote sensing data, and generating fused remote sensing data of the region of interest based on all pieces of sub fused remote sensing data and modal weights. With the present disclosure, data correction processing is performed on each of the pieces of initial remote sensing data and fusion processing is performed on the remote sensing data having different modal types, any combination of modal remote sensing data can be used, and fused remote sensing data with higher richness and accuracy in representing the ground feature information is generated, improving the availability of the fused remote sensing data to facilitate the fused remote sensing data adapting to various remote sensing interpretation tasks, and improving the efficiency of earth observation.