Uncertainty Correlation Weighted Multimodal Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multimodal fusion techniques fail to effectively account for inter-modality correlations and model uncertainty, leading to sub-optimal performance in data processing applications such as medical diagnosis and autonomous vehicle object detection.
Innovation Solution
A joint multimodal fusion method that employs an uncertainty and correlation weighted (UCW) approach, where weights are assigned to prediction outputs from different machine learning models based on estimated uncertainty and correlations between modalities, using a pairwise modality correlation matrix to minimize expected error in fused predictions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing multimodal fusion techniques are used, then the system can combine data from different modalities, but the accuracy of fused predictions is sub-optimal due to failure to account for inter-modality correlations and model uncertainty
Solution Approach 1:
The patent changes the parameters of the fusion process by introducing uncertainty weights and correlation matrices that dynamically adjust the fusion based on model confidence and inter-modality relationships. Instead of fixed fusion weights, the system computes adaptive weights using uncertainty estimates from individual models and correlation information between modalities, thereby improving prediction accuracy while accounting for reliability factors.
2Adaptability or versatility
If multiple machine learning models operate on different modalities, then complementary information can be captured, but the complexity of integrating and weighting these models increases
Solution Approach 1:
The patent segments the fusion process into distinct computational steps: first computing uncertainty weights for each model based on its performance characteristics, then constructing a correlation matrix that captures inter-modality relationships, and finally applying these components to combine predictions. This segmentation allows the complex fusion task to be broken down into manageable operations that can be systematically executed.
Solution Approach 2:
The patent introduces uncertainty weights and correlation matrices as intermediary components that mediate between individual model predictions and the final fused result. These intermediaries simplify the integration process by pre-computing reliability metrics and relationship structures, which then guide the weighted combination of predictions from multiple modalities without requiring direct complex interactions between all models.
Data Source
AI summary
A joint multimodal fusion computer model architecture is provided that receives prediction output data from a machine learning (ML) computer model set comprising a plurality of different subsets of ML computer models operating on input data of different modalities and generating different prediction outputs. Prediction outputs are fused by executing an uncertainty and correlation weighted (UCW) joint multimodal fusion operation on the prediction outputs to generate a fused output providing multimodal prediction output data. The UCW joint multimodal fusion operation applies different weights to different ones of prediction outputs from the different subsets of ML computer models operating on input data of different modalities. The different weights are determined based on an estimation of uncertainty in each of the different subsets of ML computer models and an estimate of a correlation between different modalities.