Confidence-Weighted Feature Fusion for Multimodal Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multimodal fusion learning methods suffer from low accuracy in classification results due to the significant differences among various modalities, as they assign equal weights to all modalities without considering their individual contributions.
Innovation Solution
A method and apparatus that acquire confidences for each modality, representing classification and prediction probabilities, and perform weighted fusion based on these confidences to enhance the accuracy of classification and prediction results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If equal weights are assigned to all modalities in multimodal fusion learning, then the fusion process is simple, but the classification accuracy deteriorates due to significant differences among modalities
Solution Approach 1:
The patent applies local quality by assigning different weights to different modalities based on their individual characteristics and contribution to the classification task. Instead of uniform treatment, each modality receives a weight proportional to its reliability and informational value, thereby optimizing the fusion process for maximum classification accuracy while maintaining reasonable complexity
Solution Approach 2:
The patent changes the parameter of weight assignment from equal fixed values to dynamic values determined by confidence scores. The weight of each modality is adjusted according to its confidence level, allowing the system to adaptively prioritize more reliable modalities and improve overall classification performance
2Measurement precision
If confidence-based weighted fusion is performed on modalities, then classification accuracy is improved, but the system complexity increases due to additional confidence acquisition and weight calculation
Solution Approach 1:
The patent implements self-service by having each modality automatically provide its own confidence score, which is then used to determine its weight in the fusion process. The system uses the inherent quality indicators of each modality to self-regulate their contribution, reducing the need for external intervention and complex manual tuning
Solution Approach 2:
The patent introduces feedback mechanisms where confidence scores from each modality are used to adjust weights dynamically. The system continuously evaluates the performance and reliability of each modality and adjusts the fusion weights accordingly, creating a closed-loop system that adapts to changing conditions and maintains optimal accuracy
Data Source
AI summary
The present subject matter relates to the field of machine learning and provides a classification and prediction method and apparatus, a device, a storage medium, and a computer program product. At least two pieces of data are acquired for a specified classification task. Each piece of data corresponds to one modality. A confidence corresponding to each of at least two modalities is acquired. The confidence indicates a classification and prediction probability of the modality in the specified classification task. Weighted fusion on data features of the at least two pieces of data is performed based on the confidence corresponding to each of the at least two modalities to obtain a fused feature. A prediction is performed according to the fused feature to obtain a classification and prediction result corresponding to the specified classification task.