Weighted Deep Fusion Architecture for Multi-Modal Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to multi-modal classification systems struggle to effectively integrate and exploit correlation information from heterogeneous data sources, leading to suboptimal performance due to noise, irregular sampling, and heterogeneity in the data.
Innovation Solution
The proposed automatic weighted deep fusion architecture employs a two-stage fusion approach, combining embedding level fusion and multi-modal discriminative level fusion, to capture both correlated and independent information across different modalities, and determines weights for concatenated features and representations based on their impact on prediction performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If simple concatenation operation is used to integrate multi-modal features, then the system complexity is reduced, but the ability to identify and exploit correlation information among modalities deteriorates
Solution Approach 1:
The patent segments the feature fusion process into two distinct levels: embedding level fusion and feature level fusion. This segmentation allows the system to handle different types of correlations at appropriate granularities - embedding level for basic feature integration and feature level for discriminative correlation exploitation, thereby resolving the contradiction between simplicity and correlation preservation.
Solution Approach 2:
The patent introduces a hierarchical dimension to the fusion process by operating at multiple levels (embedding level and feature level). This dimensional transformation enables the system to capture correlations that simple concatenation misses, as the hierarchical structure allows progressive refinement of feature representations while maintaining computational feasibility.
2Speed
If conventional feature selection from multiple deep architectures is used, then the processing speed is improved, but the discriminative power of multi-modal representations deteriorates
Solution Approach 1:
The patent performs preliminary action by conducting embedding level fusion before feature level fusion. This preliminary integration of embeddings from multiple modalities ensures that subsequent feature extraction operates on already-correlated representations, thereby maintaining processing speed while enhancing discriminative power through progressive correlation exploitation.
Solution Approach 2:
The patent introduces an intermediary mechanism - the embedding level fusion layer - that mediates between raw multi-modal inputs and the final feature representations. This intermediary processes and correlates features from different modalities before they reach the discriminative feature extraction stage, thereby improving both speed and discriminative power.
3Adaptability or versatility
If heterogeneous data from multiple independent sources is integrated, then the comprehensiveness of assessment is improved, but the difficulty of integration and noise handling increases
Solution Approach 1:
The patent implements a universal fusion architecture that can handle multiple modalities (images, text, audio, etc.) through the same embedding level and feature level fusion mechanisms. This multi-functional design enables comprehensive integration of heterogeneous data sources while maintaining consistent processing complexity, as the same structural framework adapts to different data types without requiring modality-specific complex processing.
Data Source
AI summary
A method, a computer program product, and a computer system fuse features for multi-modal classifications for a plurality of modality inputs. The method includes receiving a request indicative of the modality inputs to be selected. The method includes performing an embeddings level fusion operation to concatenate features from the modality inputs. The method includes performing a multi-modal discriminative feature level fusion operation that integrates feature representations learned by applying different network structures on the modality inputs. The method includes determining weights of the concatenated features and the feature representations based on a measure of the concatenated features and the feature representations indicative of affecting a final prediction performance. The method includes generating fused features for the modality inputs based on the concatenated features, the feature representations, and the weights. The method includes generating a response to the request based on the fused features. The method includes transmitting the response.