Multilayer Multimodal Fusion for Video Classification Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video classification techniques using deep neural networks face challenges in recognizing unconstrained videos due to task-dependent video representations, multiple streams of information, and large intra-class variations, with short-term information being insufficient for capturing complex events.
Innovation Solution
A multilayer and multimodal fusion framework for deep neural networks that processes training video image data using multiple layers of convolutional neural networks (CNNs) and combines different modalities of input data to generate classification output, employing spatial discriminative factors to compute modified aggregated feature descriptors for improved classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional deep neural networks are used for video classification, then feature extraction is achieved, but classification accuracy deteriorates due to insufficient discriminative capability of extracted features
Solution Approach 1:
The patent segments the video classification task into multiple sub-tasks by processing different temporal segments (short-term and long-term) separately through distinct neural network paths, then fusing their outputs. This allows each path to specialize in capturing specific temporal patterns, improving overall classification accuracy while maintaining feature discriminative capability.
Solution Approach 2:
The patent introduces a temporal dimension to feature extraction by processing video frames at multiple temporal resolutions and aggregating features across time. This transforms 2D spatial features into 3D spatio-temporal features, enhancing the discriminative capability and enabling better classification accuracy.
2Device complexity
If short-term information is used for video classification, then computational complexity is reduced, but classification accuracy deteriorates due to insufficient temporal context
Solution Approach 1:
The patent divides the temporal processing into separate modules: one for short-term information extraction and another for long-term temporal evolution modeling. This segmentation allows the system to process information efficiently while capturing both immediate and temporal context, resolving the contradiction between complexity and accuracy.
Solution Approach 2:
The patent performs preliminary feature extraction and representation learning on short-term video segments before performing more computationally intensive temporal evolution analysis. This preliminary action reduces the complexity of subsequent processing while maintaining the benefits of temporal context for accurate classification.
3Measurement precision
If multiple streams of information are processed simultaneously, then classification accuracy is improved, but device complexity increases
Solution Approach 1:
The patent merges multiple information streams (spatial features, temporal features, action features, object features) through a unified fusion mechanism that integrates their representations. This combining approach maintains the accuracy benefits of processing multiple streams while reducing the complexity of managing them separately through shared computational resources.
Solution Approach 2:
The patent designs a universal feature fusion module that can handle multiple types of information streams (visual, temporal, semantic) through a single integrated architecture. This multi-functional design improves accuracy by considering all information streams while avoiding the complexity of separate specialized processors for each stream type.
Data Source
AI summary
A method, computer readable medium, and system are disclosed for classifying video image data. The method includes the steps of processing training video image data by at least a first layer of a convolutional neural network (CNN) to extract a first set of feature maps and generate classification output data for the training video image data. Spatial classification accuracy data is computed based on the classification output data and target classification output data and spatial discrimination factors for the first layer are computed based on the spatial classification accuracies and the first set of feature maps.