Video Classification Model With Non-Local Temporal Feature Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video classification methods using three-dimensional convolutional neural networks are limited by the richness of content information between frames, leading to less accurate classification due to reliance on pixel-based feature maps.
Innovation Solution
A video classification model construction method that incorporates a non-local module in a three-dimensional convolutional neural network to extract long-distance dependency features, combined with depthwise separable convolution layers to reduce parameter count and improve feature extraction, and an improved cross-entropy loss function to focus on difficult samples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pixel-based feature maps are used in three-dimensional convolutional neural networks, then the network structure is simple, but the content information richness between frames is limited, resulting in lower classification accuracy
Solution Approach 1:
The patent introduces non-local feature extraction that operates across temporal dimensions beyond immediate adjacent frames. The non-local module captures long-distance temporal dependencies by aggregating features from multiple time steps, effectively adding a temporal dimensionality to the feature extraction process that transcends the local pixel-based approach's limitations.
Solution Approach 2:
The patent segments the feature extraction process into distinct components: local pixel-based feature extraction, non-local temporal feature extraction, and fused feature representation. This segmentation allows the system to process different types of information through specialized pathways and then combine them, preserving both local detail and global temporal context.
2Productivity
If traditional convolutional layers are used for feature extraction, then the implementation is straightforward, but the computational burden is high and parameter count is large
Solution Approach 1:
The patent segments the convolutional operation into depthwise convolution and pointwise convolution stages. The depthwise convolution applies separate filters to each input channel, reducing parameter count significantly. The pointwise convolution then combines these channel-specific features, achieving feature extraction with fewer parameters while maintaining effectiveness.
Solution Approach 2:
The patent changes the parameter structure by using depthwise separable convolution instead of standard convolution. This transformation reduces the number of parameters from O(k^3 * c^2) in standard convolution to O(k^3 * c + c^2) in separable convolution, where k is kernel size and c is channel count, significantly reducing model complexity.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
Disclosed are a video classification model construction method and apparatus, a video classification method and apparatus, and a device and a medium. The video classification model construction method is executed by the computer device, and comprises: obtaining training set data, the training set data comprising multiple video frames; inputting the training set data into a classification model to be trained, said classification model comprising a plurality of convolution layer groups and an output layer group, and some of the convolution layer groups comprising non-local modules; extracting long-distance dependence features of the multiple video frames by means of the non-local modules, the long-distance dependence features being used for representing an association relationship between spatial positions contained in the multiple video frames; outputting the long-distance dependence features to the output layer group, and outputting a prediction classification result of the multiple video frames; and training said classification model by means of a preset prediction loss function, until parameters in said classification model converge, so as to obtain a video classification model.