Video Classification Using Temporal Audio-Visual Feature Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video classification methods fail to accurately represent video content due to their inability to consider temporal variations in both visual and audio features, leading to reduced classification accuracy.
Innovation Solution
A method that processes videos by obtaining sequences of visual-signal and acoustic-signal features using convolutional neural networks (CNN) and gated recurrent units (GRU), respectively, and synthesizes these features to generate target-signal features for improved classification predictions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If feature-averaging manner is used to transform frame-level features to video-level features, then the transformation process is simple, but temporal variations in visual and audio features are not considered, leading to reduced classification accuracy
Solution Approach 1:
The patent segments the video processing into distinct temporal stages by extracting features from individual video frames at different time points, then processes these temporal segments separately through the predicting model before aggregation, allowing temporal variations to be preserved and analyzed
Solution Approach 2:
The patent introduces dynamic temporal processing by using a predicting model that processes visual and audio features at different time points (t and t-1) separately, capturing the dynamic changes in features over time rather than treating all frames statically, thereby improving classification accuracy
2Device complexity
If only visual features are processed without audio features, then the processing complexity is reduced, but the representation of video content is incomplete, reducing classification accuracy
Solution Approach 1:
The patent merges visual and audio feature processing into a unified predicting model framework, where both modalities are extracted and processed simultaneously through coordinated sub-models, allowing the system to leverage complementary information from both visual and audio channels for improved classification accuracy
Solution Approach 2:
The patent creates a multi-functional predicting model that handles both visual and audio feature processing through shared architectural components (CNN and GRU sub-models), enabling the system to process multiple types of signal features using a universal framework that reduces overall processing complexity
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present application disclose a video classification method, comprising: acquiring a video to be processed, wherein the video comprises at least one video frame, and each video frame corresponds to one time point; acquiring, by means of a video classification prediction model, a visual signal feature sequence corresponding to the video; acquiring, by means of the video classification prediction model, an audio signal feature sequence corresponding to the visual signal feature sequence; generating a target signal feature sequence according to the visual signal feature sequence and the audio signal feature sequence; and acquiring, by means of the video classification prediction model, a classification prediction result corresponding to the target signal feature sequence, wherein the classification prediction result is for predicting a video type of the video. The embodiments of the present application further disclose a server. The embodiments of the present application take into consideration feature variation of both visual features and audio features in the time dimension, thereby improving video classification.