Multi-Frame Attention for Continuous Video Action Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing action recognition methods in videos suffer from low accuracy due to the discontinuity of actions when predicted based on individual frames, leading to split continuity of actions.
Innovation Solution
Perform feature extraction on multiple video frames across dimensions to obtain multi-channel feature patterns, determine attention weights based on similarities between these patterns, and use these weights to correlate actions across frames, enhancing recognition accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If action recognition is performed based on individual video frames independently, then processing simplicity is improved, but action recognition accuracy deteriorates due to loss of action continuity
Solution Approach 1:
The patent merges multiple video frames into a unified processing framework by extracting features from each frame and then fusing them through attention mechanisms. The multi-channel feature patterns from different frames are combined to maintain action continuity while enabling comprehensive recognition, thus resolving the contradiction between processing simplicity and recognition accuracy.
Solution Approach 2:
The patent introduces dynamic attention weights that adaptively adjust the importance of different video frames based on their relevance to the action being recognized. This dynamic mechanism allows the system to flexibly process frames with varying significance, maintaining continuity while optimizing processing efficiency and accuracy.
2Stability of the object's composition
If attention weights are determined based on similarity between every two multi-channel feature patterns, then action continuity is improved, but computational complexity increases
Solution Approach 1:
The patent segments the computational task by first extracting features independently for each video frame to create multi-channel feature patterns, and then applying attention mechanisms to weigh these segmented features. This segmentation approach maintains action continuity through proper weighting while reducing overall computational complexity compared to processing all frames simultaneously.
3Adaptability or versatility
If feature extraction is performed across multiple dimensions for each video frame, then recognition capability is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary feature extraction across multiple dimensions for each video frame before applying the attention mechanism. By pre-processing and extracting comprehensive features in advance, the system enhances recognition capability while the subsequent attention-based filtering reduces the time required for final action determination.
Data Source
AI summary
The present subject matter discloses an action recognition method, apparatus and device, a storage medium, and a computer program product, belonging to the field of image recognition. Multiple video frames in a target video are obtained. Feature extraction is performed on the multiple video frames respectively according to multiple dimensions to obtain multiple multi-channel feature patterns. Each video frame corresponds to one multi-channel feature pattern. Each channel represents one dimension. An attention weight of each multi-channel feature pattern is determined based on a similarity between every two multi-channel feature patterns. The attention weight is used for representing a degree of correlation between a corresponding multi-channel feature pattern and an action performed by an object in the target video. A type of the action is determined based on the multiple multi-channel feature patterns and the determined multiple attention weights.