Video Action Segmentation With Local-Global Attention Transformers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing action segmentation methods struggle to capture long-term context in video sequences while efficiently managing computational resources, leading to suboptimal performance and prohibitive computational demands, especially in long videos.
Innovation Solution
A method that combines local and global attention transformers to process local and non-successive frames, allowing for efficient capture of long-term context without excessive resource consumption, using scaled-dot product attention to generate enhanced feature maps for action classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If attention transformers process the whole video sequence simultaneously to capture long-term context, then the long-term context modeling capability is improved, but computational requirements become prohibitive
Solution Approach 1:
The patent divides the video sequence into multiple local temporal windows and processes each window separately through attention transformers. This segmentation allows the model to capture long-term context across different windows while reducing the computational burden of processing the entire sequence simultaneously. The local windows are then aggregated to form the global representation.
2Loss of information
If the size of local attention window is increased to capture more context, then the contextual information is improved, but computational resources are exceeded
Solution Approach 1:
The patent segments the video sequence into multiple local temporal windows of manageable size, allowing attention transformers to process each window efficiently. This segmentation strategy enables the model to capture contextual information within each window while avoiding the computational explosion that would result from processing the entire sequence in one large window.
Solution Approach 2:
The patent transitions from a single-dimension approach (one large attention window) to a multi-dimensional approach (multiple smaller windows arranged temporally). By organizing windows in sequence and aggregating their outputs, the model achieves comprehensive context coverage without the computational complexity of a single large window.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
A computer-implemented method for action segmentation of a video sequence, the method comprising: selecting a plurality of local segments (113) from the video sequence, each local segment comprising a plurality of successive frames of the video sequence; (S12) processing each local segment with a first attention transformer (114) in order to generate first enhanced feature maps (116) modelling relations between local segments (113); (S20) selecting a plurality of global segments (123) from the first enhanced feature maps (116), each global segment comprising a plurality of non-successive frames of the first enhanced feature maps (116); (S22) processing each global segment with a second attention transformer (124) in order to generate second enhanced feature maps (126) modelling relations between global segments (123); and (S30) assigning an action class to each frame of the video sequence based on the second enhanced feature maps (126).