Method and apparatus for video recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transformer-based models for video recognition face computational constraints due to full space-time attention complexity of O(T2S2), making them impractical for resource-constrained devices and less efficient than 3D convolutional models.
Innovation Solution
The method restricts time attention to a local temporal window and uses spatial-only attention within this window, combined with efficient space-time mixing through channel shifting, reducing complexity to linear in the temporal term and maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If full space-time attention is performed across all spatial and temporal dimensions, then video recognition accuracy is improved, but computational complexity increases to O(T2S2) making it impractical for resource-constrained devices
Solution Approach 1:
The patent segments the video processing into two distinct components: spatial attention is applied to individual frames while temporal attention is applied separately to sequences of frame representations. This segmentation breaks the O(T2S2) complexity into manageable parts, allowing spatial and temporal processing to be performed independently rather than jointly across all dimensions simultaneously.
Solution Approach 2:
The patent transforms the problem from joint spatio-temporal attention to separate spatial and temporal attention mechanisms by changing the dimensional processing approach. Instead of attending across T×S dimensions simultaneously, the model processes spatial dimensions S first to get frame representations, then processes temporal dimensions T on these representations, effectively decomposing the dimensional complexity.
2Adaptability or versatility
If transformers are adapted from spatial to spatio-temporal domain by performing self-attention simultaneously across all spatial and temporal dimensions, then video understanding capability is improved, but memory and computational constraints make the model impractical
Solution Approach 1:
The patent segments video understanding into spatial feature extraction followed by temporal feature extraction. Spatial self-attention is performed on each frame independently to capture spatial relationships, then temporal self-attention is performed on the sequence of spatial representations to capture temporal dynamics. This segmentation reduces memory requirements compared to holding all spatio-temporal tokens in memory simultaneously.
Solution Approach 2:
The patent changes the dimensional processing order by separating spatial and temporal attention into distinct processing stages. Instead of a single spatio-temporal attention layer requiring O(T2S2) memory, the model uses sequential processing: first spatial attention O(S2) per frame, then temporal attention O(T2) on frame representations, reducing peak memory consumption significantly.
3Measurement precision
If each frame or patch is analyzed separately at each time point, then processing detail is improved, but the amount of processing required increases making it unsuitable for resource-constrained devices
Solution Approach 1:
The patent segments the processing task into spatial processing at each time point followed by temporal processing across time points. Spatial attention maintains detailed processing within each frame by attending across all spatial positions, while temporal attention operates on compressed frame representations rather than individual patches, reducing overall processing load while preserving necessary detail.
Solution Approach 2:
The patent changes the processing approach by separating spatial and temporal dimensions into sequential processing steps. Spatial attention operates at each time point to capture detailed spatial relationships, then temporal attention operates on the temporal dimension of frame representations. This dimensional separation avoids the combinatorial explosion of processing every patch at every time point jointly.
Data Source
AI summary
Broadly speaking, the present techniques generally relate to a method and apparatus for video recognition, and in particular relate to a computer-implemented method for performing video recognition using a transformer-based machine learning, ML, model. Put another way, the present techniques provide new methods of image processing in order to automatically extract feature information from a video.