3D Pose Attention for Low-Motion Activity Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for person activity recognition in videos fail to accurately identify activities with similar appearances and low motion, as they do not effectively exploit spatial and temporal attention mechanisms.
Innovation Solution
A method involving obtaining 3D poses from video frames, using separate spatial and temporal attention weights to modulate feature maps, and performing convolution to classify activities, utilizing neural networks like I3D, LSTM, and attention mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D convolutional neural networks (I3D) are used for activity recognition, then diverse activities can be recognized with high accuracy, but the method does not exploit the salient part of the video in time and space
Solution Approach 1:
The patent divides the video processing into separate spatial and temporal attention mechanisms. Spatial attention weights are computed to identify important spatial regions in each frame, while temporal attention weights are computed to identify important time instants. This segmentation allows the model to focus on salient parts independently in both dimensions rather than treating them jointly.
Solution Approach 2:
The patent applies local quality by creating attention weight maps that highlight specific local regions in space and time. The spatial attention mechanism generates weight maps for different spatial locations, and the temporal attention mechanism generates weights for different time steps. This allows the model to adaptively emphasize important local features while suppressing less relevant areas.
2Measurement precision
If attention mechanism is applied to focus on salient parts, then recognition of activities with similar appearances and low motion is improved, but the computational complexity increases
Solution Approach 1:
The patent segments the attention mechanism into two independent modules: spatial attention and temporal attention. This segmentation reduces complexity by allowing each module to operate independently on its respective dimension, rather than requiring a complex joint spatiotemporal attention mechanism. The spatial attention operates on spatial dimensions while temporal attention operates on the time dimension separately.
Solution Approach 2:
The patent applies partial action by using attention mechanisms selectively rather than processing all video data uniformly. The attention weights allow the model to focus computational resources only on salient spatial regions and temporal instants, rather than processing the entire video sequence with equal computational effort. This partial processing reduces overall computational complexity while maintaining accuracy.
3Measurement precision
If separate spatial and temporal attention weights are used to modulate feature maps, then activities with similar appearances and low motion are recognized more accurately, then the processing time increases
Solution Approach 1:
The patent segments the attention computation into parallel spatial and temporal branches. Spatial attention weights are computed based on spatial features while temporal attention weights are computed based on temporal features. This segmentation enables parallel processing of spatial and temporal dimensions, reducing the sequential processing time compared to a unified approach.
Solution Approach 2:
The patent applies partial action by computing attention weights only for salient regions and time instants rather than processing the entire feature map uniformly. The attention mechanism selectively focuses computational resources on important areas, reducing the effective processing volume and thereby reducing processing time despite the added complexity of attention computation.
Data Source
AI summary
A device and a method for recognizing person activity in a sequence of frames (100) comprising: obtaining a set of consecutives 3D poses (103),obtaining a feature map (102), obtaining a vector of spatiotemporal features, obtaining a matrix of spatial attention weights, obtaining a matrix of temporal attention weights (110), modulating (106) the feature map using the matrix of spatial attention weights to obtain a spatially modulated feature map, modulating (111) the feature map using the vector of temporal attention weights to obtain a temporally modulated feature map, performing a convolution (114) of the spatially modulated feature map and of the temporally modulated feature map to obtain a convoluted feature map, performing a classification (115) using the convoluted feature map so as to determine the activity of the person in the video.