Action Recognition on Resource-Constrained Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI models for action recognition in videos require prohibitively large computational resources, making them impractical for deployment on resource-constrained devices like smartphones due to high computational costs and latency, which hinders their practical usage and deployment.
Innovation Solution
The proposed method employs a Temporal Cross-Resolution (TCR) module that splits feature tensors into full and low temporal resolution streams, allowing for efficient temporal downsampling while maintaining accuracy through feature reconstruction loss, enabling a dual-pathway neural network architecture that reduces GPU memory requirements and FLOP count.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If 3D CNNs are used for action recognition, then temporal information is naturally captured, but computational cost becomes prohibitive
Solution Approach 1:
The patent segments the video processing into independent 2D CNN frame processing followed by separate temporal aggregation. Instead of using 3D convolutions that process spatiotemporal information jointly, the method divides the computation into spatial (2D CNN per frame) and temporal (aggregation layer) segments, reducing computational complexity while maintaining temporal sensitivity
Solution Approach 2:
The patent extracts temporal information separately from spatial features. Independent 2D CNNs extract spatial features from each frame, and then a temporal aggregation mechanism (such as temporal shift module or temporal pooling) extracts temporal patterns from the sequence of spatial features, separating the computational tasks of spatial and temporal analysis
2Power
If independent 2D CNN forward passes are used for each frame, then computational cost is reduced, but temporal sensitivity is lost
Solution Approach 1:
The patent introduces a temporal aggregation layer as an intermediary between independent 2D CNN frame processing and final action recognition. This intermediary component (such as temporal shift module, temporal pooling, or recurrent network) takes the spatial features from multiple frames and aggregates temporal information, enabling temporal sensitivity without requiring computationally expensive 3D convolutions
3Measurement precision
If standard single model baseline is trained, then action recognition accuracy is achieved, but training requires 4 top-spec GPUs and several days
Solution Approach 1:
The patent segments the model training into independent 2D CNN training for spatial feature extraction and separate temporal aggregation training. This segmentation allows for more efficient parallelization and reduces the computational burden per training step, enabling faster convergence with fewer GPU resources while maintaining or improving accuracy
4Measurement precision
If higher computational cost models are used, then action recognition accuracy improves, but latency and battery consumption increase
Solution Approach 1:
The patent changes the architectural parameters of the neural network to use lightweight 2D CNNs with temporal aggregation instead of heavy 3D CNNs. This parameter change (model architecture selection) achieves a favorable trade-off between accuracy and computational cost, reducing latency and battery consumption while maintaining competitive action recognition accuracy on edge devices
Data Source
AI summary
Broadly speaking, the present techniques relate to a method and apparatus for performing action recognition, and in particular to a computer-implemented method for performing action recognition on resource-constrained or lightweight devices such as smartphones. The ML model may be adjusted to achieve required accuracy and efficiency levels, while also taking into account the computational capability of the apparatus that is being used to implement the ML model. One way is to adjust the number of channels assigned to the first set of channels, i.e. the full temporal resolution channels. Another way is to adjust the point in the ML model where the temporal pooling layer or layers are applied.