Spatiotemporal Enhancement Network for Low-Cost Video Action Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video action recognition methods face challenges such as insufficient long-term modeling ability, high computational cost, and poor representation of spatial and temporal features, leading to suboptimal performance and efficiency.
Innovation Solution
A spatiotemporal enhancement network (STEN) based on MobileNet V2 is introduced, incorporating a spatiotemporal enhancement module to enhance spatial and temporal feature extraction, using a lightweight architecture with a residual block and 1D convolutional kernels to improve recognition accuracy and reduce computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If two-stream structure with optical flow is used, then action recognition accuracy is improved, but computational cost increases
Solution Approach 1:
The patent extracts only the essential temporal information from video data by using 3D convolution kernels that operate directly on RGB frames, eliminating the need for separate optical flow computation while retaining the temporal modeling capabilities needed for action recognition
Solution Approach 2:
The patent uses 3D convolution kernels to directly capture temporal dynamics from RGB video frames, creating an efficient copy of the temporal information extraction process that avoids the computationally expensive optical flow calculation step while maintaining recognition performance
2Loss of information
If 3D convolutional neural network is used, then temporal information extraction is improved, but computational cost and processing time increase
Solution Approach 1:
The patent segments the video processing task into efficient 3D convolution operations that process temporal and spatial information in an integrated manner, avoiding the computational burden of full 3D CNN architectures while maintaining temporal information extraction capabilities
Solution Approach 2:
The patent changes the computational parameters by using optimized 3D convolution kernel sizes and configurations that reduce the number of operations required while preserving the ability to extract meaningful temporal patterns from video sequences
3Productivity
If 2D convolutional neural network is used, then computational efficiency is improved, but long-term temporal modeling ability deteriorates
Solution Approach 1:
The patent transitions from 2D convolution operations to 3D convolution operations, adding the temporal dimension to the feature extraction process. This dimensional change enables the network to capture long-term temporal dependencies while maintaining computational efficiency through optimized 3D kernel implementations
4Loss of information
If deep 3D convolutional neural network with larger receptive field is used, then temporal information extraction is improved, but device complexity increases
Solution Approach 1:
The patent employs a nested architecture where 3D convolution kernels are integrated within a streamlined network structure. The 3D kernels are nested within efficient convolutional layers that progressively build temporal understanding without requiring excessive network depth or complexity
Data Source
AI summary
The invention provides a spatiotemporal enhancement network based video action recognition method, including: S1. equally partitioning a video into T time periods, and randomly sampling one frame from each time period, to obtain an input sequence with T frames of image; S2. preprocessing the image sequence acquired in S1; S3. taking a tensor obtained in S2 as an input, inputting the tensor into a spatiotemporal enhancement network, by which spatial and temporal features are extracted; and S4. activating and normalizing the spatial and temporal features by softmax, averaging the normalized spatial and temporal features along a time dimension, obtaining classification scores of behaviors in videos through transformation, and then taking a label corresponding to the highest score as the classification result. A spatiotemporal enhancement module is embedded in a spatial network, so that better classification accuracy can be obtained by a deep learning based video behavior recognition system herein.