Spatio-Temporal Fusion for Behavior Recognition Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Behavior recognition based on video data faces challenges with high calculation amounts and low recognition accuracy due to large weights and inefficiencies in existing methods.
Innovation Solution
A behavior recognition method that divides video data into clips, performs frame and optical flow extraction, followed by spatio-temporal convolution processing using a neural network with specific layers like ConvLIF and LIF, to reduce calculation and improve accuracy through spatial and temporal fusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional behavior recognition methods are used, then recognition accuracy can be maintained, but calculation amount and model weight become excessively large
Solution Approach 1:
The patent segments the behavior recognition process into two independent streams: spatial stream for processing frame images and temporal stream for processing optical flow images. Each stream has its own convolutional neural network structure, allowing independent optimization and reducing overall computational complexity while maintaining recognition accuracy through fused prediction results.
Solution Approach 2:
The patent extracts and processes spatial features and temporal features separately through dedicated convolutional networks. By taking out the temporal dimension processing into a separate stream that handles optical flow, the model reduces the computational burden on the main spatial recognition pathway while preserving important temporal behavior information.
2Measurement precision
If traditional behavior recognition methods are used, then recognition accuracy can be maintained, but model weight becomes excessively large
Solution Approach 1:
The model weights are segmented into two separate sets: one for the spatial convolutional network and another for the temporal convolutional network. This segmentation allows each network to be more compact and specialized, reducing the total parameter count compared to a monolithic network while maintaining equivalent or improved recognition performance.
Solution Approach 2:
Temporal processing is extracted into a separate stream that operates on optical flow data independently from the spatial stream. This extraction eliminates the need for large 3D convolutional kernels that would process spatiotemporal data jointly, significantly reducing model weight while preserving temporal behavior patterns.
3Productivity
If frame extraction only is performed, then processing speed can be improved, but time series information processing capability is insufficient
Solution Approach 1:
Optical flow images serve as an intermediary representation that captures temporal motion information between frames. By introducing this intermediary, the model can process time series behavior patterns effectively without needing to process every raw frame, thus maintaining processing speed while recovering temporal information that would otherwise be lost.
Data Source
AI summary
A behavior recognition method and system, including: dividing video data into a plurality of video clips, performing frame extraction processing on each video clip to obtain frame images, and performing optical flow extraction on the frame images to obtain optical flow images; performing feature extraction on the frame images and the optical flow images to obtain feature maps of the frame images and the optical flow images; performing spatio-temporal convolution processing on the feature maps of the frame images and the optical flow images, and determining a spatial prediction result and a temporal prediction result; fusing the spatial prediction results of all the video clips to obtain a spatial fusion result, and fusing the temporal prediction results of all the video clips to obtain a temporal fusion result; and performing two-stream fusion on the spatial fusion result and the temporal fusion result to obtain a behavior recognition result.