Two-Stream 3D CNN Action Recognition for High-Resolution Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network systems for video processing face challenges in achieving accurate classification of motion and action recognition, particularly in higher spatial resolution videos, with high computational and memory requirements.
Innovation Solution
A neural network system that combines a first and second 3D spatio-temporal convolutional neural network to process image and optical flow data, utilizing a data combiner to enhance classification accuracy, and employs an inflated 2D neural network architecture for training, with optional pre-training on different datasets to improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D spatio-temporal convolutional neural networks are used for video processing, then classification accuracy is improved, but computational complexity and memory requirements increase
Solution Approach 1:
The patent divides the video processing task into two separate streams: a spatial stream that processes appearance information from video frames, and a temporal stream that processes motion information from optical flow data. Each stream uses its own 3D spatio-temporal convolutional neural network, allowing independent optimization and processing. This segmentation reduces the computational burden on a single network while maintaining high classification accuracy through fusion of both streams.
Solution Approach 2:
The patent extends 2D convolutional neural networks to 3D spatio-temporal networks by adding the temporal dimension. This allows the networks to process video data in three dimensions (height, width, time) simultaneously, capturing both spatial features and temporal dynamics. The dimensional extension enables accurate action recognition while the modular two-stream architecture manages the associated computational complexity.
2Measurement precision
If higher spatial resolution videos are processed, then classification accuracy is improved, but computational requirements increase
Solution Approach 1:
By segmenting the processing into spatial and temporal streams, the patent allows each stream to operate at optimized resolutions. The spatial stream can process high-resolution appearance data, while the temporal stream processes motion data that may require less computational energy. This division enables high accuracy processing of high spatial resolution videos while managing energy consumption through specialized processing paths.
3Measurement precision
If longer temporal receptive fields are used, then action recognition accuracy is improved, but processing time increases
Solution Approach 1:
The temporal stream is specifically designed to handle motion information over time with its own 3D spatio-temporal convolutional network. This dedicated temporal processing path can efficiently capture long-term dependencies and temporal patterns without bottlenecking the overall processing speed. The separation allows the temporal stream to optimize for long receptive fields while the spatial stream handles immediate appearance features, reducing overall processing time.
Data Source
Figure 1
Figure 2a
Figure 2b
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing video data. An example system receives video data and generates optical flow data. An image sequence from the video data is provided to a first 3D spatio-temporal convolutional neural network to process the image data in at least three space-time dimensions and to provide a first convolutional neural network output. A corresponding sequence of optical flow image frames is provided to a second 3D spatio- temporal convolutional neural network to process the optical flow data in at least three space- time dimensions and to provide a second convolutional neural network output. The first and second convolutional neural network outputs are combined to provide a system output.