Pose-Enabled RGB-Flow Networks with Distilled Low-Latency Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning models for video activity recognition face a trade-off between increasing activity recognition performance and reducing latency during execution, as processing multiple data representations leads to increased latency.
Innovation Solution
A Pose Empowered RGB-Flow Net (PERF-Net) model processes video data into spatial, temporal, and pose input streams, using separate neural networks for each stream, and employs distillation training to transfer knowledge from multiple teacher networks to a student network, allowing it to classify activities efficiently by processing only the spatial input stream during inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple data representations (spatial, temporal, pose streams) are processed to increase activity recognition performance, then classification accuracy is improved, but execution latency increases
Solution Approach 1:
The model segments the processing of multiple data representations by using separate neural networks for each input stream (spatial, temporal, and pose). This modular architecture allows independent processing of each stream, enabling the system to maintain high recognition performance while optimizing execution time by selectively processing streams during inference based on their contribution to the final prediction.
2Reliability
If multiple teacher neural networks process different input streams during training, then knowledge transfer to student network is improved, but computational cost during training increases
Solution Approach 1:
The distillation training framework creates copies of the teacher network's knowledge by having multiple teacher neural networks process different input streams and transfer their learned representations to a student network. This copying mechanism allows the student network to acquire comprehensive knowledge from multiple specialized teachers, improving reliability while the modular structure helps manage computational costs during training.
Data Source
AI summary
A method includes receiving video data that includes a series of frames of image data. Here, the video data is representative of an actor performing an activity. The method also includes processing the video data to generate a spatial input stream including a series of spatial images representative of spatial features of the actor performing the activity, a temporal input stream representative of motion of the actor performing the activity, and a pose input stream including a series of images representative of a pose of the actor performing the activity. Using at least one neural network, the method also includes processing the temporal input stream, the spatial input stream, and the pose input stream. The method also includes classifying, by the at least one neural network, the activity based on the temporal input stream, the spatial input stream, and the pose input stream.