3D CNN Human Behavior Classification Using 2D Keypoint Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning approaches for identifying human behavior in digital videos suffer from low accuracy and computational inefficiency, making them unsuitable for real-time applications such as video surveillance and health monitoring.
Innovation Solution
A method using a two-dimensional convolutional neural network to identify key points in video frames, followed by a three-dimensional convolutional neural network to process temporal sequences, enabling efficient behavior classification through encoded representations and machine-learned models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If recurrent neural networks are used to process video frames temporally, then behavior identification accuracy is improved, but computational intensity increases significantly
Solution Approach 1:
The patent divides the behavior recognition task into two independent stages: (1) keypoint detection using a 2D CNN that processes each frame individually, and (2) behavior classification using a 3D CNN that processes the extracted keypoint trajectories. This segmentation allows the computationally intensive temporal processing to be performed only on the essential keypoint data rather than the entire video frame, significantly reducing overall computational requirements while maintaining accuracy.
Solution Approach 2:
The patent extracts salient keypoints from video frames using a 2D CNN, which identifies and tracks important body joints and their trajectories across frames. By extracting only these critical keypoint information rather than processing all pixel data through temporal models, the system reduces computational intensity while preserving the temporal dynamics necessary for accurate behavior recognition.
2Productivity
If 2D CNN is used for spatial processing of video frames, then processing efficiency is improved, but temporal understanding capability deteriorates
Solution Approach 1:
The patent transitions from processing only 2D spatial frames to extracting 3D spatio-temporal information by tracking keypoint trajectories across multiple frames. The 3D CNN processes data with dimensions corresponding to spatial coordinates and temporal sequences, effectively adding the time dimension to the processing pipeline. This allows the system to maintain the efficiency of CNN-based processing while achieving comprehensive spatio-temporal understanding.
3Measurement precision
If optical flow module is combined with 2D CNN for behavior recognition, then motion information is improved, but system complexity increases
Solution Approach 1:
Instead of using complex optical flow algorithms that compute dense motion fields, the patent employs a simpler approach: a 2D CNN detects and tracks discrete keypoints (body joints) across frames, and a 3D CNN processes the trajectories of these copied keypoint positions. This copying of essential motion information at the keypoint level achieves accurate motion representation with significantly reduced system complexity compared to dense optical flow methods.
Data Source
AI summary
A method, processing system and processor-readable medium for classifying human behavior based on a sequence of frames of a digital video. A 2D convolutional neural network is used to identify key points on a human body, such as human body joints, visible within each frame. An encoded representation of the key points is created for each video frame. The sequence of encoded representations corresponding to the sequence of frames is processed by a 3D CNN trained to identify human behaviors based on key point positions varying over time.