2D Skeleton Activity Classification With Gesture-Action RNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for classifying human activities using skeleton data are challenging due to the need for 3D data, which is difficult to extract and requires specialized hardware, and lack of contextual information, making them computationally intensive and less portable across domains.
Innovation Solution
A system using 2D skeleton data processed by a skeleton preprocessor and deep recurrent neural networks (RNNs) for gesture and action classification, including a gesture classifier and an action classifier, to identify probable gestures and actions efficiently and in real-time, without relying on 3D data or raw video background information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D skeleton data is used for activity classification, then measurement precision is improved, but device complexity and ease of manufacture worsen due to requiring specialized hardware
Solution Approach 1:
The patent creates a 2D projection copy of the 3D skeleton data that preserves all necessary information for activity classification. Instead of requiring complex 3D sensors, the system uses standard cameras to capture 2D projections of human poses, which are then processed through neural networks to achieve accurate activity recognition without specialized hardware.
2Reliability
If skeleton data processing methods are made more robust to real-world variations, then reliability is improved, but device complexity and computational requirements worsen
Solution Approach 1:
The patent segments the activity classification task into two distinct stages: gesture classification (identifying individual poses) and action classification (identifying sequences of gestures). This segmentation allows each neural network to specialize in specific aspects of the problem, improving robustness to variations while keeping computational requirements manageable through efficient 2D skeleton processing.
Solution Approach 2:
The system performs preliminary processing of skeleton data to extract key features and transform them into a standardized format before classification. This preliminary action includes normalizing 2D skeleton coordinates and preparing them for neural network input, which enhances robustness to real-world variations in lighting, distance, and camera angles without requiring complex computational resources during actual classification.
3Adaptability or versatility
If activity classifiers directly consume video data, then adaptability is improved, but training difficulty and loss of information worsen
Solution Approach 1:
The patent extracts essential skeletal information from video data, separating it from unnecessary visual details such as clothing, background, and lighting conditions. By taking out only the relevant 2D joint positions and using them for classification, the system achieves adaptability across different domains while reducing training data requirements and avoiding information loss associated with processing complete video sequences.
4Ease of operation
If skeleton data processing is simplified for portability, then ease of operation is improved, but measurement precision worsens due to lack of contextual information
Solution Approach 1:
The patent processes 2D skeleton data through multiple dimensional transformations, including temporal dimensions (sequences of frames) and hierarchical dimensions (gesture level and action level classification). This multi-dimensional approach preserves measurement precision by capturing contextual information across time and classification levels, while maintaining portability through the use of standard 2D camera inputs rather than requiring 3D data.
Data Source
AI summary
An activity classifier system and method that classifies human activities using 2D skeleton data. The system includes a skeleton preprocessor that transforms the 2D skeleton data into transformed skeleton data, the transformed skeleton data comprising scaled, relative joint positions and relative joint velocities. The system also includes a gesture classifier comprising a first recurrent neural network that receives the transformed skeleton data, and is trained to identify the most probable of a plurality of gestures. The system also has an action classifier comprising a second recurrent neural network that receives information from the first recurrent neural networks and is trained to identify the most probable of a plurality of actions.