2D Skeleton Activity Classification With Hierarchical RNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for classifying human activities from skeleton data are challenging due to the need for 3D data, which is difficult to extract and requires specialized hardware, and lack contextual information, making them computationally intensive and less portable across domains.
Innovation Solution
A system using 2D skeleton data processed by a skeleton preprocessor and two deep recurrent neural networks (RNNs) - a gesture classifier and an action classifier - to identify gestures and actions efficiently, leveraging relative joint positions and velocities, and incorporating contextual object information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D skeleton data is used for activity classification, then measurement precision is improved, but device complexity and ease of manufacture worsen due to requiring specialized hardware
Solution Approach 1:
The patent uses 2D skeleton data as a simplified copy or representation of 3D skeleton data. Instead of requiring complex 3D depth sensors, the system processes 2D joint position data from standard RGB cameras, maintaining sufficient accuracy for activity classification while dramatically reducing hardware complexity and cost.
2Adaptability or versatility
If skeleton data processing methods are made more general to handle real-world variations, then adaptability is improved, but reliability worsens due to robustness issues
Solution Approach 1:
The patent segments the activity classification task into two distinct hierarchical levels: gesture classification (identifying individual hand or body gestures) and action classification (identifying overall activities). This segmentation allows each classifier to specialize and become more robust within its specific domain, while the hierarchical structure provides adaptability to handle diverse real-world variations through composition of specialized components.
3Productivity
If deep neural networks are used for activity classification, then productivity is improved through automated classification, but use of energy worsens due to computational intensity
Solution Approach 1:
The patent divides the deep neural network processing into two separate specialized networks: a gesture classifier network and an action classifier network. This segmentation allows each network to be computationally more efficient by focusing on specific classification tasks, reducing overall energy consumption compared to a single large network, while maintaining high automation productivity.
Solution Approach 2:
The gesture classifier performs preliminary classification of individual gestures before the action classifier processes overall activities. This preliminary action allows the system to filter and organize data in stages, reducing the computational burden on subsequent processing and lowering overall energy requirements while maintaining automated classification productivity.
Data Source
AI summary
An activity classifier system and method that classifies human activities using 2D skeleton data. The system includes a skeleton preprocessor that transforms the 2D skeleton data into transformed skeleton data, the transformed skeleton data comprising scaled, relative joint positions and relative joint velocities. The system also includes a gesture classifier comprising a first recurrent neural network that receives the transformed skeleton data, and is trained to identify the most probable of a plurality of gestures. The system also has an action classifier comprising a second recurrent neural network that receives information from the first recurrent neural networks and is trained to identify the most probable of a plurality of actions.