Skeleton Action Detection via Dynamic RNN Correlation Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current skeleton-based action detection methods face challenges in accurately detecting actions from videos due to limitations in feature learning, such as fixed joint grouping which reduces the accuracy of correlations between joints and depends on experiential correlations rather than actual dependencies.
Innovation Solution
An end-to-end fully connected skeleton-based action detection scheme using a recurrent neural network (RNN) is proposed, where joint locations are mapped to features by RNN neurons without fixed grouping, allowing the network to learn correlations automatically and improve accuracy by focusing on highly correlated subsets of joints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If fixed joint grouping is used in skeleton-based action detection, then the model structure is simplified and training is easier, but the accuracy of detecting correlations between joints deteriorates
Solution Approach 1:
The patent divides the skeleton into multiple joints that can be independently processed by separate RNN neurons. Instead of fixing joints into predetermined groups, each joint is treated as an independent input that can be dynamically correlated with other joints through the learning process, allowing the model to segment the problem into manageable parts while maintaining flexibility in learning correlations.
Solution Approach 2:
The patent implements dynamic joint grouping through RNN neurons that automatically learn which joints are correlated for each action type. The model dynamically adjusts the grouping and weighting of joints based on the input data, rather than using fixed predetermined groups. This allows the correlation structure to adapt dynamically to different action types and individual variations.
2Device complexity
If experiential correlations are used to group joints, then the model construction is simplified, but the detection accuracy depends on pre-defined knowledge rather than actual data dependencies
Solution Approach 1:
The patent enables the model to self-organize and automatically learn joint correlations from the training data without requiring manual specification of joint groups based on expert knowledge. The RNN neurons automatically discover which joints are correlated for each action type through the learning process, allowing the system to serve itself rather than relying on pre-defined experiential correlations.
Solution Approach 2:
The patent incorporates feedback mechanisms through the RNN architecture where the model continuously adjusts its understanding of joint correlations based on prediction errors and training feedback. The loss function provides feedback that guides the model to improve its correlation detection, allowing it to refine its understanding of which joints are relevant for each action type iteratively during training.
Data Source
AI summary
In implementations of the subject matter described herein, an action detection scheme using a recurrent neural network (RNN) is proposed. Joint locations for a skeleton representation of an observed entity in a frame of a video and a predefined action label for the frame are obtained to train a learning network including RNN elements and a classification element. Specifically, first weights for mapping the joint locations to a first feature for the frame generated by a first RNN element in a learning network and second weights for mapping the joint locations to a second feature for the frame generated by a second RNN element in the learning network are determined based on the joint locations and the predefined action label. The first and second weights are determined by increasing a first correlation between the first feature and a first subset of the joint locations and a second correlation between the second feature and the first subset of the joint locations. Based on the joint locations and the predefined action label, a parameter for a classification element included in the learning network is also determined by increasing a probability of the frame being associated with the predefined action label. The probability is generated by the classification element at least based on the first and second features.