Unified Human Motion Prediction via Shared Latent Space
Solution Overview
Problem
Existing human motion prediction technologies face challenges in accurately predicting future human motions due to the complexity of human intentions and environmental influences, and they rely on costly 3D annotations, limiting their application.
Innovation Solution
A unified human motion prediction system that bridges the gap between 2D and 3D poses by encoding both modalities into a shared latent space, allowing for the prediction of future 2D/3D poses without requiring paired annotations, and leveraging 2D knowledge for 3D tasks and vice versa during training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 3D skeleton pose prediction is used for human motion prediction, then prediction accuracy is improved, but data collection cost and device complexity increase due to requirement of sophisticated motion capture systems
Solution Approach 1:
The patent introduces 2D pose estimation as an intermediary approach that bridges the gap between simple image input and accurate 3D pose output. By using 2D pose sequences as input and 3D pose sequences as output, the system achieves accurate motion prediction without requiring complex motion capture systems during data collection, thus resolving the contradiction between prediction accuracy and device complexity
Solution Approach 2:
The patent transitions from requiring direct 3D pose input to using 2D pose input, effectively moving the problem to a different dimensional representation. This dimensional change allows the system to achieve 3D prediction accuracy while using simpler 2D annotations for training, thereby reducing the complexity of data collection equipment needed
2Device complexity
If 2D pose is used as input for human motion synthesis, then device complexity is reduced, but prediction accuracy deteriorates due to loss of 3D spatial information
Solution Approach 1:
The patent performs preliminary training using 2D pose sequences as input and 3D pose sequences as output, establishing a mapping relationship before actual prediction. This preliminary action allows the system to learn how to transform 2D observations into accurate 3D predictions, thereby maintaining prediction accuracy while using simpler 2D data for training
Solution Approach 2:
The patent creates a copied or transformed representation of 3D pose information through the 2D pose sequence input. By training the system to predict 3D poses from 2D poses, it effectively copies the essential motion information from the simpler 2D representation into the prediction output, maintaining accuracy without requiring complex input data
3Device complexity
If existing methods only provide one-way path from 2D to 3D, then model complexity is reduced, but adaptability deteriorates due to strict constraint on input and output modalities
Solution Approach 1:
The patent designs a unified human motion prediction model that can handle multiple tasks and modalities through a single architecture. The model can predict both 2D and 3D pose sequences from various inputs, making it universally applicable to different human motion analysis tasks. This multi-functionality significantly improves adaptability while maintaining reasonable model complexity through shared computational layers
Data Source
AI summary
A system and a method for performing unified human motion prediction (HMP) are provided. In operation, the system provides a unified HMP module, which includes an encoder configured to encode a plurality of existing two-dimensional (2D) and three-dimensional (3D) pose sequences into a shared latent space V and a decoder configured to predict a plurality of predicted future 2D and 3D pose sequences based on the encoded shared latent space V, thus forming four pathways of knowledge flow. The system then trains the unified HMP module and the shared latent space V in a weakly supervised manner with input features X with actual future 2D and 3D pose sequences by minimizing a loss between the predicted future 2D and 3D pose sequences and the actual future 3D and 3D pose sequences. Once the training is complete, the unified HMP may be performed using the trained encoded shared latent space V.