3D Human Pose Simulation for Low-Latency Physical Plausibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for generating 3D human poses in virtual reality and augmented reality applications are computationally intensive and require complex optimization, leading to high latency and approximation errors, making them unsuitable for real-time interactive applications.
Innovation Solution
A simulation-based approach integrating image-based kinematic inference and physics-based dynamics modeling, using a learnable kinematic pose refinement unit and dynamics-based control generation unit, coupled with reinforcement learning to achieve accurate and physically-plausible pose estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If trajectory optimization-based methods are used to generate 3D human poses, then physical plausibility of motion is improved, but computational complexity and latency increase significantly
Solution Approach 1:
The system segments the pose estimation problem into two independent modules: a kinematic pose estimation module that processes video frames to extract body keypoints, and a dynamics-based control module that generates physically plausible motions. This segmentation allows each module to be optimized independently, reducing overall computational complexity while maintaining physical plausibility through the dynamics module.
Solution Approach 2:
The patent replaces the traditional trajectory optimization mechanical system with a dynamics-based control system that uses learned dynamics models. This substitution eliminates the need for complex batch optimization while preserving physical plausibility through physics-informed dynamics equations, significantly reducing computational requirements for real-time operation.
2Reliability
If trajectory optimization is applied to entire motion sequences, then physical consistency is improved, but processing time and latency increase
Solution Approach 1:
The system performs preliminary action by pre-training dynamics-based control policies offline using trajectory optimization on synthetic data. During real-time operation, these pre-learned policies are applied directly without requiring batch processing of motion sequences, achieving both physical consistency and low latency through the separation of offline training and online execution phases.
3Ease of manufacture
If simple differentiable physics models are used for trajectory optimization, then computational tractability is improved, but approximation errors increase
Solution Approach 1:
The patent introduces an intermediary dynamics model that bridges simple differentiable physics and complex non-differentiable physics simulators. This intermediary model uses learned dynamics representations that capture complex physical interactions while remaining computationally tractable, reducing approximation errors without sacrificing computational efficiency.
Data Source
AI summary
A method for simulating a solid body animation of a subject includes retrieving a first frame that includes a body image of a subject. The method also includes selecting, from the first frame, multiple key points within the body image of the subject that define a hull of a body part and multiple joint points that define a joint between two body parts, identifying a geometry, a speed, and a mass of the body part to include in a dynamic model of the subject, based on the key points and the joint points, determining, based on the dynamic model of the subject, a pose of the subject in a second frame after the first frame in a video stream, and providing the video stream to an immersive reality application running on a client device.