Simulated Control for 3D Human Poses in VR

The system segments the pose generation process into two distinct components: a kinematic pose refinement network that handles real-time pose prediction from video frames, and a separate dynamics-based control generation unit that ensures physical plausibility. This segmentation allows each component to be optimized independently, with the kinematic part providing real-time performance and the dynamics part providing physical accuracy, thereby resolving the contradiction between computational complexity and physical plausibility.

The patent introduces an intermediary reinforcement learning agent that acts as a mediator between the kinematic pose refinement network and the final pose output. This agent learns to generate dynamics-based control signals that adjust the kinematic poses to satisfy physical constraints, thereby ensuring physical plausibility without requiring complex trajectory optimization, thus reducing computational complexity while maintaining reliability.

The system performs preliminary action by pre-training the reinforcement learning agent offline to learn the mapping from kinematic poses to dynamics-based control signals. During real-time operation, the pre-trained agent can rapidly generate physically-plausible poses without performing complex trajectory optimization, thereby achieving physically-grounded motion with minimal processing time and enabling real-time applications.

The patent substitutes the traditional mechanical trajectory optimization system with a data-driven reinforcement learning agent. Instead of solving complex optimization problems at runtime, the system uses the pre-trained neural network agent to directly predict dynamics-based control signals, replacing the computationally intensive mechanical optimization process with a faster neural network inference process while maintaining physical plausibility.

The reinforcement learning agent serves as an intermediary that bridges the gap between simple differentiable physics models and advanced physics simulators. The agent learns to compensate for the approximation errors of simple physics models by generating dynamics-based control signals that are corrected during training using rewards based on advanced physics simulations, thereby achieving high precision without sacrificing optimization tractability.

The system changes the parameters of the physics model by using simple differentiable physics models for the core optimization process while incorporating learned correction parameters from the reinforcement learning agent. These learned parameters adjust the simple physics model outputs to match the accuracy of advanced physics simulators, thereby maintaining optimization tractability while reducing approximation errors.