Grounded Human Motion Generation With Scene-Text Feature Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating diverse and semantically consistent human motions in 3D scenes based on textual descriptions is challenging due to the joint modeling of 3D scenes and human motion, which is costly and time-consuming, and traditional methods exhibit a bias towards central motions.
Innovation Solution
A method involving a pre-trained vision-language model and a U-Net scene encoder to generate human motion by fusing scene and text features, using a conditional motion generator to predict motion parameters for a parametric human body model, with open vocabulary scene-and-text contexts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional methods are used to generate human motions in 3D scenes based on textual descriptions, then the joint modeling of 3D scene and human motion can be achieved, but the process becomes costly and time-consuming
Solution Approach 1:
The patent segments the motion generation process into distinct modules: a text encoder that processes textual descriptions, a scene encoder that processes 3D scene representations, and a motion generator that synthesizes motions. This segmentation allows each component to be optimized independently and enables efficient processing by avoiding the need to re-process entire scenes and texts for each motion generation task.
Solution Approach 2:
The patent pre-processes textual descriptions and scene representations into embedded representations before motion generation. By encoding texts and scenes in advance and storing their features, the system can quickly retrieve and combine these pre-computed features during motion generation, significantly reducing the time and computational cost required for each new motion generation task.
2Ease of manufacture
If traditional methods generate human motions centered within the scene, then the motion generation process is simplified, but the motion diversity and adaptability to different scene contexts are reduced
Solution Approach 1:
The patent introduces dynamic motion placement by allowing the motion generator to produce motions at different locations and orientations within the scene based on the scene embedding and text embedding. Instead of fixed central positioning, the system dynamically adjusts motion parameters (position, orientation, direction) according to the specific scene context and textual instructions, enabling versatile motion placement while maintaining processing efficiency through the embedded representation framework.
Data Source
AI summary
In an embodiment, a method for human motion generation with open vocabulary scene-and-text context is provided. The method involves receiving an input that includes a 3D point cloud of a scene containing a goal object with a natural language instruction related to the goal object. A text tokenizer is applied to the text to obtain tokenized text, and a text encoder from a pre-trained vision-language model generates text features. First scene features are generated by applying a pre-trained U-Net scene encoder to the 3D point cloud, which are down sampled to obtain second scene features. A conditional latent is obtained by fusing the second scene features with the text features. A conditional motion generator predicts motion parameters for a parametric human body model over a specific time duration. Finally, 3D human meshes for multiple motion frames are obtained based on the motion parameters and the parametric human body model.