Human Pose Rendering With 3D Priors for Novel Views
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems struggle to accurately predict unseen views of individuals due to the variance in human populations and require computationally intensive test-time optimization, which is often infeasible.
Innovation Solution
A generalizable approach using explicit body priors, multi-view geometry, and learnable rendering to synthesize unique human views without test-time optimization, employing a multi-step processing pipeline that extracts three-dimensional features, maps them to a target space, and aggregates them for rendering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If test-time optimization is used to synthesize views of unseen individuals, then view synthesis accuracy is improved, but computational cost and time increase significantly
Solution Approach 1:
The system performs preliminary actions by extracting and storing 3D body priors and appearance features during training time. These pre-computed representations are then used during inference to quickly synthesize views of unseen individuals without requiring test-time optimization, thus resolving the contradiction between accuracy and computational time.
Solution Approach 2:
The patent uses 3D body priors as template copies that represent average human anatomy. These priors are then warped and adjusted to match specific individuals' appearances from 2D images, allowing the system to generate accurate views of unseen people by copying and adapting existing 3D templates rather than optimizing from scratch.
2Measurement precision
If models are trained on specific individuals to provide accurate views, then view prediction accuracy is improved, but adaptability to unseen individuals deteriorates
Solution Approach 1:
The system trains a universal 3D body prior model that can represent any human individual. This single multi-functional model serves all individuals by using a shared 3D human body template that can be adapted to different people through 2D image input, eliminating the need for separate models for each individual while maintaining accuracy.
Solution Approach 2:
The patent changes parameters by adjusting the 3D body prior representations to match specific individuals' characteristics. Instead of training separate models, the system modifies the parameters of a universal 3D prior using 2D images from target individuals, allowing accurate view synthesis for unseen people through parameter adaptation rather than retraining.
3Measurement precision
If test-time optimization is performed for each individual, then rendering accuracy is improved, but device complexity and computational requirements increase
Solution Approach 1:
The system performs all complex computations during training time to extract and store 3D body priors and appearance features. During inference, the system simply needs to query and combine these pre-computed representations, significantly reducing the computational complexity required at runtime while maintaining high rendering accuracy.
Solution Approach 2:
The patent replaces the mechanical optimization process with a neural network-based approach. Instead of using traditional optimization algorithms that iteratively adjust parameters, the system uses learned 3D body priors and neural rendering functions to directly generate accurate views, substituting complex optimization mechanics with pre-learned neural representations.
Data Source
AI summary
Systems, methods, and other embodiments described herein relate to improving view synthesis of humans using a generalizable approach without test-time optimization. In one embodiment, a method includes acquiring target information and sensor data of a surrounding environment that includes a person. The target information defines a target space that includes a target pose and a target camera view. The method includes extracting appearance features of the person from the sensor data. The method includes mapping the appearance features into the target space, including aggregating the appearance features into an aggregated feature map. The method includes rendering the target camera view of the person in the target pose according to the aggregated feature map. The method includes providing the target camera view.