3D Face Model Generation Using RGBD Point Cloud Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for generating 3D human face models in virtual reality struggle to create realistic representations that accurately capture facial details and expressions from various viewing angles, leading to discrepancies between the model and the actual face, which is undesirable for realistic viewing and virtual reality applications.
Innovation Solution
A virtual reality-based apparatus that uses RGBD data to generate a 3D human face model by aligning point clouds with a pre-stored mean-shape face model, refining the shape based on differences, and applying blend shapes to achieve accurate and realistic facial expressions, utilizing principles like affine transformations and Laplacian deformations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a 3D face model is constructed using conventional modeling software and manual design, then the model can be created with basic structure, but the facial details and expressions do not accurately resemble the actual user's face
Solution Approach 1:
The patent uses depth maps and images captured from the user's actual face to create a point cloud that serves as a direct copy of the user's facial geometry. This point cloud is then used to generate the 3D face model, ensuring accurate reproduction of facial details without manual modeling.
Solution Approach 2:
The patent replaces the manual mechanical modeling process with an automated computational system that uses depth sensing technology and point cloud processing algorithms to automatically generate the 3D face model from captured images and depth maps.
2Manufacturing precision
If multiple viewing angles are captured to improve 3D model accuracy, then facial details from different perspectives can be included, but the complexity of data capture and processing increases significantly
Solution Approach 1:
The depth sensing device is designed to capture depth information from multiple viewing angles simultaneously using a single integrated system. The device can obtain depth maps and images from various perspectives without requiring multiple separate capture systems, simplifying the overall setup while maintaining high 3D model accuracy.
3Productivity
If the 3D face model structure is simplified for easier processing, then computational load is reduced, but the realism and detail of the facial representation deteriorates
Solution Approach 1:
The patent segments the facial geometry into a point cloud representation, where the face is divided into numerous discrete points that capture detailed surface information. This segmented point cloud structure allows for efficient computational processing while preserving high-fidelity facial features, as each point can be independently processed and rendered.
4Adaptability or versatility
If real-time or near-real-time expression matching is implemented, then the 3D model can dynamically reflect the user's actual expressions, but the computational complexity and processing time increase
Solution Approach 1:
The patent pre-processes the depth maps and images to create a detailed point cloud representation of the user's face and expressions. This preliminary processing captures the facial geometry and expression data in advance, allowing the system to quickly generate and update 3D face models in real-time or near-real-time without excessive computational delay during actual use.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A virtual reality-based apparatus and method to generate a 3D human face model includes, storage of a 3D face model that is an existing 3D face model or a mean-shape face model in at least a neutral expression. A point cloud of the face of the first user is generated based on the plurality of color images and depth information of the face of the first user. A first 3D face model of the first user having neutral expression is generated by a shape-based model-fitment on the stored 3D face model. A shape of the first 3D face model is refined based on a difference between the first 3D face model, the shape-based model-fitment, and the generated point cloud. The display of the refined first 3D face model is controlled to exhibit a minimum deviation from the shape and appearance of the face of the first user.