Volumetric Multi-View Face Mesh Inference for Consistent Topology
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for 3D face reconstruction and registration are inefficient, requiring manual clean-up and parameter tuning, and struggle with noise, outliers, and extreme facial expressions, leading to inaccurate and time-consuming mesh generation.
Innovation Solution
A volumetric sampling approach for 3D mesh generation using multi-view images, employing a progressive mesh generation network that iteratively upsamples and refines vertex positions, ensuring topologically consistent meshes across facial identities and expressions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If multi-view stereo (MVS) techniques are used for 3D reconstruction, then surface coverage and fidelity are improved, but manual clean-up time and processing complexity increase due to noise and outliers
Solution Approach 1:
The patent applies preliminary action by performing automated cleaning operations before mesh registration. The system identifies and removes artifacts, noise, and outliers from 3D scans using automated algorithms, eliminating the need for subsequent manual clean-up. This preliminary processing ensures that the mesh data is ready for registration without requiring time-consuming manual intervention.
Solution Approach 2:
The patent replaces manual mechanical clean-up operations with automated computational algorithms. Instead of requiring operators to manually inspect and clean 3D scan data, the system uses automated outlier detection and noise removal algorithms that process the data computationally, significantly reducing processing time and eliminating manual labor.
2Reliability
If non-rigid registration is used to warp 3D face model to scans, then expression consistency is improved, but processing time and computational complexity increase
Solution Approach 1:
The patent applies preliminary action by performing automated cleaning and preprocessing of 3D scan data before the non-rigid registration step. By removing noise, outliers, and artifacts in advance, the registration algorithm receives cleaner input data, which reduces computational complexity and processing time while maintaining expression consistency.
Solution Approach 2:
The patent changes parameters by optimizing the non-rigid registration algorithm to work more efficiently with pre-cleaned data. The system adjusts registration parameters and uses simplified models that achieve the same expression consistency with reduced computational requirements, thereby decreasing processing time.
3Reliability
If 3DMM inference is used for robustness, then reliability is improved, but accuracy for extreme expressions deteriorates due to linear model constraints
Solution Approach 1:
The patent extracts and removes the limiting linear constraints of traditional 3DMM models. By eliminating the linear model constraints, the system can represent extreme facial expressions more accurately while maintaining robustness through alternative regularization techniques that do not rely on linear assumptions.
Solution Approach 2:
The patent combines multiple approaches into a composite solution that integrates the robustness of 3DMM with the accuracy of non-linear modeling. The system uses a hybrid approach that leverages the stability of statistical models while incorporating non-linear components to accurately represent extreme expressions, achieving both robustness and high accuracy.
4Manufacturing precision
If photometric refinement is applied to fit mesh to ground-truth surfaces, then fitting accuracy is improved, but artifacts are introduced by fitting unwanted regions like facial hair
Solution Approach 1:
The patent extracts and removes unwanted regions such as facial hair, eyebrows, and other non-facial elements before performing photometric refinement. By separating these harmful elements from the facial surface data, the refinement process only fits the actual facial features, preventing artifacts while maintaining high fitting accuracy for relevant regions.
Solution Approach 2:
The patent converts the potential harm of fitting unwanted regions into a benefit by using the presence of these regions as indicators for masking. The system identifies regions like facial hair that should not be fitted and applies selective masking, thereby preventing artifacts while maintaining accurate fitting for genuine facial features.
Data Source
AI summary
Systems and methods are provided for generating topologically consistent meshes across various subjects, objects, and/or various facial expressions using a volumetric representation. In one example, a progressive mesh generation network is configured to embed the topological structure of a subject or an object in a feature volume sampled from a geometry-aware local features. Further, a coarse-to-fine iterative architecture facilitates dense and accurate facial mesh predictions using a consistent mesh topology. In another example, one or more high-quality asset maps may be generated from a final base mesh.