Markerless Facial Capture via Deformable Mesh and Differentiable Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional facial expression capture techniques require actors to wear cumbersome head-mounted cameras and numerous markers, limiting their freedom and creating distracting conditions during performances.
Innovation Solution
A method and system for capturing facial expressions without a head-mounted camera, using a three-dimensional parameterized deformable model and a differentiable renderer to iteratively deform a mesh, minimizing differences between a 3D render and captured footage, allowing for the transfer of expressions to a computer-generated character.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional marker-based facial capture techniques are used, then facial expression capture accuracy is improved, but actor comfort and freedom of movement deteriorate due to cumbersome head-mounted cameras and numerous markers
Solution Approach 1:
The patent removes the head-mounted camera from the system, extracting the disturbing element that constrained the actor. Instead, standard production cameras positioned in the scene capture the performance, eliminating the burden on the actor while maintaining capture capability through alternative computational methods.
Solution Approach 2:
The patent replaces the mechanical marker-based tracking system with a computational approach using neural networks and image processing. The physical markers and head-mounted camera are substituted with algorithms that analyze standard video footage to extract facial expression data, eliminating the need for actors to wear specialized equipment.
2Measurement precision
If head-mounted cameras and numerous markers are used, then facial expression capture quality is improved, but filming conditions and production flexibility deteriorate
Solution Approach 1:
The patent makes the facial capture system universal by using standard production cameras that are already present in any film or video production. The same cameras used for capturing the scene also capture the facial performance, eliminating the need for specialized capture equipment and allowing the technique to be applied in any production environment.
Solution Approach 2:
The patent creates a computational model (digital twin) of the actor's face that replicates facial expressions from the captured video. This virtual copy allows facial expression data to be extracted and transferred to CGI characters without requiring the physical presence of markers or specialized cameras during production.
3Measurement precision
If markers are positioned on the actor's face, then facial expression tracking accuracy is improved, but the natural appearance and realism of the performance deteriorate
Solution Approach 1:
The patent completely removes markers from the actor's face, extracting the element that compromised performance naturalness. Facial expression data is captured purely through analysis of the actor's appearance in standard video footage, allowing actors to perform without any visible tracking devices.
Solution Approach 2:
The patent employs color-based or appearance-based detection methods to track facial features without physical markers. By analyzing changes in pixel values, lighting, and shading on the actor's face, the system extracts expression data while maintaining the natural appearance of the performance.
Data Source
AI summary
A method of transferring a facial expression from a subject to a computer generated character that includes receiving a plate with an image of the subject's facial expression, a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters, a model of a camera rig used to capture the plate, and a virtual lighting model that estimates lighting conditions when the image on the plate was captured. The method can solve for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the deformable model with a differentiable renderer and shape from shading techniques, using, as inputs, the three-dimensional parameterized deformable model, the model of the camera rig and the virtual lighting model over a series of iterations to infer geometry of the facial expression and generate a final facial mesh using the set of parameter values of the deformable model which result in a facial expression that closely matches the expression of the subject in the plate.