Facial Animation Retargeting via Physics-Based Soft Body Simulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current facial animation retargeting techniques lack efficiency and effectiveness in changing facial identities while maintaining performance and avoiding visual artifacts, particularly in handling physical effects like lip contact and collisions.
Innovation Solution
The method involves generating facial expression and identity codes in latent spaces, converting spatial points to a canonical space, and using simulator control values to create a simulated soft body, allowing for collision-free retargeting without learning collision handling, thus focusing on muscle-driven expression activations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional facial animation retargeting techniques are used, then facial identity can be changed, but visual artifacts appear and physical effects like lip contact and collisions are not properly handled
Solution Approach 1:
The patent extracts collision handling from the neural network learning process and places it in the physics-based simulation stage. The network only learns muscle-driven expression activations, while the simulator separately handles collisions and physical constraints, eliminating the trade-off between learning complexity and physical accuracy.
Solution Approach 2:
The patent introduces a physics-based simulator as an intermediary between the neural network output and the final facial animation. The simulator acts as a mediator that enforces physical constraints and collision handling on the muscle-driven activations, ensuring physically accurate results without requiring the network to learn these complex physical interactions.
2Reliability
If collision handling is learned by the neural network, then physical accuracy improves, but training data requirements and model complexity increase
Solution Approach 1:
The patent extracts collision handling from the neural network learning process and places it in the physics-based simulation stage. The network only learns muscle-driven expression activations, while the simulator separately handles collisions and physical constraints, eliminating the trade-off between learning complexity and physical accuracy.
Solution Approach 2:
The patent replaces the machine learning approach to collision handling with a physics-based simulation approach. Instead of learning collision patterns from data, the system uses deterministic physics simulations with collision constraints, eliminating the need for extensive training data while maintaining physical accuracy.
3Manufacturing precision
If detailed physical effects are simulated, then animation quality improves, but computational complexity and processing time increase
Solution Approach 1:
The patent segments the facial animation pipeline into distinct components: neural network for muscle activation prediction, physics simulator for collision handling, and rendering for final output. Each component handles specific tasks with appropriate computational resources, improving overall efficiency while maintaining quality.
Solution Approach 2:
The patent performs collision detection and constraint application as preliminary actions during the simulation phase, before final rendering. This preliminary handling of physical constraints prevents the need for complex corrective computations during rendering, reducing overall computational complexity while maintaining animation quality.
Data Source
AI summary
One embodiment of the present invention sets forth a technique for retargeting a facial expression to a different facial identity. The technique includes generating, based on an input target facial identity, a facial identity code in an input identity latent space. The technique further includes converting a spatial input point from an input facial identity space of the input target facial identity to a canonical-space point in a canonical space. The technique still further includes generating one or more canonical simulator control values based on the facial identity code, an input source facial expression, and the canonical-space point. The technique still further includes generating a simulated active soft body based on one or more identity-specific control values, wherein each identity-specific control value corresponds to one or more of the canonical simulator control values and is in an output facial identity space associated with an output target facial identity.