Neural Network Face Animation Synthesis for Real-Time Mobile Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current face animation synthesis techniques on mobile devices either generate non-photorealistic faces quickly or are time-consuming and not suitable for real-time processing, failing to provide realistic head turns and face animation effectively.
Innovation Solution
A method and system that utilize neural networks to generate realistic head turns and face animation synthesis by receiving frames of a source video, determining target identity information, and modifying target images based on source pose parameters, allowing for real-time processing on mobile devices without the need for internet connection or server-side resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If morphable face models are used for face animation synthesis, then generation speed is fast, but photorealism is poor
Solution Approach 1:
The patent introduces an intermediary neural network model that acts as a bridge between the source video frames and target image. This intermediary network learns the mapping relationship and generates photorealistic results while maintaining real-time performance, resolving the contradiction between speed and quality.
Solution Approach 2:
The patent transforms the face animation synthesis problem into a parameter estimation problem. By estimating pose parameters, expression parameters, and identity parameters, the system achieves both fast generation and photorealistic results. The key insight is changing from direct pixel manipulation to parameter-based transformation.
2Manufacturing precision
If traditional face animation synthesis techniques are used, then photorealism can be achieved, but computation time is too long for real-time processing on mobile devices
Solution Approach 1:
The patent segments the face animation synthesis task into three independent parameter estimation sub-tasks: pose estimation, expression estimation, and identity estimation. This segmentation allows each sub-task to be processed efficiently and independently, enabling real-time performance on mobile devices while maintaining photorealism.
Solution Approach 2:
The patent replaces traditional mechanical/algorithmic face warping and blending methods with a neural network-based parameter estimation approach. This substitution enables the system to achieve photorealistic results with significantly reduced computation time suitable for mobile devices.
Data Source
AI summary
Provided are systems and methods for realistic head turns and face animation synthesis. An example method may include receiving frames of a source video with the head and the face of a source actor. The method may then proceed with generating sets of source pose parameters that represent positions of the head and facial expressions of the source actor. The method may further include receiving at least one target image including the target head and the target face of a target person, determining target identity information associated with the target face, and generating an output video based on the target identity information and the sets of source pose parameters. Each frame of the output video can include an image of the target face modified to mimic at least one of the positions of the head of the source actor and at least one of facial expressions of the source actor.