Mobile Face Animation Synthesis Using Neural Network Pose Transfer
Solution Overview
Problem
Current face animation synthesis techniques on mobile devices are either not photorealistic or time-consuming, making them unsuitable for real-time applications on regular mobile devices without server-side computational resources.
Innovation Solution
A method and system for realistic head turns and face animation synthesis on mobile devices, utilizing a computing device to generate sets of source pose parameters and key points from a source video, which are applied to a target video to mimic the source actor's facial expressions and head movements, using neural networks and machine learning models to create a photorealistic output without internet connection or server-side resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If morphable face models are used for face animation synthesis, then generation speed is improved, but photorealism deteriorates
Solution Approach 1:
The patent changes the underlying mathematical model parameters from linear morphable face models to non-linear neural network transformations. The neural network learns complex non-linear mappings from source to target face space, enabling photorealistic results while maintaining real-time performance on mobile devices through optimized computation graphs and tensor operations.
Solution Approach 2:
The patent replaces traditional mechanical morphable face model computations with neural network-based image transformation. Instead of using explicit 3D mesh deformation and linear blending, the system uses learned neural representations that directly transform source face images to target face images with desired expressions and poses.
2Manufacturing precision
If traditional face animation synthesis techniques are used, then photorealism is improved, but computation time increases
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing neural network weights and transformation parameters during an offline training phase using large datasets of facial images. During the online execution phase on mobile devices, only lightweight inference operations are needed, dramatically reducing real-time computation time while maintaining high photorealism.
Solution Approach 2:
The patent segments the face animation synthesis process into distinct stages: (1) pose estimation from source video frames, (2) neural network-based face transformation, and (3) rendering. Each stage is optimized independently, with the neural network component being the core computational bottleneck that is addressed through model compression and hardware acceleration.
3Power
If server-side computational resources are used, then processing power is improved, but device independence deteriorates
Solution Approach 1:
The patent extracts the essential computational functionality from server-side cloud processing and relocates it to mobile device-based neural network inference. By implementing the neural network model directly on the device through optimized libraries (TensorFlow Lite, Core ML, etc.), the system achieves serverless operation that maintains high processing power for face animation synthesis while enabling device independence and privacy preservation.
Data Source
AI summary
Provided are systems and methods for realistic head turns and face animation synthesis. An example method includes receiving a source frame of a source video, where the source frame includes a head and a face of a source actor, generating source pose parameters corresponding to a pose of the head and a facial expression of the source actor; receiving a target image including a target head and a target face of a target person, determining target identity information associated with the target head and the target face of the target person, replacing source identity information in the source pose parameters with the target identity information to obtain further source pose parameters, and generating an output frame of an output video that includes a modified image of the target face and the target head adopting the pose of the head and the facial expression of the source actor.