Robust Video-Based Neural Facial Animation for Mobile 3D Avatars
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for animating avatars in online platforms face limitations due to insufficient computing resources on mobile devices, leading to issues such as facial jitter, lack of responsiveness, and poor translation of visual cues, preventing many users from enjoying robust facial animations.
Innovation Solution
A system utilizing fully convolutional networks and overloaded output convolutional neural networks to automatically generate robust facial animations for 3D avatars, capable of detecting faces and refining facial expressions and head poses in real-time, even on resource-constrained devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional avatar animation methods are used, then device compatibility is maintained, but facial animation quality deteriorates due to insufficient computing resources
Solution Approach 1:
The system segments the avatar animation process into distinct components: face detection module, facial expression recognition module, and avatar rendering module. This allows each component to be optimized independently and processed efficiently on mobile devices with limited resources.
Solution Approach 2:
The system uses 2D facial landmark detection from video frames as a copy or representation of the user's actual facial movements, which is then mapped to control a 3D avatar. This copying approach reduces computational complexity compared to full 3D facial scanning while maintaining animation quality.
2Speed
If real-time facial animation is implemented, then responsiveness is improved, but computational load increases causing device performance degradation
Solution Approach 1:
The system processes video frames at optimized intervals rather than continuously analyzing every frame. The face detection and landmark extraction are performed periodically at rates that maintain animation responsiveness while reducing overall computational energy consumption on mobile devices.
Solution Approach 2:
The system performs preliminary face detection and bounding box identification on lower-resolution preview frames before processing high-resolution animation frames. This preliminary action filters out frames without faces and prepares initial parameters, reducing the computational load for subsequent detailed processing.
3Measurement precision
If robust facial animation is achieved, then animation accuracy is improved, but system complexity increases making implementation difficult
Solution Approach 1:
The system introduces an intermediary preprocessing stage that detects facial bounding boxes and key landmarks before full facial expression analysis. This intermediary step simplifies the input for the main animation generation network, reducing its complexity while maintaining detection accuracy through the two-stage approach.
4Manufacturing precision
If multi-parameter facial tracking is implemented, then animation realism is improved, but processing time increases reducing frame rate
Solution Approach 1:
The system segments facial parameter extraction into parallel independent tasks: head pose estimation, facial landmark detection, and expression intensity measurement. These segmented tasks process different aspects of facial anatomy simultaneously, reducing total processing time while maintaining comprehensive facial tracking accuracy.
Data Source
AI summary
Implementations described herein relate to methods, systems, and computer-readable media to generate animations for a 3D avatar from input video captured at a client device. A camera may capture video of a face while a trained face detection model and a trained regression model output a set of FACS weights, head poses, and facial landmarks to be translated into the animations of the 3D avatar. Additionally, a higher level-of-detail may be intelligently selected based upon user preferences and/or computing conditions at the client device.