Facial Landmark Generation for Real-Time AR Head-Turn Synthesis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current face animation synthesis techniques are either not photorealistic or too time-consuming for real-time processing on mobile devices, and existing messaging systems struggle to efficiently process and render augmented reality content on power and resource-constrained devices.
Innovation Solution
A messaging system that includes a neural network-based face detection system for facial landmark generation, enabling real-time face animation synthesis and augmented reality content rendering on mobile devices, using convolutional neural networks for efficient image processing and power management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional face animation synthesis techniques are used, then photorealism can be achieved, but processing time becomes too long for real-time applications
Solution Approach 1:
The system performs preliminary actions by detecting facial landmarks and extracting pose parameters (yaw, pitch, roll) in advance before rendering. The neural network pre-processes the input image to identify key facial features and head orientation, so that when rendering is needed, the heavy computational work has already been done, enabling real-time photorealistic synthesis
Solution Approach 2:
The face is segmented into multiple key landmarks (eyes, nose, mouth, jawline) that are detected independently. This segmentation allows the system to process only critical facial features rather than the entire face uniformly, reducing computational complexity while maintaining photorealism in the synthesized output
2Measurement precision
If complex neural network processing is used for face detection and landmark generation, then accurate facial synthesis can be achieved, but power consumption increases on mobile devices
Solution Approach 1:
The system extracts only the essential facial landmarks and pose parameters needed for synthesis, rather than processing the entire facial image at full resolution. By taking out only the critical features (key points of eyes, nose, mouth, jaw), the neural network achieves accurate facial representation with significantly reduced computational load and power consumption
Solution Approach 2:
The neural network applies different processing quality to different facial regions - focusing computational resources on key landmarks that most affect synthesis accuracy (eyes, mouth, nose) while using lighter processing for less critical areas. This local quality approach maintains overall facial accuracy while reducing total power consumption
3Speed
If real-time processing is implemented on mobile devices, then latency is reduced, but processing accuracy and photorealism may deteriorate
Solution Approach 1:
The system performs preliminary detection of facial landmarks and pose parameters before the actual synthesis rendering. This preliminary action separates the computationally intensive detection phase from the rendering phase, allowing real-time processing speed while maintaining synthesis accuracy through pre-computed accurate facial feature data
Solution Approach 2:
The system changes parameters by detecting head pose (yaw, pitch, roll angles) and using these parameters to transform the facial synthesis accordingly. By working with compact pose parameters rather than full-resolution facial images throughout processing, the system achieves real-time performance without sacrificing synthesis accuracy
Data Source
AI summary
The subject technology receives frames of a source media content, the frames of the source media content including representations of a head and a face of a source actor. The subject technology generates, based at least in part on the frames of the source media content, sets of source pose parameters. The subject technology receives at least one target image, the at least one target image including representations of a target head and a target face of a target entity. The subject technology provides the sets of source pose parameters to a neural network to determine facial landmarks for head turns and facial expressions. The subject technology generates, based at least in part on the sets of source pose parameters and the facial landmarks for head turns and facial expressions, an output media content. The subject technology provides augmented reality content based at least in part on the output media content for display on a computing device.