Real-time Face AR Rendering via Segmented Depth Map
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality systems face challenges in rendering depth-accurate effects on a user's face in real-time, especially on devices with limited computational capabilities, as they require significant processing resources and often sacrifice detail or perform offline due to complexity.
Innovation Solution
The method involves generating a global facial depth map and luminance map from a captured image, segmenting it into smaller areas, and using least square minimization to create a detailed depth map for each segment, allowing for real-time rendering of visual effects that preserve three-dimensional details like wrinkles and facial expressions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If photo-realistic facial effects are rendered with high detail, then rendering quality is improved, but computational complexity and processing requirements increase significantly
Solution Approach 1:
The captured image is divided into multiple segments, and a detailed depth map is generated for each segment independently using least square minimization. This segmentation approach reduces the computational complexity of processing the entire face at once while maintaining high rendering detail quality through localized processing.
2Measurement precision
If detailed depth maps are generated for the entire face, then rendering accuracy is improved, but processing time increases making real-time rendering impractical
Solution Approach 1:
The face image is segmented into multiple regions, and detailed depth maps are generated for each segment in parallel. This segmentation enables real-time processing by dividing the computationally intensive task into smaller, manageable units that can be processed simultaneously, maintaining depth map accuracy while reducing total processing time.
Solution Approach 2:
Instead of generating a detailed depth map for the entire face at once, the system generates detailed depth maps for partial segments of the face. This partial action approach achieves sufficient rendering accuracy for visual effects while significantly reducing processing time to enable real-time performance.
3Manufacturing precision
If multiple images and significant processing capabilities are used, then rendering quality is improved, but device requirements become impractical for mobile devices
Solution Approach 1:
The system segments the face image and generates detailed depth maps for each segment using computationally efficient least square minimization. This segmentation approach enables high rendering quality to be achieved on mobile devices with limited processing capabilities by avoiding the need for multiple images and complex processing clusters.
Solution Approach 2:
The system creates a detailed depth map representation that can be used for rendering visual effects without requiring multiple captured images. This single-image-based approach with generated depth maps reduces device requirements while maintaining rendering quality suitable for mobile platforms.
4Speed
If complex visual effects are rendered in real-time, then responsiveness is improved, but computational resources required become impractical
Solution Approach 1:
The rendering process is segmented by processing facial segments independently and in parallel. This segmentation enables real-time rendering of complex visual effects by distributing computational workload across multiple segments, achieving high rendering speed without requiring impractical computational resources.
Data Source
AI summary
Embodiments provide techniques for rendering augmented reality effects on an image of a user's face in real time. The method generally includes receiving an image of a face of a user. A global facial depth map and a luminance map are generated based on the captured image. The captured image is segmented into a plurality of segments. For each segment in the plurality of segments, a displacement energy of the respective segment is minimized using a least square minimization of a linear system for the respective segment. The displacement energy is generally defined by a relationship between a detailed depth map, the global facial depth map and the luminance map. The detailed depth map is generated based on the minimized displacement energy for each segment in the plurality of segments. One or more visual effects are rendered over the captured image using the generated detailed depth map.