Avatar Texture Synthesis for Realistic Mobile Facial Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing avatar systems are computationally intensive and do not provide nuanced facial representations or emotional states in realistic lighting, particularly on mobile devices.
Innovation Solution
A hybrid approach using trained autoencoders and camera feed data to generate avatars, breaking the process into smaller algorithms for face portions like eyes and mouth, reducing computational resources, and employing neural networks for texture prediction and lighting adjustment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If existing avatar systems use general and graphics processors for real-time rendering, then rendering quality and realism are improved, but computational resource consumption increases making it incompatible with mobile devices
Solution Approach 1:
The patent segments the avatar rendering process into distinct components: geometry generation from expression latents, texture synthesis using neural networks, and lighting computation. This segmentation allows each component to be optimized independently, reducing overall computational requirements while maintaining rendering quality.
Solution Approach 2:
The system performs preliminary actions by pre-training neural networks on large datasets of facial expressions and lighting conditions. These pre-trained models can then be deployed on mobile devices to generate realistic avatars without requiring heavy computational resources during runtime, as the intensive learning work was done beforehand.
2Use of energy by moving object
If existing avatar systems use simplified rendering methods, then computational resources are reduced for mobile devices, but the ability to communicate nuanced facial representations and emotional states is lost
Solution Approach 1:
The patent changes parameters by using latent variable representations that encode complex facial expressions and emotional states in a compressed form. These latents capture nuanced information efficiently, allowing rich facial representations to be transmitted and rendered with reduced computational overhead compared to full-resolution image processing.
Solution Approach 2:
The system introduces neural networks as intermediaries that bridge between compact latent representations and realistic visual output. These neural networks act as mediators that can generate high-fidelity facial expressions from low-dimensional latents, preserving emotional nuance while avoiding the computational burden of processing full-resolution facial images directly.
3Loss of information
If existing avatar systems process complete facial images, then comprehensive facial information is maintained, but computational complexity increases making real-time processing difficult on mobile devices
Solution Approach 1:
The patent extracts only the essential facial information needed for avatar rendering by using landmoat-based expression latents that capture key facial features and expressions. This extraction approach removes unnecessary computational complexity while preserving the essential facial information required for realistic avatar representation.
Data Source
AI summary
Rendering an avatar for a user in a communication session includes obtaining enrollment data associated with the user. For each of one or more frames of the communication session, a set of expression latents is obtained for the user. A first texture for a first portion of the face of the user is generated based on the enrollment data and expression latents. A combined albedo map is generated based on one or more identity textures from the enrollment data and the first texture. A target texture is generated based on the one or more identity textures and the combined albedo map for a particular frame of the one or more frames.