Neural Shading for Photorealistic Mobile Avatars
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing avatar systems are computationally intensive and do not provide nuanced facial representations or emotional states in realistic lighting, particularly on mobile devices.
Innovation Solution
A machine learning-based shading technique using a shading encoder and decoder to generate photorealistic avatars by applying shading latents to identity textures, generating neural maps, and combining them with multipass rendering to create a target texture.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional avatar rendering systems are used, then photorealistic lighting and facial expressions can be achieved, but computational requirements become excessively high for mobile devices
Solution Approach 1:
The system performs preprocessing of identity textures and extraction of shading latents before real-time rendering. Identity textures are pre-computed from user images, and shading latents are extracted from captured images in advance, enabling the neural shading network to operate with reduced computational load during live avatar rendering on mobile devices
Solution Approach 2:
The patent introduces shading latents as an intermediary representation between captured images and final avatar rendering. These compact latent vectors encode lighting and shading information, serving as a mediator that reduces the complexity of direct neural network processing while preserving photorealistic lighting effects
2Loss of information
If existing avatar systems are used, then basic facial representation is achieved, but nuanced facial expressions and emotional states cannot be communicated
Solution Approach 1:
The system transforms facial expression representation by changing from discrete animation parameters to continuous neural latent vectors. The neural shading network processes shading latents that capture subtle variations in lighting and texture, enabling nuanced facial expressions and emotional states to be represented and transmitted without requiring complex explicit parameter control
3Manufacturing precision
If high-performance processors are used for avatar rendering, then realistic lighting can be achieved, but the system cannot run on mobile devices
Solution Approach 1:
The patent extracts and separates the computationally intensive neural shading computations from the mobile device. The neural shading network is trained and executed on powerful servers, while only the lightweight inference process runs on mobile devices. This extraction enables realistic lighting effects to be achieved without requiring mobile devices to have high-performance processors
Solution Approach 2:
The system uses pre-trained neural shading networks and pre-computed identity textures as copies that can be deployed to mobile devices. Instead of running full rendering pipelines locally, mobile devices use copied models and pre-processed data from server environments, enabling access to photorealistic avatar rendering capabilities on resource-constrained devices
Data Source
AI summary
Rendering an avatar for a user in a communication session includes obtaining one or more identity textures for the user for the session and obtaining, throughout the session, shading latents for the user. The shading latents are derived from lighting and other information and are utilized as input into a decoder to obtain one or more neural maps. A target texture is generated by warping the identity textures based on the neural maps. The target texture is used for rendering the avatar.