3D User Representation Relighting for Accurate Appearance Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques fail to accurately represent a user's current appearance due to discrepancies in lighting conditions between image capture and real-time environments, leading to inaccurate and dishonest representations in electronic devices.
Innovation Solution
A combined user representation is generated using a first user representation from live frame-specific 3D data and a second user representation from PiFU texture data, adjusted to match current lighting conditions through color grading and relighting, enabling a more accurate and honest portrayal during communication sessions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a device provides an avatar representation of a user based on images obtained minutes, hours, days, or years before, then the device can provide a user representation, but the representation does not accurately represent the user's current (real-time) appearance and lighting conditions
Solution Approach 1:
The system performs preliminary actions by capturing and processing user images in advance to create a base 3D model and texture atlas. This pre-processing enables real-time representation generation without requiring complete real-time data capture, thus resolving the contradiction between having current representations and avoiding time delays.
Solution Approach 2:
The user representation is segmented into multiple components: a 3D mesh model for geometry, a texture atlas for surface appearance, and separate lighting condition data. This segmentation allows each component to be processed and updated independently, enabling efficient real-time updates of appearance and lighting without requiring complete reprocessing.
2Measurement precision
If the system uses detailed 3D mesh and texture data to represent the user, then the representation accuracy is improved, but the computational requirements and bandwidth increase
Solution Approach 1:
The system creates a simplified copy of the user's appearance through a 3D mesh model that captures essential geometric features, and a texture atlas that stores surface appearance information. This copying approach maintains high representation accuracy while significantly reducing data complexity compared to transmitting or processing complete high-resolution image sequences.
Solution Approach 2:
The system transitions from 2D images to a 3D parameter space by creating a 3D mesh model with vertices and faces. This dimensional transformation allows efficient storage and manipulation of user geometry using compact mathematical representations (vertex positions, normal vectors, texture coordinates) rather than large image datasets.
Data Source
AI summary
Various implementations disclosed herein include devices, systems, and methods that adjust a combined user representation via light normalization techniques. For example, a process may include obtaining a first user representation of at least a first portion of a user generated via a first technique based on the user in a first lighting condition in a first physical environment. The process may further include obtaining a second user representation of at least a second portion of the user, the second user representation being generated by generating and delighting an initial user representation based on a lighting representation of a second physical environment having a second lighting condition, and producing the second user representation by relighting the delighted initial user representation based on the first lighting condition. The process may further include generating combined user representation based on the first user representation and the second user representation.