3D User Representation via Segmented 3D Data and Frame-Specific Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques fail to accurately and efficiently represent users in real-time within electronic devices, often using outdated avatar representations that do not reflect current appearances, such as not showing a user smiling or not accounting for changes like a beard.
Innovation Solution
The use of a set of values representing a 3D shape and appearance of a user's face, defined relative to multiple points on a surface, which combines predetermined 3D data with frame-specific data to generate accurate and up-to-date user representations, utilizing techniques like pixel-aligned implicit functions (PIFu) and Monge-Kantorovich Color Transfer for color adjustment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing techniques use outdated avatar representations, then device complexity is reduced, but user representation accuracy deteriorates
Solution Approach 1:
The system segments user representation into two parts: a predetermined 3D model (from enrollment data) and frame-specific updates (from real-time sensor data). This segmentation allows the system to maintain accuracy by updating only the necessary portions while keeping the overall structure simple and efficient.
Solution Approach 2:
The system performs preliminary action by creating a predetermined 3D user representation during enrollment. This pre-established model serves as a foundation that can be quickly updated with minimal real-time processing, thus improving accuracy without significantly increasing device complexity.
2Measurement precision
If 3D mesh or 3D point cloud is used for accurate user representation, then user representation accuracy is improved, but computational requirements and bandwidth increase
Solution Approach 1:
The system extracts only the essential depth information needed for accurate user representation, rather than processing complete 3D meshes or point clouds. By taking out and processing only the critical depth values, the system maintains accuracy while significantly reducing computational requirements and bandwidth usage.
3Adaptability or versatility
If depth values are defined relative to a single camera location, then integration with existing systems is improved, but user representation accuracy deteriorates
Solution Approach 1:
The system transitions from defining depth relative to a single camera location to defining depth relative to multiple points on a surface. This dimensional change allows the system to achieve higher accuracy by considering multiple reference points while maintaining compatibility with existing single-camera systems through the use of curviplanar surfaces.
Data Source
AI summary
Various implementations disclosed herein include devices, systems, and methods that generate a combined 3D representation of a user. For example, a process may include obtaining a first three-dimensional (3D) representation of a first portion of a user. The process may further include obtaining a sequence of frame-specific second 3D representations in a period of time, each of the frame-specific second 3D representations represent a second portion of the user. The process may further include generating a combined 3D representation of the user for the period of time by modifying the first 3D representation with a respective frame-specific second 3D representation.