Non-Riggable 3D Face Modeling for Realistic Video Avatars
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video conferencing systems lack the ability to create a non-riggable model of a person's face for generating avatars, which are essential for providing a realistic and immersive virtual presence in 3D video conference environments.
Innovation Solution
A system and method for generating a non-riggable model of a person's face using neural networks to estimate and update 3D participant representation information based on gaze direction, head pose, and expressions, allowing for the creation of avatars that accurately reflect the participant's appearance and gaze within a virtual 3D environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional video conferencing systems are used, then system simplicity is maintained, but the ability to create realistic 3D avatar representations is lost
Solution Approach 1:
The patent introduces an intermediary processing system that captures video input and transforms it into 3D avatar representations. This intermediary layer includes neural networks and rendering systems that bridge the gap between simple video conferencing and complex 3D visualization, allowing traditional systems to gain advanced capabilities without complete redesign
Solution Approach 2:
The patent creates a digital copy (avatar) of the participant's appearance and characteristics. By generating a 3D model that replicates the participant's visual features, the system enables realistic representation without requiring the actual participant to be physically present, thus maintaining system simplicity while improving representation accuracy
2Reliability
If detailed 3D facial models are created, then avatar realism is improved, but data transmission bandwidth requirements increase
Solution Approach 1:
The patent transforms detailed 3D facial models into a compressed parameter representation. Instead of transmitting complete high-resolution 3D mesh data, the system extracts and transmits key parameters (such as shape coefficients, texture parameters, and pose information) that can be used to reconstruct the avatar at the receiving end, significantly reducing bandwidth requirements while maintaining realism
Solution Approach 2:
The patent extracts only the essential features and parameters needed to create a realistic avatar representation, separating the critical information from the redundant data. This extraction process removes unnecessary geometric details and focuses on transmitting only the most important facial characteristics, thereby reducing data transmission requirements
3Manufacturing precision
If non-riggable models are used, then model accuracy is improved, but the ability to animate and control expressions is reduced
Solution Approach 1:
The patent implements a dynamic parameter system that allows the non-riggable model to adapt and change expressions through parameter modulation rather than rigid structural control. By using neural networks to dynamically adjust facial parameters based on captured expressions, the system maintains high model accuracy while gaining the versatility to animate and control facial expressions naturally
Data Source
AI summary
A method for creating a non-riggable model of a face of a person, the method includes obtaining video and depth information regarding the face of the person, wherein different images of the video are acquired by a camera at different camera locations; and for each image of the different images repeating the steps of: separating face information from background information; determining translation and rotation parameters that represent the different camera locations; and generating the non-riggable model of the face of the person based on the face information and the translation and rotation parameters.