Avatar Eyeball Texture Generation via Neural Network Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality (AR)/virtual reality (VR) devices struggle to accurately render an avatar's gaze, as the eyes are overly dependent on the rest of the facial expression and lack a separate model for the eyeballs, resulting in an inability to reproduce a real gaze.
Innovation Solution
A method is introduced to render an avatar's eyes using a separated 3D mesh and eyeball texture, where a neural network generates the eyeball texture based on detected keypoints and facial mesh, allowing for independent control of the gaze direction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the avatar's eyes are rendered using the entire facial expression model, then the facial expression reproduction is comprehensive, but the gaze accuracy deteriorates because the eyes become overly dependent on the rest of the facial expression
Solution Approach 1:
The patent divides the facial expression model into separate components: a facial expression model for the overall face and an independent eye model for the eyes. This segmentation allows the eye model to independently control gaze direction while the facial expression model handles other facial movements, thereby improving gaze accuracy without excessive complexity.
Solution Approach 2:
The eye model is extracted from the overall facial expression model as a separate, independent component. This extraction enables the eyes to be controlled independently from the rest of the facial expression, allowing accurate gaze reproduction while maintaining comprehensive facial expression capabilities.
2Ease of manufacture
If the avatar's eyes are rendered as textures only, then the rendering process is simple, but the ability to reproduce real gaze deteriorates
Solution Approach 1:
The eye rendering is segmented into two parts: a texture component for the basic eye appearance and a separate eyeball mesh component for the spherical shape and gaze direction. This segmentation maintains rendering simplicity through texture usage while enabling accurate gaze reproduction through the independent eyeball mesh transformation.
3Adaptability or versatility
If the eyeball model is coupled with the facial expression model, then the overall facial animation is coordinated, but the independence of gaze control deteriorates
Solution Approach 1:
The model structure is segmented into a facial expression model and an independent eye model. The eye model receives both facial expression inputs and independent gaze direction inputs, allowing it to adapt to both coordinated facial animation and independent gaze control requirements without excessive structural complexity.
Solution Approach 2:
The eye model is designed to be multi-functional: it responds to facial expression inputs for coordinated animation and to independent gaze direction inputs for gaze control. This universality allows the same model structure to handle both coordination and independence requirements.
Data Source
AI summary
In one embodiment, a system may capture one or more images of a user using one or more cameras, the one or more images depicting at least an eye and a face of the user. The system may determine a direction of a gaze of the user based on the eye depicted in the one or more images. The system may generate a facial mesh based on depth measurements of one or more features of the face depicted in the one or more images. The system may generate an eyeball texture for an eyeball mesh by processing the direction of the gaze and the facial mesh using a machine-learning model. The system may render an avatar of the user based on the eyeball mesh, the eyeball texture, the facial mesh, and a facial texture.