HMD NIR Face Mesh Estimation Without Companion Cameras
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR systems face challenges in generating precise mesh models of a user's face for avatars due to obstructed views caused by head-mounted displays, which complicates the use of traditional RGB image-based training methods and increases costs with companion devices.
Innovation Solution
Utilizing near-infrared (NIR) cameras integrated within the HMD to capture face images, training a machine learning (ML) model to estimate mesh models by converting synthetic HMD user images to NIR, and comparing predicted mesh models with training models to enhance mesh estimation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional RGB image-based training methods are used for mesh estimation, then color information is available for training, but the HMD obstructs the view of the user's face making accurate mesh generation difficult
Solution Approach 1:
The HMD uses its own integrated NIR cameras to capture facial images for mesh estimation, eliminating the need for external companion devices. The system serves itself by leveraging resources already present in the device structure.
Solution Approach 2:
The patent changes the wavelength parameter of light from visible RGB spectrum to near-infrared spectrum. This allows the HMD to capture facial images through the device obstruction since NIR light has different penetration and reflection properties, enabling mesh estimation without requiring external devices.
2Measurement precision
If companion devices are used to capture unobstructed views of the user's face, then mesh estimation accuracy improves, but system cost and complexity increase
Solution Approach 1:
The HMD system captures its own facial data using integrated NIR cameras, eliminating the need for separate companion devices. This self-service approach reduces manufacturing costs by removing additional hardware requirements while maintaining mesh estimation functionality.
Solution Approach 2:
The patent extracts the facial capture function from external companion devices and relocates it to the HMD itself using NIR imaging. This extraction eliminates the need for separate devices, reducing overall system cost and complexity while preserving the essential function of capturing facial data for mesh generation.
3Ease of operation
If NIR cameras are used instead of RGB cameras, then unobstructed facial views can be obtained through the HMD, but color information is lost
Solution Approach 1:
The patent changes the operational parameter from visible light detection to near-infrared detection. This parameter change allows the system to penetrate HMD obstructions and capture facial geometry information, sacrificing color data but gaining the ability to obtain clear facial views through the device.
Solution Approach 2:
The patent converts the limitation of NIR imaging (lack of color information) into a benefit by focusing on geometric precision. The monochromatic NIR images provide high-contrast facial structure data that, when processed through ML models, yields accurate mesh estimates without the interference of color variations and lighting conditions.
4Ease of manufacture
If synthetic NIR HMD user images are generated through ML model training, then the model can be trained without real NIR data, but training accuracy may be affected
Solution Approach 1:
The patent performs preliminary action by generating synthetic NIR HMD user images before actual model training. These synthetic images, created by converting RGB images to NIR domain and simulating HMD capture conditions, prepare the training data in advance, enabling the ML model to learn from realistic synthetic data that mimics actual capture scenarios.
Solution Approach 2:
The patent creates copies of real facial data in synthetic form. By generating synthetic NIR images that replicate the characteristics of real HMD-captured facial images, the system provides sufficient training data without requiring actual captured images, maintaining training effectiveness while avoiding data privacy and availability issues.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables precise mesh model generation for avatars without the need for additional devices, improving immersion and reducing costs by leveraging NIR imaging and ML for accurate face representation in XR environments.
Implementation Method 1
obtain a set of near infrared (NIR) images of a first face from a set of cameras on a head mounted device (HMD)
Data Source
AI summary
Systems and techniques are described for performing mesh estimation using head mounted display (HMD) images. For example, a computing device can obtain a set of near infrared (NIR) images of a first face from a set of cameras on a head mounted device (HMD) worn on the first face. The computing device can predict, using a machine learning (ML) model, a set of parameters. The set of parameters describe a mesh model of the first face based on the set of NIR images. The computing device can generate, using the ML model, the mesh model of the first face based on the predicted set of parameters.