XR Device Face 3D Reconstruction with Glasses Removal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
XR devices face challenges in accurately generating 3D images of users wearing glasses or sunglasses, as these accessories are often misinterpreted as part of the face, leading to inaccuracies in the generated 3D content.
Innovation Solution
The XR device determines if objects other than the face are present in the 2D image, removes them, generates a mask image to identify the type of object, and combines pre-defined 3D images specific to that object type with the user's 3D face image to create a more accurate 3D representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the XR device generates 3D image based on user's 2D image acquired by camera sensor, then the device can provide XR content, but the accuracy of user's 3D image/video is reduced when glasses are recognized as part of the face
Solution Approach 1:
The patent segments the image processing into distinct stages: initial object detection, classification phase to identify glasses versus facial features, and separate processing paths for different object types. This segmentation allows the system to correctly differentiate glasses from facial features and apply appropriate processing to each, resolving the misclassification issue that degraded 3D image accuracy
Solution Approach 2:
The patent introduces an intermediary classification module that acts as a mediator between the object detection stage and the 3D generation stage. This intermediary analyzes detected objects to determine whether they are glasses or facial features, and routes them to appropriate processing paths. This intermediary prevents direct misclassification and ensures accurate 3D image generation by properly distinguishing between different object types
2Measurement precision
If the device processes and removes objects from 2D image before generating 3D image, then the accuracy of 3D image is improved, but the device complexity increases due to additional processing steps
Solution Approach 1:
The patent applies preliminary action by performing object classification and determination before the main 3D image generation process. The system pre-identifies and categorizes objects in the 2D image, determining which are glasses versus facial features before proceeding with 3D reconstruction. This preliminary classification simplifies the overall process by preventing errors early, rather than requiring complex post-processing corrections
Solution Approach 2:
The patent applies local quality by implementing different processing strategies for different regions and object types within the image. Facial features undergo one type of processing while detected glasses undergo a different processing path. This localized, differentiated processing approach improves accuracy for each object type without requiring the entire system to be overly complex, as each region receives appropriately tailored processing
Data Source
AI summary
A method for providing XR content includes obtaining a first 2D image representing a user's face, determining whether an object which is not the user's face is included in the first 2D image, in response to determining that the object is included in the acquired first 2D image, generating a second 2D image representing the user's face without the object based on the first 2D image, generating a first 3D image corresponding to the generated second 2D image, generating a mask image representing a difference between the first 2D image and the second 2D image, determining a type of object based on the generated mask image, obtaining a 3D preset including one or more 3D images corresponding to the determined type of object, generating a second 3D by combining at least one of one or more 3D images with the first 3D image and providing XR content including the generated second 3D image.