Fingernail Segmentation for Hand-Centric XR Virtual Object Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
XR systems face challenges in processing large amounts of extraneous real-world scene image data, leading to a breakdown in the illusion of wearing virtual objects when a user's hand is further away, due to limited processing power.
Innovation Solution
The XR system captures video frame data, detects the user's hand, generates a cropping boundary box, and processes cropped video frame data to generate a 3D model of the hand, allowing for the display of virtual objects that appear to be worn by the user, using hand-tracking and fingernail segmentation techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the XR system processes all real-world scene image data to maintain high measurement precision for hand detection, then the detection accuracy improves, but the processing time increases significantly
Solution Approach 1:
The patent segments the image processing task by first detecting the hand region and then performing detailed fingernail segmentation only on the cropped hand region. This divides the large-scale scene processing into a preliminary hand detection stage and a focused fingernail segmentation stage, reducing the computational burden on the second stage while maintaining overall detection precision.
Solution Approach 2:
The patent performs preliminary hand detection and cropping before conducting detailed fingernail segmentation. By预先 identifying and isolating the hand region in the full scene, the system prepares the data in advance for the subsequent detailed analysis, avoiding the need to process the entire scene at high resolution and thus reducing processing time.
2Manufacturing precision
If the XR system uses detailed fingernail segmentation to improve virtual object alignment, then the manufacturing precision of the virtual try-on effect improves, but the device complexity increases
Solution Approach 1:
The patent implements a two-stage segmentation approach: first segmenting the hand region from the full scene, then segmenting the fingernails from the cropped hand region. This hierarchical segmentation breaks down the complex task of precise fingernail-level virtual object alignment into manageable steps, reducing the computational complexity of each individual segmentation operation while achieving high overall precision.
Solution Approach 2:
The patent applies different levels of processing quality to different regions: full-scene hand detection uses moderate precision, while fingernail segmentation uses high precision only on the localized hand region. This local quality approach ensures high manufacturing precision where needed (on the fingernails) without applying excessive processing complexity to the entire scene.
3Productivity
If the XR system crops video frames to reduce data volume, then the processing speed improves, but the loss of information about hand position and context increases
Solution Approach 1:
The patent performs preliminary hand detection in the full scene to establish the hand's position and context before cropping. This preliminary action captures essential spatial information about where the hand is located in the overall scene, preserving contextual information even after the subsequent cropping operation reduces the data volume for detailed processing.
Data Source
AI summary
An extended Reality (XR) system provides methodologies for displaying virtual objects in a hand-centric XR experience. The XR system provides an XR user interface of an XR system to a user. The XR system captures video frame data of a hand of the user and detects the hand of the user based on the video frame data and a hand-detecting model. The XR system generates a cropping boundary box based on the detection of the hand and the video frame data and generates cropped video frame data based on the cropping boundary box and the video frame data. The XR system generates a 3D model of a portion of the hand of the user based on the cropped video frame data and a virtual object based on the 3D model of the portion of the hand of the user and a 3D texture. The XR displays the virtual object in the XR user interface.