Fingernail Segmentation for Precise Hand Tracking in XR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
XR systems face processing challenges when tracking user hands due to the large amount of extraneous real-world scene data, leading to a broken illusion of wearing virtual objects, especially when the hand is further away, as they struggle to compute the location of hand portions accurately.
Innovation Solution
The XR system captures video frame data, detects the hand, generates a cropping boundary box, and processes cropped video frame data to generate a 3D model of the hand, allowing for precise virtual object placement and display, using hand-tracking and fingernail segmentation models to enhance hand-centric XR experiences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the XR system processes the entire real-world scene data to track the hand, then the tracking coverage is comprehensive, but the processing time increases and the location accuracy decreases when the hand is far away
Solution Approach 1:
The patent divides the real-world scene into multiple image zones (first image zone and second image zone) based on hand detection results. By segmenting the processing area and only applying detailed hand portion location computation to relevant zones, the system reduces overall processing time while maintaining accuracy where needed.
Solution Approach 2:
The patent applies different processing qualities to different image zones. The first image zone (where hand portions are detected) receives high-precision processing with detailed location computation, while the second image zone uses standard processing. This local quality differentiation maintains measurement precision for hand tracking while reducing total processing time.
2Reliability
If the XR system processes the entire real-world scene data to track the hand, then the tracking is comprehensive, but the device complexity increases
Solution Approach 1:
The patent segments the image processing pipeline into distinct stages: hand detection in the first image zone, virtual object placement determination, and selective processing of the second image zone. This segmentation reduces device complexity by organizing complex processing into manageable, modular components.
Solution Approach 2:
The patent performs preliminary hand detection and virtual object placement determination before processing the entire scene. By pre-identifying hand locations and determining where virtual objects should be placed, the system simplifies subsequent processing steps and reduces overall system complexity while maintaining tracking reliability.
3Productivity
If the XR system uses simplified processing for distant hands, then the processing speed increases, but the illusion of wearing virtual objects breaks
Solution Approach 1:
The patent maintains high processing quality specifically in the first image zone where hand portions are detected, even when hands are distant. This localized high-quality processing ensures accurate hand portion location computation and proper virtual object placement, maintaining the illusion of wearing virtual objects while allowing faster processing in other zones.
Solution Approach 2:
The patent performs preliminary detection of hand portions and determination of virtual object placement before final rendering. This preliminary action ensures that even distant hands receive appropriate processing attention, maintaining the illusion while enabling efficient overall processing through subsequent optimizations.
Data Source
AI summary
An extended Reality (XR) system provides methodologies for displaying virtual objects in a hand-centric XR experience. The XR system provides an XR user interface of an XR system to a user. The XR system captures video frame data of a hand of the user and detects the hand of the user based on the video frame data and a hand-detecting model. The XR system generates a cropping boundary box based on the detection of the hand and the video frame data and generates cropped video frame data based on the cropping boundary box and the video frame data. The XR system generates a 3D model of a portion of the hand of the user based on the cropped video frame data and a virtual object based on the 3D model of the portion of the hand of the user and a 3D texture. The XR displays the virtual object in the XR user interface.