XR Hand and Tool Tracking via Machine Vision Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR systems face inefficiencies when users try to interact with virtual objects while holding physical tools, as these tools interfere with hand tracking and require users to set them down, disrupting the natural interaction process.
Innovation Solution
A machine vision-based system that differentiates between a user's hand and tools, allowing the system to track both and enable tool-based interactions by identifying tools through image processing and machine learning, enabling users to interact with virtual objects without putting down physical tools.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine vision tracking is used to track user's hand, then hand tracking accuracy is improved, but physical tools in the hand interfere with tracking and reduce reliability
Solution Approach 1:
The system segments the hand-tool assembly into distinct components: the user's hand and the physical tool. Machine vision algorithms identify and track each separately by detecting anatomical landmarks on the hand and geometric features on the tool, allowing independent tracking of both objects even when the tool occludes parts of the hand.
Solution Approach 2:
The system introduces an intermediary computational model that predicts hand pose from partial observations. When the tool blocks the hand, the model uses visible fingers and contextual information to infer the complete hand position and orientation, maintaining tracking reliability despite occlusion.
2Adaptability or versatility
If users hold physical tools to interact with virtual objects, then interaction versatility is improved, but hand tracking is disrupted and ease of operation deteriorates
Solution Approach 1:
The system merges the tracking data of the physical tool and the user's hand into a unified interaction model. The tool's position and orientation are combined with hand pose estimation to create a comprehensive representation of the user's intent, enabling natural tool-based interactions with virtual objects without requiring the user to set down the tool.
Solution Approach 2:
The tracking system is designed to handle multiple interaction modes universally: direct hand gestures when no tool is present, and tool-mediated gestures when a tool is held. The system automatically adapts its interpretation of user intent based on whether a tool is detected, providing a seamless multi-functional interaction experience.
Data Source
AI summary
In an example in accordance with the present disclosure, an extended reality system is described. The extended reality system includes an imaging system to present virtual objects to a user. The extended reality system also includes a machine vision tracking system. The machine vision tracking system includes a camera to capture images of a user's hand a tool grasped in the user's hand. The machine vision tracking system also includes a processor. The processor 1) tracks a position and orientation of the user's hand in physical space, 2) identifies the tool grasped in the user's hand, and 3) tracks a position and orientation of the tool in physical space. The extended reality system also includes an XR controller to manipulate the XR environment based on the position and orientation of the tool and the position and orientation of the hand.