XR Virtual Touch Control With Multi-View Fingertip Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual keyboards and touch controls in extended reality (XR) systems face challenges in accurately determining whether a trigger fingertip has touched a function region due to visual blocking and lack of tactile feedback, leading to inaccurate input and the inability to perform blind typing.
Innovation Solution
A method using a system with at least two cameras in an XR wearable device to track the position of a trigger fingertip relative to trigger determination points on a palm, calculating ratios of positional information from multiple video streams to confirm actual touch without physical sensors, providing tactile feedback through sound, vibration, or mechanical means.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If visual detection and calculation is used to determine whether a trigger fingertip touches a function region, then the system can operate without physical sensors, but the determination accuracy deteriorates due to visual blocking by hands and fingers
Solution Approach 1:
The patent introduces an intermediary computational model (joint recognition model or human hand detection model) that indirectly infers touch status by analyzing hand joint positions and movements, rather than directly detecting fingertip contact. This mediator enables touch determination without physical sensors while compensating for visual blocking issues through algorithmic reasoning about hand anatomy and motion patterns.
Solution Approach 2:
The patent replaces the mechanical sensor-based detection system with a visual-computational system. Instead of using physical sensors on gloves or rings to detect touch, the system uses camera-based visual detection combined with computational models to infer touch status from hand joint positions, thereby eliminating the need for wearable mechanical devices.
2Measurement precision
If sensor gloves or sensor rings are used to accurately determine fingertip touch, then measurement precision improves, but device complexity and user burden increase due to requiring wearable auxiliary devices
Solution Approach 1:
The patent enables the system to determine touch status using only the existing camera equipment already present in the XR device, without requiring additional wearable sensors. The computational models process visual data from the built-in cameras to infer fingertip contact, allowing the system to serve itself with its own resources rather than depending on external auxiliary devices.
Solution Approach 2:
The patent extracts the essential touch detection function from the physical sensor hardware and implements it through software-based computational models processing visual data. By separating the detection logic from physical sensors, the system achieves accurate touch determination without requiring wearable sensor gloves or rings, removing the harmful dependency on auxiliary devices.
3Ease of operation
If virtual keys are assigned on palm and fingers, then tactile feedback is improved through physical contact, but the problem of fingertip visually blocking function regions persists
Solution Approach 1:
The patent shifts the detection approach from two-dimensional visual image analysis to three-dimensional spatial reasoning by incorporating hand joint detection models that understand the three-dimensional structure and movement of hand joints. This dimensional enhancement allows the system to accurately determine fingertip position and contact status even when the fingertip visually blocks the function region, by inferring position from joint kinematics rather than direct visual observation of the fingertip itself.
Data Source
AI summary
A method in XR technology to determine whether an actual touch of a function region during virtual typing or touch control performed on the palm or on a real object is achieved, by marking preset points on the palm; assigning a function region to each preset point; setting two trigger determination points WL and WR; acquiring N number of video streams with parallax; tracking and determining whether a trigger fingertip P is located between the two trigger determination points in all corresponding N number of images bearing a same time from the N number of video streams; in each of the N number of images, calculate a ratio being a difference in X-axis value between P and WR to a difference in X-axis value between WL and P; only when all ratios in all images are the same, the trigger fingertip P is determined to have touched the function region.