Hand-Tracking Stabilization with Temporal Filtering for AR Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing user interfaces for augmented reality head-wearable devices face challenges in accurately selecting and interacting with virtual 3D objects due to hand pose changes during gestures like pinching, which can disrupt the selection process.
Innovation Solution
An improved hand-tracking model is trained to make hand poses invariant to movement during pinch gestures, combined with temporal smoothing and filtering of skeletal 3D model features, and a hysteresis function to stabilize target selection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If hand pose tracking is used for selecting virtual 3D objects, then user interaction capability is improved, but hand pose changes during gestures cause selection instability
Solution Approach 1:
The system performs preliminary actions by training the hand-tracking model to recognize invariant features before selection occurs, and by pre-filtering skeletal data to remove noise that would cause selection instability during gestures
Solution Approach 2:
The patent introduces intermediary components including a temporal smoothing filter that processes skeletal 3D model features and a hysteresis function that acts as a mediator between hand pose detection and target selection, stabilizing the selection process during gestures
2Adaptability or versatility
If hand pose changes are allowed for natural gestures, then gesture expressiveness is improved, but target selection accuracy deteriorates
Solution Approach 1:
The patent applies local quality by making the hand-tracking model selectively invariant to specific features that change during gestures (such as finger positions) while maintaining sensitivity to other features that indicate intent (such as hand orientation and position), thereby preserving gesture expressiveness while improving selection accuracy
Solution Approach 2:
The system uses feedback mechanisms through temporal smoothing of skeletal data and hysteresis functions that consider previous selection states, allowing the system to maintain accurate target selection even when hand poses change during natural gestures
Data Source
AI summary
An Augmented Reality (AR) system provides stabilization of hand-tracking input data. The AR system provides for display a user interface of an AR application. The AR system captures, using one or more cameras of the AR system, video frame tracking data of a gesture being made by a user while the user interacts with the AR user interface. The AR system generates skeletal 3D model data of a hand of the user based on the video frame tracking data that includes one or more skeletal 3D model features corresponding to recognized visual landmarks of portions of the hand of the user. The AR system generates targeting data based on the skeletal 3D model data where the targeting data identifies a virtual 3D object of the AR user interface. The AR system filters the targeting data using a targeting filter component and provides the filtered targeting data to the AR application.