Augmented Reality Hand Tracking via Multi-Coordinate Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality technologies face challenges in providing a natural and intuitive way for users to interact with virtual 3D scenes using handheld devices, particularly in accurately tracking hand movements and gestures without markers, and in enabling seamless manipulation of objects both near and far.
Innovation Solution
The implementation of a user device with multiple coordinate systems (object-to-device, hand-to-device, and virtual cursor) that utilizes a 3D sensing device to track hand movements, allowing for two mapping modes (GoGo and Magic Hand) to control a virtual hand or cursor in three dimensions, enabling direct manipulation of objects through translational and rotational movements, and incorporating gesture and posture detection for precise interactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If marker-based tracking is used to track hand movements, then tracking reliability is improved, but ease of operation deteriorates
Solution Approach 1:
The patent extracts and removes the marker element from the tracking system, transitioning from marker-based to markerless hand tracking. This allows the system to track hand movements and gestures directly using the camera and image processing algorithms, eliminating the need for users to attach or wear markers, thereby improving ease of operation while maintaining tracking reliability through advanced computer vision techniques.
2Ease of operation
If touch interaction is used to manipulate virtual objects, then ease of operation is improved, but adaptability deteriorates
Solution Approach 1:
The patent implements a dynamic interaction system that adapts between different interaction modes based on detected hand gestures. The system can switch between touch-based interaction when the hand contacts the device screen and gesture-based interaction when the hand is in free space. This dynamic adaptation allows the system to maintain ease of operation through touch while gaining versatility through gesture recognition, enabling users to manipulate virtual objects in various ways depending on the context.
3Measurement precision
If 3D sensing device is added to track hand movements, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent introduces an intermediary processing layer that translates complex 3D sensing data into simplified control commands for the virtual hand. The system uses image processing algorithms and coordinate transformation mechanisms as intermediaries to convert raw camera or depth sensor data into meaningful hand position and gesture information. This intermediary layer abstracts the complexity of 3D sensing, allowing the system to achieve high measurement precision while keeping the overall device architecture manageable through modular processing stages.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
A method for spatial interaction in Augmented Reality (AR) includes displaying an AR scene that includes an image of a real-world scene, a virtual target object, and a virtual cursor. A position of the virtual cursor is provided according to a first coordinate system within the AR scene. A user device tracks a pose of the user device relative to a user hand according to a second coordinate system. The second coordinate system is mapped to the first coordinate system to control movements of the virtual cursor. In a first mapping mode, virtual cursor movement is controlled to change a distance between the virtual cursor and the virtual target object. In a second mapping mode, virtual cursor movement is controlled to manipulate the virtual target object. User input is detected to control which of the first mapping mode or the second mapping mode is used.