3D Virtual Object Translation Using Pinch Gesture Offset Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR systems lack effective input modalities for translating virtual objects, as touch-based and haptic devices are disruptive and not optimal for immersive experiences, and hand-tracking methods for rotating virtual objects are inadequate without tracking both hands.
Innovation Solution
Implementing a pinch gesture detection system using machine learning to translate virtual objects by determining a selection location within the object's boundaries and applying an offset vector based on the pinch gesture's movement, allowing continuous translation without diverting user attention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If touch-based or haptic devices are used for input, then user interaction capability is provided, but user immersion is disrupted and attention must be diverted from the virtual environment
Solution Approach 1:
The patent replaces physical touch-based or haptic input devices with a camera-based vision system that tracks hand gestures in the air. The camera captures images, and machine learning algorithms detect hand poses and gestures without requiring physical contact, thereby substituting mechanical input with optical sensing and computational analysis.
Solution Approach 2:
The patent introduces an intermediary computational system that translates natural hand gestures into digital commands. The camera and machine learning model act as intermediaries between the user's natural hand movements and the virtual environment, allowing interaction without direct physical contact with input devices.
2Ease of operation
If hand-tracking is used to rotate virtual objects, then gesture-based control is achieved, but the method is inadequate when both hands need to be tracked simultaneously
Solution Approach 1:
The patent segments the hand-tracking problem into distinct processing stages: image capture, hand detection, pose estimation, and gesture recognition. By dividing the complex task of tracking multiple hands into modular components, the system can handle multiple hands independently and combine their information without overwhelming computational complexity.
Solution Approach 2:
The patent transitions from 2D image coordinates to 3D hand pose estimation by inferring depth and spatial relationships from camera images. This dimensional transformation allows the system to track hands in three-dimensional space and accurately determine gestures even when both hands are present, resolving the limitation of previous 2D-based approaches.
3Measurement precision
If traditional input devices are used, then precise input control is achieved, but the system requires additional hardware that increases complexity
Solution Approach 1:
The patent makes the camera serve multiple functions: it captures the virtual environment for rendering, detects hand gestures for input, and tracks object positions. This multi-functionality eliminates the need for separate input devices while maintaining precision through the same optical sensing system used for augmented reality rendering.
Solution Approach 2:
The system uses its own camera and processing resources to detect hand gestures and interpret user intent, making the XR system self-sufficient for input detection without requiring external controllers or additional sensors. The machine learning model processes the camera data to automatically distinguish between hand gestures and the virtual environment.
Data Source
AI summary
An extended Reality (XR) system that translates virtual objects in an XR user interface using gestures is provided. The XR system displays a virtual object to a user in an extended reality interface, detects a pinch gesture and its selection location within the virtual object's boundaries, and determines an offset vector from the virtual object's center point to the pinch gesture's selection location. The XR system generates a translated virtual object using the offset vector and a current location of the pinch gesture, and displays the translated virtual object to the user.