XR System UI Rotation via Stable-Point Hand Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Interacting with virtual objects, such as system user interfaces (UIs), in conventional artificial reality (XR) environments is complex due to fixed screen positions, limited haptic feedback, and difficulty in accessing and performing fine-grained interactions, leading to unsatisfactory user experiences.
Innovation Solution
The interaction facilitation system renders the system UI as a 3D virtual element, tracks the user's position and hand gestures, and rotates the UI based on a stable point, enabling smooth transitions between direct touch and ray casting modes of interaction, and adjusts display modes based on distance to enhance usability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the screen is fixed at a defined distance in the XR environment, then the system UI structure is simple and stable, but the user experience deteriorates due to difficulty in accessing and performing fine-grained interactions
Solution Approach 1:
The system UI transitions from a fixed static screen to a dynamic 3D virtual object that can be freely moved, rotated, and resized within the XR environment. The UI responds to user gestures by changing its position, orientation, and scale, enabling natural interaction while maintaining system stability through programmed behavior rules.
Solution Approach 2:
The patent introduces gesture recognition and spatial tracking as intermediary mechanisms between the user and the system UI. These intermediaries translate physical hand movements into digital interactions, enabling fine-grained control without requiring complex direct manipulation of virtual elements.
2Illumination intensity
If the screen size is small in the XR environment, then the system UI structure is compact, but the user experience deteriorates due to poor content visibility
Solution Approach 1:
The system UI expands from a 2D screen concept to a 3D virtual object that occupies spatial volume in the XR environment. This dimensional transformation allows the UI to maintain compactness in terms of data structure while providing large visual presence and excellent visibility through spatial expansion and perspective rendering.
Solution Approach 2:
The UI size dynamically adapts based on distance from the user and interaction context. When the user approaches or performs fine-grained interactions, the UI enlarges to improve visibility. When distant, it maintains a compact form, optimizing both visibility and space efficiency.
3Ease of operation
If the system UI is rendered as a 3D virtual element that can be moved and rotated, then the ease of operation improves through natural grasping and rotation, but the device complexity increases due to tracking and transformation requirements
Solution Approach 1:
The system UI performs self-adjustment based on user interaction. When grasped, it automatically rotates to face the user and positions itself optimally. This self-service behavior reduces the need for complex control systems while maintaining natural interaction, as the UI adapts to user actions rather than requiring precise pre-programmed responses.
Solution Approach 2:
The system implements continuous feedback loops where the UI's position, orientation, and size are constantly adjusted based on real-time tracking of user hand position and gesture. This feedback mechanism enables natural interaction while managing complexity through reactive rather than proactive control.
4Adaptability or versatility
If multiple interaction modes (direct touch and ray casting) are implemented, then the adaptability improves for different distances, but the device complexity increases due to mode switching logic
Solution Approach 1:
The interaction mode dynamically transitions between direct touch and ray casting based on the distance between the user and the system UI. This automatic mode switching adapts to user behavior without requiring manual selection, providing versatility while managing complexity through distance-based rules rather than multiple manual controls.
Data Source
AI summary
A computer implemented method for facilitating system user interface (UI) interactions in an artificial reality (XR) environment is provided. The method includes rendering the system UI in the XR environment as a 3D virtual element. The method further includes tracking a position of a hand of a user and a pre-defined stable point on the user. The method further includes identifying, based on the tracking, that the hand has grasped a portion of the system UI and, in response, rotating the position of the system UI around the grasped portion of the system UI such that a line, between the stable point and the surface of the system UI, is moved, to be perpendicular or at a predefined angle from perpendicular to the surface of the system UI, as the user moves the system UI via the grasped portion.