3D Virtual Cursor Pose Estimation and Volumetric Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Navigating and interacting with three-dimensional virtual environments is challenging due to difficulties in identifying and effectively using interactive elements, especially when combined with two-dimensional user interface elements, leading to inefficiencies and frustration.
Innovation Solution
A system and method using a head-mounted display device that receives hand pose inputs, estimates the pose of a virtual input source, identifies virtual object surfaces, and renders frames with pixel color changes based on distance to a virtual input line, providing visual feedback to enhance interaction in three-dimensional virtual environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional two-dimensional user interface elements are carried over to three-dimensional virtual environments, then familiar interaction patterns are maintained, but user efficiency and ease of operation deteriorate due to difficulty in identifying and interacting with elements in complex three-dimensional spaces
Solution Approach 1:
The patent extends traditional two-dimensional cursor interaction into three-dimensional space by introducing a volumetric selection region that radiates from the cursor position in multiple directions. This allows users to interact with virtual objects in 3D environments using familiar cursor-based operations while naturally accommodating spatial depth, thereby resolving the contradiction between maintaining traditional UI patterns and improving interaction efficiency in three-dimensional spaces.
2Speed
If handheld motion controllers are used to improve speed and dexterity, then interaction speed increases, but device complexity and difficulty of detecting and measuring increase due to operating away from the virtual camera axis
Solution Approach 1:
The patent introduces a virtual cursor as an intermediary between the handheld motion controller and the virtual environment. The cursor is positioned along the virtual camera axis, providing a stable reference point that is easier to detect and measure. The controller operates the cursor indirectly, allowing users to benefit from both the speed and dexterity of motion controllers while maintaining accurate positioning through the camera-aligned cursor mediator.
3Ease of operation
If visual feedback is provided continuously to help users locate interactive elements, then ease of operation improves, but loss of information increases due to potential obscuration of the virtual environment view
Solution Approach 1:
The patent implements visual feedback that activates periodically or on-demand based on user interaction states rather than continuously. For example, the volumetric selection region can be displayed when the cursor approaches potential interaction targets or when interaction mode is activated, and hidden or dimmed when not needed. This periodic display provides necessary guidance information while minimizing obscuration of the virtual environment, resolving the contradiction between visibility and view clarity.
Data Source
Figure 1
Figure 2A~2B
Figure 2C
AI summary
Devices, systems, and methods for interacting with a three-dimensional virtual environment, including receiving an input associated with a change in pose of a user's hand; estimating, based on at least the input, a first pose in the virtual environment for an input source associated with the hand; identifying a surface of a virtual object in the virtual environment; rendering a frame depicting elements of the virtual environment, the frame including a pixel rendered for a position on the surface; determining a distance between the position and a virtual input line extending through a position of the first pose and in a direction of the first pose; changing a pixel color rendered for the pixel based on the distance between the position and the virtual input line; and displaying the frame including the pixel with the changed pixel color to the user via a head-mounted display device.