Two-Handed Natural User Interface Gesture Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural user interfaces, particularly for augmented reality head-mounted displays, face challenges in accurately determining user intent and spatially perceiving gestures relative to interface elements, as actions intended for control can correspond to non-interface actions, and the apparent location of interface elements within the user's field of view can be difficult to accurately perceive.
Innovation Solution
The implementation of two-handed interactions where one hand performs a context-setting gesture to define the context for dynamic actions performed by the other hand, utilizing image sensors like depth cameras to detect gestures and establish a virtual interaction coordinate system for precise control of user interface elements, with the context-setting hand providing a real-world reference for making dynamic gestures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If single-hand gestures are used for controlling user interface elements, then the operation is simple, but it is difficult to distinguish intended interactions from non-interface actions and spatial perception is inaccurate
Solution Approach 1:
The gesture recognition process is segmented into two distinct phases: context-setting gesture (defining the interaction plane) and dynamic action gesture (performing the actual control). This segmentation allows the system to first establish a reference frame and then interpret subsequent gestures within that frame, improving spatial perception accuracy while maintaining operational simplicity.
Solution Approach 2:
The context-setting gesture acts as a preliminary action that defines the interaction plane and coordinate system before the dynamic action gesture is performed. By establishing the reference frame in advance, the system can accurately perceive the spatial relationship between gestures and interface elements, resolving the ambiguity between intended and non-interface actions.
2Reliability
If context-setting gesture is required before dynamic action, then spatial perception and intent expression improve, but the interaction sequence becomes more complex
Solution Approach 1:
The interaction system transitions from a static single-gesture model to a dynamic two-phase model where the first gesture (context-setting) establishes the reference frame and the second gesture (dynamic action) performs the control. This dynamic approach improves reliability by clearly distinguishing user intent while the sequential nature minimizes time loss compared to more complex simultaneous multi-gesture systems.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Two-handed interactions with a natural user interface are disclosed. For example, one embodiment provides a method comprising detecting via image data received by the computing device a context-setting input performed by a first hand of a user. and sending to a display a user interface positioned based on a virtual interaction coordinate system, the virtual coordinate system being positioned based upon a position of the first hand of the user. The method further includes detecting via image data received by the computing device an action input performed by a second hand of the user, the action input performed while the first hand of the user is performing the context-setting input, and sending to the display a response based on the context-setting input and an interaction between the action input and the virtual interaction coordinate system.