Bimanual Hand Region Mapping for Virtual Element Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for interacting with virtual elements on displays, particularly in augmented and virtual reality environments, lack efficient and intuitive methods for bimanual control using hand gestures, leading to suboptimal user experience and interaction efficiency.
Innovation Solution
A bimanual control system utilizing mapped hand regions and advanced computer vision algorithms to track and interpret hand gestures, enabling precise interaction with virtual elements on wearable devices like eyewear, by correlating hand movements with display positions in real-time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional input systems are used for interacting with virtual elements, then device complexity is reduced, but ease of operation and interaction efficiency deteriorate
Solution Approach 1:
The patent replaces traditional mechanical input systems (buttons, switches, touchscreens) with a vision-based hand gesture recognition system. Cameras capture hand movements and computer vision algorithms interpret these gestures to control virtual elements, eliminating the need for physical interaction interfaces while improving ease of operation in VR/AR environments
Solution Approach 2:
The patent introduces an intermediary system consisting of cameras and computer vision processing between the user and virtual elements. This intermediary translates physical hand gestures into digital control commands, enabling intuitive interaction without direct contact with display surfaces or traditional input devices
2Ease of operation
If hand gesture tracking is implemented for bimanual control, then ease of operation improves, but measurement precision and detection difficulty worsen
Solution Approach 1:
The patent segments the hand into distinct regions (palms, fingers, joints) and tracks each segment independently using computer vision. This segmentation enables precise measurement of hand position and orientation while maintaining intuitive gesture-based control, as each segment's movement can be detected and translated into corresponding control actions
Solution Approach 2:
The patent maps three-dimensional hand gestures in physical space to two-dimensional control interfaces on display surfaces. By capturing hand position, orientation, and shape in 3D space and projecting these onto 2D virtual elements, the system achieves both measurement precision and operational intuitiveness across dimensional boundaries
3Speed
If real-time hand tracking is performed, then interaction speed improves, but use of energy and processing requirements worsen
Solution Approach 1:
The patent implements partial tracking by focusing computational resources on detecting and tracking only the relevant hand regions and gestures needed for control, rather than processing entire video frames in full resolution. This selective approach maintains real-time response speed while reducing overall energy consumption and processing requirements
Data Source
AI summary
Example systems, devices, media, and methods are described for controlling the presentation of one or more virtual or graphical elements on a display in response to bimanual hand gestures detected by an eyewear device that is capturing frames of video data with its camera system. An image processing system detects a first hand and defines a mapped region relative to a surface of the detected first hand. The system presents a virtual target icon at a position within or near the mapped region. The image processing system detects a series of bimanual hand shapes, including the detected first hand and at least one fingertip of a second hand. In response to determining whether the detected series of bimanual hand shapes matches a predefined hand gesture, the system executes a selecting action that includes presenting one or more graphical elements on the display.