3D Interface Control via Hand Motion and Gaze Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional interactive displays, such as those using multi-touch and hover methods, are limited to 2D interactions and fail to provide an immersive and intuitive way to operate 3D user interfaces, lacking unity and precision in controlling 3D objects.
Innovation Solution
An interface control apparatus and method that utilizes a receiver to capture depth images from a sensor, processing this information to generate motion and gaze data, which is then used to control a 2D or 3D graphical user interface, enabling direct interaction with 3D objects by correlating hand motions and gaze direction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If multi-touch method is used for interaction, then direct interaction is achieved where gaze and operation space correspond, but only 2D interaction is provided which cannot intuitively operate 3D objects
Solution Approach 1:
The patent transitions from 2D touch interaction to 3D spatial interaction by introducing depth information through time-of-flight sensing. Users can interact with 3D objects in virtual space by performing hand gestures in the air, adding a third dimension (depth) to the traditional 2D touch interface. This enables intuitive manipulation of 3D objects while maintaining direct interaction where gaze and operation space correspond.
2Adaptability or versatility
If hover method is used for interaction, then interaction with 3D objects is enabled, but operation is not immersive and unified as it relies only on hand information
Solution Approach 1:
The patent merges multiple interaction modalities by simultaneously tracking both hand gestures and gaze direction. The system combines hand position information from time-of-flight sensing with gaze information from camera-based eye tracking, creating a unified 3D interaction interface that is both immersive and precise. This combination allows users to naturally point at and manipulate 3D objects using both hand movements and eye direction.
3Device complexity
If conventional 2D UI control is used, then simple control is achieved, but precision and unity in controlling 3D objects is lacking
Solution Approach 1:
The patent introduces an intermediary mapping layer that translates physical hand gestures in 3D space into precise digital commands for controlling 3D objects. The time-of-flight sensor captures hand position in three dimensions, and this spatial information is mapped to the virtual 3D environment, providing precise control while maintaining natural gesture-based operation. This intermediary layer bridges the physical and digital domains with high precision.
Data Source
AI summary
In an apparatus and method for controlling an interface, a user interface (UI) may be controlled using information on a hand motion and a gaze of a user without separate tools such as a mouse and a keyboard. That is, the UI control method provides more intuitive, immersive, and united control of the UI. Since a region of interest (ROI) sensing the hand motion of the user is calculated using a UI object that is controlled based on the hand motion within the ROI, the user may control the UI object in the same method and feel regardless of a distance from the user to a sensor. In addition, since positions and directions of view points are adjusted based on a position and direction of the gaze, a binocular 2D/3D image based on motion parallax may be provided.