3D Interface Gaze-Gated Hand Gestures for Low-Burden Control

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for interacting with virtual and augmented reality environments are cumbersome, inefficient, and place a significant cognitive burden on users, often requiring multiple inputs and leading to energy waste, especially in battery-operated devices.

Innovation Solution

Implementing computer systems with display generation components and input devices such as cameras and touch-sensitive surfaces that utilize micro-gestures, hand and eye tracking, and gaze detection to facilitate intuitive interactions, reducing the need for multiple inputs and enhancing user control and safety.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional input methods (multiple inputs, controllers, joysticks) are used to interact with virtual reality environments, then user control capability is achieved, but cognitive burden increases and interaction efficiency decreases

Engineering Contradiction:

Improveinteraction efficiencyVSAvoidcognitive burden

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent replaces physical input devices (controllers, joysticks, touch screens) with a camera-based vision system that tracks hand gestures and eye movements. This substitution eliminates the need for mechanical interaction interfaces, allowing users to interact with virtual environments through natural gestures and gaze, thereby reducing cognitive burden and improving interaction efficiency.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system enables users to interact with virtual objects using their own body movements (hand gestures, eye tracking) without requiring external controllers or complex input mechanisms. The camera system automatically captures and interprets these natural movements, making the interaction process self-service and intuitively understandable.

Inventive Principle:

Principle #25Self-service

2Measurement precision

If multiple inputs are required to achieve desired outcomes in augmented reality environments, then interaction precision is improved, but interaction time increases and energy consumption rises

Engineering Contradiction:

Improveinteraction precisionVSAvoidinteraction time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary action by continuously tracking hand gestures and eye movements in real-time, preparing the vision system to immediately recognize and respond to interaction intents. This continuous pre-processing of movement data eliminates the need for multiple sequential input steps, allowing users to achieve desired outcomes with fewer actions and reduced interaction time.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The camera-based vision system acts as an intermediary that translates natural hand gestures and eye movements directly into virtual environment commands. This intermediary eliminates the need for multiple input steps by interpreting the user's intent from continuous movement data, thereby reducing interaction time while maintaining precision.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Adaptability or versatility

If complex manipulation methods are used for virtual objects, then interaction capability is enhanced, but error rate increases and user experience deteriorates

Engineering Contradiction:

Improveinteraction capabilityVSAvoiderror rate

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent replaces complex mechanical manipulation methods with vision-based gesture recognition. The camera system captures hand movements and eye tracking data, translating these natural gestures into virtual object manipulation commands. This substitution simplifies the interaction process, reducing the likelihood of errors while maintaining versatile interaction capability.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system provides visual feedback by displaying the user's hand gestures and eye movements in real-time within the virtual environment. This feedback mechanism allows users to verify their interaction intent before execution, reducing errors and improving the reliability of virtual object manipulation while maintaining adaptability.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS20260044242A1Devices, methods, and graphical user interfaces for interacting with three-dimensional environments

Publication Date: 2026.02.12 APPLE INC

AI summary

A computer-system displays a three-dimensional environment, including displaying a representation of a physical environment. While displaying the three-dimensional environment including the representation of the physical environment, the computer system detects a first gesture performed by a hand of a user. In response to detecting the first gesture: if the user's gaze is directed to a representation of the hand of the user in the three-dimensional environment, the computer system displays a system user interface in the three-dimensional environment; and if the user's gaze is not directed to the representation of the hand of the user in the three-dimensional environment, the computer system forgoes displaying the system user interface in the three-dimensional environment.