3D Input Interface Using Gaze-Hand Tracking to Cut Interaction Errors

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods and interfaces for processing inputs in three-dimensional environments, such as augmented and virtual reality, are cumbersome, inefficient, and prone to errors, leading to a significant cognitive burden on users and inefficient energy usage, particularly in battery-operated devices.

Innovation Solution

The system uses gaze and hand tracking to provide consistent input identifiers, distinguishes between direct and indirect manipulation, and conditionally provides gesture information based on user readiness and object behavior parameters, reducing the number and nature of user inputs and enhancing user interface efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If conventional input processing methods are used in three-dimensional environments, then the system can receive inputs from multiple input devices, but the interface becomes cumbersome and creates significant cognitive burden on users

Engineering Contradiction:

Improvesupport for different input mechanismsVSAvoiduser interaction simplicity

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent introduces an intermediary input processing layer that sits between multiple input devices and the three-dimensional environment. This layer receives inputs from various devices (cameras, controllers, touch surfaces), processes them through a unified framework, and delivers consistent input identifiers to the virtual environment, thereby simplifying user interaction while maintaining multi-device support

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The input processing system is designed with universal functionality to handle multiple types of inputs (gaze, hand gestures, controller inputs, touch inputs) through a single unified interface. The system provides consistent input identifiers regardless of the input device used, making the interface adaptable to different input mechanisms while maintaining ease of operation

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Productivity

If the system processes all input information efficiently, then user interaction becomes more responsive, but energy consumption increases significantly in battery-operated devices

Engineering Contradiction:

Improveinput processing efficiencyVSAvoiddevice energy consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary processing of input data by establishing a unified input framework beforehand. Input identifiers are pre-defined and standardized, allowing the system to quickly match and process inputs without extensive real-time computation. This preliminary structuring enables efficient input handling while reducing energy consumption during actual interaction

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent changes the parameter of input representation by using consistent input identifiers across different input devices. This parameter change allows the system to process inputs more efficiently by comparing standardized identifiers rather than analyzing raw input data from each device, thereby improving productivity while reducing energy consumption

Inventive Principle:

Principle #35Parameter changes

3Measurement precision

If the system provides comprehensive gesture information for all inputs, then input accuracy improves, but the complexity of input processing increases

Engineering Contradiction:

Improveinput identification accuracyVSAvoidinput processing system complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system extracts only the essential information needed for input identification by using consistent input identifiers. Instead of processing comprehensive gesture information from all input devices, the system extracts and delivers only the necessary input identifiers to the three-dimensional environment, thereby maintaining input accuracy while reducing processing complexity

Inventive Principle:

Principle #2Taking out (Extraction)

Data Source

PatentUS20250362783A1Devices, Methods, and Graphical User Interfaces for Processing Inputs to a Three-Dimensional Environment

Publication Date: 2025.11.27 APPLE INC

AI summary

While a view of an environment is visible via a computer system, the computer system detects an input, including detecting that a gaze of a user is directed toward the environment. In response, the computer system initiates an interaction with a target of the input, and updates the view of the environment to indicate the initiated interaction. The target is determined based on a location in the environment to which the user's gaze is directed while the input is detected, and the initiated interaction is associated with an input identifier. The computer system detects a continuation of the input, including detecting movement of a hand of the user, and, in response, continues the interaction with the target of the input and updates the view of the environment to indicate the continued interaction, based on the movement of the user's hand. The continued interaction is associated with the same input identifier.