AR Controller Input Using Gaze and Hand Tracking Feedback

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for interacting with augmented and virtual reality environments are cumbersome, inefficient, and create a significant cognitive burden on users, requiring multiple inputs and providing insufficient feedback, leading to energy wastage, particularly in battery-operated devices.

Innovation Solution

Implementing computer systems with improved user interfaces that utilize touchpads, cameras, eye-tracking, and hand-tracking components to reduce the number and complexity of user inputs, provide intuitive feedback, and optimize device responses, thereby enhancing interaction efficiency and conserving power.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional input methods are used in augmented reality environments, then users can interact with virtual objects, but the interaction becomes cumbersome and creates significant cognitive burden

Engineering Contradiction:

Improveease of interactionVSAvoidinput complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The system automatically tracks user gaze and hand movements to identify selected virtual objects and intended actions, eliminating the need for users to manually navigate menus or provide multiple sequential inputs. The interface serves itself by continuously monitoring user behavior and interpreting intentions without requiring explicit command sequences.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces traditional mechanical input devices (buttons, joysticks, touchscreens) with optical tracking systems (cameras, eye-tracking, hand-tracking) that capture user intent through natural movements and gestures, substituting physical interaction mechanics with optical detection and computational interpretation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If multiple input steps are required to achieve desired outcomes, then precise control is possible, but interaction time increases and energy is wasted

Engineering Contradiction:

Improveinteraction efficiencyVSAvoidinteraction time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The system continuously pre-processes and analyzes user gaze direction, hand position, and movement trajectories in advance, preparing predictions of user intent before explicit selection or command is given. This allows the system to anticipate and prepare for upcoming interactions, reducing the time required for actual execution.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The tracking and interpretation systems operate continuously rather than in discrete steps, maintaining constant monitoring of user behavior and virtual object states. This continuous operation eliminates idle time between input steps and ensures that the system is always ready to respond to user intentions as they arise.

Inventive Principle:

Principle #20Continuity of useful action

3Loss of information

If detailed feedback is provided for user inputs, then user understanding improves, but system processing load and energy consumption increase

Engineering Contradiction:

Improvefeedback sufficiencyVSAvoidenergy consumption

Core Design Contradiction:

Loss of informationVSUse of energy by moving object

Solution Approach 1:

The system provides feedback selectively focused on the specific virtual object currently being interacted with, rather than providing comprehensive system-wide status information. Visual highlights, selection indicators, and action confirmations are localized to the relevant object or interface element, reducing overall processing requirements while maintaining user understanding.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system provides just enough feedback to confirm user understanding and guide interaction, without overwhelming the user with excessive information. Feedback is calibrated to the minimum necessary level—providing confirmation of selection, anticipated action, and result—thereby reducing processing load while maintaining sufficient user awareness.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20250377760A1Devices, methods, and graphical user interfaces for controller input

Publication Date: 2025.12.11 APPLE INC

AI summary

In some embodiments, a computer system controls a user interface element based on directional control input. In some embodiments, a computer system performs an operation in connection with a virtual content item based on input focus. In some embodiments, a computer system reduces the visual prominence of virtual content that is obscuring visibility of one or more objects. In some embodiments, a computer system reduces the visual prominence of virtual content that is obscuring visibility of one or more objects. In some embodiments, a computer system scrolls through paginated content based on directional control input. In some embodiments, a computer system reduces visibility of a first portion of a hand without reducing visibility of a second portion. In some embodiments, a computer system reduces a visual prominence of a portion of virtual content to increase a visibility of one or more hands.