Extended Reality Media Interfaces for Gaze and Hand Tracking

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing user interfaces for interacting with augmented and virtual reality environments are cumbersome, inefficient, and complex, leading to a significant cognitive burden on users and excessive energy consumption, particularly in battery-operated devices.

Innovation Solution

Implementing computer systems with improved methods and interfaces that utilize cameras, touch-sensitive displays, eye-tracking, and hand-tracking components to provide intuitive and efficient interactions, including stereoscopic media capture and gaze-based targeting, to reduce the number and nature of user inputs and enhance feedback.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional user interfaces are used for augmented and virtual reality environments, then basic functionality can be achieved, but user interaction becomes cumbersome and cognitively demanding

Engineering Contradiction:

Improveuser interaction efficiencyVSAvoidinterface complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The system provides visual feedback by displaying a capture guide that changes appearance based on the detected hand gesture state. This feedback mechanism helps users understand the connection between their inputs and device responses, making the interaction more intuitive and reducing cognitive burden without requiring complex multi-step input sequences

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system automatically detects hand gestures and translates them into capture operations without requiring explicit user commands. The gesture-based interface allows the system to interpret user intent automatically, reducing the number of inputs needed and simplifying the interaction process

Inventive Principle:

Principle #25Self-service

2Reliability

If multiple input steps are required to achieve desired outcomes, then precise control can be obtained, but interaction time and energy consumption increase

Engineering Contradiction:

Improvecontrol precisionVSAvoidinteraction time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system establishes a direct mapping between hand gestures and capture operations, preparing the interface to execute actions immediately upon gesture detection. This preliminary configuration of the gesture-to-action mapping eliminates the need for multiple sequential input steps, reducing interaction time while maintaining precise control through gesture-based input

Inventive Principle:

Principle #10Preliminary action

3Productivity

If conventional input methods are used, then basic media capture can be achieved, but feedback insufficiency leads to errors and rework

Engineering Contradiction:

Improvecapture efficiencyVSAvoidfeedback information

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The capture guide provides continuous visual feedback that changes appearance based on the detected hand gesture state and capture progress. This feedback loop allows users to understand the current state of the capture operation and adjust their gestures accordingly, improving capture efficiency and reducing errors without requiring additional input steps

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12450854B2User interfaces for capturing media and manipulating virtual objects

Publication Date: 2025.10.21 APPLE INC

AI summary

Electronic devices provide extended reality experiences. In some embodiments, a media capture user interface is displayed, including a capture guide. In some embodiments, gaze information is used for targeting. In some embodiments, a virtual object is manipulated.