Depth-Aware Media Captions for Lower-Input AR/VR Interfaces

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for interacting with augmented and virtual reality environments are cumbersome, inefficient, and place a significant cognitive burden on users, often requiring multiple inputs and providing insufficient feedback, leading to errors and energy wastage, particularly in battery-operated devices.

Innovation Solution

The system employs improved user interfaces that reduce the number and complexity of user inputs by using touch-sensitive displays, eye-tracking, hand-tracking, and tactile output generators to provide intuitive interactions, including depth-based and attention-based display of user interface elements with varying visual appearances.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional user interfaces are used in augmented reality environments, then users can interact with virtual objects, but the interaction becomes cumbersome and requires multiple inputs

Engineering Contradiction:

Improveuser interaction efficiencyVSAvoidnumber of inputs required

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The system provides immediate visual feedback by displaying captions and user interface elements directly at the depth location of media content portions. This feedback mechanism allows users to understand the connection between their inputs and device responses without requiring multiple sequential inputs, thereby improving interaction efficiency while reducing operational complexity

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent introduces captions and user interface elements as intermediary visual indicators that mediate between user inputs and system responses. These intermediaries provide contextual information about media content depth and guide user interactions, reducing the cognitive burden and number of inputs needed to achieve desired outcomes

Inventive Principle:

Principle #24Intermediary (Mediator)

2Loss of information

If conventional user interfaces provide limited feedback, then system complexity is reduced, but user understanding of input-output connections deteriorates

Engineering Contradiction:

Improvefeedback sufficiencyVSAvoidinteraction efficiency

Core Design Contradiction:

Loss of informationVSProductivity

Solution Approach 1:

The system enhances feedback by displaying captions and user interface elements that visually represent the depth of media content portions. This rich feedback provides users with comprehensive information about their interaction context, improving both user understanding and interaction efficiency simultaneously rather than creating a trade-off

Inventive Principle:

Principle #23Feedback

3Reliability

If multiple inputs are required to achieve desired outcomes, then system control is improved, but time to complete tasks increases

Engineering Contradiction:

Improveinput accuracyVSAvoidtask completion time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary action by displaying captions and user interface elements that indicate the depth and characteristics of media content portions before user interaction. This preliminary visual information guides users to provide accurate inputs on the first attempt, reducing the need for multiple corrective inputs and thereby decreasing task completion time while maintaining reliability

Inventive Principle:

Principle #10Preliminary action

4Measurement precision

If complex manipulation of virtual objects is required, then precision of interaction is improved, but cognitive burden on users increases

Engineering Contradiction:

Improveinteraction precisionVSAvoidcognitive burden

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

The system reduces cognitive burden by providing continuous visual feedback through captions and user interface elements that display the depth of media content portions. This feedback eliminates the need for complex mental models of system state, allowing users to interact with precision while maintaining low cognitive load

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system enables self-service by automatically displaying relevant information about media content depth and characteristics without requiring users to perform complex manipulation or remember system rules. The interface serves itself by providing contextual information that guides accurate interaction naturally

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS12449946B2Methods for displaying user interface elements relative to media content

Publication Date: 2025.10.21 APPLE INC

AI summary

In some embodiments, a computer system displays a caption for a media item at different depths depending on the depth of the portion of the media item over which the caption is displayed. In some embodiments, a computer system displays a user interface element that includes information associated with the media item at different locations relative to the media item depending on attention of the user. In some embodiments, a computer system displays a user interface element that includes information associated with the media item with different visual appearances depending on visual characteristics of the portion of the media item over which the user interface element is displayed.