Eye- and Hand-Tracking for Efficient Virtual Object Control

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods for interacting with virtual/augmented reality environments are cumbersome, inefficient, and complex, requiring multiple inputs and leading to a significant cognitive burden on users, particularly in battery-operated devices.

Innovation Solution

The system employs eye-tracking and hand-tracking components to enhance interactions with selectable user interface elements, slider elements, and virtual objects in three-dimensional environments, providing intuitive and efficient input methods through gaze detection and hand gestures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional input methods (touch, click, keyboard) are used to interact with virtual objects in AR/VR environments, then the system can process and respond to user inputs, but the interaction becomes cumbersome, inefficient, and cognitively burdensome requiring multiple inputs

Engineering Contradiction:

Improveease of interactionVSAvoidcomplexity of input methods

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent replaces mechanical input devices (touchscreens, keyboards, mice) with a voice-activated virtual assistant system. Users speak natural language commands which are processed by speech recognition algorithms, eliminating the need for physical interaction with complex input devices. This substitution of mechanical systems with acoustic field-based interaction directly improves ease of operation while reducing the perceived complexity for the user.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The virtual assistant acts as an intermediary layer between the user and the system. Instead of directly manipulating virtual objects through complex gestures or touch sequences, users speak commands to the virtual assistant, which then interprets and executes the desired actions. This intermediary simplifies the interaction model by translating natural language into system commands, reducing cognitive burden and improving ease of use.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If multiple input steps are required to achieve a desired outcome in AR/VR environments, then the system can perform complex operations, but the process takes longer than necessary and wastes energy

Engineering Contradiction:

Improveefficiency of interactionVSAvoidtime for interaction

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The virtual assistant system is pre-configured with knowledge of system capabilities and commands. When activated, it can immediately process and execute commands without requiring users to navigate through multiple menus or perform sequential input steps. The system performs preliminary setup of the interaction framework, allowing users to jump directly to the desired action through speech, significantly reducing interaction time and improving productivity.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent extracts the essential function of interaction from complex multi-step input sequences and condenses it into single-voice-command operations. By taking out the core intent from elaborate gesture or touch sequences and implementing it through direct voice recognition, the system eliminates unnecessary intermediate steps, reducing both time loss and improving operational efficiency.

Inventive Principle:

Principle #2Taking out (Extraction)

3Use of energy by moving object

If traditional input devices and methods are used in battery-operated devices, then the device can function, but energy is wasted due to prolonged interaction times and inefficient processes

Engineering Contradiction:

Improveenergy consumptionVSAvoidefficiency of input methods

Core Design Contradiction:

Use of energy by moving objectVSEase of operation

Solution Approach 1:

The patent replaces energy-intensive mechanical input processing (touchscreen controllers, keyboard matrix scanning, gesture sensor processing) with acoustic signal processing. Voice commands can be recognized and processed more quickly than interpreting complex touch or gesture sequences, reducing the time processors, displays, and other energy-consuming components remain active. This substitution directly reduces overall energy consumption while maintaining or improving ease of operation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The virtual assistant enables users to skip through multiple interaction steps by processing voice commands that directly trigger desired actions. Instead of requiring users to navigate through menus, select options, and confirm actions in sequence, a single well-formed voice command can bypass intermediate steps and execute the final action immediately. This rushing through of unnecessary steps reduces the duration of energy consumption during interaction tasks.

Inventive Principle:

Principle #21Skipping (Rushing through)

Data Source

PatentEP4217835B1Methods for interacting with virtual controls and/or an affordance for moving virtual objects in virtual environments

Publication Date: 2025.10.22 APPLE INC

Figure 7A
Figure 7B
Figure 7C

AI summary

In some embodiments, an electronic device enhances interactions with virtual objects in a three-dimensional environment. In some embodiments, an electronic device enhances interactions with selectable user interface elements. In some embodiments, an electronic device enhances interactions with slider user interface elements. In some embodiments, an electronic device moves virtual objects in a three-dimensional environment and facilitates accessing actions associated with virtual objects.