Screenless Object Selection via Head Pose and Hand Gestures

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Extended reality devices can be intrusive or difficult to wear, obstructing interaction with physical environments.

Innovation Solution

A system that enables user interaction with physical environments using head pose and hand gestures without the need for display technology, utilizing audio headsets with cameras to capture image data and determine head direction.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If extended reality devices are used to provide immersive virtual content, then user interaction with virtual content is improved, but obstruction of the physical environment and difficulty of wearing the device increases

Engineering Contradiction:

Improveinteraction with virtual contentVSAvoidobstruction of physical environment

Core Design Contradiction:

Adaptability or versatilityVSObject-affected harmful factors

Solution Approach 1:

The patent removes the display screen from the extended reality device, extracting the visual output component that causes physical obstruction. Users interact with virtual content through hand gestures and voice commands while the device captures and processes visual data without presenting it on a screen, thereby eliminating the barrier between user and physical environment while maintaining virtual interaction capabilities

Inventive Principle:

Principle #2Taking out (Extraction)

2Ease of operation

If screen-based interaction is used for object selection, then user interface functionality is improved, but power consumption and device complexity increases

Engineering Contradiction:

Improveobject selection functionalityVSAvoidpower consumption

Core Design Contradiction:

Ease of operationVSUse of energy by moving object

Solution Approach 1:

The patent replaces the mechanical/optical system of display screens and visual interfaces with gesture recognition and voice command systems. The device uses cameras to capture hand movements and processes these gestures to control virtual content and select objects, eliminating the need for power-intensive display components while maintaining full object selection functionality

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Measurement precision

If eye tracking technology is implemented for object selection, then selection precision is improved, but device complexity and power consumption increases

Engineering Contradiction:

Improveobject selection precisionVSAvoideye tracking system complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent uses the camera system already present for environmental awareness to capture images of the user's hands and gestures. Instead of implementing a separate eye tracking system, the device processes hand gesture images to determine selection intent, achieving accurate object selection through a simpler, multi-purpose imaging system that reduces overall device complexity

Inventive Principle:

Principle #26Copying

Data Source

PatentUS20250199606A1Screenless Object Selection with Head Pose and Hand Gestures

Publication Date: 2025.06.19 APPLE INC

AI summary

A technique for providing user selection includes obtaining first sensor data from a first device worn on a head, and obtaining second sensor data from a second device worn on the head. The first and second sensor data are collected in a same time frame. A head position is determined for the head based on the first sensor data and the second sensor data. User input is received based on an input gesture. In response to the user input, an object in the local environment is identified based on the head position and selection input.