HMD Sensor Fusion for Physical Awareness in Immersive VR

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Users immersed in computer-generated reality experiences through head-mounted displays often struggle to detect and respond to physical surroundings, making it difficult for others to attract their attention effectively.

Innovation Solution

Incorporating image sensors and microphones into head-mounted displays to detect individuals within proximity, determine their view angle and speech direction, and recognize audio triggers or familiar faces, triggering alerts to the user.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If head-mounted displays present immersive images and audio signals, then the immersion experience is improved, but the user's ability to detect events in physical surroundings deteriorates

Engineering Contradiction:

Improveimmersion experienceVSAvoiddetection of physical events

Core Design Contradiction:

Adaptability or versatilityVSDifficulty of detecting and measuring

Solution Approach 1:

The system continuously monitors the physical environment using image sensors and microphones, and provides feedback to the user through alerts when relevant events are detected. This feedback loop allows the user to remain immersed while still being aware of important physical occurrences.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The head-mounted display acts as an intermediary between the physical environment and the user's immersive experience. Sensors capture physical events, the processing apparatus analyzes them, and the display presents filtered information to the user, mediating between full immersion and physical awareness.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If head-mounted displays provide immersive computer-generated reality, then the virtual experience quality is improved, but social interaction awareness deteriorates

Engineering Contradiction:

Improvevirtual experience qualityVSAvoidsocial cues information

Core Design Contradiction:

Adaptability or versatilityVSLoss of information

Solution Approach 1:

The system detects social cues such as speech signals and facial expressions in the physical environment, and provides feedback to the user through alerts. This maintains social awareness without compromising the immersive virtual experience quality.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The head-mounted display mediates between the immersive virtual reality and the physical social environment. It selectively extracts and presents social information from the physical world, allowing users to maintain both virtual immersion and social awareness simultaneously.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11828940B2System and method for user alerts during an immersive computer-generated reality experience

Publication Date: 2023.11.28 APPLE INC

AI summary

Systems and methods for computer-generated reality user hailing are described. Some implementations may include accessing sensor data captured using one or more sensors; detecting a person within a distance of a head-mounted display based on the sensor data; detecting a hail event based on the sensor data; and responsive to the hail event, invoking an alert using the head-mounted display.