XR Streaming Cinematic Fallback for High-Latency Immersion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

High latency environments in extended reality (XR) streaming can cause degraded experiences, motion sickness, and loss of balance for users, particularly in video games, due to system throttling and reduced responsiveness.

Innovation Solution

Implementing automatic cinematic mode fallback by switching XR applications to a 2D presentation on a virtual or real-world display, pausing the game, enabling AI agent control, transitioning to dual-stick navigation, and using video pass-through when high latency is detected, with the system reverting to immersive mode upon latency reduction.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If streaming XR applications are used to provide immersive experience, then user engagement is improved, but latency increases causing motion sickness and loss of balance

Engineering Contradiction:

Improveimmersive experienceVSAvoiduser stability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system dynamically switches between XR immersive mode and 2D cinematic mode based on detected latency conditions. When high latency is detected, the system transitions from 3D immersive rendering to 2D presentation, allowing adaptive response to changing network conditions while maintaining user experience quality

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

A latency detection and management system acts as an intermediary between the XR application and display system. This intermediary monitors latency conditions and mediates the transition between immersive and cinematic modes, preventing motion sickness by intervening before latency causes harmful effects

Inventive Principle:

Principle #24Intermediary (Mediator)

2Temperature

If system throttling occurs due to overheating, then device temperature is reduced, but responsiveness decreases causing degraded experience

Engineering Contradiction:

Improvedevice temperatureVSAvoidsystem responsiveness

Core Design Contradiction:

TemperatureVSSpeed

Solution Approach 1:

The system changes presentation parameters from 3D immersive mode to 2D cinematic mode when thermal throttling is detected. This parameter change reduces computational load and maintains responsiveness during thermal constraints, preventing degraded user experience while allowing the device to cool

Inventive Principle:

Principle #35Parameter changes

3Reliability

If 2D cinematic mode is used during high latency, then motion sickness is reduced, but immersive experience is lost

Engineering Contradiction:

Improveuser stabilityVSAvoidimmersive experience

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The system dynamically adjusts presentation mode based on real-time latency detection, switching between 2D cinematic and 3D immersive modes. This dynamic adaptation ensures user stability when needed while preserving immersive experience when conditions allow

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS20250360402A1Automatic cinematic mode fallback for high latency connections during XR streaming

Publication Date: 2025.11.27 SONY INTERACTIVE ENTERTAINMENT LLC

AI summary

Techniques for streaming (and non-streaming) XR applications that, when high latency is detected, automatically and naturally falls back on a cinematic experience in VR. For example, the game may be shown on a movie theater screen in a locally rendered environment where the game is played.