Split Rendering for Extended Reality Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current XR technologies face challenges in providing low latency and high-quality experiences due to the conflict between mobile XR devices' limited power and the need for high-performance hardware, leading to nausea and dislocation issues with existing cloud-based streaming solutions that introduce high latency.
Innovation Solution
A split latency approach is implemented, where background or distant imagery is rendered by a server and transformed on the user's device, while fast-moving or near imagery is rendered locally, minimizing latency by differentially computing and communicating data based on imagery type and viewpoint, and using metadata to facilitate correct image association.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If cloud-based streaming systems are used to render XR content remotely, then device complexity and local hardware requirements are reduced, but latency increases to 60-200 milliseconds causing nausea and dislocation
Solution Approach 1:
The patent segments the rendering process into two distinct parts: server-side rendering for background/distant imagery and client-side rendering for foreground/near imagery. This segmentation allows the system to leverage cloud computing power for complex scenes while maintaining low latency for critical interactive elements by processing them locally on the user's device.
Solution Approach 2:
The patent applies different rendering quality and processing locations based on the spatial characteristics of imagery. Distant background elements are rendered remotely with less urgency, while near foreground elements that require immediate responsiveness are rendered locally with higher priority, ensuring that critical visual information maintains low latency regardless of overall system load.
2Productivity
If high-performance hardware is used to achieve high frame rates and low latency, then XR experience quality improves, but device weight and power consumption increase making it cumbersome
Solution Approach 1:
The patent divides the computational workload between server and client devices. The server handles the majority of rendering computations for background and distant imagery, while the client device only processes foreground/near imagery and performs compositing. This segmentation enables mobile devices to deliver high-quality XR experiences without requiring high-performance local hardware that would increase weight and power consumption.
3Loss of time
If all imagery is rendered locally on mobile XR devices, then latency is reduced, but device power consumption increases and battery life decreases
Solution Approach 1:
The patent segments the rendering responsibilities based on imagery type and latency requirements. Only foreground/near imagery that requires immediate responsiveness is rendered locally on the mobile device, while background/distant imagery is rendered on the server. This selective local rendering minimizes power consumption while maintaining low latency for critical visual elements.
Solution Approach 2:
The patent applies local rendering only where it is most needed - for foreground/near imagery that directly impacts user interaction and immersion. Background imagery can tolerate higher latency and is therefore rendered remotely, allowing the mobile device to conserve battery power while still delivering responsive XR experiences for critical visual elements.
4Productivity
If high-end XR technology is deployed to achieve immersive experiences, then experience quality improves, but accessibility to mass market decreases due to high costs
Solution Approach 1:
The patent segments the computational workload to enable cloud-based XR rendering, allowing high-quality graphics processing to occur on remote servers rather than requiring expensive local hardware. This enables mass-market mobile devices to access premium XR experiences by leveraging cloud computing resources, significantly improving accessibility while maintaining experience quality.
Data Source
AI summary
A method for reducing the effects of latency in an extended reality (XR) experience is disclosed. The method includes: i) distinguishing between two types of imagery to be rendered; ii) differentially computing and communicating data, dependent upon the imagery type, such that when the imagery is of: a) a first type, which is background or distant imagery, it is rendered in a known frame of reference by a server and sent to a user's device from the server where it is transformed to a second frame of reference by the user's device and re-rendered; and b) a second type, which is fast-moving imagery, fast-changing imagery, or imagery very close to a user's point of view relative to the first type imagery, it is rendered locally by the user's device; and iii) compositing the first type imagery and the second type imagery, by the user device.