Split Rendering of XR Graphic Layers for Delay Variation Tolerance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR applications face significant latency and delay variations due to network congestion, leading to queuing delays and frame loss, which cannot be effectively mitigated by current techniques like bitrate adaptation and frame dropping, especially in volatile 5G networks.
Innovation Solution
A split rendering process that divides rendered images into layers based on Quality of Experience (QoE) importance, allowing for adaptive encoding and decoding, where the client device uses buffered data to substitute missing layers and provides feedback to the server for optimized layer transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If bitrate adaptation is used to prevent queueing delays, then video stream quality is maintained, but transient periods occur where video rate exceeds throughput causing delays
Solution Approach 1:
The video stream is segmented into multiple independent graphic layers (foreground, background, etc.) that can be prioritized and transmitted independently. This allows the system to ensure delivery of critical layers while allowing non-critical layers to be delayed or dropped without affecting overall video functionality.
Solution Approach 2:
Different quality levels are applied to different graphic layers based on their importance. Critical foreground layers maintain high quality and are prioritized for transmission, while background layers use lower quality settings. This localized quality management prevents transient delays from affecting the overall user experience.
2Loss of time
If fast reaction to congestion is implemented using L4S mechanism, then queue build-up is reduced, but resource utilization decreases and user experience deteriorates
Solution Approach 1:
The system dynamically adjusts the encoding rate and quality of individual graphic layers based on real-time network conditions. When congestion is detected, less critical layers are reduced in quality or dropped, while critical layers maintain their quality. This dynamic adaptation maximizes resource utilization while preventing queue delays.
Solution Approach 2:
The system implements feedback mechanisms where the client device reports which graphic layers were received and their quality. The server uses this feedback to adjust the encoding and prioritization of layers in subsequent transmissions, optimizing both queue delay prevention and resource utilization continuously.
3Loss of time
If frames are dropped at transport queue to eliminate queuing delays, then latency is reduced, but frame loss occurs requiring I-frames which increase bandwidth consumption
Solution Approach 1:
By segmenting the video into independent graphic layers, the system can drop or delay non-critical layers without requiring complete frame retransmission. This avoids the need for I-frames and reduces bandwidth consumption while maintaining acceptable latency.
Solution Approach 2:
The system uses previously transmitted graphic layers as references or copies to reconstruct delayed or lost layers at the client side, avoiding the need for retransmission of complete frames and reducing bandwidth requirements.
4Loss of time
If encoding quality is reduced to match network throughput, then queueing delays are prevented, but video quality and user experience decrease
Solution Approach 1:
Different quality levels are applied to different graphic layers based on their perceptual importance. Critical foreground layers maintain high encoding quality even when network throughput is limited, while background layers use lower quality settings. This ensures acceptable video quality for critical content while preventing queueing delays.
Solution Approach 2:
The system transmits only the most critical graphic layers when network conditions are poor, accepting that non-critical layers will be delayed or dropped. This partial transmission approach prevents queueing delays while maintaining acceptable quality for the essential portions of the video stream.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An improved split rendering process of the present disclosure mitigates the impact of delay variation in XR application with remote rendering applications. A visual scene is split rendered to generate graphic layers from 3D objects in the visual scene. The server node groups and sorts the graphic layers based on QoE importance to create graphic layer groups, encodes each graphic layer group into a composite video frame and appends metadata to the composite video frame. The encoded video frame is then transmitted in sorted order based on quality rank to a client device (e.g., an HMD worn by a user) where the video frame is decoded and displayed. The client device further sends feedback to the server indicating the graphic layer groups that were timely received.