Server-Side XR Image Composition for Lower Client Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional extended-reality systems face challenges with high bandwidth requirements and computational complexity due to simultaneous rendering applications, especially in remote XR setups where image composition at the client device leads to motion-to-photon latency and non-feasible upscaling of depth camera images.
Innovation Solution
A system and method that centralizes image composition on a server, using a real-world depth map and VR images to generate a single VR image, reducing bandwidth and computational load by transmitting only the composed image to the display apparatus.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If composition is performed at the client device, then real-time processing is achieved, but motion-to-photon latency increases and computational complexity becomes non-feasible
Solution Approach 1:
The system performs pre-composition of multiple VR images at the server before transmission to the client device. By completing the composition operation in advance on the server side, the system eliminates the need for real-time composition at the client, thereby reducing motion-to-photon latency while maintaining real-time display performance.
2Adaptability or versatility
If multiple rendering applications are supported simultaneously, then application versatility increases, but bandwidth requirements grow linearly
Solution Approach 1:
The system merges multiple separate VR image streams from different rendering applications into a single composed image at the server. Instead of transmitting multiple independent image streams to the client, the server combines them using depth maps and compositing operations, then transmits only the final composed image, thereby reducing bandwidth consumption while supporting multiple applications simultaneously.
3Measurement precision
If depth camera images are upscaled at the client HMD, then image resolution is improved, but computational complexity becomes non-feasible
Solution Approach 1:
The system extracts the computationally intensive upscaling operation from the client device and relocates it to the server. The server performs high-resolution upscaling of depth camera images using its greater computational resources, then transmits the already-upscaled images to the client, eliminating the need for the client HMD to perform complex upscaling operations.
4Speed
If composition is performed at the client device, then real-time processing is achieved, but device complexity increases due to additional computational requirements
Solution Approach 1:
The system introduces the server as an intermediary between the rendering applications and the client device. The server acts as a mediator that performs pre-composition and image processing operations, receiving multiple VR images from rendering applications, combining them with depth information, and delivering the composed result to the client. This intermediary approach reduces the computational burden on the client device while maintaining real-time performance.
Data Source
AI summary
A system includes at least one server that is communicably coupled to at least one display apparatus. The at least one server is configured to obtain a real-world depth map corresponding to a target pose, and obtain a plurality of virtual-reality (VR) images and a plurality of VR depth maps for the target pose, wherein the plurality of VR images are generated by respective ones of a plurality of rendering applications that are executing at the at least one server. Further, the at least one server is configured to composite the plurality of VR images to generate a single VR image, based on the real-world depth map and the plurality of VR depth maps; and send the single VR image to the at least one display apparatus.