XR Image Reprojection for Lower-Bandwidth Stereo Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high computational expense and large bandwidth requirements for rendering and streaming two XR images for each eye in extended reality applications pose a challenge, particularly in XR devices like head-mounted displays, leading to inefficient bandwidth usage.
Innovation Solution
An XR streaming method that integrates an XR application instance with a computer device, employing reprojection techniques to render a single XR image per frame, which is then reprojected to the appropriate eye positions, reducing bandwidth by up to 50% through techniques like machine-learning or algorithm-based reprojection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If two XR images are rendered and streamed for each eye, then image quality for both eyes is maintained, but streaming bandwidth requirement increases
Solution Approach 1:
The patent uses reprojection technology to generate the second eye's image from the first eye's rendered image. Instead of rendering two separate high-quality images, the system renders one high-quality image and creates a copy/derivative for the second eye through geometric transformation and reprojection, significantly reducing bandwidth while maintaining acceptable image quality for both eyes
Solution Approach 2:
The patent merges the rendering process for both eyes into a single rendering operation. By rendering only one image and using reprojection to generate the other eye's view, the system combines what would traditionally be two separate rendering and streaming operations into one, reducing the total bandwidth requirement while preserving image quality through intelligent image generation
2Reliability
If two XR images are rendered for each frame, then realistic 3D experience is enabled, but computational expense increases
Solution Approach 1:
The system creates a copy of the rendered image and applies reprojection transformations to generate the second eye's view. This copying approach with geometric transformation requires significantly less computational power than performing two complete rendering cycles, while still maintaining the 3D experience quality needed for realistic visualization
Solution Approach 2:
The system performs preliminary rendering of a single reference image that serves as the basis for generating both eye views. By completing the expensive rendering operation once and then using less computationally intensive reprojection operations, the system reduces overall computational expense while maintaining the ability to deliver realistic 3D experiences
3Quantity of substance
If a single XR image is rendered per frame, then streaming bandwidth is reduced, but image quality for individual eyes may deteriorate
Solution Approach 1:
The system renders one high-quality reference image and uses reprojection to generate the second eye's image. The reprojection process preserves important image quality characteristics by using geometric transformations rather than simple duplication, ensuring that both eyes receive images of sufficient quality while bandwidth is reduced by transmitting only one full-resolution image
Solution Approach 2:
The system changes the viewing parameters of the single rendered image through reprojection transformations to create the second eye's view. By adjusting parameters such as viewpoint position and projection angle rather than re-rendering the entire scene, the system maintains image quality while significantly reducing the bandwidth required for streaming
Data Source
Figure 1~2
Figure 3
AI summary
An extended reality, XR, streaming method of streaming XR images from an XR application instance (26) to an XR device (12) is described. The XR application instance (26) is integrated into a computer device (14) that is provided separately from the XR device (12). The XR streaming method comprises the steps of - rendering, by the XR application instance (26), an XR image to be displayed on the XR device (12) with respect to a predetermined reference position, wherein the predetermined reference position has a predetermined relation to a momentary position of the XR device (12); - streaming, by the XR application instance (26), the rendered XR image to the XR device (12); and - reprojecting, by the XR device (12), the rendered XR image to a first eye position and/or reprojecting, by the XR device (12), the rendered XR image to a second eye position, such that a first XR image for the first eye and a second XR image for the second eye are obtained based on the rendered XR image. Further, an XR projection system (10) is described.