Foviation and HDR Streaming for AR/VR Bandwidth Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in efficiently streaming high-quality video data to a wide variety of client devices due to bandwidth and computing power limitations, leading to significant time lags and adverse impacts on user experience in applications like augmented reality and virtual reality.
Innovation Solution
The technique involves remapping the dynamic range of high dynamic range (HDR) source images to foviated images based on a viewer's view direction, preserving high detail in the foveal vision area while compressing non-foveal areas, and using spatially differentiated display mapping to minimize bandwidth usage and enhance rendering efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If high-quality video data is streamed to support seamless user experience in AR/VR applications, then image quality and user experience are improved, but bandwidth requirements and computing power consumption increase significantly
Solution Approach 1:
The patent applies local quality by encoding video content at different quality levels for different regions of the display. The foveal region (center of vision) receives high-quality encoded content, while peripheral regions receive lower-quality content. This spatially differentiated encoding maintains high user experience quality in the important foveal area while significantly reducing overall bandwidth requirements.
Solution Approach 2:
The patent segments the display area into multiple regions based on human visual perception characteristics, specifically dividing the display into a foveal region and peripheral regions. This segmentation allows the system to apply different encoding strategies to different segments, optimizing the balance between quality and bandwidth consumption.
2Manufacturing precision
If high-quality video data is processed and rendered in real-time, then image quality is improved, but time lags between user gaze direction and rendered content increase
Solution Approach 1:
The patent reduces rendering time lag by applying local quality differentiation to the rendering process. High-quality rendering is applied only to the foveal region where the user is looking, while peripheral regions are rendered with lower quality and faster processing. This approach maintains high image quality where needed while significantly reducing overall rendering time and eliminating perceptible lags.
Solution Approach 2:
The patent applies partial action by rendering only the necessary portions of the display at high quality. Instead of rendering the entire display at maximum quality, the system focuses computational resources on the foveal region, providing partial rendering coverage that is sufficient for maintaining user experience while reducing processing time.
3Quantity of substance
If video data is compressed to reduce bandwidth usage, then bandwidth requirements are reduced, but image quality and detail preservation deteriorate
Solution Approach 1:
The patent resolves this contradiction by applying local quality to the compression process. The foveal region is encoded with high quality and minimal compression, preserving image detail where the user is looking. Peripheral regions are encoded with lower quality and higher compression ratios. This spatially adaptive compression maintains acceptable image quality while significantly reducing overall bandwidth usage.
Solution Approach 2:
The patent applies partial action to compression by focusing high-quality encoding only on the foveal region rather than the entire display. This allows the system to use excessive compression (lower quality) for peripheral regions while maintaining adequate quality in the important foveal area, achieving the right balance between bandwidth savings and quality preservation.
Data Source
AI summary
First foviated images are streamed to a streaming client. The first foviated images with first image metadata sets are used to generate first display mapped images for rendering to a viewer at first time points. View direction data is collected and used to determine a second view direction of the viewer at a second time point. A second foviated image and a second image metadata set are generated from a second HDR source image in reference to the second view direction of the viewer and used to generate a second display mapped image for rendering to the viewer at the second time point. The second image metadata set comprises a display management metadata portions for adapting a focal-vision and peripheral-vision image portions to corresponding image portions in the second display mapped image. The focal-vision display management metadata portion is generated with a predicted light adaptation level of the viewer for the second time point. The second foviated image and the second image metadata set are transmitted to the video streaming client.