Foviated HDR Video Streaming with View-Adaptive Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video streaming technologies face challenges in efficiently streaming high-quality video data to a wide variety of client devices due to bandwidth and computing power limitations, leading to significant time lags and adverse user experiences, especially in applications like augmented and virtual reality.
Innovation Solution
The technique involves remapping the dynamic range of high dynamic range (HDR) source images to foviated images based on a viewer's view direction, preserving high detail in the foveal vision area and compressing non-foveal areas, allowing for reduced bandwidth usage and seamless rendering of high-quality video content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If high-quality HDR video data is streamed to support seamless user experience, then image quality and user experience are improved, but bandwidth requirements and computing power increase significantly
Solution Approach 1:
The video content is segmented into foveal region (high detail) and peripheral regions (compressed detail), allowing selective transmission of high-quality data only where needed, thus reducing overall bandwidth requirements while maintaining perceived image quality
Solution Approach 2:
Different quality levels are applied to different regions of the video frame: high-quality HDR data is transmitted for the foveal region where the viewer focuses, while compressed or lower-resolution data is used for peripheral regions, optimizing the trade-off between quality and bandwidth
2Manufacturing precision
If high-quality video processing is performed to maintain seamless rendering, then image quality is improved, but time lags increase adversely impacting user experience
Solution Approach 1:
The computationally intensive processing is extracted and performed on the server side before streaming, allowing the client device to receive pre-processed video data with reduced processing requirements and minimal latency
Solution Approach 2:
Video processing and compression are performed in advance on the server based on predicted or actual viewing conditions, so that when video data is streamed to the client, minimal additional processing is needed, reducing time lags
3Quantity of substance
If video data is compressed to reduce bandwidth usage, then bandwidth requirements are reduced, but image quality deteriorates
Solution Approach 1:
Different compression levels are applied to different regions: the foveal region maintains high quality with minimal compression, while peripheral regions use higher compression ratios, achieving overall bandwidth reduction without noticeable quality loss in the viewer's focus area
Data Source
AI summary
First foviated images are streamed to a streaming client. The first foviated images with first image metadata sets are used to generate first display mapped images for rendering to a viewer at first time points. View direction data is collected and used to determine a second view direction of the viewer at a second time point. A second foviated image and a second image metadata set are generated from a second HDR source image in reference to the second view direction of the viewer and used to generate a second display mapped image for rendering to the viewer at the second time point. The second image metadata set comprises a display management metadata portions for adapting a focal-vision and peripheral-vision image portions to corresponding image portions in the second display mapped image. The focal-vision display management metadata portion is generated with a predicted light adaptation level of the viewer for the second time point. The second foviated image and the second image metadata set are transmitted to the video streaming client.