Video Encoding Rate Control for Low Latency VR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Virtual reality (VR) and mixed reality (MR) systems face challenges in maintaining low latency and high frame rates over wireless connections, especially with varying channel capacities, which affects the quality of immersive experiences.
Innovation Solution
A video encoding system that performs wavelet transforms on pixel data to decompose it into frequency bands, uses block-based encoding, and implements rate control methods to adjust quantization parameters for each slice, optimizing bandwidth usage and minimizing latency by prioritizing frequency bands and utilizing gaze tracking for foveated regions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional video encoding is used over wireless connections, then bandwidth can be utilized, but latency increases and frame rates cannot be maintained at low latency levels (4-5 milliseconds)
Solution Approach 1:
The video frame is divided into multiple slices, and each slice is further decomposed into frequency bands using wavelet transform. This segmentation allows independent encoding and transmission of different frequency components, enabling prioritized transmission of low-frequency bands that contain most visual information, thereby reducing overall encoding latency while maintaining frame rate
Solution Approach 2:
The system applies foveated rendering based on gaze tracking data, where high-resolution encoding is applied only to the foveal region (center of visual attention) and lower resolution to peripheral regions. This local quality differentiation reduces the amount of data requiring low-latency transmission while maintaining perceived visual quality, thus improving frame rate without sacrificing user experience
2Manufacturing precision
If high-resolution video is encoded and transmitted, then visual quality is improved, but bandwidth consumption increases and latency increases
Solution Approach 1:
The system dynamically adjusts quantization parameters for different frequency bands based on available bandwidth and latency requirements. Low-frequency bands use finer quantization (higher quality) while high-frequency bands use coarser quantization (lower quality), optimizing the balance between visual quality and transmission latency
Solution Approach 2:
Gaze tracking data is used in advance to identify the foveal region before encoding. This preliminary action allows the encoder to pre-determine which regions require high resolution and which can be downsampled, reducing overall data volume and transmission latency before the video is even transmitted
3Speed
If rate control is applied to reduce latency, then transmission speed is improved, but video quality degrades
Solution Approach 1:
Different quality levels are applied to different frequency bands and spatial regions. Low-frequency bands maintain high quality while high-frequency bands use lower quality encoding. Similarly, the foveal region receives high-quality encoding while peripheral regions use lower quality, optimizing overall perceived quality at reduced bitrates
Solution Approach 2:
The system uses feedback from gaze tracking and channel capacity monitoring to dynamically adjust encoding parameters. When channel capacity is high, more bits are allocated to maintain quality; when capacity is low or latency is critical, bits are reallocated from less important frequency bands or peripheral regions, maintaining quality where it matters most
4Manufacturing precision
If complex encoding processes are used to maintain quality, then video quality is preserved, but processing time increases and latency increases
Solution Approach 1:
The encoding process is segmented into independent stages: wavelet transform for frequency decomposition, slice-based processing, and parallel encoding of different frequency bands. This segmentation allows computationally intensive operations to be distributed and parallelized, reducing overall processing time while maintaining quality through careful management of each segment
Solution Approach 2:
The system extracts and processes only the most visually important components (low-frequency bands and foveal regions) at high quality, while using simplified encoding for less important components (high-frequency bands and peripheral regions). This extraction approach maintains perceived quality while significantly reducing total processing complexity and time
Data Source
AI summary
A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands for a slice of a frame may be buffered so that complexity statistics may be calculated across the frequency bands prior to encoding. The statistics may then be used by a rate control component in determining quantization parameters for the frequency bands for modulating the rate in the encoder for the current slice. The quantization parameters for the frequency bands may be calculated jointly to optimize the quality of the displayed frames after decoder reconstruction and wavelet synthesis on a receiving device. Information about one or more previously processed frames may be used in combination with the statistics for a current slice in determining the quantization parameters for the current slice.