VR Video Decoding via CPU-GPU Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current virtual reality systems face challenges in providing immersive experiences due to bandwidth, storage, and processing limitations, particularly in decoding high-resolution video data at 90 frames per second for each eye, which is beyond the capabilities of traditional CPU-based decoding methods.
Innovation Solution
The proposed solution involves joint decoding by both the central processing unit (CPU) and graphics processing unit (GPU), where the CPU performs initial decoding tasks like Huffman entropy decoding and the GPU handles de-quantization and color conversion, with the video stream being encoded to facilitate parallel processing and reduce memory access latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If CPU-based decoding methods are used, then decoding can be performed with existing hardware, but decoding throughput is too slow to meet virtual reality requirements
Solution Approach 1:
The patent divides the decoding process into distinct segments: the CPU performs initial decoding tasks (Huffman entropy decoding, inverse run length encoding, de-zigzagging) to generate a partially-decoded bitstream, then transfers control to the GPU for parallel processing of de-quantization, inverse DCT, and color conversion. This segmentation allows each processor to handle tasks suited to its architecture, achieving the required 1166 megapixels per second throughput while maintaining decoding accuracy.
2Speed
If high-resolution video data is decoded at 90 frames per second, then immersive virtual reality experience is provided, but bandwidth and storage requirements increase significantly
Solution Approach 1:
The patent applies preliminary action by having the CPU perform initial decoding operations (Huffman entropy decoding, inverse run length encoding, de-zigzagging) before the main parallel processing stage. This preliminary processing reduces the complexity of the data that needs to be handled in subsequent stages, enabling efficient high-frame-rate playback while managing bandwidth and storage requirements.
3Productivity
If joint CPU and GPU decoding is implemented, then decoding throughput increases, but system complexity increases
Solution Approach 1:
The patent segments the decoding workload between CPU and GPU based on task characteristics. The CPU handles serial tasks that require sequential processing (entropy decoding, de-zigzagging), while the GPU handles parallel tasks that benefit from its multi-core architecture (de-quantization, inverse DCT, color conversion). This clear segmentation reduces system complexity by assigning tasks according to processor strengths rather than requiring one processor to handle all operations.
Solution Approach 2:
The patent introduces an intermediary mechanism where the CPU generates a partially-decoded bitstream that serves as input to the GPU. This intermediary data format allows the two processors to work together efficiently, with the CPU preparing data in a form that optimizes GPU processing, thereby managing system complexity through structured data flow between components.
Data Source
AI summary
A virtual reality or augmented reality experience of a scene may be decoded for playback for a viewer through a combination of CPU and GPU processing. A video stream may be retrieved from a data store. A first viewer position and/or orientation may be received from an input device, such as the sensor package on a head-mounted display (HMD). At a processor, the video stream may be partially decoded to generate a partially-decoded bitstream. At a graphics processor, the partially-decoded bitstream may be further decoded to generate viewpoint video of the scene from a first virtual viewpoint corresponding to the first viewer position and/or orientation. The viewpoint video may be displayed on a display device, such as screen of the HMD.