Viewport Bitstream Splitting for Immersive Video Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Immersive video streaming faces latency and bandwidth inefficiencies due to the need to download and render all frames when the viewport changes, especially in virtual reality applications, leading to delays and degraded user experience.
Innovation Solution
The method involves splitting viewport bitstreams into smaller fragments using different Group-of-Pictures (GOP) structures and tile-based encoding, allowing for efficient transmission and rendering of only the necessary frames within the user's field of view, with fallback low-resolution streams for improved random-access and reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all frames are transmitted and rendered when viewport changes, then complete viewport coverage is achieved, but latency increases and bandwidth efficiency decreases
Solution Approach 1:
The video stream is segmented into tiles at the encoder level, allowing the receiver to selectively decode and render only the tiles within the current viewport. This segmentation enables partial rendering rather than requiring all frames to be processed, directly reducing latency while maintaining viewport coverage.
Solution Approach 2:
The patent extracts and transmits only the necessary viewport-related tile data rather than complete frame data. By taking out only the relevant portions needed for the current viewport, the system reduces the amount of data that must be transmitted and rendered, thereby reducing latency without sacrificing viewport coverage completeness.
2Reliability
If all frames are transmitted when viewport changes, then no rendering gaps occur, but bandwidth usage increases
Solution Approach 1:
The video content is divided into spatial segments (tiles) that can be independently transmitted and decoded. This allows the system to send only the specific tiles needed for the current viewport rather than transmitting all frame data, reducing bandwidth consumption while maintaining rendering continuity through proper tile sequencing and fallback mechanisms.
Solution Approach 2:
The patent implements partial action by transmitting only the minimum necessary tile data required for current viewport rendering rather than complete frame data. This partial transmission approach reduces bandwidth usage while maintaining sufficient rendering continuity through strategic selection of which tiles to transmit based on viewport position and motion prediction.
3Manufacturing precision
If high-resolution encoding is used for all tiles, then rendering quality is improved, but computational complexity and bandwidth increase
Solution Approach 1:
The patent applies local quality by encoding tiles at different resolutions based on their importance and visibility. Tiles within the current viewport are encoded at high resolution for optimal rendering quality, while tiles outside the viewport or less critical tiles use lower resolution encoding. This differential encoding approach maintains rendering quality where needed while reducing overall encoding complexity and bandwidth requirements.
Data Source
AI summary
A method, computer program, and computer system is provided for splitting viewport bitstreams. A first coded video bitstream is at least partially encoded using a first Group-of-Pictures (GOP) structure at a first resolution. A second coded video bitstream structured into a plurality of tiles is encoded using a second GOP structure at a second resolution, whereby the second GOP structure includes fewer coded pictures than the first GOP structure. A streaming bitstream for decoding or rendering is created using the first coded video bitstream and the second coded bitstream.