Dual Bitstream GOP Switching for Viewport-Driven Camera Changes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding techniques face challenges in efficiently switching between multiple camera views in augmented reality (AR), virtual reality (VR), and volumetric video applications without introducing visual artifacts or significant latency, especially when viewer positions or orientations change, leading to inefficient bandwidth usage and inconsistent image quality.
Innovation Solution
Implementing a dual bitstream structure for each camera, comprising a primary image stream with GOP structures and a secondary image stream of reconstructed I-frames, allowing seamless switching between camera views by using reconstructed I-frames to set up correct coding contexts for predictive decoding, thereby maintaining GOP integrity and reducing visual artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional video coding techniques are used for switching between multiple camera views, then bandwidth usage is reduced, but visual artifacts and latency increase during viewport changes
Solution Approach 1:
The patent segments the video stream into multiple independent bitstreams, each corresponding to a specific camera view with its own complete GOP structure. This segmentation allows the decoder to independently switch between views without relying on inter-view prediction, thereby maintaining image quality consistency during viewport changes while efficiently using bandwidth by transmitting only the necessary view data.
2Adaptability or versatility
If viewport switching is performed during GOP structure, then viewer experience is improved, but coding context consistency is lost causing visual artifacts
Solution Approach 1:
The patent prepares complete GOP structures for each camera view in advance, ensuring that each view has its own independent coding context ready before switching is needed. This preliminary preparation allows the decoder to seamlessly switch between views during viewport changes without losing coding context consistency, thereby preventing visual artifacts while maintaining adaptability to viewer movements.
3Ease of operation
If multiple complete I-frame streams are transmitted for each camera view, then view switching is simplified, but bandwidth consumption increases significantly
Solution Approach 1:
The patent applies local quality by providing complete GOP structures with I-frames only at the positions where viewport switching occurs, rather than transmitting complete I-frame streams for every camera view. This localized approach simplifies view switching by ensuring coding context availability at switch points while significantly reducing bandwidth consumption by eliminating redundant I-frames in non-switching regions.
Data Source
AI summary
A multi-view image stream encoded with primary and secondary image is accessed. Each primary image stream comprises groups of pictures (GOPs). Each secondary image stream comprises I-frames generated from a corresponding primary image stream. Viewpoint data collected in real time is received from a recipient decoding device to indicate that the viewer's viewpoint has changed from a specific time point. A camera is selected based on the viewer's changed viewpoint. It is determined whether the specific time point corresponds to a non-I-frame in a GOP of a primary image stream of the selected camera. If so, an I-frame from a secondary image stream corresponding to the primary image stream is transmitted to the recipient decoding device.