Mixed NAL Unit Constraints for Adaptive Sub-Picture Video Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding systems struggle with efficiently encoding and decoding mixed pictures containing both intra-random access point (IRAP) and non-IRAP sub-pictures, as they require all NAL units of a single picture to have the same type, leading to inefficiencies in bandwidth and resource usage, particularly in virtual reality applications where different sub-pictures may have varying display likelihoods.
Innovation Solution
A flag, referred to as mixed_nalu_types_in_pic_flag, is introduced to indicate whether a picture contains both IRAP and non-IRAP sub-pictures, constraining the picture to exactly two NAL unit types, allowing the decoder to treat sub-pictures differently based on this flag, and ensuring efficient decoding and display.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If all NAL units of a single picture must have the same type, then decoding simplicity is maintained, but coding efficiency and adaptability are reduced
Solution Approach 1:
The picture is segmented into multiple sub-pictures, each with its own NAL unit type (IRAP or non-IRAP), allowing different coding efficiencies while maintaining overall picture structure. This segmentation enables the system to adapt to varying display likelihoods of different sub-pictures without requiring all NAL units to have the same type.
Solution Approach 2:
Different sub-pictures within the same picture are assigned different NAL unit types based on their specific requirements and display likelihoods. This local differentiation allows each sub-picture to be coded optimally for its intended use, improving overall adaptability while the flag mechanism keeps the complexity manageable.
2Reliability
If more IRAP pictures are transmitted for sub-pictures likely to be viewed, then decoding reliability is improved, but network resources and bandwidth are consumed
Solution Approach 1:
Different sub-pictures are assigned different NAL unit types based on their display likelihood. Sub-pictures likely to be viewed can use IRAP type for reliable decoding, while less likely sub-pictures use non-IRAP type to save bandwidth. This local differentiation optimizes the balance between reliability and bandwidth usage.
Solution Approach 2:
The system dynamically adjusts the NAL unit type assignment based on the specific requirements of each sub-picture and its likelihood of being displayed. This dynamic approach allows the system to optimize resource allocation in real-time, transmitting more IRAP pictures only when needed for sub-pictures with high display probability.
3Productivity
If sub-pictures are coded into separate sub-bitstreams, then coding efficiency is improved, but device complexity increases
Solution Approach 1:
The picture is divided into multiple sub-pictures that can be coded into separate sub-bitstreams, allowing independent optimization of each sub-picture. This segmentation improves coding efficiency by enabling different compression levels and types for different sub-pictures while the flag mechanism helps manage the complexity of handling multiple bitstreams.
Solution Approach 2:
The flag mechanism serves multiple functions: it indicates mixed NAL unit types, guides decoder behavior, and enables proper processing of different sub-picture types. This multi-functionality helps manage complexity by consolidating control information into a single flag rather than requiring separate control mechanisms for each sub-picture type.
Data Source
AI summary
A video coding mechanism is disclosed. The mechanism includes receiving a bitstream comprising a flag and a plurality of sub-pictures associated with a picture. The plurality of sub-pictures are contained in a plurality of video coding layer (VCL) network abstraction layer (NAL) units. The mechanism determines that VCL NAL units of one or more of the sub-pictures of the picture all have a first particular value of NAL unit type and other VCL NAL units in the picture all have a different second particular value of NAL unit type based on a value of the flag. One or more of the sub-pictures are decoded based on the first particular value of NAL unit type and the second particular value of NAL unit type. One or more of the sub-pictures are forwarded for display as part of a decoded video sequence.