Prioritized Video Decoding for Early Regional Frame Output
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding standards lack the ability to provide fast access to prioritized data in the decoded pixel buffer, limiting applications that require immediate access to specific parts of a video frame, such as virtual reality and omnidirectional content rendering.
Innovation Solution
A modified hypothetical reference decoder model that allows prioritized decoding and output of parts of a picture by assigning prioritization information to decoding units (DUs), enabling partial reconstruction of a picture before full decoding is complete, with DUs having no intra-coding dependencies.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If the entire picture is decoded before outputting any parts, then decoding completeness is ensured, but access time to specific parts of the picture increases
Solution Approach 1:
The picture is divided into multiple independent decoding units (DUs) that can be decoded separately. Each DU represents a self-contained portion of the picture that does not depend on other DUs for decoding, allowing selective decoding and early output of specific regions without waiting for the entire picture to be decoded.
Solution Approach 2:
High-priority decoding units are decoded and output before lower-priority DUs are fully processed. The system performs preliminary decoding on selected DUs based on priority information, allowing applications to access decoded data earlier in the decoding process while maintaining the option to continue decoding remaining units.
2Productivity
If all decoding units are decoded in parallel, then decoding speed increases, but memory requirements and processing complexity increase
Solution Approach 1:
Different decoding units are assigned different priority levels based on their importance to the application. Instead of treating all DUs uniformly, the system applies quality differentiation by decoding high-priority DUs with higher resource allocation and faster processing, while lower-priority DUs receive standard processing, optimizing the balance between speed and complexity.
Solution Approach 2:
The system performs partial decoding by processing only the necessary high-priority DUs first, rather than decoding the entire picture. This partial action approach provides sufficient decoded data for time-sensitive applications without incurring the full cost of complete picture decoding, thereby improving productivity while controlling complexity.
3Stability of the object's composition
If different video objects are packed in the same picture, then time synchronization is achieved, but fine-grained access to individual objects is lost
Solution Approach 1:
Each video object within the picture is encapsulated as a separate decoding unit with its own priority identifier. This segmentation allows the system to maintain time synchronization by keeping all objects in the same picture frame while enabling fine-grained access by selecting and decoding only the specific DU containing the desired video object based on its priority level.
Solution Approach 2:
Priority information acts as an intermediary mechanism that bridges the contradiction between time synchronization and fine access. The priority metadata associated with each DU provides the interface for applications to request specific video objects by priority level, enabling selective access to individual objects while maintaining the synchronized structure of the complete picture.
Data Source
AI summary
Methods and apparatus for processing an encoded representation of a sequence of pictures are disclosed, wherein the method includes: allocating memory for a current picture of the sequence of pictures in a decoded picture buffer, DPB; receiving a bitstream comprising an encoded representation of the current picture, and storing the encoded representation as a plurality of decoding units DUs, in a coded picture buffer, CPB, each of the plurality of DUs comprising an encoded set of pixels, wherein no coding dependency exists between coded pixels of different DUs of the current picture; receiving prioritization information associated with each of the plurality of DUs stored in the CPB, the prioritization information defining a decoding order in which the DUs of the current picture are being decoded; decoding one or more of said plurality of DU's into one or more decoded DU's, in accordance with said decoding order, each decoded DU comprising a set of pixels, each set of pixels being associated with a different area in the current picture; storing the sets of pixels in the DPB, a set of pixels being stored at a location in the allocated memory of the DPB that corresponds to a position of the area in the current picture; and, outputting as a partial output, one or more sets of pixels of decoded DUs of the current picture in accordance with the decoding order, while other DUs of the current picture are being decoded or received by the decoding apparatus.