Video Multiplexing Apparatus Using Decoding Units for Ultra-Low Delay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing MPEG-2 Systems standard for digital television broadcasting experiences significant codec delay, which is not suitable for applications requiring ultra-low delays, such as videoconferencing and remotely controlled robots, as it results in delays of around 100 milliseconds or more.
Innovation Solution
The proposed solution involves a video multiplexing apparatus and multiplexed video decoding apparatus that utilize advanced video coding standards like HEVC, which introduce the concept of Decoding Units (DUs) to allow for earlier decoding and display of video data by including decode time information in the packet headers, enabling ultra-low delay encoding and decoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If MPEG-2 Systems standard is used for video multiplexing, then compatibility with existing digital television broadcasting systems is maintained, but codec delay becomes significant (100 milliseconds or more)
Solution Approach 1:
The video picture is divided into multiple sub-pictures, and each sub-picture is further divided into decoding units (DUs). This segmentation allows the decoder to process and display portions of the picture earlier without waiting for the entire picture to be decoded, thereby reducing codec delay while maintaining compatibility with MPEG-2 Systems standard multiplexing structures.
Solution Approach 2:
The encoder performs preliminary processing by dividing the picture into sub-pictures and DUs, and embeds decode time information in the packet headers in advance. This allows the decoder to start decoding and displaying sub-pictures as soon as their data is received, without waiting for the complete picture decoding process, thus reducing overall delay.
2Loss of time
If HEVC with Decoding Units is used to reduce codec delay, then near-real-time video processing is achieved, but device complexity increases
Solution Approach 1:
The picture is segmented into sub-pictures and decoding units (DUs), allowing incremental decoding and display. This segmentation enables the decoder to process smaller units independently and display them earlier, reducing codec delay while distributing the computational workload across multiple smaller processing tasks rather than one large complex task.
Solution Approach 2:
The decoder performs partial decoding actions by processing individual DUs and sub-pictures independently rather than waiting for complete picture decoding. This partial action approach allows early display of decoded portions, reducing perceived delay while managing complexity through incremental processing.
3Ease of manufacture
If picture-level decoding is used, then decoding simplicity is maintained, but display delay increases due to waiting for complete picture decoding
Solution Approach 1:
The picture is divided into sub-pictures and decoding units (DUs), which are the smallest decodable entities. This segmentation allows the decoder to process and display individual DUs as soon as their data is received, eliminating the need to wait for complete picture decoding while maintaining relatively simple decoding operations for each small unit.
Solution Approach 2:
The decoding process is transformed from a single-dimensional picture-level operation to a multi-dimensional hierarchical structure with pictures containing sub-pictures containing DUs. This dimensional change enables parallel processing and earlier display of decoded portions without significantly increasing the complexity of individual decoding operations.
Data Source
AI summary
A video multiplexing apparatus includes: a control unit which determines first decode/display time information and second decode/display time information for each picture in video data and for each encoding unit of first media data; a video packet generating unit which appends the first and second decode/display time information for the picture to encoded data of the video data; and a media packet generating unit which appends the first and second decode/display time information for the encoding unit to encoded data of the first media data. The first decode/display time information is used when the decoding unit of the video data is a picture, and the second decode/display time information is used when the decoding unit is a sub-picture. The picture display time based on the second decode/display time information is earlier than the picture display time based on the first decode/display time information.