Real-Time VVC Decoding With CTU-Level Multithreading
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding systems, such as VVC, face limitations in fully utilizing CPU processing capabilities due to inefficiencies in parallelization techniques like Wavefront Parallel Processing (WPP) and Overlapped Wavefront (OWF), leading to coding losses and memory access issues, which hinder real-time decoding on multi-core systems.
Innovation Solution
A Multi-threaded (MT) framework for VVC decoding that employs CTU-level parallel processing, dynamic thread allocation, and interleaved filtering stages to optimize parallelism without compromising coding efficiency, utilizing CTU rows and temporal layers for improved CPU and GPU processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If Wavefront Parallel Processing (WPP) is used for VVC decoding, then parallel processing capability is improved, but coding efficiency deteriorates due to coding losses
Solution Approach 1:
The decoding process is segmented into independent CTU row groups that can be processed in parallel. Each thread handles a specific CTU row group, allowing simultaneous processing while maintaining coding accuracy by avoiding unnecessary data partitioning that would cause coding losses.
Solution Approach 2:
The patent transitions from traditional WPP that processes data sequentially along the wavefront to a parallel processing approach that operates across multiple dimensions simultaneously - specifically by dividing work at the CTU row group level and processing multiple rows in parallel, thereby improving productivity without sacrificing coding efficiency.
2Speed
If Overlapped Wavefront (OWF) is used for VVC decoding, then processing speed is improved, but memory access issues worsen
Solution Approach 1:
The video data is segmented into distinct CTU row groups that are processed independently in parallel. This segmentation eliminates memory access conflicts by ensuring that each processing thread operates on its own dedicated data segment, thereby maintaining reliability while improving processing speed.
Solution Approach 2:
The patent introduces an intermediary data structure organization where reference data is pre-prepared and organized for each CTU row group before parallel processing begins. This intermediary preparation eliminates memory access issues during the actual parallel decoding by ensuring all necessary data is readily available to each thread without conflicts.
3Productivity
If fine-grained parallelization is applied to decoding stages, then CPU and GPU utilization is maximized, but system complexity increases
Solution Approach 1:
The decoding system is segmented into distinct functional stages (CABAC decoding, motion vector generation, inter/intra coding unit reconstruction, inverse reshaping, in-loop filtering) that can be independently parallelized. Each stage can be processed by separate thread pools, maximizing resource utilization while keeping each individual stage relatively simple and manageable.
Solution Approach 2:
The patent implements dynamic thread allocation where the number of threads assigned to each decoding stage is adjusted based on the specific characteristics of the video content and available computational resources. This dynamic approach allows the system to maximize utilization without requiring a fixed complex architecture, as the system adapts to different scenarios.
Data Source
AI summary
Apparatus and methods for implementing a real-time Versatile Video Coding (VVC) decoder use multiple threads to address the limitation with existing parallelization techniques and fully utilizes the available CPU computation resource without compromising on the coding efficiency. The proposed Multi-threaded (MT) framework uses CTU level parallel processing techniques without compromising on the memory bandwidth. Picture level parallel processing separates the sequence into temporal levels by considering the picture's referencing hierarchy. Embodiments are provided using various optimization techniques to achieve real-time VVC decoding on heterogenous platforms with multi-core CPUs, for those bitstreams generated using a VVC reference encoder with a default configuration.