Hybrid Video Codec Switching Across Temporal Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding technologies face challenges in efficiently handling both large motion vectors and small motion scenarios, with traditional codecs excelling in capturing large motion while neural codecs outperforming in small motion, leading to suboptimal performance in random access scenarios.
Innovation Solution
A combined codec approach that utilizes conventional encoders for lower temporal layers and neural network-based encoders for higher temporal layers, leveraging the strengths of both to achieve state-of-the-art video coding performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional codecs are used for video coding, then large motion scenarios are handled well, but small motion scenarios show suboptimal performance
Solution Approach 1:
The patent changes the coding parameter by switching between conventional codec and neural network codec based on the temporal layer identifier. For lower temporal layers (first set), conventional coding is used which excels at large motion. For higher temporal layers (second set), neural network-based coding is used which outperforms in small motion scenarios. This dynamic parameter change resolves the contradiction by adapting the coding method to the specific motion characteristics of different temporal layers.
2Reliability
If neural network-based codecs are used for all temporal layers, then small motion scenarios improve, but overall coding efficiency decreases due to suboptimal performance on large motion
Solution Approach 1:
The patent applies local quality by using different coding methods for different temporal layers. Lower temporal layers (which typically contain larger motion) use conventional coding optimized for their characteristics. Higher temporal layers (which typically contain smaller motion) use neural network coding optimized for their characteristics. This localized application of different coding qualities resolves the contradiction by ensuring each temporal layer is coded with the most appropriate method for its specific motion characteristics.
3Reliability
If a single coding method is used for all temporal layers, then device complexity is reduced, but video coding performance suffers due to inability to optimize for different motion scenarios
Solution Approach 1:
The patent introduces dynamics by making the codec structure adaptive rather than static. The combined codec dynamically selects between conventional and neural network coding methods based on the temporal layer identifier of each picture. This dynamic adaptation allows the system to optimize performance for different motion scenarios while maintaining a unified codec structure that manages complexity through conditional logic rather than requiring completely separate coding systems.
Data Source
AI summary
An example device for decoding video data includes a processing system comprising one or more processors implemented in circuitry and configured to: determine that a first temporal layer identifier of a first picture of the video data is included in a first set of temporal layers; in response to the first temporal layer identifier being included in the first set of temporal layers, decode blocks of the first picture on a block by block basis; determine that a second temporal layer identifier of a second picture of the video data is included in a second set of temporal layers, the second set of temporal layers being higher than the first set of temporal layers; and in response to the second temporal layer identifier being included in the second set of temporal layers, execute a neural network-based video decoder to decode the second picture.