Selective Neural Intra Prediction for Coding Complexity Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding systems face challenges in efficiently combining block-based prediction with neural network-based prediction, particularly in determining when to apply neural network-based prediction to optimize video block processing.
Innovation Solution
A device determines conditions for applying neural network-based prediction to video blocks, using a matrix with adjustable weights, and switches between neural network-based and block-based prediction tools based on complexity thresholds, including the use of template-based multiple reference line (TMRL) and decoder side intra mode derivation (DIMD) tools.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural network-based prediction is applied to all video blocks, then prediction accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent changes the parameter of prediction method selection by introducing a flag in the bitstream that indicates whether neural network-based prediction should be applied. This allows dynamic adjustment of processing complexity based on content characteristics and system capabilities, resolving the contradiction by enabling high accuracy only when needed rather than universally applying the complex neural network approach to all blocks.
Solution Approach 2:
The system dynamically switches between block-based prediction and neural network-based prediction based on conditions determined during encoding. The encoder evaluates whether to apply neural network prediction for each block and signals this decision to the decoder, allowing the system to adapt processing complexity to actual content requirements rather than using a fixed approach.
2Device complexity
If block-based prediction is used, then processing complexity is reduced, but prediction accuracy deteriorates
Solution Approach 1:
The patent segments the video processing into two distinct paths: block-based prediction for standard blocks and neural network-based prediction for selected blocks. By dividing the processing approach based on block characteristics and encoder decisions, the system can maintain low complexity for most blocks while achieving high accuracy for blocks that benefit from neural network processing.
Solution Approach 2:
The patent introduces an intermediary mechanism (the prediction mode flag in the bitstream) that mediates between the simple block-based prediction and the complex neural network-based prediction. This intermediary allows the decoder to know when to apply which method, enabling the system to achieve high accuracy where needed without universally increasing complexity.
3Productivity
If neural network-based prediction is selectively applied, then video coding efficiency is improved, but device complexity increases
Solution Approach 1:
The encoder performs preliminary evaluation to determine which blocks should use neural network-based prediction and signals this information in advance through the bitstream. This preliminary action allows the decoder to prepare appropriately without needing to perform complex analysis, improving coding efficiency while distributing complexity primarily to the encoder side.
Solution Approach 2:
The system uses feedback from the encoder's analysis of block characteristics to guide the decoder's prediction method selection. The encoder evaluates content properties and sets appropriate flags in the bitstream, creating a feedback loop that optimizes coding efficiency while managing device complexity through intelligent signal transmission rather than universal complex processing.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Video coding may selectively use neural network-based intra prediction in combination with advanced prediction tools. In examples, a device (e.g., encoder or decoder) may obtain a current video block. For example, the current video block may be associated with a block-based intra prediction tool. The device may determine whether a condition for applying a neural network-based prediction to the current video block is met. The device may perform a prediction for the current video block based at least on whether the condition for applying the neural network-based prediction to the current video block is met. For example, if the condition is met, the prediction may be performed using at least a first neural network, and, if the condition is not met, the prediction may be performed using the block-based intra prediction tool. The device may decode or decode the current video block based at least on the prediction.