Matrix Intra Prediction Downsampling for Lower Codec Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The computational complexity and latency issues in matrix based intra prediction (MIP) for video coding are exacerbated by the need for averaging and shift operations in deriving reduced boundary samples, and misalignment of boundary samples with the matrix multiplication unit (MMU) output, which increases encoder and decoder load.
Innovation Solution
The proposed solution aligns reduced boundary samples with the MMU output by downsampling input boundary samples using filters centered on the MMU output or selecting aligned samples, reducing the number of averaged samples, and deriving samples in parallel for matrix multiplication and interpolation to minimize complexity and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If boundary samples are averaged to derive reduced boundary samples, then prediction accuracy is improved, but computational complexity and latency increase
Solution Approach 1:
The patent extracts only the necessary boundary samples for prediction without performing full averaging operations. Specifically, it uses a subset of boundary samples (e.g., corner samples or samples at specific positions) directly for matrix multiplication, eliminating the need to average all boundary samples while maintaining sufficient prediction accuracy.
Solution Approach 2:
The patent segments the boundary sampling process into distinct stages: first identifying and extracting specific boundary samples, then using them in matrix multiplication, and finally performing interpolation only where necessary. This segmentation allows the system to avoid unnecessary averaging operations while maintaining prediction quality.
2Measurement precision
If boundary samples are averaged to derive reduced boundary samples, then prediction accuracy is improved, but encoder and decoder latency increase
Solution Approach 1:
The patent extracts only the necessary boundary samples for prediction without performing full averaging operations. Specifically, it uses a subset of boundary samples (e.g., corner samples or samples at specific positions) directly for matrix multiplication, eliminating the need to average all boundary samples while maintaining sufficient prediction accuracy.
Solution Approach 2:
The patent performs preliminary identification and extraction of boundary samples before the matrix multiplication operation. By pre-selecting the samples that will be used (such as corner samples or samples at specific positions), the system avoids performing averaging operations during the main prediction pipeline, thereby reducing latency.
3Manufacturing precision
If two-step derivation process is used for reduced boundary samples, then alignment with MMU output is improved, but encoder and decoder latency increase
Solution Approach 1:
The patent merges the boundary sample derivation and alignment operations into a single step. Instead of first deriving reduced boundary samples and then separately aligning them with MMU output, the system performs both operations simultaneously by directly selecting samples that are already aligned with the MMU output positions.
Solution Approach 2:
The patent performs preliminary identification and extraction of boundary samples before the matrix multiplication operation. By pre-selecting the samples that will be used (such as corner samples or samples at specific positions), the system avoids performing averaging operations during the main prediction pipeline, thereby reducing latency.
4Measurement precision
If maximum number of boundary samples are used in average operation, then prediction accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent extracts only the necessary boundary samples for prediction without performing full averaging operations. Specifically, it uses a subset of boundary samples (e.g., corner samples or samples at specific positions) directly for matrix multiplication, eliminating the need to average all boundary samples while maintaining sufficient prediction accuracy.
Solution Approach 2:
The patent applies partial averaging or selective sampling instead of averaging all boundary samples. By using only the necessary subset of samples (e.g., 4 corner samples out of potentially 64 boundary samples), the system achieves sufficient prediction accuracy with significantly reduced computational complexity.
Data Source
AI summary
A prediction unit for an encoder or decoder implements matrix based intra prediction. Input boundary samples for a current block are downsampled to obtain reduced boundary samples for matrix multiplication and/or linear interpolation, or both. In one embodiment, downsampling is performed in a manner that aligns the reduced boundary samples with an output of a matrix multiplication unit of the prediction unit. In other embodiments, downsampling is performed without averaging. The embodiments reduce the complexity of the prediction unit and the latency of the encoder or decoder.