Neural Network In-Loop Filter With Residual Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video coding technologies face challenges in reducing distortion during compression, particularly in scaling neural network filter outputs and combining multiple models effectively, which affects video quality and bitrate efficiency.
Innovation Solution
The implementation of neural network filter models trained for in-loop filtering, with residual scaling and adaptive inference block sizes, allows for better performance by applying different neural network filters to samples with varying characteristics and combining their outputs using weighted sums based on specific criteria.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If neural network filter output is directly applied to unfiltered samples, then filtering is performed, but the distortion reduction effectiveness is insufficient
Solution Approach 1:
The patent applies a scaling function to modify the residual output parameters from the neural network filter before adding them to the unfiltered sample. This parameter transformation optimizes the filter output to achieve better distortion reduction while maintaining filtering effectiveness.
2Ease of manufacture
If a fixed inference block size is used for all samples, then processing is simplified, but filtering performance varies for different sample characteristics
Solution Approach 1:
The patent dynamically determines the inference block size based on sample characteristics such as boundary strength and activity levels. This adaptive approach allows the system to simplify processing for uniform samples while applying more detailed filtering where needed, balancing simplicity and performance.
Solution Approach 2:
The patent applies different filtering strategies to different regions of the video data based on their characteristics. Samples with higher boundary strength or activity are processed with different block sizes than uniform regions, optimizing filtering performance for each local area while maintaining overall system efficiency.
3Adaptability or versatility
If multiple neural network filter models are used for different sample characteristics, then filtering adaptability improves, but system complexity increases
Solution Approach 1:
The patent segments the filtering process into different modes based on sample characteristics such as boundary strength and activity. By dividing the processing into distinct categories (e.g., strong boundary mode, weak boundary mode, high activity mode, low activity mode), the system can select appropriate filter models for each segment, improving adaptability while managing complexity through structured organization.
Data Source
AI summary
A method implemented by a video coding apparatus. The method includes applying an output of a neural network (NN) filter to an unfiltered sample of a video unit to generate a residual, applying a scaling function to the residual to generate a scaled residual, adding another unfiltered sample to the scaled residual to generate a filtered sample, and converting between a video media file and a bitstream based on the filtered sample that was generated. A corresponding video coding apparatus and non-transitory computer readable medium are also disclosed.