Neural Network Loop Filtering With Multi-Scale Residual Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding and decoding technologies face challenges in improving filtering effects while reducing the operation complexity of neural network loop filters (NNLFs) to enhance coding and decoding efficiency.
Innovation Solution
Implementing a neural network loop filter (NNLF) with residual units containing sequentially connected residual blocks, each with multiple parallel convolution layers of different kernel sizes, to extract multi-scale receptive field features, thereby improving filtering effects and reducing operation complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a neural network loop filter (NNLF) is used to improve filtering effects, then the filtering quality is improved, but the operation complexity increases
Solution Approach 1:
The NNLF is divided into multiple residual blocks, where each residual block contains parallel convolution layers with different kernel sizes. This segmentation allows the filter to process different spatial scales independently, improving filtering quality while distributing computational complexity across modular units that can be optimized separately.
Solution Approach 2:
The patent introduces multi-scale receptive fields by using parallel convolution layers with different kernel sizes (e.g., 3x3, 5x5, 7x7). This adds a dimensional aspect of scale variation to the filtering process, enabling the network to capture features at multiple spatial resolutions simultaneously, which improves filtering effectiveness without linearly increasing complexity.
2Manufacturing precision
If multiple parallel convolution layers with different kernel sizes are used to extract multi-scale features, then the filtering effect is improved, but the computational complexity increases
Solution Approach 1:
Multiple parallel convolution layers with different kernel sizes are merged into a single residual block structure. The outputs of these parallel layers are combined through element-wise addition or concatenation followed by a fusion convolution layer. This merging strategy allows the network to extract multi-scale features simultaneously while sharing computational resources and reducing redundant calculations compared to separate processing streams.
Solution Approach 2:
The patent varies the kernel sizes (parameters) of parallel convolution layers to create different receptive fields. By changing this key parameter across parallel branches (e.g., 3x3, 5x5, 7x7 kernels), the network achieves multi-scale feature extraction. The use of depth-wise separable convolutions and group convolutions further optimizes these parameter variations to reduce computational complexity while maintaining the multi-scale capability.
3Manufacturing precision
If loop filtering is performed on the reconstructed image to improve image quality, then the distortion is reduced, but the coding and decoding efficiency decreases
Solution Approach 1:
The NNLF is integrated into the loop filtering stage, performing filtering actions preliminary to the coding and decoding processes. By pre-filtering the reconstructed image within the decoding loop, the network reduces distortion before subsequent coding operations, thereby improving overall image quality without requiring additional post-processing filtering steps that would further reduce efficiency.
Solution Approach 2:
Traditional mechanical loop filters (such as deblocking filters or sample adaptive offset filters) are replaced with a neural network-based filtering system. This substitution transitions from rule-based, hand-crafted filtering mechanisms to learning-based adaptive filtering that automatically optimizes filtering parameters and strategies, achieving superior image quality improvement while maintaining or enhancing coding and decoding efficiency through the network's ability to process multiple operations in parallel.
Data Source
Figure 1
Figure 2~3
Figure 4~7
AI summary
Provided in the embodiments of the present application are a filtering method, an encoding method, a decoding method, an apparatus, a computer-readable medium and an electronic device. The filtering method based on a neural network comprises: generating input data of a neural network filter on the basis of an image to be filtered, wherein the input data at least includes the image to be filtered; inputting the input data into the neural network filter, wherein the neural network filter includes a residual unit for extracting image feature information, the residual unit includes a plurality of residual blocks which are connected in sequence, and a first residual block among the plurality of residual blocks includes a plurality of convolution layers which are arranged in parallel and have different convolution kernel sizes; and acquiring a filtered image which is output by means of the neural network filter for the image to be filtered. By means of the technical solution provided in the embodiments of the present application, the effect of filtering can be improved while reducing the operation complexity of a neural network filter, thereby facilitating an improvement in the encoding and decoding efficiency of a video.