In-loop Neural Network Video Coding Spatial Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding technologies face challenges in efficiently enhancing image quality due to the complexity of neural networks and the mismatch between training and testing data when applied as in-loop filters.
Innovation Solution
The proposed solution involves using a neural network as an in-loop filter in video encoding and decoding systems, where the video frame is spatially divided into portions, and distinct parameter sets are applied to each portion based on learned error statistics, with a multi-pass training process to account for re-application of the neural network during in-loop operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a neural network is applied as an in-loop filter to enhance image quality, then image quality is improved, but computational complexity increases
Solution Approach 1:
The video frame is spatially divided into multiple portions, and the neural network is applied to each portion separately with distinct parameter sets. This segmentation allows the system to process different regions with appropriate complexity levels, improving overall image quality while managing computational load through localized processing rather than applying a single complex model to the entire frame
Solution Approach 2:
Different parameter sets are used for different spatial portions of the video frame based on learned error statistics. This local quality approach enables the neural network to adapt its processing characteristics to specific regions, enhancing image quality in areas that need it most while reducing unnecessary computation in regions with lower error characteristics
2Productivity
If a neural network is trained with standard procedures, then training efficiency is improved, but mismatch between training and testing data occurs
Solution Approach 1:
A multi-pass training process is employed where the neural network is trained multiple times with different parameter sets, and the output of one pass becomes the input for the next pass. This preliminary action during training simulates the in-loop operation conditions, ensuring that the trained model accurately reflects the actual testing scenario and eliminates the mismatch between training and deployment
Solution Approach 2:
The multi-pass training process incorporates feedback loops where the neural network output is fed back as input for subsequent passes. This feedback mechanism during training ensures that the model learns to handle the re-application of its own output, accurately simulating the in-loop filtering operation and eliminating the training-testing data mismatch
Data Source
AI summary
A method for video decoding includes receiving a video frame reconstructed based on data received from a bitstream. The method further includes extracting, from the bitstream, a first syntax element indicating whether a spatial partition for partitioning the video frame is active. The method also includes, responsive to the first syntax element indicating that the spatial partition for partitioning the video frame is active, determining a configuration of the spatial partition for partitioning the video frame, determining a plurality of parameter sets of a neural network, and applying the neural network to the video frame. The video frame is spatially divided based on the determined configuration of the spatial partition for partitioning the video frame into a plurality of portions, and the neural network is applied to the plurality of portions in accordance with the determined plurality of parameter sets.