Feature Map Translation for Neural Network In-Loop Video Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding technologies face challenges in effectively addressing coding artifacts and mismatches between post-loop and in-loop data processing, limiting the coding gain and quality of reconstructed video frames.
Innovation Solution
Incorporating a convolutional neural network (CNN) as an in-loop filter within the video coding system, which applies feature map translation mechanisms to compensate for these mismatches, allowing for improved image restoration and subsequent prediction processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional in-loop filtering methods are used, then the processing speed is maintained, but the coding gain and quality of reconstructed video frames are limited
Solution Approach 1:
The patent replaces traditional mechanical filtering systems with a neural network-based in-loop filter. The neural network processes feature maps to reduce coding artifacts and improve reconstruction quality, substituting conventional signal processing mechanisms with intelligent learning-based systems that achieve superior coding gain while maintaining processing speed.
2Manufacturing precision
If neural network-based in-loop filtering is applied, then the quality of reconstructed frames is improved, but the device complexity increases
Solution Approach 1:
The patent segments the neural network into multiple layers, each processing specific feature maps at different stages of the video coding pipeline. This segmentation allows the complex neural network functionality to be distributed across manageable layers, reducing the complexity burden on any single component while maintaining overall system performance.
Solution Approach 2:
The neural network-based in-loop filter is designed to handle multiple functions within a single integrated system. It processes various types of feature maps, applies different filtering operations, and adapts to different video content characteristics, making the system more versatile while consolidating complexity into a unified multi-functional component rather than requiring separate specialized systems.
3Manufacturing precision
If feature map translation is applied to compensate for mismatches, then the coding artifacts are reduced, but the processing time increases
Solution Approach 1:
The patent applies feature map translation and neural network processing during the in-loop filtering stage, which occurs before the final reconstruction and encoding of video frames. This preliminary action allows the system to pre-process and correct coding artifacts early in the pipeline, reducing the computational burden on subsequent stages and minimizing overall processing time while achieving effective artifact reduction.
Data Source
AI summary
A video coding method that uses a neural network to perform in-loop filtering is provided. The video coder provides a neural network filter by spatially shifting a feature map of the neural network. The video coder reconstructs a current block of pixels of a current picture of a video based on prediction residuals generated or received by the video coder. The video coder filters samples of the reconstructed current block by using the neural network filter. The video coder encodes or decodes a subsequent block of the video by inter- or intra-prediction based on the filtered samples of the current block.