Jointly Trained Neural Networks for Video Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video encoding and decoding methods rely on separate optimization of in-loop and out-of-loop filters, leading to sub-optimal filtering and increased complexity, as they are optimized independently using different metrics.
Innovation Solution
The use of jointly trained multi-part neural networks for both in-loop and out-of-loop filtering, where a first neural network optimizes the reconstructed image quality for coding reference and a second neural network optimizes visual quality for display, leveraging a shared architecture to reduce computational complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If separate optimization of in-loop and out-of-loop filters is used, then filtering can be performed independently for each function, but the filtering performance becomes sub-optimal and device complexity increases
Solution Approach 1:
The patent combines separate in-loop and out-of-loop filter optimizations into a single joint training process. The encoder and decoder both apply the same neural network filters (CNNs) for in-loop filtering during encoding and out-of-loop filtering during decoding, eliminating the need for separate optimization processes and reducing system complexity while improving overall filtering performance.
Solution Approach 2:
The neural network filters are designed to serve multiple functions: they perform in-loop filtering to improve coding reference quality during encoding, and out-of-loop filtering to enhance visual quality during decoding. This multi-functional approach allows a single filtering system to address both coding efficiency and visual quality requirements without increasing device complexity.
2Productivity
If jointly trained multi-part neural networks are used for both in-loop and out-of-loop filtering, then filtering efficiency is improved and complexity is reduced, but training and implementation becomes more challenging
Solution Approach 1:
The joint training process is segmented into distinct phases: first, the neural network is trained for in-loop filtering to optimize coding reference quality; then, the same network is fine-tuned for out-of-loop filtering to optimize visual quality. This segmentation of the training process makes the complex joint optimization more manageable while maintaining filtering efficiency.
Solution Approach 2:
The joint training approach uses feedback from both encoding performance metrics and decoded visual quality metrics to optimize the neural network filters. The system iteratively adjusts filter parameters based on feedback from both the encoder's rate-distortion optimization and the decoder's visual quality assessment, achieving high filtering efficiency while managing training complexity through structured feedback loops.
Data Source
AI summary
A method for video decoding is disclosed. A first reconstructed version of an image block of an encoded video is accessed. The first reconstructed version of said image block is filtered by a first neural network to form a second reconstructed version of said image block to be used as reference. The second reconstructed version of said image block is filtered by a second neural network to form a third reconstructed version of said image block to be displayed. The first and second neural networks are trained jointly.