Neural Network In-Loop Filtering for Machine-Oriented Video Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding technologies struggle to effectively encode videos for machine tasks, as they often prioritize human visual characteristics over machine-specific requirements, leading to inefficiencies in compression and quality.
Innovation Solution
Implementing a neural network in-loop filter (NNLF) that can be dynamically enabled or disabled at various coding levels, with parameters signaled in the bitstream or derived from other VCM-coded processes, and applied in conjunction with other filters to enhance video quality for machine tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional video coding technologies are used that prioritize human visual characteristics, then subjective quality for human users is maintained, but compression efficiency and quality for machine tasks deteriorate
Solution Approach 1:
The patent applies different filtering strategies to different regions or aspects of the video data based on machine task requirements. The NNLF is selectively applied to enhance features relevant to machine vision tasks (such as object detection, segmentation, tracking) while maintaining compatibility with traditional coding structures. This allows the system to optimize for machine-specific quality metrics rather than uniform human visual quality across all video content.
Solution Approach 2:
The patent modifies coding parameters and filtering characteristics to better suit machine task requirements. By introducing the NNLF with learnable parameters and adjusting coding settings based on machine task performance metrics, the system transforms the optimization criteria from human visual system characteristics to machine-specific performance requirements, thereby improving measurement precision for machine tasks.
2Measurement precision
If a neural network in-loop filter is applied to enhance video quality for machine tasks, then compression efficiency and machine task performance improve, but system complexity increases
Solution Approach 1:
The neural network filter is trained offline in advance on representative video data and machine task requirements. This preliminary training phase allows the NNLF to learn optimal filtering parameters and characteristics before deployment. During actual video coding and decoding, the pre-trained filter applies learned transformations without requiring complex real-time optimization, thereby reducing runtime system complexity while maintaining high machine task performance.
Solution Approach 2:
The NNLF acts as an intermediary component between the traditional video coding pipeline and the machine task processing stage. It serves as a bridge that translates coded video data into a form more suitable for machine vision tasks without requiring fundamental changes to either the coding standards or the machine task algorithms. This intermediary approach isolates the complexity within a dedicated module rather than propagating it throughout the entire system.
3Adaptability or versatility
If neural network parameters are signaled in the bitstream for each coding level, then adaptability to different coding scenarios improves, but bitstream overhead increases
Solution Approach 1:
The patent segments the parameter signaling into different levels and scopes. Rather than signaling complete NNLF parameters at every coding level, the system divides parameters into those that are globally applicable (signaled once) and those that are locally specific (signaled selectively). This segmentation reduces redundant parameter transmission while maintaining the ability to adapt to different coding scenarios through hierarchical parameter control.
Solution Approach 2:
The patent applies partial parameter signaling based on the actual needs of different coding scenarios. Instead of always signaling the full set of NNLF parameters, the system signals only the necessary subset of parameters for each specific coding context. This partial action approach reduces bitstream overhead by avoiding transmission of parameters that would not affect the current coding scenario, while still maintaining adaptability when needed.
Data Source
AI summary
This disclosure relates generally to video coding, and more particularly to in-loop filtering for video coding for machine tasks based on neural networks. For example, utilization of one or more neural network in-loop filter (NNLF) may be determined (either enabled or disabled) at various coding levels. Such determination may be based on output of one or more other VCM-codec-related processed. The usages of the NNLF may be signaled in the bitstream or may be implicitly derived from the output of the one or more other VCM-coded-related processes during decoding process in a decoder or during in-loop decoding process of an encoder. The signaling may be provided at a corresponding constructed syntax structure at one the various coding levels. The NNLF may be applied at various orders with one or more other in-loop or post filters.