Content-Adaptive Neural Post-Filtering for Machine Video Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding technologies face challenges in optimizing intra prediction and motion compensation for efficient compression, particularly in scenarios where video content is adapted for machine analysis, leading to suboptimal bit usage and decoding performance.
Innovation Solution
Implementing a post-filtering neural network (NN) in video encoders and decoders that is trained based on metadata for machine tasks, using Supplemental Enhancement Information (SEI) messages to adaptively apply post-filtering parameters, such as bounding boxes and object masks, to enhance video content for machine consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional video coding techniques are used for machine analysis, then general video compression is achieved, but machine task performance is suboptimal
Solution Approach 1:
The patent applies preliminary action by training a post-filtering neural network in advance based on metadata for machine tasks before actual video decoding. The trained NN and its parameters are then reused during decoding, avoiding the need for real-time training while optimizing machine task performance. This is evident in the training phase where the NN learns from labeled data and is subsequently deployed for inference during video processing.
Solution Approach 2:
The patent utilizes parameter changes by adapting post-filtering parameters dynamically based on machine task requirements. The system selects and applies different post-filtering parameters from the trained NN depending on the specific machine task (e.g., object detection, segmentation), allowing optimization of video content for different machine vision applications without changing the fundamental coding structure.
2Reliability
If post-filtering neural network is trained and applied, then machine task performance improves, but computational overhead increases
Solution Approach 1:
The computationally intensive training of the post-filtering neural network is performed in advance during an offline phase, separate from the actual video decoding process. This preliminary training allows the system to store pre-computed parameters that can be efficiently applied during runtime, significantly reducing the computational energy required during actual video processing while maintaining improved decoding performance.
3Measurement precision
If video content is optimized for machine analysis, then machine task accuracy improves, but video quality for human viewing may deteriorate
Solution Approach 1:
The patent applies local quality by selectively applying post-filtering operations to specific regions or aspects of the video content that are relevant to machine tasks. Rather than uniformly processing the entire video, the system can apply different filtering strategies to different areas, optimizing for machine analysis where needed while preserving overall video quality for human viewing. This is achieved through content-aware processing guided by the trained neural network.
Data Source
AI summary
Aspects of the disclosure provide methods and apparatuses, for video decoding and encoding. The apparatus includes processing circuitry configured to receive an image/video comprising one or more blocks and metadata for a machine task associated with the image/video. The metadata specifies neural network post-filtering characteristics for machine consumption. The processing circuitry decodes a first post-filtering parameter in the image/video corresponding to the one or more blocks to be reconstructed. The first post-filtering parameter applies to a block in the one or more blocks and has been updated by a post-filtering module in a post-filtering neural network (NN) that is trained based on a training dataset and the metadata. The processing circuitry determines the post-filtering NN in a video decoder corresponding to the one or more blocks based on the first post-filtering parameter, and decodes the block based on the determined post-filtering NN corresponding to the block and the metadata.