Video Coding With Decoder-Side Neural Weight Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video encoding technologies struggle to maintain quality consistency between human and machine perception, as machine learning algorithms often require different compression standards than human-approved quality metrics.
Innovation Solution
Implement an encoder's decoder-side neural network (en-DSNN) for decoding and post-processing, with iterative compressing and decompressing using a weight-update codec, and adapting neural networks based on input and output signals to optimize compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional video compression is optimized for human perceptual quality, then human viewing quality is improved, but machine analysis accuracy deteriorates
Solution Approach 1:
The patent applies different quality standards to different parts of the video data by introducing a dual-branch neural network architecture. One branch processes video for human perceptual quality with emphasis on visual fidelity, while the other branch processes the same video for machine analysis quality with emphasis on semantic information preservation. This allows each branch to optimize its own quality metrics independently.
Solution Approach 2:
The video coding system is segmented into two separate processing streams: a human-oriented decoding stream and a machine-oriented decoding stream. Each stream has its own neural network model trained for specific tasks (e.g., video quality assessment for humans, object detection or event detection for machines), allowing simultaneous optimization for both human and machine consumption from the same compressed bitstream.
2Productivity
If compression bitrate is reduced, then transmission efficiency is improved, but reconstruction quality deteriorates
Solution Approach 1:
The patent employs feedback mechanisms where the encoder receives information about the decoded output quality and adjusts encoding parameters accordingly. Neural networks analyze the reconstructed video and provide feedback signals that guide the encoder to optimize compression settings, allowing the system to maintain reconstruction quality while improving transmission efficiency through adaptive parameter adjustment.
Solution Approach 2:
The system dynamically changes encoding parameters such as quantization levels, transform block sizes, and neural network processing intensity based on content characteristics and desired quality targets. By adaptively adjusting these parameters, the system can achieve better transmission efficiency without significant loss in reconstruction quality for specific machine tasks.
3Measurement precision
If neural network complexity is increased, then machine task accuracy is improved, but computational cost deteriorates
Solution Approach 1:
The patent applies partial action by implementing lightweight neural network variants that perform only the specific machine tasks required (e.g., object detection, motion analysis) rather than full general-purpose video processing. This allows the system to achieve sufficient machine task accuracy with reduced computational complexity and lower energy consumption, avoiding excessive processing for tasks that don't require full neural network capability.
Data Source
AI summary
The embodiments relate to a method for encoding, comprising receiving a representation (1005) of input media to be encoded: encoding the representation (1005) to generate encoded bitstreams to be delivered to a decoder, and using encoder's de-coder-side neural network (en-DSNN) for decoding the encoded representation and/or post-processing the decoded representation: adapting (1030) the en-DSNN based at least on the representation or a signal derived from the representation and on an output of the en-DSNN or a signal derived from an output of the en-DSNN, thus obtaining a weight-update (1040) as a result: performing one or more iterations of compressing and decompressing the weight-update (1040) by using a weight-update codec: selecting values of one or more configuration parameters of the weight-update codec and using the selected values for a final compression of the weight-update (1040); and signaling the compressed weight update to a decoder.