Neural Video Coding With Sliding-Window Rate-Distortion Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
End-to-end neural video coders (E2E-NVCs) face challenges in handling longer sequences of P-frames, leading to increased distortion and inferior rate-distortion tradeoffs compared to conventional video coders like HEVC and VVC, which affects user experience.
Innovation Solution
Implementing sliding-window rate-distortion optimization techniques using an image analysis neural network to modify frames within a sliding window, optimizing distortion across a sequence of P-frames to match conventional coders without increasing distortion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If E2E-NVC trains on a smaller number of frames to reduce bit rate, then compression efficiency is improved, but distortion increases as the decoder proceeds through the sequence of P-frames
Solution Approach 1:
The patent applies dynamics by making the training window size variable rather than fixed. The encoder dynamically adjusts the number of frames included in the training window based on temporal complexity metrics. When temporal complexity is high, the window expands to include more frames for better prediction; when complexity is low, the window contracts to maintain efficiency. This dynamic adaptation resolves the contradiction by allowing the system to optimize between bit rate and distortion on a frame-by-frame basis rather than using a static small window throughout the entire sequence.
Solution Approach 2:
The patent changes the parameter of training window size from a fixed small value to a variable parameter that adapts to local temporal complexity. By modifying this parameter dynamically based on measured temporal characteristics of the video sequence, the system can increase distortion tolerance in simple regions (maintaining low bit rate) while allocating more training frames in complex regions (reducing distortion), thus resolving the trade-off between compression efficiency and quality.
2Productivity
If E2E-NVC uses a limited number of training frames, then processing speed is improved, but the rate-distortion tradeoff becomes inferior to conventional coders
Solution Approach 1:
The system dynamically adjusts the training window size based on temporal complexity, allowing faster processing in simple regions and more thorough optimization in complex regions. This dynamic approach maintains high overall processing speed while improving the rate-distortion tradeoff where it matters most, unlike conventional coders that use a fixed small window throughout.
Solution Approach 2:
The patent applies local quality by allowing different effective training window sizes for different regions of the video sequence based on local temporal complexity. Complex regions receive more training frames for better optimization, while simple regions use fewer frames for speed. This localized adaptation resolves the contradiction by ensuring high rate-distortion performance only where needed, maintaining overall productivity.
3Manufacturing precision
If I-frames are inserted more frequently to increase decoded video quality, then video quality is improved, but bit rate increases due to inferior rate-distortion tradeoff
Solution Approach 1:
Rather than frequently inserting I-frames (which increases bit rate), the system dynamically adjusts the training window size to improve prediction accuracy within P-frame sequences. This dynamic adaptation maintains video quality by optimizing the training process locally, avoiding the need for frequent quality-resetting I-frames and their associated bit rate penalty.
Solution Approach 2:
The dynamic training window acts as an intermediary mechanism between I-frames and P-frames. Instead of relying solely on frequent I-frames to maintain quality, the adaptive window size provides continuous optimization within P-frame sequences, serving as a mediator that improves quality without requiring the high bit rate cost of frequent I-frame insertion.
Data Source
AI summary
A device for encoding video data can be configured to encode a set of input frames of the video data with an image analysis neural network to generate a corresponding set of output frames, wherein the set of output frames includes a first output frame and additional output frames, wherein the first output frame temporally precedes the additional output frames; determine a modification to the first output frame that results in an optimized amount of distortion between the set of input frames and decoded versions of the corresponding set of output frames; update the first output frame based on the determined modification; and output a bitstream that includes the updated first output frame.