Deep Neural Network Inter-Frame Prediction for Video Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional video coding methods face challenges in efficiently compressing video data, especially with complex motion in dynamic scenes, as they rely on block-based motion vectors that fail to handle non-translational motions effectively, leading to errors and inefficiencies in bandwidth and storage requirements.
Innovation Solution
A Deep Neural Network (DNN)-based model is employed for video coding that generates virtual reference frames from adjacent frames for inter-frame prediction, using optical flow estimation and detail enhancement modules to handle complex motion implicitly and improve frame quality, reducing redundancy and artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional block-based motion vectors are used for inter-frame prediction, then the video coding process is simple and computationally efficient, but complex non-translational motions cannot be handled effectively leading to prediction errors and reduced compression quality
Solution Approach 1:
The patent replaces the traditional mechanical block-based motion estimation system with a deep neural network-based optical flow estimation system. The DNN model learns complex motion patterns from training data and generates pixel-level flow maps that accurately represent non-translational motions such as rotation, scaling, and deformation, thereby improving prediction accuracy without relying on rigid block-based methods
Solution Approach 2:
The patent transforms the motion representation from discrete block vectors to continuous pixel-level optical flow fields. By changing the parameter granularity from block-level to pixel-level and using learned features instead of hand-crafted motion models, the system achieves superior handling of complex motions while maintaining computational feasibility through efficient DNN inference
2Manufacturing precision
If uncompressed video is stored and transmitted, then video quality is preserved without loss, but bandwidth and storage requirements become prohibitively high
Solution Approach 1:
The patent introduces an intermediate optical flow representation as a mediator between the reference frames and the prediction process. The learned optical flow fields serve as a compact intermediary that captures essential motion information, enabling efficient compression while maintaining high reconstruction quality through accurate motion compensation
Solution Approach 2:
The patent changes the compression approach from compressing raw pixel data to compressing learned feature representations and optical flow fields. By transforming the video data into a more compact latent space using DNNs and encoding only the essential motion and residual information, the system achieves high compression ratios while preserving video quality
3Productivity
If block-based hybrid prediction framework with handcrafted tools is used, then the coding structure is simple and standardized, but efficiency is limited in handling dynamic scenes with complex motion
Solution Approach 1:
The patent replaces handcrafted block-based prediction tools with learned deep neural network models that automatically adapt to complex motion patterns. The DNN-based optical flow estimation and detail enhancement modules learn optimal prediction strategies from training data, achieving superior compression efficiency for dynamic scenes with non-translational motions
Solution Approach 2:
The patent performs preliminary action by pre-training the DNN model on large video datasets to learn general motion patterns and scene dynamics before actual video encoding. This pre-learning phase enables the model to handle complex motions efficiently during inference, improving compression efficiency without requiring complex runtime processing
Data Source
AI summary
Video coding using neural network based inter-frame prediction is performed by generating a current reference frame by generating intermediate flows based on two input frames, performing backward warping of the input frames to generate reconstruction frames, and generating a fusion map and a residual map based on the input frames, the intermediate flows and the reconstruction frames. The video coding method further includes outputting an enhanced frame or a virtual reference picture by generating a feature map with different levels, based on the current reference frame, a first reference frame and a second reference frame, generating a predicted frame based on aligned features from the generated feature map by refining the current reference frame, the first reference frame, and the second reference frame, generating a final residual based on the predicted frame, and computing the enhanced frame as an output by adding the final residual to the current reference frame.