Temporal Deformable Convolution for Video Frame Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional video coding methods struggle with compression artifacts and instability due to complex motion in videos, especially with non-translational motions and dynamic scenes, which are not effectively handled by traditional block-based motion vectors and learning-based optical flow methods.
Innovation Solution
A Deep Neural Network (DNN) with Temporal Deformable Convolutions (TDC) is used to process video frames, determining feature maps and offset maps to generate aligned frames, thereby stabilizing and improving visual quality by handling arbitrary and complex motions without explicit motion estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If traditional block-based motion vectors are used, then device complexity is reduced, but motion handling accuracy deteriorates for non-translational motions
Solution Approach 1:
The patent replaces traditional mechanical block-based motion vector methods with a deep learning-based optical flow estimation system. The neural network processes pixel-level information to generate continuous motion fields, substituting discrete block-matching mechanics with continuous data-driven prediction, thereby achieving superior accuracy for complex non-translational motions while maintaining computational feasibility through optimized network architecture
Solution Approach 2:
The patent transitions from block-based discrete motion representation to pixel-level continuous motion fields. By estimating optical flow at sub-pixel precision and representing motion as continuous vector fields rather than discrete block vectors, the system captures nuanced non-translational motions (rotation, scaling, deformation) that block-based methods cannot represent, effectively adding dimensional continuity to the motion estimation process
2Measurement precision
If learning-based optical flow methods are used, then motion estimation accuracy is improved, but reliability deteriorates at object boundaries
Solution Approach 1:
The patent applies preliminary deforming operations to reference frames using estimated motion fields before performing block-matching. By pre-aligning frames according to optical flow estimates, the subsequent block-matching operates on already-partially-corrected data, reducing boundary errors and improving reliability. This two-stage approach (optical flow pre-deformation followed by refined block-matching) combines the strengths of both methods while mitigating their individual weaknesses
Solution Approach 2:
The patent introduces an intermediate deforming step that acts as a mediator between optical flow estimation and final motion compensation. The estimated optical flow field serves as an intermediary transformation that deforms reference frames into a common motion-corrected space, facilitating more accurate block-matching and reducing boundary artifacts. This intermediate processing stage bridges the gap between continuous optical flow and discrete block-based coding
3Manufacturing precision
If image-based denoising methods are applied to videos, then visual quality is improved, but stability deteriorates due to changing video conditions
Solution Approach 1:
The patent merges spatial denoising operations with temporal motion compensation in a unified video processing framework. By combining optical flow-based temporal alignment with spatial filtering operations, the system leverages information across multiple frames while maintaining temporal consistency. The merged approach processes videos as spatiotemporal volumes rather than independent images, ensuring stable and coherent results across varying video conditions
Solution Approach 2:
The patent implements dynamic adaptation of processing parameters based on local motion characteristics. The system adjusts filtering strength, block sizes, and motion compensation parameters according to detected motion complexity and scene dynamics. In high-motion regions, the system reduces filtering intensity to avoid artifacts, while in static regions, it applies stronger denoising, creating a dynamically adaptive processing pipeline that maintains stability across varying video conditions
Data Source
AI summary
A method, apparatus and storage medium for performing video coding are provided. The method includes obtaining a plurality of image frames in a video sequence; determining a feature map for each of the plurality of image frames and determining an offset map based on the feature map; determining an aligned feature map by performing a temporal deformable convolution (TDC) on the feature map and the offset map; and generating a plurality of aligned frames based on the aligned feature map.