Adaptive Convolutional Kernel for Video Frame Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video frame interpolation methods face challenges in ensuring accuracy of optical flows due to large motions and occlusions, making it difficult to generate high-quality interpolation frames.
Innovation Solution
A video frame interpolation apparatus and method that calculates bidirectional optical flow, performs kernel and weight estimation using convolutional layers, and generates adaptive local convolutional kernels and weight coefficients to synthesize high-quality interpolation frames by converting and synthesizing frames with an adaptive conversion layer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If bidirectional optical flow is calculated for frame interpolation, then motion information is captured, but accuracy deteriorates due to large motions and occlusions
Solution Approach 1:
The patent divides the frame interpolation task into multiple stages: bidirectional optical flow calculation, kernel estimation, weight estimation, and frame synthesis. By segmenting the process, each component can be optimized independently to handle large motions and occlusions better
Solution Approach 2:
The patent employs adaptive local convolutional kernels that are estimated for each pixel position based on local image characteristics. This allows the interpolation to adapt to local variations in motion and occlusion patterns, improving accuracy in different regions of the frame
2Measurement precision
If adaptive local convolutional kernels are generated using convolutional layers, then information from neighboring pixels is fully utilized, but computational complexity increases
Solution Approach 1:
The patent performs kernel estimation and weight estimation as preliminary steps before actual frame synthesis. The convolutional layers pre-compute adaptive kernels and weights based on input frames, so that during synthesis, these pre-computed values are directly applied, reducing real-time computational burden
Solution Approach 2:
The patent uses convolutional operations to efficiently compute adaptive kernels by copying and adapting filter patterns across different spatial locations. This leverages the inherent parallelism of convolution to reduce computational complexity compared to direct pixel-wise calculations
Data Source
AI summary
Embodiments of this disclosure provide a video frame interpolation apparatus and method. The method includes calculating a bidirectional optical flow between a first frame and a second frame; an performing kernel and weight estimation according to the first frame and the second frame. An adaptive local convolutional kernel is generated by using a convolutional layer and a weight coefficient is generated by using another convolutional layer. A conversion on the first frame and the second frame is performed by using an adaptive conversion layer according to the bidirectional optical flow, the weight coefficient and the adaptive local convolutional kernel, so as to generate a conversion frame. A frame synthesis on the first frame, the second frame and the conversion frame is performed to generate an interpolation frame between the first frame and the second frame.