Video Frame Interpolation Using Iterative Optical Flow and Depth Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video frame interpolation methods have low accuracy due to the low accuracy of individual sub-modules, resulting in a low-quality interpolation frame.
Innovation Solution
A video frame interpolation method that involves obtaining initial optical flow maps through iterative residual optical flow estimation, up-sampling these maps using a pixel adaptive convolution joint process, and using depth estimation models to generate depth maps and context feature maps, which are then synthesized into an output frame using a pixel adaptive convolution method.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing video frame interpolation methods use multiple parallel sub-modules, then the processing capability is improved, but the accuracy of each module is low resulting in low quality interpolation frames
Solution Approach 1:
The patent divides the frame interpolation task into distinct processing stages: optical flow estimation, depth map generation, and frame synthesis. Each stage is handled by specialized modules that can be independently optimized, allowing high processing capability while maintaining accuracy through targeted improvements in each segment rather than relying on generic parallel sub-modules.
Solution Approach 2:
The patent introduces depth maps as an intermediary element between optical flow estimation and frame synthesis. These depth maps provide additional geometric information that mediates the transformation process, enabling more accurate interpolation by incorporating 3D spatial relationships rather than relying solely on 2D optical flow constraints.
2Speed
If traditional optical flow estimation is used, then the processing speed is maintained, but the accuracy of motion estimation is insufficient leading to poor interpolation quality
Solution Approach 1:
The patent performs preliminary depth estimation and generates depth maps before the final frame synthesis stage. This preliminary action provides accurate 3D geometric information that constrains and guides the subsequent optical flow-based interpolation, improving motion estimation accuracy without requiring complex real-time calculations during the main processing stage.
Solution Approach 2:
The patent transforms the problem from 2D optical flow estimation to 3D spatial transformation by incorporating depth maps. This parameter change from 2D to 3D space allows for more accurate motion representation by accounting for depth variations, enabling high precision motion estimation while maintaining processing efficiency through optimized 3D warping operations.
3Device complexity
If simple up-sampling is applied to optical flow maps, then the computational complexity is reduced, but the resolution and accuracy of optical flow information is insufficient
Solution Approach 1:
The patent replaces traditional mechanical up-sampling methods with a learning-based approach using a depth estimation neural network. This substitution achieves high-resolution optical flow information by learning complex spatial relationships from data, providing superior resolution and accuracy compared to simple interpolation methods while maintaining reasonable computational complexity through efficient network architecture.
Data Source
AI summary
The present disclosure relates to the field of information display, and specifically to a method, device, computer readable storage medium, and electronic device for video frame interpolation. The method comprises: obtaining, based on two input frames, two initial optical flow maps corresponding to the two input frames; optimizing the initial optical flow maps to obtain target optical flow maps; obtaining an interpolation frame kernel, two depth maps and two context feature maps based on the two input frames; obtaining an output frame using a frame synthesis method based on the target optical flow maps, the depth maps, the context feature maps, and the interpolation frame kernel.