CNN Video Frame Interpolation Resolves Motion Artifacts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional video frame interpolation methods suffer from artifacts such as halos, ghosts, and break-ups due to insufficient quality in motion estimation, motion interpolation, and occlusion reasoning, which are not effectively addressed by existing technologies.
Innovation Solution
A CNN-based framework that combines pre-trained convolutional neural networks with simple hand-crafted modules for video frame interpolation, including a coarse CNN for initial feature extraction and refinement, and a motion-compensated frame interpolation module for synthesizing middle non-existing frames, reducing artifacts and improving interpolation quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional hand-engineered modules are used for VFI, then the system is easier to implement and understand, but the quality of motion estimation, occlusion reasoning, and frame interpolation deteriorates, leading to artifacts such as halos, ghosts, and break-ups
Solution Approach 1:
The VFI system is segmented into distinct functional modules: motion estimation module, occlusion reasoning module, and frame interpolation module. Each module can be independently optimized - some using conventional hand-engineered methods and others using CNNs - allowing flexible implementation while maintaining high quality output
Solution Approach 2:
The system allows dynamic selection of processing methods by changing parameters - whether to use conventional algorithms or CNN-based approaches for specific modules. This enables adaptation between speed/complexity and quality requirements based on application needs
2Manufacturing precision
If CNN-based components are used to replace hand-engineered modules, then the quality of motion estimation and occlusion reasoning improves, but the device complexity and computational requirements increase
Solution Approach 1:
The system divides the VFI process into separate modules, allowing CNNs to be applied only to specific functions where they provide the most benefit (motion estimation, occlusion reasoning) while keeping other parts simpler. This reduces overall complexity compared to a fully CNN-based approach
Solution Approach 2:
Instead of applying CNNs to all VFI modules, the system uses them partially - only for motion estimation and occlusion reasoning where they provide significant quality improvement. This balances computational complexity with quality enhancement
3Manufacturing precision
If fully CNN-based VFI is implemented, then interpolation quality improves and artifacts are reduced, but the processing time and computational energy consumption increase
Solution Approach 1:
The processing pipeline is segmented into stages with varying computational requirements. Conventional modules handle tasks that can be processed quickly, while CNNs are applied selectively to complex tasks like motion estimation, optimizing the balance between quality and processing time
Solution Approach 2:
The system performs preliminary motion estimation and occlusion detection using efficient methods before proceeding to the computationally intensive frame interpolation stage. This preliminary processing reduces the complexity of subsequent steps and overall processing time
Data Source
AI summary
A system of convolutional neural networks (CNNs) that synthesize middle non-existing frames from pairs of input frames includes a coarse CNN that receives a pair of images acquired at consecutive points of time, a registration module, a refinement CNN, an adder, and a motion-compensated frame interpolation (MC-FI) module. The coarse CNN outputs from the pair of images a previous feature map, a next feature map, a coarse interpolated motion vector field (IMVF) and an occlusion map, the registration module uses the coarse IMVF to warp the previous and next feature maps to be aligned with pixel locations of the IMVF frame, and outputs registered previous and next feature maps, the refinement CNN uses the registered previous and next feature maps to correct the coarse IMVF, and the adder sums the coarse IMVF with the correction and outputs a final IMVF.