Panoramic Video Frame Interpolation With Corrected Optical Flow
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing panoramic video frame interpolation methods consume excessive system memory and have low estimation accuracy for frame-interpolated video frames.
Innovation Solution
A method and apparatus that perform downsampling, optical flow estimation, inverse transformation, optical flow correction, and upsampling operations on panoramic video frames using neural networks to reduce memory consumption and improve estimation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If classical algorithms based on neural networks (Flownet-S, Flownet-C, Lite-flownet) are used for frame interpolation, then the fluency of slow-motion panoramic video is improved, but system memory consumption increases and estimation accuracy decreases
Solution Approach 1:
The patent segments the frame interpolation process into distinct modules: optical flow estimation, inverse transformation, and frame synthesis. Each module processes specific aspects of the interpolation independently, allowing for optimized memory usage in each segment rather than loading entire classical neural network models into memory.
Solution Approach 2:
The patent introduces optical flow graphs as intermediary representations that capture motion information between frames. These optical flow graphs serve as compact mediators that enable frame interpolation without requiring large neural network models to be loaded into system memory, thus reducing memory consumption while maintaining fluency.
2Speed
If classical algorithms based on neural networks are used for frame interpolation, then the fluency of slow-motion panoramic video is improved, but estimation accuracy of frame-interpolated video frames decreases
Solution Approach 1:
The patent implements feedback mechanisms where optical flow estimation results are refined through inverse transformation and verification against the original frames. This iterative feedback process improves the accuracy of motion estimation and consequently enhances the precision of frame interpolation without requiring larger neural networks.
Solution Approach 2:
The patent replaces the mechanical system of large-scale neural network computations with an optical flow-based computational approach. By using optical flow graphs and inverse transformation mathematics instead of heavy neural network inference, the system achieves both fluency and accuracy without the memory overhead of classical algorithms.
3Quantity of substance
If downsampling and upsampling operations are performed on panoramic video frames, then system memory consumption is reduced, but processing complexity increases
Solution Approach 1:
The patent performs downsampling as a preliminary action before optical flow estimation and processing. By reducing the resolution of input frames beforehand, the system decreases memory requirements for subsequent processing steps. The preliminary downsampling ensures that all subsequent operations work with compact data representations, offsetting the added complexity through systematic preprocessing.
Data Source
AI summary
The present invention provides a panoramic video frame interpolation method, including outputting a pre-post frame image optical flow graph and a post-pre frame image optical flow graph; calculating an image optical flow graph before frame interpolation and an image optical flow graph after frame interpolation; obtaining a downsampling pre-frame image after transformation and a downsampling post-frame image after transformation; outputting a pre-post frame image correction optical flow graph, a post-pre frame image correction optical flow graph and an image occlusion relationship graph; obtaining a downsampling pre-frame image after correction and a downsampling post-frame image after correction; obtaining an upsampling image optical flow graph before frame interpolation and an upsampling image optical flow graph after frame interpolation; and calculating a frame interpolation image corresponding to a frame interpolation position.