Video Frame Interpolation Using Semantic and Depth Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video frame interpolation methods based on optical flow and motion compensation have low computational efficiency and fail to effectively address blocking issues and significant content changes, resulting in poor interpolation effects.
Innovation Solution
The method involves obtaining visual semantic feature maps and depth maps of adjacent video frames, using these to generate frame interpolation information, and inserting interpolated frames between original frames, leveraging deep convolutional networks to improve calculation efficiency and handle large motion and scene changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If optical flow and motion compensation methods are used for video frame interpolation, then the interpolation process can be performed, but the computational efficiency is low and blocking issues occur
Solution Approach 1:
The patent replaces traditional optical flow and motion compensation methods with a deep learning-based convolutional neural network approach. The CNN model directly processes input frames to generate interpolated frames, substituting the complex iterative optimization processes of optical flow with a trained neural network that performs inference, thereby improving computational efficiency while maintaining or enhancing interpolation quality
Solution Approach 2:
The patent transforms the interpolation problem from a parameter-optimization approach (optical flow vectors, motion compensation parameters) to a feature-based approach using deep learning. The CNN extracts and processes visual semantic features and depth information, changing the fundamental parameters used for interpolation from motion vectors to learned feature representations, which resolves both efficiency and quality issues
2Reliability
If traditional frame interpolation methods are used, then the process is simpler, but they fail to address blocking issues and significant content changes
Solution Approach 1:
The patent introduces depth maps as an additional dimensional information source beyond the traditional 2D image data. By incorporating depth information from stereo vision or depth sensors, the system adds a third dimension (depth/Z-axis) to the interpolation process, enabling better handling of occlusions, occluded regions, and significant content changes that 2D methods cannot resolve, while the modular architecture manages the increased complexity
3Productivity
If deep convolutional networks are used to improve computational efficiency, then calculation speed increases, but the system complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-training the convolutional neural network model offline using large datasets. The complex learning and optimization processes are performed in advance during model training, so that during actual video interpolation, the pre-trained model can efficiently process frames without requiring complex real-time optimization, thus achieving high calculation efficiency while managing system complexity through separation of training and inference phases
Data Source
AI summary
A method for video frame interpolation, a related electronic device and a storage medium is disclosed. A video is obtained. An (i−1)th frame and an ith frame of the video are obtained. Visual semantic feature maps and depth maps of the (i−1)th frame and the ith frame are obtained. Frame interpolation information is obtained based on the visual semantic feature maps and the depth maps. An interpolated frame between the (i−1)th frame and the ith frame is generated based on the frame interpolation information and the (i−1)th frame and is inserted between the (i−1)th frame and the ith frame.