Video Frame Interpolation via Optical Flow and Occlusion Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current frame rate up-conversion methods based on motion compensation interpolation suffer from blocking effects, 'shielding', and 'exposure' problems due to the assignment of motion vectors across object edges and inaccuracies in motion estimation.
Innovation Solution
The method employs optical flow to calculate bidirectional motion vectors between adjacent frames, judges their reliability, updates unreliable vectors in occlusion regions, and maps frames using a pixel-based approach to eliminate blocking effects and handle occlusion regions effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If block matching algorithm is used for motion compensation interpolation, then frame rate can be increased, but blocking effect occurs at object edges
Solution Approach 1:
The patent segments the block matching process by dividing blocks into interior blocks and edge blocks. Interior blocks use traditional block matching, while edge blocks use optical flow method to calculate motion vectors independently, preventing the blocking effect at object edges while maintaining frame rate improvement.
Solution Approach 2:
The patent applies different motion estimation methods to different regions: block matching for interior blocks and optical flow for edge blocks. This local differentiation ensures that regions prone to blocking effects receive specialized processing, improving overall image quality while maintaining productivity gains.
2Device complexity
If block matching algorithm assigns same motion vector to entire block, then processing is simplified, but shielding and exposure problems occur
Solution Approach 1:
The patent segments blocks into interior and edge blocks, applying appropriate motion estimation methods to each. This segmentation allows the system to handle complex occlusion scenarios at edges while maintaining simpler processing for interior regions, balancing complexity and reliability.
Solution Approach 2:
The patent introduces an occlusion detection mechanism that acts as an intermediary between motion estimation and frame interpolation. This intermediary identifies shielding and exposure problems and triggers appropriate handling procedures, improving reliability without significantly increasing overall system complexity.
3Loss of energy
If frame skip is used to reduce data, then bandwidth requirements are reduced, but motion discontinuity and image quality degradation occur
Solution Approach 1:
The patent performs preliminary motion estimation and occlusion detection on skipped frames before actual interpolation. By pre-calculating motion vectors and identifying potential problems areas, the system can perform more accurate frame interpolation, recovering image quality while maintaining bandwidth savings from frame skipping.
Solution Approach 2:
The patent replaces simple temporal interpolation with optical flow-based motion estimation for edge blocks. This substitution provides more accurate motion compensation that accounts for complex movements and occlusions, thereby improving image quality after frame skip compression.
Data Source
AI summary
A method and system for video frame interpolation based on an optical flow method is disclosed. The process includes calculating bidirectional motion vectors between two adjacent frames in a frame sequence of input video by using the optical flow method, judging reliabilities of the bidirectional motion vectors between the two adjacent frames, and processing a jagged problem and a noise problem in the optical flow method; marking “shielding” and “exposure” regions in the two adjacent frames, and updating an unreliable motion vector; with regard to the two adjacent frames, according to marking information about the “shielding” and “exposure” regions and the bidirectional motion vector field, mapping front and back frames to an interpolated frame to obtain a forward interpolated frame and a backward interpolated frame; synthesizing the forward interpolated frame and the backward interpolated frame into the interpolated frame; repairing a hole point in the interpolated frame to obtain a final interpolated frame. Since the optical flow method is based on pixels, disclosed method and system are more accurate and do not have blocking effect and other problems.