Video Frame Interpolation Using Motion Vectors and Occlusion Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating video frames requires significant memory, time, and computing resources, which can be improved.
Innovation Solution
The use of Bi-directional motion vector (BiMV) and Uni-directional motion vector (UniMV) interpolation techniques, which generate interpolated frames based on frame data including RGB color images, depth buffers, and geometric motion vectors, using bilinear sampling operations and occlusion/dis-occlusion masks to optimize pixel interpolation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional video frame generation methods are used, then video frames can be generated, but significant memory, time, and computing resources are consumed
Solution Approach 1:
The patent pre-calculates and stores motion vectors between consecutive video frames before interpolation is needed. By having motion information ready in advance, the actual frame interpolation process can proceed more efficiently without performing complex motion analysis during real-time generation, thus reducing computing resources while maintaining productivity
Solution Approach 2:
The patent replaces traditional complex optical flow computation with a simplified motion vector-based approach. Instead of using resource-intensive mechanical-like iterative optimization algorithms, the system uses pre-computed discrete motion vectors combined with bilinear sampling, substituting the heavy computational mechanism with a lighter mathematical operation that achieves similar interpolation results with fewer resources
2Reliability
If frame interpolation is performed to generate intermediate frames, then video smoothness is improved, but computational complexity increases
Solution Approach 1:
The patent segments the interpolation process into distinct stages: motion vector calculation, occlusion mask generation, and bilinear sampling. By dividing the complex interpolation task into these manageable segments, each handling a specific aspect of the problem, the overall computational complexity is reduced while maintaining interpolation accuracy through specialized processing at each stage
Solution Approach 2:
The patent introduces motion vectors as intermediary data structures that bridge the input frames and the interpolated output. These motion vectors serve as a compact representation of pixel displacement that simplifies the interpolation calculation, acting as a mediator that reduces the direct computational burden between source and target frames while preserving accuracy
3Use of energy by moving object
If simple interpolation methods are used, then computational resources are reduced, but interpolation accuracy deteriorates due to scene changes and occlusions
Solution Approach 1:
The patent applies local quality by generating occlusion masks that identify specific regions where simple interpolation would fail due to occlusions or scene changes. In these localized problematic areas, the system uses motion-compensated sampling with occlusion handling, while in normal areas it uses simpler methods. This selective approach maintains precision where needed while reducing computing resources overall
Solution Approach 2:
The patent dynamically adjusts the interpolation strategy based on local image content characteristics. By analyzing motion vectors and occlusion patterns, the system adapts its interpolation method for each region, switching between simple and complex approaches as needed. This dynamic adaptation maintains precision in challenging areas while conserving computing resources in straightforward regions
Data Source
AI summary
Apparatuses, systems, and techniques to generate interpolated video frames. In at least one embodiment, an interpolated video frame is generated based, at least in part, on a first set of pixel data sampled from a first video frame, and a second set of pixel data sampled from a second video frame based, at least in part, on a set of forward pointing motion vectors from the first video frame to the second video frame.