3D Frame Interpolation for Multiview Video
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current frame rate up-conversion methods for videos, especially in multiview applications, face challenges such as motion judder and blur, particularly in moving areas, and are computationally intensive, which degrades visual quality and requires significant resources.
Innovation Solution
The method involves determining 3D position data using a calibration matrix and depth maps to calculate 3D rotation and translation matrices, allowing for interpolation in 3D space, which simplifies the process of generating interpolated frames and reduces computational complexity by performing interpolation once in 3D, enabling accurate 2D projections for multiview displays.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional frame rate up-conversion methods are used, then the frame rate is increased, but motion judder and blur occur in moving areas
Solution Approach 1:
The patent transitions from 2D frame interpolation to 3D space interpolation by utilizing depth information and 3D coordinates. Instead of interpolating frames directly in 2D image space, the method reconstructs objects in 3D space using depth maps and calibration matrices, then projects them back to 2D. This dimensional change enables accurate handling of moving objects by preserving their spatial structure, thereby eliminating motion judder and blur while maintaining high frame rates.
Solution Approach 2:
The patent changes the parameter space from 2D pixel coordinates to 3D spatial coordinates by introducing depth information. By representing objects in 3D space with coordinates (X, Y, Z) instead of 2D coordinates (x, y), the system can accurately track and interpolate moving objects across frames. This parameter transformation allows for precise motion compensation and eliminates the visual artifacts associated with conventional 2D interpolation methods.
2Speed
If multiview video frame rate up-conversion is performed, then all views are up-converted, but computational resources required become significant
Solution Approach 1:
The patent merges the interpolation process across multiple views by performing a single 3D interpolation in unified space. Instead of independently interpolating each view (which would require separate computational resources), the method reconstructs all views from a common 3D representation. This consolidation reduces redundant calculations and significantly lowers the computational burden while maintaining high frame rates for all multiview content.
Solution Approach 2:
By moving the interpolation operation to 3D space, the patent creates a view-independent representation of the scene. Once objects are interpolated in 3D, they can be projected to any number of views without additional interpolation computations. This dimensional transformation decouples the frame rate up-conversion from the number of views, making multiview processing computationally efficient.
Data Source
Figure 1
Figure 2A~2C
Figure 3
AI summary
The invention provides a method and apparatus for up-converting the frame rate of a digital video signal, the method comprising: receiving a digital video signal containing a first frame and a second frame; finding in one of the received frames, matches for objects in the other of the received frames; utilising 3 dimensional position data in respect of the objects within the frames to determine 3 dimensional movement matrices for the matched objects; using the 3 dimensional movement matrices, determining the position of the objects in a temporally intermediate frame and thereby generating an interpolated frame, temporally between the first and second frame.