Depth-Aware Mesh Warping for Video Frame Interpolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video frame interpolation technologies cause local image distortion and degradation in user visual experience due to discontinuous depth values in predicted frames, leading to noticeable differences between predicted and real frames.
Innovation Solution
An image processing method that calculates a motion vector and creates an initial mesh, subdivides it based on depth values to identify points, and performs image warping on the second image using the target mesh to generate a predicted frame, ensuring continuous depth information and avoiding local stretching.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If image warping is performed on the second real frame to generate a predicted frame, then the frame rate and smoothness of the video are improved, but local image distortion occurs in regions with discontinuous depth values, causing noticeable differences between predicted and real frames
Solution Approach 1:
The patent divides the image into multiple regions based on depth value continuity. Regions with continuous depth values are processed separately from regions with discontinuous depth values. This segmentation allows the image warping operation to be applied selectively, avoiding distortion in regions where depth values are discontinuous while maintaining the frame rate improvement benefit.
Solution Approach 2:
The patent applies different processing strategies to different regions of the image based on their depth value characteristics. Regions with continuous depth values undergo standard image warping, while regions with discontinuous depth values receive special handling to prevent local distortion. This local differentiation preserves image quality where needed while maintaining overall productivity.
2Device complexity
If image warping is performed without mesh subdivision, then the processing complexity is reduced, but local stretching and distortion occur in regions with discontinuous depth values
Solution Approach 1:
The patent segments the mesh into multiple sub-meshes based on depth value continuity. By dividing the original mesh into smaller, depth-consistent segments, the patent achieves better image continuity without requiring excessive processing complexity. Each sub-mesh can be processed independently with appropriate warping operations.
Solution Approach 2:
The patent introduces an additional dimension of organization by creating a hierarchical mesh structure with sub-meshes. This dimensional organization allows the system to handle depth discontinuities by processing different depth regions separately, improving image continuity while keeping the overall processing complexity manageable through structured organization.
Data Source
AI summary
Embodiments of this application provide an image processing method and an electronic device. In this method, when a predicted frame is calculated based on a first real frame and a second real frame, a motion vector corresponding to the first real frame and the second real frame is first calculated, and an initial mesh Mesh corresponding to the second real frame is created. Then, subdivision points are identified based on depth values of pixels, the initial Mesh is subdivided based on the subdivision points to obtain a target Mesh, and then image warping is performed on the second real frame based on the target Mesh and the motion vector, to generate the predicted frame of image.