View Synthesis Using Depth Map Discontinuity Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing view synthesis methods in 3D video are inadequate in handling occlusion and disocclusion, particularly sensitive to noise and dependent on video content, and fail to accurately match occluded pixels in synthesized views.
Innovation Solution
The method calculates occluded or disoccluded pixels based on the 3D geometry of depth changes and camera positions, eliminating the need for pixel search and reducing noise sensitivity by using a closed-form approach that maps depth discontinuities to virtual views.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If photometry-based estimation of occlusions is used, then occlusion detection can be performed, but significant errors are induced because reference frame pixels cannot be accurately matched in the target frame
Solution Approach 1:
The patent replaces photometry-based intensity matching with a geometry-based approach using depth maps and camera models. Instead of matching pixel intensities (which fails for occluded pixels), the system uses geometric relationships between cameras and depth information to determine occlusion, substituting the mechanical intensity-matching process with a geometric calculation process that is not sensitive to noise or content variations.
Solution Approach 2:
The patent introduces a depth map as an intermediary element between the reference frame and target frame. The depth map provides geometric information about 3D objects, which serves as a mediator to determine occlusion relationships without directly comparing pixel intensities. This intermediary allows the system to handle occluded pixels accurately by using depth-based geometric reasoning rather than direct pixel matching.
2Difficulty of detecting and measuring
If smart search mechanism is used to detect occluded and disoccluded pixels, then occlusion can be detected, but the method is sensitive to noise and depends on video content
Solution Approach 1:
The patent replaces the smart search mechanism (which iteratively searches for occluded pixels based on intensity differences) with a closed-form geometric calculation. By using depth maps and camera projection models, the system directly computes occlusion relationships through geometric equations, eliminating the need for iterative search and removing sensitivity to noise and content variations that plague search-based methods.
Solution Approach 2:
The patent changes the fundamental parameters used for occlusion detection from pixel intensity values (which are noisy and content-dependent) to depth values and camera geometric parameters. This parameter transformation fundamentally alters the detection process, making it robust to noise and content variations while maintaining accuracy through geometric consistency.
3Device complexity
If linear interpolation is used for view synthesis, then simple calculation is achieved, but it cannot handle situations involving 3D objects that occlude surrounding areas
Solution Approach 1:
The patent segments the view synthesis process into two distinct stages: (1) depth-based occlusion detection using depth maps and camera geometry, and (2) pixel value synthesis using linear interpolation. By separating the occlusion detection function from the pixel synthesis function, the system can accurately identify occluded regions using geometric reasoning while maintaining the simplicity of linear interpolation for pixel value calculation. This segmentation allows the simple interpolation method to work correctly within the context of geometrically-determined occlusion masks.
Data Source
Figure 1a~1c
Figure 2~3
AI summary
Control of view synthesis of a 3D scene is described. A method comprises detecting discontinuities in a depth map that comprises depth values corresponding to a view point of a reference camera (C1). The detection comprises calculation of shifts for neighbouring pixels of the depth map, the shifts being associated with a change of viewpoint from the reference camera to a virtual camera (Cv). The detected discontinuities are then analyzed, which comprises identifying increase of depths associated with the change of viewpoint from the reference camera to the virtual camera. Areas of disocclusion (108) associated with the viewpoint of the virtual camera are then identified, the areas being delimited by positions of the identified increase of depths associated with the change of viewpoint from the reference camera to the virtual camera. The identified areas of disocclusion are then provided to a view synthesis process.