Multi-Pass View Synthesis for Coherent Depth Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-camera systems face issues with coherent depth map generation across multiple views, leading to artifacts and low compression efficiency due to incoherent depth maps, especially when using cameras with large baselines or capturing natural content.
Innovation Solution
A multi-pass view synthesis technique that reorders and processes views based on relative position and orientation, performs multiple synthesis passes, and masks and merges intermediate results to achieve coherent and complete view synthesis, minimizing artifacts and improving compression efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If local depth estimation techniques are used to estimate depth maps from neighboring views, then depth maps can be generated for each view independently, but the resultant depth maps lack coherency across all views
Solution Approach 1:
The patent segments the view synthesis process into multiple passes, where in the first pass only a subset of views are used for initial synthesis, and in subsequent passes additional views are progressively incorporated. This segmentation allows the system to handle coherence requirements systematically across different view subsets rather than attempting to process all views simultaneously.
Solution Approach 2:
The patent performs preliminary view synthesis using a subset of views before incorporating all available views. By first generating coherent depth maps and synthesized views from a manageable subset, the system establishes a coherent foundation that can then be progressively extended to include more views without compromising overall coherence.
2Loss of information
If all available views are used for view synthesis, then complete information can be obtained, but artifacts appear in occluded regions due to incoherent depth maps
Solution Approach 1:
The patent divides the view synthesis process into multiple passes where views are processed in incremental groups. In each pass, only coherent depth maps from currently processed views are used for synthesis, preventing artifacts from incoherent depth maps from affecting the final output while still progressively incorporating all views for completeness.
Solution Approach 2:
The patent uses masks generated during synthesis to identify and correct artifacts. The mask indicates regions where artifacts are present, and the system uses this feedback to adjust the synthesis process, particularly in occluded regions, to eliminate harmful artifacts while preserving complete information.
3Productivity
If incoherent depth maps are used in multi-view coding schemes, then compression efficiency decreases due to difficulty in finding matched blocks, but processing all views maintains redundancy
Solution Approach 1:
The patent segments the view processing into passes where coherent depth maps are established first, enabling effective motion vector and block matching in subsequent passes. This segmentation allows the encoder to achieve compression efficiency by using coherent depth information without requiring all views to be processed simultaneously.
Solution Approach 2:
The patent changes the parameter of depth map coherence by processing views in multiple passes, transforming incoherent depth maps into coherent depth maps progressively. This parameter change enables the system to achieve both compression efficiency and complete information utilization without excessive bandwidth consumption.
4Measurement precision
If sharp transitions in depth values are generated for edge regions in synthetic content, then depth maps accurately represent scene geometry, but halo artifacts appear when rendering using all available views
Solution Approach 1:
The patent segments the rendering process into multiple passes where depth maps are progressively refined. In each pass, only a subset of views is used, allowing sharp depth transitions to be maintained for geometric accuracy while progressively blending results to eliminate halo artifacts that would appear if all views were used simultaneously.
Data Source
AI summary
An embodiment of an image processor for immersive video includes technology to re-order patches from a plurality of views based on one or more of relative position and orientation related information for a desired synthesized view, select a set of views to be used in each view synthesis pass, perform two or more view synthesis passes for the synthesized view to provide two or more intermediate view synthesis results, and mask and merge the two or more intermediate view synthesis results to provide a final view synthesis result. Other embodiments are disclosed and claimed.