Auxiliary Data for View Synthesis Artifact Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for rendering 3D images from 2D videos and depth maps face challenges in texture and depth consistency, leading to artifacts such as ghosting and disocclusion, especially when view points change, due to inaccurate depth maps and compression errors.
Innovation Solution
The use of auxiliary data including shape images and partial occlusion data to facilitate view synthesis in 2D, stereo, and autostereoscopic displays, with an artifacts-aware method that refines depth-texture consistency and reduces artifacts through morphing and fly-over effects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional 2D plus depth format is used for view synthesis, then bandwidth and storage requirements are reduced, but visual artifacts such as ghosting and disocclusion increase due to inaccurate depth maps and compression errors
Solution Approach 1:
The patent segments the scene into multiple layers including foreground objects, background, and depth discontinuity regions. By dividing the scene representation into distinct layers with different processing strategies, the system can apply targeted refinement techniques to critical regions while maintaining compression efficiency, thereby reducing artifacts without proportionally increasing overall data volume
Solution Approach 2:
The patent applies local quality enhancement by identifying and separately processing regions with depth discontinuities and occlusions. These critical regions receive additional refinement treatment including boundary detection and localized depth correction, while other regions use standard compression, achieving artifact reduction where needed without uniformly increasing data requirements
2Reliability
If accurate depth maps are used to reduce artifacts, then view synthesis quality improves, but data volume and processing complexity increase
Solution Approach 1:
The patent performs preliminary refinement of depth maps during the encoding stage, pre-correcting depth discontinuities and occlusion boundaries before view synthesis. By addressing potential artifact sources in advance during encoding, the system reduces the computational burden during decoding and view generation while maintaining high visual quality
Solution Approach 2:
The patent extracts and separately processes critical depth information related to object boundaries and depth discontinuities. By isolating and refining only the essential depth data needed for artifact reduction rather than processing entire high-precision depth maps, the system achieves quality improvement with reduced processing complexity
3Reliability
If sophisticated inpainting algorithms are applied to fill occluded areas, then disocclusion artifacts are reduced, but processing time and computational resources increase
Solution Approach 1:
The patent applies inpainting selectively only to disoccluded regions identified through depth map analysis, rather than processing entire images. By limiting the inpainting operation to only the necessary occluded areas, the system achieves effective artifact reduction with minimal processing time and computational resource consumption
Data Source
AI summary
Original or compressed Auxiliary Data, including possibly major depth discontinuities in the form of shape images, partial occlusion data, associated tuned and control parameters, and depth information of the original video(s), are used to facilitate the interactive display and generation of new views (view synthesis) of conventional 2D, stereo, and multi-view videos in conventional 2D, 3D (stereo) and multi-view or autostereoscopic displays with reduced artifacts. The partial or full occlusion data includes image, depth and opacity data of possibly partially occluded areas to facilitate the reduction of artifacts in the synthesized view. An efficient method is used for extracting objects at partially occluded regions as defined by the auxiliary data from the texture videos to facilitate view synthesis with reduced artifacts. Further, a method for updating the image background and the depth values uses the auxiliary data after extraction of each object to reduce the artifacts due to limited performance of online inpainting of missing data or holes during view synthesis.