Texture Hole Filling via Depth Clustering and Directional Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In depth image-based rendering, the synthesis of virtual views often results in 'holes' due to occluded background areas and depth discontinuities, leading to noticeable visual artifacts as the number of views encoded or transmitted is reduced, which existing methods fail to adequately address.
Innovation Solution
The proposed solution involves generating a texture hole mask to identify and fill texture holes in synthesized views by clustering bordering pixels into background and foreground clusters, predicting pixel values from candidate directions, and propagating these values into the hole regions using digital signal processors or graphics processing units for parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the number of views encoded or transmitted is reduced, then data amount is reduced, but hole size and number in synthesized views increase
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing depth information and geometric data before actual rendering. The system pre-processes scene understanding, object segmentation, and depth mapping so that when views are synthesized from reduced sets, the pre-computed geometric relationships enable accurate warping and back-projection without requiring full view data, thus reducing hole sizes while maintaining lower data quantity.
Solution Approach 2:
The patent introduces intermediary structures including depth maps, geometric models, and virtual camera parameters as mediators between original views and synthesized views. These intermediaries enable the system to reconstruct missing background areas by warping existing views according to pre-computed geometric relationships, filling holes without requiring transmission of all original views, thus reducing data amount while controlling hole size.
2Loss of time
If the number of views encoded or transmitted is reduced, then transmission time is reduced, but visual quality deteriorates due to increased holes
Solution Approach 1:
The patent performs preliminary scene analysis, depth estimation, and geometric model construction before actual video transmission. By pre-computing the relationships between multiple views and their corresponding depth information, the system can efficiently synthesize high-quality virtual views from a reduced set of transmitted views, minimizing transmission time while maintaining visual quality through accurate geometric warping and hole filling.
Solution Approach 2:
The patent implements feedback mechanisms where the system continuously monitors synthesized view quality and adjusts rendering parameters, warping strengths, and hole filling algorithms in real-time. This feedback loop allows optimization of visual quality during transmission by adapting to specific scene characteristics and view configurations, ensuring high quality even with reduced view sets.
3Measurement precision
If depth discontinuities are present in depth images, then depth information is preserved, but holes are created in synthesized views
Solution Approach 1:
The patent segments the scene into distinct depth regions based on depth discontinuities and uses these segments to guide the warping and back-projection processes. By identifying and respecting depth boundaries, the system can accurately preserve depth information at discontinuities while using appropriate hole filling strategies for different depth regions, thus maintaining depth accuracy while controlling hole area through region-specific processing.
Solution Approach 2:
The patent applies different processing strategies for different local regions of the image based on depth characteristics. Areas with depth discontinuities receive specialized handling that preserves depth accuracy, while continuous regions use standard warping and hole filling. This local quality approach allows the system to maintain depth information accuracy at discontinuities while minimizing hole area in other regions through optimized rendering parameters for each local zone.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
Bordering pixels delineating a texture hole region in an image are identified. Depth values of the bordering pixels are recorded. The depth values are automatically clustered into two depth value clusters with a depth value threshold separating the two depth value clusters. A subset of bordering background pixels is identified in the bordering pixels as those with depth values in one of the two depth value clusters that is declared as a background depth value cluster. The subset of bordering background pixels is used to predict texture hole pixel values in the texture hole region based on multiple candidate prediction directions. Quality indicator values are computed for the multiple candidate prediction directions and used to select a specific candidate prediction direction for filling in final texture hole pixel values in the texture hole region of the image.