Hole Filling Using Estimated Spatio-Temporal Background
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hole filling methods in 3D video and virtual reality systems face challenges in accurately separating background and foreground regions, leading to geometric distortion and blurring, especially when there is restricted information, and existing methods struggle to effectively estimate background regions in hidden areas using spatio-temporal information.
Innovation Solution
A hole filling method using estimated spatio-temporal background information that involves extracting a background codebook for non-overlapping patches from color and depth images, estimating temporal and spatial background information, and merging these to perform primary and secondary hole filling actions, including ghost removal and in-painting with a priority function that incorporates depth terms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If in-painting method is used to fill holes by using characteristics of unknown region and neighboring known region, then hole filling effectiveness is improved, but limitation remains when there is restricted information regarding background region and foreground region in hidden region
Solution Approach 1:
The patent applies preliminary action by estimating background regions in preceding images before performing hole filling in the current image. The system pre-processes temporal information from previous frames to create background estimates that are then used during the hole filling process, ensuring that background information is prepared in advance for regions that will be hidden or occluded.
Solution Approach 2:
The patent uses an intermediary approach by introducing a background estimation mechanism that acts as a mediator between the limited information available in the current frame and the hole filling process. The estimated background information from preceding images serves as an intermediary source that provides missing background details without requiring direct observation of the hidden regions.
2Ease of manufacture
If codebook with fixed threshold value is used to select codeword corresponding to background information, then processing is simplified, but effectiveness and speed of estimating background pixels in preceding image is limited
Solution Approach 1:
The patent applies dynamics by replacing the fixed threshold value with a dynamic adaptive threshold mechanism. The threshold is automatically adjusted based on the statistical properties of the image data, such as the distribution of depth values or intensity values. This allows the system to adapt to different scenes and lighting conditions, improving both the accuracy and speed of background pixel estimation without manual intervention.
3Adaptability or versatility
If depth-image-based rendering with 3D warping is used to compose virtual-viewpoint image, then free viewpoint is achieved, but holes are created in the virtual-viewpoint image
Solution Approach 1:
The patent uses copying by replicating background information from preceding images to fill the holes created during 3D warping. Instead of attempting to generate entirely new content for the hole regions, the system copies relevant background information from temporal neighbors (previous frames) and spatial neighbors (surrounding pixels), then merges this copied information with the warped image data to reconstruct the missing regions.
Data Source
AI summary
Provided are a hole filling method using estimated spatio-temporal background information and a recording medium and apparatus for performing the same. The present disclosure is providing a hole filling method using estimated spatio-temporal background information, the hole filling method being capable of estimating temporal background information, estimating spatial background information, and merging the temporal background information and the spatial background information to perform a hole filling process, and a recording medium and apparatus for performing the same.