Multi-view Rendering Hole Restoration via Temporal Neighboring Images
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generating multi-view 3D images is challenging due to difficulties in capturing and transmitting data, especially with physical limitations in photography systems, and existing methods degrade image quality with holes and cracks caused by depth estimation inaccuracies and binocular disparity differences.
Innovation Solution
An image processing apparatus and method that uses a processor to generate output view images through image warping with reference view images and binocular disparity information, and includes units for neighboring image-based hole restoration, binocular disparity crack detection, and optimal patch search to restore holes and cracks by scaling pixels and searching for similar patches from neighboring images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If image warping is used to generate output view images from reference view images and binocular disparity information, then multi-view 3D images can be generated with fewer input views, but holes and cracks are generated in the output images due to depth estimation inaccuracies
Solution Approach 1:
The patent applies preliminary action by detecting holes and cracks in the output view image before final rendering, and by preparing reference pixels from temporally adjacent frames in advance. The hole and crack detection unit identifies defective regions beforehand, and the restoration unit uses pre-prepared reference data from neighboring frames to restore these regions before the final image is displayed, thus preventing quality degradation in the generated multi-view images
Solution Approach 2:
The patent uses temporally adjacent frames as an intermediary to restore holes and cracks. Instead of directly copying pixels from spatially adjacent reference views (which would propagate depth errors), the system uses reference pixels from frames taken at different times but from the same viewpoint. This temporal intermediary approach allows restoration without introducing additional spatial distortion or depth estimation errors
2Adaptability or versatility
If depth information is used for image warping to generate binocular disparity, then 3D depth perception is achieved, but depth estimation inaccuracies cause holes and cracks in the generated images
Solution Approach 1:
The system performs preliminary detection of holes and cracks caused by depth estimation errors before final image generation. By identifying these defective regions in advance using the detection unit, and preparing restoration data from temporally adjacent frames, the system can correct depth-related artifacts before they affect the final 3D image quality
Solution Approach 2:
The patent introduces temporally adjacent frames as an intermediary source for restoring depth-related artifacts. Instead of relying solely on spatially adjacent reference views that share the same depth estimation errors, the system uses reference pixels from frames captured at different times but from the same viewpoint, thereby obtaining more accurate depth information without the propagation of spatial depth errors
3Manufacturing precision
If multiple reference views are used to improve image quality, then more data must be captured and transmitted, but storage and transmission difficulties increase
Solution Approach 1:
The system performs preliminary processing by detecting holes and cracks in the output view before final rendering, and by pre-selecting and preparing reference pixels from temporally adjacent frames. This preliminary action allows the system to work with a single reference view while achieving restoration quality that would otherwise require multiple reference views, thus reducing the quantity of data that needs to be captured, stored, and transmitted
Solution Approach 2:
The patent transitions from spatial dimension to temporal dimension by using frames from temporally adjacent time points as reference data. Instead of relying on multiple spatially separated reference views (which would increase data volume), the system exploits the temporal dimension by capturing and using reference pixels from frames taken at different times but from the same viewpoint, thereby maintaining image quality without increasing spatial data requirements
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An apparatus and method for restoring a hole generated in multi-view rendering are provided. A hole in an output view may be restored using temporally neighboring images.