Depth Map Alignment Using Color Image Guidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-view imaging systems face challenges in generating high-quality depth-view images due to lower resolution and noise in depth sensors, particularly in time-of-flight cameras, which affect the alignment and up-sampling of depth maps, especially in mobile environments with limited power consumption and non-Lambertian surfaces.
Innovation Solution
A method that involves non-local denoising and projection alignment techniques, using cross-bilateral filtering and cost-volume filtering to enhance depth map resolution and accuracy, while avoiding explicit non-uniform resampling and z-ordering, by leveraging high-resolution color images for confidence weighting and filtering.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If depth map is filtered before alignment with colour image, then noise reduction is improved, but it is not possible to filter depth map using available colour image until it is aligned with a colour view
Solution Approach 1:
The patent performs alignment of the depth map to the colour image view before filtering operations. This preliminary alignment ensures that subsequent filtering using colour image guidance is effective, as the depth and colour data are now in corresponding spatial relationships. The method establishes proper geometric correspondence first, then applies filtering to reduce noise while preserving depth discontinuities at object boundaries.
2Measurement precision
If depth map is up-sampled to match colour camera resolution, then resolution is improved, but non-regular-to-regular grid resampling is needed which complicates the process
Solution Approach 1:
The patent transforms the depth map from the depth camera's native resolution to the colour camera's higher resolution grid through systematic up-sampling. This dimensionality change in resolution space is achieved by establishing geometric correspondence between the two views and interpolating depth values at the higher resolution grid points, converting the non-regular depth samples into a regular high-resolution depth map that matches the colour image dimensions.
3Reliability
If noise filtering is applied to depth map, then noise is reduced, but texture details and small details may be over-smoothed
Solution Approach 1:
The patent applies local quality weighting in the filtering process by using colour image gradients to identify regions with high texture detail. In areas where the colour image shows significant variation (indicating texture or edges), the filtering is weakened or disabled to preserve detail. In smooth regions without texture, full filtering strength is applied to reduce noise. This spatially adaptive filtering preserves object boundaries and texture details while removing random noise in homogeneous areas.
4Use of energy by moving object
If depth camera operates in low-powered sensing mode to conserve energy, then power consumption is reduced, but depth map quality deteriorates with increased noise and lower resolution
Solution Approach 1:
The patent uses the high-resolution colour image as an intermediary to enhance the low-quality depth map obtained from the power-constrained depth camera. The colour image provides geometric and textural information that guides the filtering and up-sampling processes, allowing the system to recover depth details that would otherwise be lost in low-powered mode. This intermediary colour data compensates for the degraded depth sensor performance.
5Measurement precision
If alignment is performed using 3D image warping with calibrated cameras, then alignment accuracy is improved, but systematic errors on non-Lambertian surfaces and hair remain
Solution Approach 1:
The patent applies feedback-based refinement by using the colour image as a reference to identify and correct systematic errors in the aligned depth map. After initial alignment through 3D warping, the method compares depth values with corresponding colour image features and adjusts depth measurements in regions where systematic errors are detected, such as non-Lambertian surfaces or hair. This feedback loop from colour to depth corrects residual alignment errors that geometric warping alone cannot resolve.
Data Source
Figure 1
Figure 2a~2b
Figure 3~5c
AI summary
Various embodiments are provided which relate to the field of image signal processing, specifically relating to the generation of a depth-view image of a scene from a set of input images of a scene taken at different cameras of a multi-view imaging system. A method comprises obtaining a frame of an image of a scene and a frame of a depth map regarding the frame of the image. A minimum depth and a maximum depth of the scene and a number of depth layers for the depth map are determined. Pixels of the image are projected to the depth layers to obtain projected pixels on the depth layers; and cost values for the projected pixels are determined. The cost values are filtered and a filtered cost value is selected from a layer to obtain a depth value of a pixel of an estimated depth map.