Confidence-Based Depth Map Hole Filling for Autonomous Vehicles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional 3D vision systems for autonomous vehicles and driver-assistance systems face challenges in providing accurate and reliable depth maps in real time, particularly due to occlusions, inadequate lighting, and motion-based blurriness, which result in 'holes' in the depth maps that can lead to false negatives and gross underestimations of object sizes.
Innovation Solution
A method using a non-uniform convolution kernel for extrapolation and interpolation principles to fill holes in depth maps, allowing for real-time or nearly real-time processing by determining a confidence mask and convolving a kernel with neighboring pixels to estimate depth values for missing data points.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If conventional stereo vision systems are used to generate depth maps, then 3D information can be obtained, but occlusions, inadequate lighting, and motion blur result in holes in the depth maps
Solution Approach 1:
The patent extracts and processes only the reliable pixels from the depth map, separating them from unreliable or missing regions. By identifying confident pixels and using them as a basis for interpolation, the system extracts useful depth information while excluding problematic areas, thereby reducing information loss without compromising overall reliability.
Solution Approach 2:
The patent introduces an intermediary processing step that uses confident pixels as mediators to infer depth values for unreliable pixels. The system uses the known depth values from confident pixels as intermediate data points to interpolate and fill in missing depth information, acting as a bridge between reliable measurements and estimated values.
2Loss of time
If real-time processing is implemented to provide timely depth information, then reaction time to hazards is improved, but processing accuracy may be compromised due to the complexity of hole-filling algorithms
Solution Approach 1:
The patent applies partial action by processing only the necessary portions of the depth map - specifically, only the unreliable pixels that need interpolation. Rather than reprocessing the entire depth map, the system selectively applies hole-filling algorithms only to regions with missing or unreliable data, reducing computational complexity while maintaining accuracy for critical areas.
Solution Approach 2:
The patent performs preliminary identification and classification of pixels into confident and unreliable categories before applying the interpolation algorithm. This preliminary sorting allows the system to prepare the data structure in advance, enabling faster real-time processing by having the confidence mask ready and organized before the actual hole-filling computation begins.
3Loss of information
If complex hole-filling algorithms are used to fill all missing depth data, then completeness of depth maps is improved, but computational complexity and processing time increase
Solution Approach 1:
The patent applies local quality by using different processing approaches for different regions of the depth map. Confident pixels are kept as-is with their original high-quality measurements, while only unreliable pixels undergo interpolation. This localized differentiation ensures that high-quality data is preserved without unnecessary processing, while still filling gaps where needed, thereby improving completeness without uniformly increasing complexity across the entire map.
Data Source
AI summary
A system and a method are provided for repairing a depth map. The repairing includes representing pixels of the depth map using a matrix H having values corresponding to depths, the pixels including hole pixels and non-hole pixels; determining a confidence mask matrix C for the depth map; for each hole pixel, determining a kernel G of neighboring pixels, the neighboring pixels having values corresponding to a predetermined distribution, and convolving the kernel G with corresponding portions of the matrix C and the matrix H to obtain an estimated depth; generating a corrected matrix Ĥ in which values for non-hole pixels are retained and in which the hole pixels have values corresponding to their estimated depths. The estimated depths may be normalized based on the matrix C. An updated depth map based on the corrected matrix Ĥ may be outputted to a control system of a vehicle.