Image and Sparse Depth Fusion for Scale-Accurate Depth Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current depth mapping technologies using a single sensor type suffer from scale ambiguity, sparse depth information, limited depth range, and limited field of view, leading to incomplete and inaccurate depth estimation.
Innovation Solution
A hybrid approach utilizing both an image sensor and a sparse depth sensor, employing a fusion network and multi-stage depth prediction to generate a high-resolution dense depth map, which includes preprocessing to reduce noise and enhance resolution, and a light-weight encoder-decoder design to reduce computational burden.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If only an RGB camera sensor is used for depth mapping, then the system complexity is reduced, but scale ambiguity and inaccurate depth estimation occur
Solution Approach 1:
The patent combines data from an RGB camera sensor and a Time-of-Flight (ToF) sensor to generate depth maps. The RGB sensor provides wide-field depth information while the ToF sensor provides accurate sparse depth measurements, and their fusion resolves scale ambiguity to achieve both reduced complexity and improved accuracy.
2Measurement precision
If only a Time-of-Flight (ToF) sensor is used for depth mapping, then the measurement precision is improved, but the field of view and depth information completeness are limited
Solution Approach 1:
The patent merges the sparse but accurate depth data from the ToF sensor with the dense but ambiguous depth data from the RGB sensor. This combination extends the effective field of view while maintaining measurement precision through data fusion and scale correction.
3Measurement precision
If a hybrid sensor system with fusion network is used, then the depth estimation accuracy is improved, but the computational burden increases
Solution Approach 1:
The patent performs preliminary actions by pre-processing the sparse depth frame from the ToF sensor before fusion, including noise reduction and coordinate transformation. This preparation reduces the complexity of the subsequent fusion operation and enables efficient processing on mobile devices.
4Loss of information
If preprocessing and fusion operations are performed on sparse depth frames, then the depth information completeness is improved, but the processing time increases
Solution Approach 1:
The patent applies partial preprocessing operations focused only on critical aspects of the sparse depth frame (noise reduction in depth values, coordinate alignment) rather than complete image processing. This selective approach recovers lost depth information while minimizing processing time for real-time applications.
Data Source
AI summary
A method of depth estimation, including, receiving an image frame, determining a relative depth map based on the image frame, receiving a sparse depth frame, preprocessing the sparse depth frame, determining a scale-adjusted relative depth map based on the relative depth map and the preprocessed sparse depth frame and fusing the relative depth map and the scale-adjusted relative depth map to produce an absolute depth map.