Depth Estimation Using LiDAR and RGB Fusion for Autonomous Vehicles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current depth sensing technologies in autonomous vehicles face challenges due to inconsistent sensor performances across different environments and modalities, leading to unreliable object detection, especially at high speeds, and difficulties in recognizing smaller objects due to limited data resolution and sampling rates.
Innovation Solution
A depth estimation apparatus that integrates multiple sensors, such as LiDAR and RGB cameras, using segmentation algorithms, data alignment, and depth estimation algorithms to generate high-resolution depth maps by transforming sparse point clouds into dense point clouds, improving detection accuracy and enabling more precise path planning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If LiDAR with fewer number of beams is used, then power consumption is reduced and cost is lowered, but detection accuracy and data resolution deteriorate
Solution Approach 1:
The patent introduces an intermediate processing stage that fuses LiDAR point cloud data with RGB camera image data. The RGB data acts as an intermediary to fill gaps in sparse LiDAR point clouds, allowing the system to use lower-beam LiDAR (reducing power consumption) while maintaining detection accuracy through data fusion and up-sampling techniques.
2Ease of manufacture
If LiDAR with fewer number of beams is used, then device cost is reduced, but object recognition reliability deteriorates
Solution Approach 1:
The patent combines data from two different sensing modalities (LiDAR point clouds and RGB images) into a composite depth map. This composite approach allows the use of lower-cost, fewer-beam LiDAR sensors while maintaining object recognition reliability through the complementary information provided by the RGB camera and the fusion algorithm.
3Device complexity
If data sampling rate is low, then device complexity is reduced, but detection reliability at high speed deteriorates
Solution Approach 1:
The patent merges LiDAR point cloud data with high-temporal-resolution RGB image data to compensate for the low sampling rate of LiDAR. By combining these data sources and applying up-sampling techniques, the system achieves reliable high-speed object detection without requiring complex high-speed LiDAR sampling.
4Ease of operation
If uniform specifications are used for detection results, then processing simplicity is maintained, but path planning precision deteriorates
Solution Approach 1:
The patent applies different processing and fusion strategies to different regions of the depth map based on local characteristics. High-confidence regions receive standard processing while uncertain regions undergo up-sampling and fusion with RGB data. This localized approach maintains processing simplicity in reliable areas while enhancing precision in critical areas for path planning.
Data Source
AI summary
In one of the exemplary embodiments, the disclosure is directed to a depth estimation apparatus including a first type of sensor for generating a first sensor data; a second type of sensor for generating a second sensor data; and a processor coupled to the first type of sensor and the second type of sensor and configured at least for: processing the first sensor data by using two stage segmentation algorithms to generate a first segmentation result and a second segmentation result; synchronizing parameters of the first segmentation result and parameters of the second sensor data to generate a synchronized second sensor data; fusing the first segmentation result, the synchronized second sensor data, and the second segmentation result by using two stage depth estimation algorithms to generate a first depth result and a second depth result.