Lidar-Image Depth Association for Occlusion-Robust Object Ranging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for depth perception in real-time applications, such as autonomous vehicle control, using stereo cameras are error-prone and slow, failing to accurately determine the depth of objects from monocular images.
Innovation Solution
A system that combines image data from a camera with LIDAR data to determine depth by scoring and sorting LIDAR points, using a weighted median calculation to provide accurate depth estimates, while accounting for occluding objects and reducing the need for multiple image sensors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If stereo cameras are used to recover depth from images, then depth information can be obtained, but the method is error-prone and too slow for real-time applications
Solution Approach 1:
The patent combines LIDAR data with monocular image data to achieve accurate depth estimation. The system projects LIDAR points onto the image plane and uses weighted median calculation to determine depth, merging the advantages of both sensing modalities to resolve the contradiction between accuracy and speed.
Solution Approach 2:
The patent replaces the mechanical stereo camera system with a combination of monocular camera and LIDAR. This substitution eliminates the need for complex stereo matching algorithms while achieving faster and more accurate depth estimation through direct LIDAR range measurements combined with image data.
2Measurement precision
If multiple image sensors are used to improve depth perception, then measurement accuracy improves, but device complexity and computational requirements increase
Solution Approach 1:
The patent merges LIDAR sensing with monocular vision to achieve depth perception accuracy comparable to or exceeding multi-camera systems. This combination allows the system to use a single camera while obtaining reliable depth information from LIDAR, thereby reducing device complexity.
Solution Approach 2:
The patent uses LIDAR data as an intermediary to provide depth information that bridges the gap between 2D monocular images and 3D scene understanding. This intermediary enables accurate depth estimation without requiring multiple image sensors, simplifying the overall system architecture.
3Reliability
If stereo camera systems are used for real-time control, then depth information is available, but computational power and energy consumption increase
Solution Approach 1:
The patent substitutes computationally intensive stereo matching algorithms with a more efficient approach that leverages direct LIDAR range measurements. This substitution dramatically reduces computational energy consumption while maintaining real-time depth availability for autonomous vehicle control.
Solution Approach 2:
The patent changes the fundamental parameter used for depth measurement from indirect stereo geometric computation to direct LIDAR range measurement. This parameter change enables real-time depth processing with minimal computational energy, as LIDAR provides direct distance measurements without requiring complex image processing.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach improves depth estimation accuracy over stereo camera techniques, reduces computational and power requirements, and enables real-time depth perception suitable for autonomous vehicle control, achieving depth estimates in approximately 6 milliseconds.
Implementation Method 1
receiving LIDAR data from a LIDAR sensor
Data Source
Figure 1A~1B
Figure 2A~2C
Figure 3
AI summary
A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest ("ROI") in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.