Targetless LiDAR-Camera Calibration Using Natural Scene Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sensor calibration techniques for autonomous vehicles require ground-truth targets or complex alignment methods, which are cumbersome and prone to errors due to differing fields-of-view and illumination conditions, especially for optical and LiDAR sensors.
Innovation Solution
A targetless calibration method using natural textures and 3D objects, preprocessing optical and LiDAR data to reduce distortion, and employing machine learning-based feature detection and matching to align sensors accurately.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If ground-truth targets with known patterns are used for calibration, then measurement precision is improved, but device complexity and ease of operation deteriorate due to careful placement requirements, specific illumination conditions, and uncluttered environment needs
Solution Approach 1:
The system uses naturally occurring environmental features (trees, poles, rocks, parked cars) as calibration references instead of requiring external ground-truth targets. The sensors automatically detect and match features in the natural scene, enabling the system to calibrate itself without human intervention for target placement and configuration.
Solution Approach 2:
The patent introduces an image processing and feature matching system as an intermediary between the sensors and environmental features. This intermediary processes optical images and LiDAR point clouds to automatically identify, extract, and match corresponding features, replacing the need for direct visual inspection and manual target placement.
2Measurement precision
If ground-truth targets are used for calibration, then measurement precision is improved, but loss of time increases due to returning to original calibration location and setting up targets
Solution Approach 1:
The calibration system operates autonomously using naturally occurring environmental features that are always present during vehicle operation. No return to calibration locations or manual target setup is required, as the system continuously performs calibration using the surrounding environment during normal vehicle operation.
Solution Approach 2:
The system performs calibration continuously during normal vehicle operation rather than requiring separate calibration sessions. By using environmental features that are naturally present during operation, the calibration process is integrated into regular vehicle use, eliminating dedicated calibration time.
3Device complexity
If depth discontinuities and edge features are used for targetless calibration, then device complexity is reduced, but measurement precision deteriorates due to alignment quality issues affected by sensor spacing and distance
Solution Approach 1:
The patent introduces a multi-step feature processing system as an intermediary that transforms raw depth discontinuities and edge features into robust calibration data. The system extracts features from both optical images and LiDAR point clouds, matches them using descriptor comparison, and computes alignment transformations, thereby improving precision beyond simple depth-edge alignment.
Solution Approach 2:
The system combines information from multiple dimensions by integrating optical image data (2D intensity information) with LiDAR point cloud data (3D spatial information). This multi-dimensional feature matching approach overcomes the limitations of single-modality alignment and improves measurement precision across varying distances and sensor spacings.
4Ease of operation
If three-dimensional LiDAR data is projected into two-dimensional surface, then ease of operation is improved, but measurement precision deteriorates due to mismatch with optical image data
Solution Approach 1:
Instead of projecting 3D LiDAR data onto a 2D surface, the patent performs reverse dimensionality transformation by projecting 2D optical image features into 3D space using LiDAR depth information. This preserves the geometric accuracy of the 3D point cloud while enabling feature matching in a common coordinate system, thereby maintaining measurement precision.
Solution Approach 2:
The patent introduces a coordinate transformation system as an intermediary that converts features from different data representations into a common 3D coordinate system. This intermediary process maintains the geometric fidelity of both optical and LiDAR data while enabling accurate feature matching without loss of precision.
Data Source
AI summary
A sensor calibration system of a vehicle is provided, comprising an optical sensor configured to generate two-dimensional optical calibration data and a LiDAR sensor configured to generate three-dimensional intensity calibration data. The system extracts one or more optical images from the optical calibration data and projects one or more portions of the intensity calibration data onto one or more optical image planes corresponding to the one or more optical images to form one or more intensity images. The system detects a set of points in the one or more intensity images and another set of points in the one or more optical images corresponding to one or more environmental features. The system creates a plurality of pairings wherein each pairing comprises corresponding points from the two sets of points and computes an alignment between the optical sensor and the LiDAR sensor based on a subset of the plurality of pairings.