Camera–LiDAR Sensor Calibration Without Cross-Modal Correspondences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sensor calibration methods for autonomous vehicles face challenges in accurately determining calibration parameters between camera and LiDAR sensors due to difficulties in obtaining reliable cross-modal correspondences, especially when straight line features are not consistently captured and photometric matching is incomplete or inaccurate, and object-based approaches require pre-trained detectors.
Innovation Solution
A method that uses image feature correspondences between camera images, assuming locally planar surfaces, to determine calibration parameters without requiring explicit cross-sensor correspondences, by constructing an optimization problem to minimize a geometric loss function based on LiDAR-generated point clouds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If correspondence-based methods are used to find calibration parameters by matching features between camera images and LiDAR point clouds, then sensor alignment can be achieved, but reliable correspondences are difficult to obtain due to inconsistent feature capture across modalities
Solution Approach 1:
The patent introduces an intermediary optimization framework that mediates between camera images and LiDAR point clouds. Instead of directly matching features across modalities, the system uses an optimization process with a geometric loss function as an intermediary step to find calibration parameters that maximize feature alignment, thereby resolving the reliability issue of direct correspondence methods
Solution Approach 2:
The patent changes the approach from directly matching features to optimizing calibration parameters through an iterative process. By adjusting parameters like rotation and translation to minimize the geometric loss function, the system achieves accurate alignment without requiring reliable direct correspondences between different sensor modalities
2Ease of manufacture
If object-based correspondence approaches are used requiring pretrained detectors for object classes in both camera images and LiDAR point clouds, then calibration can be performed using object classes, but the approach becomes restrictive and requires additional preprocessing
Solution Approach 1:
The patent extracts and removes the requirement for pre-trained object detectors from the calibration process. By focusing on geometric feature alignment through optimization rather than object-class-based correspondence, the system eliminates the need for complex detector preprocessing while maintaining calibration effectiveness
Solution Approach 2:
The patent creates a universal calibration approach that works across different sensor configurations and modalities without requiring modality-specific preprocessing. The optimization framework with geometric loss function serves multiple purposes: it handles calibration, alignment, and feature matching in a single unified process that adapts to various sensor setups
3Reliability
If correspondence-free methods using odometry trajectory are used to determine coarse alignment, then cross-sensor correspondence is avoided, but not all sensor measurements can be used simultaneously to constrain calibration parameters
Solution Approach 1:
The patent merges camera images and LiDAR point clouds into a unified optimization framework. By combining measurements from both modalities and using them simultaneously to constrain calibration parameters through the geometric loss function, the system achieves both high reliability and full utilization of available sensor data
Solution Approach 2:
The patent transitions from using odometry trajectory for coarse alignment in three-dimensional space to a six-dimensional calibration parameter space that includes both rotation and translation parameters. This dimensional expansion allows simultaneous constraint from all sensor measurements while maintaining alignment accuracy
Data Source
AI summary
In various examples, sensor configuration for autonomous or semi-autonomous systems and applications is described. Systems and methods are disclosed that may use image feature correspondences between camera images along with an assumption that image features are locally planar to determine parameters for calibrating an image sensor with a LiDAR sensor and/or another image sensor. In some examples, an optimization problem is constructed that attempts to minimize a geometric loss function, where the geometric loss function encodes the notion that corresponding image features are views of a same point on a locally planar surface (e.g., a surfel or mesh) that is constructed from LiDAR data generated using a LiDAR sensor. In some examples, performing such processes to determine the calibration parameters may remove structure estimation from the optimization problem.