Camera–LiDAR Calibration Using Locally Planar Image Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sensor calibration methods for autonomous vehicles, particularly those using correspondence-based approaches, face challenges in obtaining accurate correspondences between camera and LiDAR sensors due to the reliance on feature detection across different sensor modalities, which can be unreliable and require additional sensor-specific information.
Innovation Solution
A method that uses image feature correspondences and the assumption of locally planar surfaces to determine calibration parameters without requiring explicit cross-sensor feature correspondences, utilizing an optimization process to minimize a geometric loss function.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If correspondence-based methods are used to find calibration parameters by maximizing alignment between features in camera images and LiDAR point clouds, then calibration can be performed, but reliable and accurate correspondences are difficult to obtain
Solution Approach 1:
The patent introduces a depth map as an intermediary representation that bridges camera images and LiDAR point clouds. Instead of directly finding correspondences between image features and LiDAR features (which are unreliable), the system projects image features into the depth map space where they can be reliably matched with LiDAR-derived surfels (surface elements). This intermediary depth map space enables accurate calibration without requiring direct cross-modality feature correspondence.
Solution Approach 2:
The patent creates a simplified representation (copy) of the 3D environment using surfels derived from LiDAR data, which are 2D-like planar elements with associated depth information. These surfels serve as a simplified copy of the actual 3D surfaces, making correspondence matching more reliable than working with raw LiDAR point clouds or camera images directly. The surfel representation preserves essential geometric information while being much easier to match across sensor modalities.
2Measurement precision
If photometric matching is used to compare intensity information from LiDAR sensors to camera images, then correspondence can be established, but LiDAR intensity information is not always available or accurate
Solution Approach 1:
The patent extracts and utilizes only the geometric information (depth, surface normal, position) from LiDAR data while discarding the unreliable intensity information. By taking out the geometric components and forming surfels based solely on LiDAR's spatial measurements, the system avoids the problem of missing or inaccurate LiDAR intensity data while still achieving reliable correspondences with camera images through geometric consistency.
3Measurement precision
If object-based correspondence methods are used requiring pretrained detectors for object classes, then calibration can be performed, but the approach is restrictive and requires additional sensor-specific information
Solution Approach 1:
The patent creates a universal calibration approach using surfels that can represent any surface type without requiring object class identification. The surfel-based method is modality-agnostic and works with any combination of camera and LiDAR sensors, eliminating the need for pretrained object detectors or sensor-specific processing pipelines. This universal approach simplifies the system while maintaining calibration accuracy across diverse environments and sensor configurations.
4Reliability
If correspondence-free methods using odometry trajectory are used to determine coarse alignment, then cross-sensor correspondence is avoided, but not all sensor measurements can be used simultaneously to constrain calibration parameters
Solution Approach 1:
The patent merges the advantages of correspondence-free methods with correspondence-based methods by combining camera image features, LiDAR-derived surfels, and depth map information into a unified calibration framework. This merged approach allows simultaneous use of all sensor measurements (camera images and LiDAR data) to constrain calibration parameters while avoiding the unreliable cross-sensor correspondences through the intermediary depth map space.
Data Source
AI summary
In various examples, sensor configuration for autonomous or semi-autonomous systems and applications is described. Systems and methods are disclosed that may use image feature correspondences between camera images along with an assumption that image features are locally planar to determine parameters for calibrating an image sensor with a LiDAR sensor and/or another image sensor. In some examples, an optimization problem is constructed that attempts to minimize a geometric loss function, where the geometric loss function encodes the notion that corresponding image features are views of a same point on a locally planar surface (e.g., a surfel or mesh) that is constructed from LiDAR data generated using a LiDAR sensor. In some examples, performing such processes to determine the calibration parameters may remove structure estimation from the optimization problem.