LiDAR-Camera Alignment Using Flexible 3D-2D Control Points

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Traditional sensor alignment algorithms for autonomous vehicles face challenges in accurately aligning data from different sensors when they detect different numbers and locations of objects, leading to inconsistent sensor correlation and difficulty in determining corresponding objects.

Innovation Solution

A method that solves sensor alignment as a mathematical optimization problem using a group of Lidar-camera control points with flexible 3D-2D correspondence, employing a flexible number of control points to generate correlated sensor data and alignment correction values, including translation and rotation parameters.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If traditional sequential iterative alignment algorithms are used, then the alignment process can be performed step-by-step, but the sensor correlation becomes inconsistent when different numbers and locations of objects are detected

Engineering Contradiction:

Improvealignment process executionVSAvoidsensor correlation consistency

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent replaces the traditional sequential iterative alignment algorithm with a mathematical optimization approach. Instead of performing alignment step-by-step through sequential iterations, the system formulates the alignment problem as an optimization problem and solves it directly, eliminating the sequential dependency that causes inconsistency when object detection counts differ between sensors.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent changes the fundamental parameters of the alignment approach by introducing flexible 3D-2D correspondence requirements and using control points with variable numbers. This allows the system to handle mismatched object detections robustly by optimizing alignment parameters based on the actual number of detected objects rather than assuming fixed correspondence.

Inventive Principle:

Principle #35Parameter changes

2Quantity of substance

If data pairs from multiple sensors are combined, then more comprehensive environmental information is obtained, but difficulty arises in determining corresponding objects when detection counts mismatch

Engineering Contradiction:

Improveenvironmental informationVSAvoidobject correspondence identification

Core Design Contradiction:

Quantity of substanceVSDifficulty of detecting and measuring

Solution Approach 1:

The patent segments the alignment problem into independent control points, where each control point represents a potential correspondence between 3D LiDAR points and 2D camera points. This segmentation allows the system to process objects independently and flexibly match them based on spatial relationships rather than relying on pre-assumed correspondences, making it easier to handle mismatched detection counts.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces control points as intermediary elements that mediate between LiDAR and camera data. These control points serve as flexible intermediaries that can correspond to zero, one, or multiple objects, allowing the system to bridge the gap between sensors with different detection capabilities without requiring direct one-to-one object correspondence.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Device complexity

If a fixed number of control points are used, then the alignment algorithm has simple structure, but it cannot handle noisy or mismatched sensor inputs robustly

Engineering Contradiction:

Improvealgorithm structureVSAvoidalignment robustness to noise

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The patent makes the number of control points dynamic rather than fixed. The system automatically determines the number of control points based on the actual number of detected objects in each sensor's field of view. This dynamic adaptation allows the algorithm to handle noisy or mismatched inputs robustly by adjusting the number of control points to match the actual data, while maintaining relative simplicity through the optimization framework.

Inventive Principle:

Principle #15Dynamics

Data Source

PatentUS12586245B2Robust lidar-to-camera sensor alignment

Publication Date: 2026.03.24 GM GLOBAL TECHNOLOGY OPERATIONS LLC

AI summary

Method for sensor alignment including detecting a depth point cloud including a first object and a second object, generating a first control point in response to a location of the first object within the depth point cloud and a second control point in response to a location of the second object within the depth point cloud, capturing an image of a second field of view including a third object, generating a third control point in response to a location of the third object detected in response to the image, calculating a first reprojection error in response to the first control point and the third control point and a second reprojection error in response to the second control point and the third control point, generating an extrinsic parameter in response to the first reprojection error in response to the first reprojection error being less than the second reprojection error.