Online Stereo Rectification for Wide-Baseline Camera Misalignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Stereo camera systems in autonomous vehicles, particularly those with wide baselines, become misaligned due to vibrations and environmental factors, leading to poor performance in tasks like depth estimation and object detection, and existing calibration methods are not computationally feasible for online rectification.
Innovation Solution
An online stereo rectification process using a neural network model that predicts a rotation matrix between images, applies positional feature enhancement, computes a correlation volume, and employs differentiable rectification to calibrate the cameras, utilizing self-supervised learning to minimize vertical optical flow without ground truth data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional calibration methods are used for stereo camera systems, then initial alignment is achieved, but the system becomes misaligned over time due to vibrations and environmental factors
Solution Approach 1:
The system transitions from static offline calibration to dynamic online rectification. A neural network model continuously predicts rotation matrices between stereo images in real-time, adapting to changing environmental conditions and vibrations. The system processes incoming image streams and updates calibration parameters dynamically, ensuring sustained alignment accuracy throughout vehicle operation.
Solution Approach 2:
The system performs self-calibration without requiring external ground truth data or manual intervention. The neural network learns from the stereo image pairs themselves, using self-supervised learning where the network predicts relative camera poses and rectifies images based on its own predictions. This autonomous recalibration capability allows the system to maintain accuracy independently after initial setup.
2Measurement precision
If offline calibration methods are used, then accurate initial alignment is achieved, but real-time recalibration is not feasible due to computational complexity
Solution Approach 1:
The system replaces traditional iterative mechanical calibration procedures with a neural network-based predictive model. Instead of using computationally intensive optimization algorithms that require ground truth data, the neural network directly predicts rotation matrices from image features. This substitution enables real-time processing while maintaining calibration accuracy, as the network inference is significantly faster than traditional iterative methods.
Solution Approach 2:
The system changes the computational parameters from iterative optimization with ground truth constraints to direct prediction from image features. The neural network transforms the calibration problem from solving complex optimization equations to predicting transformation parameters directly from visual input. This parameter transformation enables real-time operation by replacing computationally heavy processes with efficient neural network inference.
3Area of stationary object
If wide baseline camera systems are used to increase coverage, then field of view is improved, but misalignment due to mounting structure deformation is exacerbated
Solution Approach 1:
The system continuously adapts to the dynamic deformation of wide baseline mounting structures. The neural network processes stereo image pairs in real-time and predicts rotation matrices that compensate for structural shifts caused by temperature gradients and vehicle vibrations. This dynamic adjustment allows the system to maintain alignment accuracy despite the increased susceptibility of wide baseline configurations to environmental deformation.
4Productivity
If self-supervised learning is used without ground truth data, then real-time operation is enabled, but training complexity increases
Solution Approach 1:
The system uses self-supervised learning where the neural network learns from the stereo image pairs themselves without requiring external ground truth data. The network predicts relative camera poses and uses these predictions to rectify images, creating its own training signal from the input data. This self-service approach enables deployment in real-time operation while the training phase, though complex, is performed offline during system initialization.
Data Source
AI summary
An autonomy computing system and a method of an autonomous vehicle for rectifying stereo images includes a memory storing computer executable instructions and a processor coupled to the memory, the processor, upon execution of the computer executable instructions, configured to: receive an image pair captured using respective cameras in the stereo camera pair; predict a rotation matrix between the first image and the second image by: extracting a first feature map and a second feature map; applying positional feature enhancement on the feature maps to derive a pair of enhanced feature maps; computing a correlation volume across the enhanced feature maps; determining a set of likely matches between the enhanced feature maps; computing a predicted relative pose; and computing the rotation matrix. The system and method further include calibrating the stereo camera pair to rectify the first image and the second image based on the rotation matrix.