Fisheye Depth Estimation Using Spatial-Temporal Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Unsupervised depth estimation for stereo fisheye cameras in autonomous driving is challenging due to their large field of view, significant rotation and translation, and the complexities of unsupervised training methods, which existing technologies struggle to address effectively.
Innovation Solution
The method employs spatial-temporal consistency by training a depth estimation model using sequences of frames and synchronized stereo camera pairs, with optional modal consistency from different sensor types, and utilizes a fisheye camera projection model to project disparity maps into point cloud maps for rectification, enabling unsupervised depth estimation directly on raw, distorted images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional supervised depth estimation methods are used, then training accuracy can be improved using ground truth labels, but the system complexity and computational requirements increase significantly
Solution Approach 1:
The system performs self-supervised learning by automatically generating depth labels from stereo image pairs without requiring external ground truth data. The depth estimation network learns to predict depth by minimizing reconstruction error between projected and actual images, enabling the system to train itself using only the input images.
Solution Approach 2:
The patent introduces an intermediary projection mechanism that maps depth predictions to synthesized images, which then serve as a bridge between the depth estimation network and the training objective. This intermediary approach allows the system to learn depth estimation without direct access to ground truth depth labels.
2Area of stationary object
If fisheye cameras with large field of view are used, then coverage area is improved, but distortion and rectification complexity increase
Solution Approach 1:
The system changes the parameter space by working directly with distorted fisheye images in their native coordinate system rather than attempting to rectify them to a standard perspective. The depth estimation and projection operations are performed using fisheye-specific projection models that accommodate the distorted geometry.
Solution Approach 2:
The patent transitions from 2D image plane operations to 3D point cloud representation and back, using an intermediate spherical coordinate system. This dimensional transformation allows the system to handle fisheye distortion naturally by operating in a coordinate space that matches the camera's optical characteristics.
3Ease of manufacture
If unsupervised training methods are used, then ground truth labels are eliminated, but training constraints and convergence difficulty increase
Solution Approach 1:
The system implements a feedback loop where depth predictions are projected back to image space and compared with the original images. The reconstruction error serves as a differentiable loss signal that guides the gradient descent optimization, providing continuous feedback to improve depth estimation accuracy during training.
Solution Approach 2:
The patent performs preliminary feature extraction and image alignment operations before the main depth estimation task. By pre-processing the images to extract relevant features and establish correspondences, the system prepares the data in a form that facilitates faster and more stable convergence during the unsupervised training process.
4Area of stationary object
If stereo fisheye camera pairs with large rotation and translation are used, then spatial coverage is improved, but epipolar geometry complexity and matching difficulty increase
Solution Approach 1:
The system adopts a dynamic approach by not fixing the epipolar geometry assumptions but instead learning the transformation parameters between stereo pairs directly from the data. The model accommodates large rotations and translations by using a flexible projection model that adapts to the specific baseline and orientation of each camera pair.
Data Source
AI summary
Methods and systems for unsupervised depth estimation for fisheye cameras using spatial-temporal (and, optionally, modal) consistency. This unsupervised depth estimation works directly on raw, distorted stereo fisheye images, such as those obtained from the four fisheye camera disposed around a vehicle in rigid alignment. Temporal consistency involves training a depth estimation model using a sequence of frames as input, while spatial consistency involves training the depth estimation model using overlapping images from synchronized stereo camera pairs. Images from different stereo camera pairs can also be used at different times. Modal consistency, when applied, dictates that different sensor types (e.g., camera, lidar, etc.) must also agree. The methods and systems of the present disclosure utilize a fisheye camera projection model that projects a disparity map into a point cloud map, which aides in the rectification of stereo pairs.