Self-Occlusion Masks for Multi-Camera Depth Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional self-supervised depth and ego-motion models are limited in their application to multi-camera settings due to self-occlusions caused by the ego vehicle's body, which prevents optimal camera placement and results in unsuitable data for learning, especially when cameras are placed on streamlined roof racks for aesthetics or aerodynamics.
Innovation Solution
A method that involves generating a hard-coded binary mask by manually segmenting self-occluded areas in images captured by a multi-camera rig, which is then used to mask the photometric loss during training, avoiding gradient back-propagation in self-occluded areas and enabling accurate depth and ego-motion estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If cameras are placed on streamlined roof rack for aesthetics or aerodynamics, then aerodynamic performance and esthetics are improved, but self-occlusion increases making depth estimation unreliable
Solution Approach 1:
The patent segments the loss function computation by introducing self-occlusion masks that divide the image into reliable and unreliable regions. The photometric loss is computed separately for non-occluded and occluded areas, allowing the model to learn from valid regions while excluding corrupted regions from gradient updates.
Solution Approach 2:
The patent applies different quality standards to different regions of the image. Non-occluded regions are used for standard photometric loss computation, while occluded regions are identified and excluded using self-occlusion masks. This local differentiation ensures that only high-quality, reliable image regions contribute to the training signal.
2Measurement precision
If conventional self-supervised methods are used, then forward-facing camera depth estimation is improved, but side-facing and rear-facing cameras with self-occlusions cannot be effectively utilized
Solution Approach 1:
The patent creates a universal self-occlusion masking framework that works across all camera orientations (forward-facing, side-facing, rear-facing). The same photometric loss function with self-occlusion masking can be applied to any camera in the multi-camera rig, making the system adaptable to diverse camera placements and configurations.
Solution Approach 2:
The patent changes the parameters of the loss function by introducing self-occlusion masks that modulate the photometric loss. This parameter modification allows the system to adapt to different camera viewpoints and occlusion patterns, enabling effective training for side and rear cameras that were previously unusable.
3Productivity
If self-occluded areas are included in training, then more data is available for learning, but gradient back-propagation becomes unsuitable leading to poor learning results
Solution Approach 1:
The patent extracts and removes self-occluded regions from the training process by generating binary masks that identify these problematic areas. The photometric loss is then computed only on non-occluded regions, effectively taking out the harmful data portions while retaining useful data for training.
Solution Approach 2:
The self-occlusion mask acts as an intermediary between the raw image data and the loss computation. This intermediate binary mask selectively gates the photometric loss signal, allowing gradients to flow only from reliable, non-occluded regions while blocking corrupted signals from occluded areas.
Data Source
AI summary
A method for self-supervised depth and ego-motion estimation is described. The method includes determining a multi-camera photometric loss associated with a multi-camera rig of an ego vehicle. The method also includes generating a self-occlusion mask by manually segmenting self-occluded areas of images captured by the multi-camera rig of the ego vehicle. The method further includes multiplying the multi-camera photometric loss with the self-occlusion mask to form a self-occlusion masked photometric loss. The method also includes training a depth estimation model and an ego-motion estimation model according to the self-occlusion masked photometric loss. The method further includes predicting a 360° point cloud of a scene surrounding the ego vehicle according to the depth estimation model and the ego-motion estimation model.