Photometric Masks for Multi-Frame Depth Estimation Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-frame depth estimation networks face accuracy issues with dynamic objects, low texture areas, and occluded objects due to reliance on cost volumes, which can lead to noisy and inaccurate feature matching.
Innovation Solution
Implement a self-supervised learning approach using a cross-attention model trained with a photometric loss from a single-frame depth estimation model to generate a cross-attention cost volume, improving depth estimation accuracy for dynamic and textureless objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If cost volumes are used for multi-frame depth estimation, then depth estimation can be performed using multiple images, but accuracy deteriorates for dynamic objects, low texture areas, and occluded objects due to noisy feature matching
Solution Approach 1:
The patent introduces photometric masks as an intermediary element that mediates between the cost volume computation and the final depth estimation. These masks selectively weight or suppress contributions from unreliable pixel pairs (those involving dynamic objects, low texture areas, or occlusions), thereby improving depth accuracy while preserving multi-frame estimation capabilities. The masks act as a filtering mechanism that allows the system to leverage multi-frame information where reliable and ignore it where unreliable.
2Extent of automation
If photometric loss from single-frame model is used for training cross-attention model, then self-supervised learning is achieved, but training complexity increases
Solution Approach 1:
The patent implements self-service by enabling the cross-attention model to train itself using photometric loss computed from its own depth predictions. The system generates its own training signals by comparing reconstructed images against original images, eliminating the need for external ground truth depth data. This self-supervised approach automates the training process while managing complexity through efficient loss computation and iterative refinement.
Data Source
AI summary
A method of estimating a depth of an environment includes receiving a current image and a previous image of the environment in a sequence of images. The method also includes extracting current image features from the current image and previous image features from the previous image using a feature extraction network. The method further includes generating a correspondence representation based on comparing the current image features to the previous image features, the correspondence representation encoding spatial relationships for depth estimation. The method also includes generating a depth estimate of the current image based on the correspondence representation. The method further includes controlling an operation of an agent based on the depth estimate.