Cross-Attention Depth Estimation for Dynamic and Occluded Scenes

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing multi-frame depth estimation networks face accuracy issues with dynamic objects, low texture areas, and occluded objects due to reliance on cost volumes that are noisy and inaccurate for feature matching.

Innovation Solution

Implement a self-supervised learning approach using a cross-attention model trained with a photometric loss from a single-frame depth estimation model to generate a cross-attention cost volume for improved depth estimation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a multi-frame network uses cost volumes to estimate depth, then depth estimation can be performed using multiple images, but the accuracy deteriorates for dynamic objects, low texture areas, and occluded objects due to noisy cost volumes

Engineering Contradiction:

Improvemulti-frame depth estimation capabilityVSAvoiddepth estimation accuracy

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent introduces photometric masks as an intermediary component that mediates between the cost volume and the depth estimation network. These masks selectively weight different regions of the cost volume based on photometric consistency, filtering out noisy regions (dynamic objects, low texture areas, occluded objects) while preserving reliable information. This allows the multi-frame network to benefit from temporal information without being degraded by unreliable feature matches in problematic regions.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If a single-frame network is used as a teacher for the multi-frame network, then the accuracy of depth estimates improves, but the complexity of the training process increases

Engineering Contradiction:

Improvedepth estimation accuracyVSAvoidtraining process complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent implements a teacher-student training framework where a single-frame depth estimation network serves as the teacher and guides the multi-frame network (student). The teacher network provides supervision signals that feedback to the student network during training, enabling the student to learn from both the teacher's predictions and the photometric loss. This feedback mechanism allows the complex multi-frame network to be trained effectively using the simpler single-frame network as a guide.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The single-frame teacher network is trained first before being used to supervise the multi-frame student network. This preliminary training establishes a baseline depth estimation capability that provides initial supervision signals. By preparing the teacher network in advance, the training process for the student network can proceed more systematically, reducing the overall training complexity compared to training both networks simultaneously from scratch.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12428022B2Photometric masks for self-supervised depth learning

Publication Date: 2025.09.30 TOYOTA JIDOSHA KK

AI summary

A method estimating a depth of an environment includes generating, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images. The method also includes generating, via the cross-attention model, a depth estimate of the current image based on the cross-attention cost volume, the cross-attention model having been trained using a photometric loss associated with a single-frame depth estimation model. The method further includes controlling an action of the vehicle based on the depth estimate.