Depth Estimation Using Semantic Features

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Monocular images fail to accurately determine depth due to pixel-based metrics, leading to difficulties in capturing geometric structures, especially for thin or small objects, resulting in reduced situational awareness and navigation challenges for robotic devices.

Innovation Solution

A depth system that incorporates a machine learning model with a novel architecture, using semantic features injected through pixel-adaptive convolutions to guide depth estimates, improving the processing of monocular images by distinguishing object boundaries and avoiding issues like bleeding and missed small objects.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If pixel-based metrics are used to derive depth from monocular images, then processing simplicity is maintained, but depth accuracy and geometric structure capture deteriorate

Engineering Contradiction:

Improveprocessing simplicityVSAvoiddepth accuracy

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The patent introduces semantic features as an intermediary between monocular images and depth estimation. These semantic features act as a mediator that guides the depth model to focus on geometric structures and object boundaries, thereby improving depth accuracy without complicating the processing pipeline. The semantic features are extracted through semantic segmentation and then injected into the depth model via guiding connections.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent transitions from traditional pixel-based 2D depth estimation to a multi-dimensional approach by incorporating semantic features that add contextual understanding. This dimensionality change allows the system to consider not just pixel intensity and position, but also semantic meaning and object relationships, significantly improving geometric structure capture and depth accuracy.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Use of energy by moving object

If traditional depth estimation methods are used, then computational resources are conserved, but ability to capture thin or small objects deteriorates

Engineering Contradiction:

Improvecomputational resource consumptionVSAvoidobject detection reliability

Core Design Contradiction:

Use of energy by moving objectVSReliability

Solution Approach 1:

The patent applies local quality by using pixel-adaptive convolutions that selectively process different regions of the image with different levels of attention. Thin or small objects receive enhanced processing through guiding connections that inject semantic features specifically at locations where these objects are likely to occur, improving detection reliability without uniformly increasing computational resources across the entire image.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent performs preliminary semantic segmentation to identify potential objects and their boundaries before conducting depth estimation. This preliminary action prepares guiding information that directs the depth model to focus computational resources on regions containing thin or small objects, thereby improving object detection reliability while optimizing computational resource consumption.

Inventive Principle:

Principle #10Preliminary action

3Measurement precision

If semantic features are injected into the depth model, then depth estimation accuracy improves, but device complexity increases

Engineering Contradiction:

Improvedepth estimation accuracyVSAvoidmodel architecture complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments the deep learning model into distinct functional components: a semantic segmentation module that extracts semantic features, a depth estimation module that processes monocular images, and guiding connections that inject semantic features into the depth model. This segmentation allows each component to be optimized independently and simplifies the overall system design while maintaining high depth estimation accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements a multi-functional architecture where the semantic segmentation module serves dual purposes: it provides semantic labels for classification tasks and simultaneously generates guiding features for depth estimation. This universality reduces device complexity by reusing the same semantic feature extraction process for multiple objectives rather than requiring separate specialized modules.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11321863B2Systems and methods for depth estimation using semantic features

Publication Date: 2022.05.03 TOYOTA JIDOSHA KK

AI summary

Systems, methods, and other embodiments described herein relate to generating depth estimates of an environment depicted in a monocular image. In one embodiment, a method includes identifying semantic features in the monocular image according to a semantic model. The method includes injecting the semantic features into a depth model using pixel-adaptive convolutions. The method includes generating a depth map from the monocular image using the depth model that is guided by the semantic features. The pixel-adaptive convolutions are integrated into a decoder of the depth model. The method includes providing the depth map as the depth estimates for the monocular image.