Attribute-Specific Joint Learning for Depth Estimation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional methods for image depth estimation, such as dual camera setups, multi-view stereo matching, and dual-pixel camera sensors, are costly and complex, and deep neural network-based single image depth estimation faces challenges like sparse annotated datasets, laborious data annotation, and difficulty in preserving sharp edges and gaps.

Innovation Solution

A learning-based model is trained using attribute-specific loss values to perform tasks like depth estimation, media restoration, and semantic segmentation by grouping training datasets based on attributes like scene understanding, depth correctness, and sharp edges, using view construction, pixelwise L1/L2, and gradient difference losses.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional methods like dual camera setups or multi-view stereo matching are used for depth estimation, then measurement precision is improved, but device complexity and cost increase

Engineering Contradiction:

Improvedepth estimation accuracyVSAvoidhardware setup complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent replaces complex mechanical hardware systems (dual camera setups, multi-view stereo matching equipment) with a software-based deep neural network model that performs depth estimation from single images, eliminating the need for specialized hardware while maintaining depth measurement capability

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent uses synthetic depth maps generated from rendered images as training data, creating artificial copies of real-world depth information that can be used to train the neural network without requiring complex physical measurement setups

Inventive Principle:

Principle #26Copying

2Device complexity

If DNN-based single image depth estimation is used, then device complexity is reduced, but manufacturing precision deteriorates due to sparse annotated datasets

Engineering Contradiction:

Improvehardware setupVSAvoiddepth map quality

Core Design Contradiction:

Device complexityVSManufacturing precision

Solution Approach 1:

The patent performs preliminary data preparation by generating synthetic training datasets with ground truth depth maps through image rendering before training the neural network, ensuring high-quality training data is available in advance to achieve accurate depth estimation

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent combines multiple data sources including synthetic rendered images, real-world photographs, and sparsely annotated depth maps into a composite training dataset, leveraging the strengths of each source to improve overall model performance and depth map quality

Inventive Principle:

Principle #40Composite materials

3Ease of manufacture

If sparsely annotated ground truth is used for training, then ease of manufacture is improved, but measurement precision deteriorates as output quality depends on ground truth quality

Engineering Contradiction:

Improvedata annotation effortVSAvoiddepth estimation accuracy

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The patent introduces synthetic rendered images as an intermediary training source that provides complete, accurate ground truth depth maps, mediating between the ease of using sparse real data and the need for high-precision training signals

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent segments the training dataset into multiple components (synthetic rendered images, real photographs, sparsely annotated data) and processes them through different training strategies, allowing each segment to contribute its strengths to the overall model performance

Inventive Principle:

Principle #1Segmentation

4Device complexity

If dual-pixel camera sensor technique is used, then device complexity is reduced, but measurement precision deteriorates due to small pixel subline

Engineering Contradiction:

Improvecamera setupVSAvoiddepth map resolution

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent replaces hardware-based depth sensing methods (dual-pixel camera sensors) with a computational approach using deep neural networks that can achieve high-resolution depth maps without being constrained by physical pixel subline limitations

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS11790228B2Methods and systems for performing tasks on media using attribute specific joint learning

Publication Date: 2023.10.17 SAMSUNG ELECTRONICS CO LTD

AI summary

A learning-based model is trained using a plurality of attributes of media. Depth estimation is performed using the learning-based model. The depth estimation supports performing a computer vision task on the media. Attributes used in the depth estimation include scene understanding, depth correctness, and processing of sharp edges and gaps. The media may be processed to perform media restoration or the media quality enhancement. A computer vision task may include semantic segmentation.