Monocular Camera Depth Estimation Using Bokeh Statistical Models

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing monocular camera systems face challenges in accurately estimating distances to subjects, particularly in regions where depth cannot be reliably calculated, leading to incomplete grasping of three-dimensional structures, and require additional devices like LiDAR, which are costly and difficult to integrate.

Innovation Solution

An image processing device that uses a first statistical model to estimate bokeh values and uncertainty levels from images affected by optical aberrations, generating a bokeh map and interpolating uncertain regions using a second statistical model to improve depth estimation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If distance estimation is performed using a monocular camera, then cost and installation complexity are reduced, but measurement precision and reliability deteriorate in certain regions

Engineering Contradiction:
Improveinstallation complexityVSAvoiddistance estimation precision
Core Design Contradiction:
Device complexityVSMeasurement precision

Solution Approach 1:

The patent introduces an intermediary statistical model that acts as a mediator between the monocular camera and the final distance estimation. The model learns the relationship between bokeh characteristics and distance, providing reliable depth information in regions where direct estimation fails, thus resolving the contradiction between simple device installation and precise measurement.

Inventive Principle:
Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent creates a virtual depth map through statistical modeling that copies the depth information pattern learned from training data. This virtual depth map serves as a substitute for direct physical measurement, enabling accurate distance estimation in regions where the monocular camera alone would fail, while maintaining the simplicity of the monocular setup.

Inventive Principle:
Principle #26Copying

2Measurement precision

If LiDAR is used to improve distance estimation accuracy, then measurement precision improves, but device complexity and cost increase

Engineering Contradiction:
Improvedistance estimation precisionVSAvoiddevice complexity
Core Design Contradiction:
Measurement precisionVSDevice complexity

Solution Approach 1:

The patent replaces the mechanical LiDAR system with a computational approach using statistical models and image processing. Instead of using active light measurement hardware, the system substitutes a learned statistical relationship between bokeh and distance, achieving similar precision goals without the complexity and cost of LiDAR hardware integration.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent uses a computationally lightweight statistical model that can be trained once and then efficiently applied to multiple images. This approach is more economical and simpler than continuously operating LiDAR hardware, providing a cost-effective solution that maintains measurement precision without requiring expensive, complex distance measuring devices.

Inventive Principle:
Principle #27Cheap short-living objects (Disposable)

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

The solution enables accurate and reliable estimation of three-dimensional structures by enhancing depth calculation in challenging regions, reducing the need for costly additional devices like LiDAR and improving the integration of depth estimation with monocular cameras.

Implementation Method 1

bokeh which occurs in a first image affected by aberration of a first optical system and varies non-linearly in accordance with a distance to a subject in the first image

Methodology Applied
Scientific EffectBokeh: Depth of Field

Data Source

PatentUS11868430B2Image processing device, method, and storage medium
Publication Date: 2024.01.09 KK TOSHIBA
  • US11868430B2 patent drawing
  • US11868430B2 patent drawing
  • US11868430B2 patent drawing

AI summary

According to one embodiment, an image processing device includes first storage and a processor. The first storage is configured to store a first statistical model generated by learning bokeh which occurs in a first image. The processor is configured to acquire a second image, acquire a bokeh value which indicates bokeh occurring in the second image and an uncertainty level which indicates a level of uncertainty for the bokeh value, generate a first bokeh map based on the bokeh value and uncertainty level, and acquire a second bokeh map obtained by interpolating a bokeh value with respect to the first bokeh map. The bokeh value and the uncertainty level are output from the first statistical model by inputting the second image into the first statistical model.