Depth Reconstruction Using Multi-View Plane Sweeping

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

A single depth capture unit using structured light techniques is unable to produce a complete depth representation of an object due to limitations in capturing all portions, leading to 'holes' in the final depth scene caused by overlapping projected patterns from multiple units.

Innovation Solution

A depth construction module that utilizes a multi-view reconstruction technique, specifically plane sweeping, to identify and fill in deficient portions in depth images by jointly processing images from multiple depth capture units, leveraging constraint relationships and confidence scores to generate accurate depth information.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Area of stationary object

If multiple depth capture units are used to capture different parts of an object, then the coverage of the depth scene is improved, but overlapping projected patterns cause interpretation failures and holes in the depth image

Engineering Contradiction:

Improvecoverage area of depth sceneVSAvoiddepth information accuracy

Core Design Contradiction:

Area of stationary objectVSReliability

Solution Approach 1:

The patent converts the harmful overlapping projected patterns into beneficial multi-view constraints. By treating the overlapping patterns as additional observation angles rather than interference, the system uses plane sweeping to evaluate multiple surface position hypotheses, where each view (including overlapping regions) provides complementary constraints that help resolve ambiguities and fill holes in the depth scene.

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Solution Approach 2:

The patent transitions from single-view depth interpretation to multi-view joint reconstruction by introducing the dimension of multiple observation angles. The plane sweeping technique evaluates hypothetical surface positions from multiple views simultaneously, adding angular dimensionality to the reconstruction process and enabling reliable depth estimation in overlapping regions where single-view methods fail.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Device complexity

If a single depth capture unit is used, then the system complexity is reduced, but it cannot produce a complete depth representation of the entire object

Engineering Contradiction:

Improvenumber of depth capture unitsVSAvoidcoverage area of depth scene

Core Design Contradiction:

Device complexityVSArea of stationary object

Solution Approach 1:

The patent merges the information from multiple depth capture units through the depth construction module, which jointly processes captured images from multiple views. By combining the observations from multiple units and applying plane sweeping across all views, the system achieves complete object coverage while managing the complexity through unified processing rather than independent processing of each unit.

Inventive Principle:

Principle #5Merging (Combining)

3Area of stationary object

If overlapping projected patterns are present, then more areas can be illuminated, but the depth capture units fail to correctly interpret the captured images

Engineering Contradiction:

Improveilluminated areaVSAvoiddepth measurement accuracy

Core Design Contradiction:

Area of stationary objectVSMeasurement precision

Solution Approach 1:

The patent introduces plane sweeping as an intermediary computational process between pattern projection and depth interpretation. Instead of directly interpreting overlapping patterns, the system uses plane sweeping to generate and evaluate multiple surface position hypotheses, mediating the transformation from ambiguous overlapping patterns to reliable depth measurements by considering constraints from multiple views.

Inventive Principle:

Principle #24Intermediary (Mediator)

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

This approach effectively combines depth information from multiple units to create a more comprehensive and accurate depth scene, reducing errors and omissions caused by overlapping patterns, thereby enhancing the completeness of the depth representation.

Implementation Method 1

The projector projects a pattern onto the surface of the object, e.g., using infrared-spectrum light

Methodology Applied

Scientific EffectInfrared radiation: Infrared Radiation

Data Source

PatentUS9536312B2Depth reconstruction using plural depth capture units

Publication Date: 2017.01.03 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A depth construction module is described that receives depth images provided by two or more depth capture units. Each depth capture unit generates its depth image using a structured light technique, that is, by projecting a pattern onto an object and receiving a captured image in response thereto. The depth construction module then identifies at least one deficient portion in at least one depth image that has been received, which may be attributed to overlapping projected patterns that impinge the object. The depth construction module then uses a multi-view reconstruction technique, such as a plane sweeping technique, to supply depth information for the deficient portion. In another mode, a multi-view reconstruction technique can be used to produce an entire depth scene based on captured images received from the depth capture units, that is, without first identifying deficient portions in the depth images.