Depth Estimation Fusion for Reflective Surfaces
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI-based image processing techniques for 3D reconstruction from 2D image data suffer from artifacts caused by non-Lambertian surfaces, leading to inaccurate depth predictions and false geometry in scenes with reflective surfaces.
Innovation Solution
The technique involves fusing predicted depth and reprojected depth values using a mask based on specific criteria to determine a fused depth, which reduces degradation from reflective surface artifacts and maintains high-frequency details, and retraining the depth estimation model with fused depths as pseudo-labels for improved fine-tuning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If predicted depth from 2D image data is used for 3D reconstruction, then high frequency details are provided, but reflection artifacts and false geometry occur on non-Lambertian surfaces
Solution Approach 1:
The patent introduces reprojected depth from a 3D reconstruction as an intermediary to mediate the harmful reflection artifacts in the predicted depth. The system fuses predicted depth and reprojected depth together, using the reprojected depth as a mediator that provides geometric consistency while reducing reflection artifacts, thereby resolving the contradiction between maintaining high-frequency details and eliminating reflection artifacts
Solution Approach 2:
The patent merges predicted depth and reprojected depth through a fusion process to create a combined depth representation. This merging combines the advantages of both depth sources: the high-frequency details from predicted depth and the artifact-reduced geometry from reprojected depth, resolving the contradiction by integrating multiple depth predictions
2Object-affected harmful factors
If reprojected depth from 3D reconstruction is used, then reflection artifacts are reduced, but high frequency details are lost
Solution Approach 1:
The patent combines reprojected depth with predicted depth through fusion, where the predicted depth contributes high-frequency details that would otherwise be lost. This merging ensures that the final depth representation retains both the artifact-reduced geometry from reprojection and the fine details from the original prediction
Solution Approach 2:
The predicted depth acts as an intermediary that preserves high-frequency details while the reprojected depth provides geometric consistency. By using both as inputs to the fusion process, the system recovers high-frequency details that would be lost in reprojected depth alone, while still benefiting from artifact reduction
3Reliability
If depth estimation model is retrained with fused depths as pseudo-labels, then fine-tuning is improved, but processing complexity increases
Solution Approach 1:
The system performs self-service by using its own output (fused depth from the fusion of predicted and reprojected depth) as training data (pseudo-labels) to retrain and fine-tune the depth estimation model. This self-improving loop enhances reliability without requiring external ground truth data, and the automation of the process manages the complexity through systematic reuse of generated data
Data Source
AI summary
This disclosure provides systems, methods, and devices for image signal processing that support artificial intelligence (AI)-based processing of image data for reconstructing 3D worlds. In a first aspect, a method of image processing includes receiving a plurality of image frames representing a scene; determining a first depth prediction for the scene based on the plurality of image frames; determining a reconstructed mesh from the plurality of image frames; determining a second depth prediction for the scene based on the reconstructed mesh; and determining a third depth prediction based on the first depth prediction and the second depth prediction. Other aspects and features are also claimed and described.