Depth Estimation Device Composites ToF and Image Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing depth estimation methods face challenges in producing high-quality depth maps due to missing pixels in ToF sensor data and inconsistencies in depth estimation models based on deep neural networks, leading to unnatural boundaries and reduced accuracy.
Innovation Solution
A depth estimation device composites depth maps from ToF sensors and image-based depth estimation models using a cost function with constraints that prioritize using existing distance values from ToF data for complete pixels and image-based values for incomplete pixels, while smoothing adjacent pixel values to enhance accuracy and consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth maps are acquired using TOF sensors, then accurate distance values are obtained, but many pixels become missing
Solution Approach 1:
The patent combines TOF sensor depth maps with image-based depth estimation models to create a composite depth map. The TOF sensor provides accurate distance values where available, while the image-based model fills in missing pixel regions, achieving both accuracy and completeness.
Solution Approach 2:
The patent applies different quality standards to different regions of the depth map. Areas with TOF data use accurate distance values, while areas without TOF data use image-based estimation, allowing each region to have the appropriate quality level for its data source.
2Stability of the object's composition
If depth estimation models based on deep neural networks are used, then consistent depth maps are output, but accurate distance values and fine textures cannot be obtained
Solution Approach 1:
The patent merges image-based depth estimation (which provides consistency) with TOF sensor data (which provides accuracy) to achieve both properties simultaneously in the composite depth map.
Solution Approach 2:
The patent creates a composite depth map that integrates multiple data sources (TOF sensor and image-based model) with different characteristics, similar to how composite materials combine materials with different properties to achieve superior overall performance.
3Loss of information
If missing pixels are supplemented using image-based depth maps, then complete depth maps are obtained, but unnatural boundaries appear between pixel regions
Solution Approach 1:
The patent uses a cost function with smoothing parameters that control the transition between different data sources. By adjusting smoothing strength and reliability weights, the patent achieves natural transitions that avoid unnatural boundaries while maintaining complete depth map coverage.
Solution Approach 2:
The patent employs a cost function that evaluates and optimizes the composite depth map based on multiple criteria including smoothness and data reliability, allowing iterative refinement to achieve natural transitions at boundaries.
4Measurement precision
If TOF sensor depth maps are directly used, then accurate distance values are obtained, but fine textures are lost
Solution Approach 1:
The patent combines TOF sensor data (providing accurate distance values) with image-based depth estimation (providing fine texture information) to create a composite depth map that preserves both properties.
Solution Approach 2:
The patent maintains different characteristics in different regions: TOF data regions provide accurate distance values while image-based regions provide fine texture information, with smooth transitions between regions.
Data Source
AI summary
Examples of a depth estimation method include acquiring a plurality of depth maps, and outputting one output depth map obtained by compositing the plurality of depth maps with a lower average difference between distance values of adjacent pixels than in a case of directly using distance values included in the plurality of depth maps.