ToF Depth Map Enhancement via Neural Network and Color Guidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current time-of-flight sensors suffer from low spatial resolution, multi-path reflections, and difficulties with strong ambient light, limiting their use in high-resolution and accurate 2D and 3D computer vision applications.
Innovation Solution
An image processing system that utilizes a trained artificial intelligence model, specifically an end-to-end trainable neural network, to generate an improved time-of-flight depth map by enriching the input depth map with features from the RAW correlation signal and processing it with co-modality guidance from aligned color images, thereby increasing resolution, accuracy, and precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional time-of-flight sensors are used to generate depth maps, then the computation can be done comparably fast, but the spatial resolution is very limited and depth accuracy is poor
Solution Approach 1:
The patent segments the depth estimation task into multiple stages: first using fast ToF correlation computation for initial depth mapping, then applying selective refinement only to regions requiring improvement. This segmentation allows the system to maintain overall computational speed while improving depth accuracy in critical areas through targeted processing.
Solution Approach 2:
The patent transforms the problem from 2D pixel-wise depth estimation to 3D spatial reasoning by leveraging depth information across multiple resolutions and applying geometric constraints. This dimensional transition enables the system to recover accurate depth measurements without requiring purely computational approaches, thus improving accuracy without sacrificing speed.
2Reliability
If time-of-flight sensors are used for depth mapping, then low-light scenarios are handled well, but multi-path reflections and strong ambient light cause measurement errors
Solution Approach 1:
The patent introduces an intermediary processing stage that mediates between the raw ToF correlation data and the final depth map. This intermediate layer filters out erroneous measurements caused by multi-path reflections and ambient light by comparing ToF data with complementary vision data, thus improving reliability without requiring the sensor itself to be modified.
Solution Approach 2:
The patent replaces the purely optical measurement mechanism with a hybrid approach that uses computational algorithms to correct optical measurement errors. By substituting post-processing computation for sensor-level filtering, the system effectively eliminates the impact of harmful factors like multi-path reflections and ambient light interference.
3Measurement precision
If the depth map resolution is increased, then more detail is captured, but the image alignment process causes additional information loss
Solution Approach 1:
The patent performs preliminary actions to preserve information before alignment occurs. By pre-processing the depth map to enhance key features and boundaries, and by preparing multiple resolution levels in advance, the system minimizes information loss that would otherwise occur during the necessary alignment process, thus maintaining high depth resolution.
4Measurement precision
If deep learning methods are applied to enrich ToF data, then resolution and accuracy can be improved, but the computational complexity and processing time increase
Solution Approach 1:
The patent applies local quality by implementing selective deep learning processing only in regions where the depth map requires enhancement. Rather than processing the entire image uniformly, the system identifies and refines only the critical regions (such as boundaries and texture-rich areas), thus improving accuracy without proportionally increasing overall computational complexity.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The system effectively corrects depth errors, recovers missing data, and resolves multi-path ambiguities, resulting in a higher-resolution time-of-flight depth map that enhances image rendering and user experience.
Implementation Method 1
By illuminating a scene with a pulsed light source, the light is reflected by objects in the scene and its roundtrip time is measured. Using the measured roundtrip time, and with knowledge of the speed of light, one can estimate the distance of reflecting objects from the camera.
Implementation Method 2
the light is reflected by objects in the scene and its roundtrip time is measured
Data Source
AI summary
An image processing system configured to receive an input time-of-flight depth map representing the distance of objects in an image from a camera at a plurality of locations of pixels in the respective image, and in dependence on that map to generate an improved time-of-flight depth map for the image, the input time-of-flight depth map having been generated from at least one correlation image representing the overlap between emitted and reflected light signals at the plurality of locations of pixels at a given phase shift, the system being configured to generate the improved time-of-flight depth map from the input time-of-flight depth map in dependence on a colour representation of the respective image and at least one correlation image.