Neural Network Depth Completion for Sparse ToF Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Time-of-flight sensors in mobile devices capture sparse depth maps due to power constraints, leading to low resolution and the need for sensor fusion techniques to obtain dense depth maps, which is inefficient.
Innovation Solution
A depth completion method using a neural network model that processes grayscale images and sparse depth maps to generate dense depth maps through nearest neighbor interpolation, Euclidean distance transform, and pixel-level addition, with optimizations like model pruning, clustering, and quantization to reduce power consumption and improve resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If ToF sensor uses fewer projector points for eye safety and low power consumption, then power consumption is reduced, but depth map resolution becomes far lower than color image resolution
Solution Approach 1:
The patent introduces a neural network model as an intermediary that processes the sparse depth map and grayscale image to generate a dense depth map. The neural network acts as a mediator that transforms the low-resolution depth information into high-resolution depth information without requiring additional physical sensors or increasing projector point density, thus maintaining low power consumption while improving resolution.
Solution Approach 2:
The patent creates a dense depth map as a computational copy of the sparse depth map by leveraging the grayscale image information. Instead of physically capturing more depth points, the system generates a high-resolution depth map by copying and refining the sparse depth information using neural network-based super-resolution techniques.
2Use of energy by moving object
If ToF sensor captures only 500-1500 dots per frame, then power consumption is reduced, but sensor fusion techniques are required which increase system complexity
Solution Approach 1:
The patent merges the depth information from the sparse depth map with the visual information from the grayscale image within a unified neural network framework. By combining these two data sources in the neural network, the system achieves dense depth map generation without requiring complex external sensor fusion techniques, thereby reducing overall system complexity while maintaining low power consumption.
3Measurement precision
If neural network model processes full-resolution images, then depth map quality is improved, but computational load and power consumption increase
Solution Approach 1:
The patent segments the processing task into two stages: first processing the sparse depth map at its native low resolution to identify key depth structures, then applying neural network refinement only to generate the dense depth map at full resolution. This segmentation allows the system to maintain high depth map quality while reducing overall computational load compared to processing full-resolution images from the start.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The method effectively generates high-precision, high-resolution dense depth maps, enhancing 3D perception and supporting tasks like 3D object detection and semantic segmentation on resource-limited devices.
Implementation Method 1
Time-of-flight (ToF) sensors are active depth sensing devices with the potential to provide more reliable scene understanding by true 3D perception
Data Source
AI summary
A depth completion method of sparse depth map includes: acquiring a grayscale image and a sparse depth map corresponding to the grayscale image; obtaining a nearest neighbor interpolation (NNI) image and a Euclidean distance transform (EDT) image based on the sparse depth map; inputting the grayscale image, the NNI image, and the EDT image into a neural network model, thereby outputting a predicted residual map; and generating a predicted dense depth map according to the predicted residual map and the NNI image.