Depth Densification With Learned Kernels and Confidence Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing algorithms struggle to accurately generate high-resolution depth maps from low-resolution depth maps and image data, particularly in scenes with flat or textureless surfaces and low illumination, leading to inaccuracies and noise in reconstructions.
Innovation Solution
A machine-learning model generates kernels based on input image data and depth maps to upsample low-resolution depth maps to high-resolution, accompanied by a confidence map to assess the quality of the output.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional MVS algorithms are used to reconstruct scenes, then the process can handle general scenes, but accuracy deteriorates significantly in scenes with textureless surfaces or low illumination
Solution Approach 1:
The patent introduces an intermediary machine learning model that mediates between the low-resolution depth map and the final high-resolution output. This intermediary component learns from training data to bridge the gap between sparse input data and accurate high-resolution output, specifically addressing the failure of traditional MVS algorithms in textureless and low illumination scenes by transferring learned patterns from training data rather than relying on direct geometric correspondence
Solution Approach 2:
The patent changes the fundamental parameters of the reconstruction approach by using learned priors from diverse training datasets instead of traditional geometric algorithms. The machine learning model parameters are trained to handle challenging scenarios (textureless surfaces, low illumination) by learning from examples, transforming the problem from one of direct geometric computation to one of statistical pattern recognition and synthesis
2Measurement precision
If learning-based algorithms are used to reconstruct depth maps, then accuracy improves in challenging scenes, but computational complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-training the machine learning model on diverse training datasets before actual use. The complex learning-based reconstruction is performed offline during training, creating a ready-to-use model that can quickly generate accurate depth maps during actual application without requiring complex real-time computations, thus reducing operational complexity while maintaining high accuracy in challenging scenes
3Quantity of substance
If low-resolution depth maps are upsampled using traditional methods, then resolution increases, but noise and gaps increase in the reconstruction
Solution Approach 1:
The patent uses copying by creating synthetic training data pairs from high-resolution ground truth depth maps and their corresponding low-resolution versions. The machine learning model learns to copy the patterns and structures from the high-resolution training data and replicate them in the low-resolution input scenarios, effectively synthesizing accurate high-resolution output from sparse input without introducing noise or gaps that would occur with traditional upsampling methods
Data Source
AI summary
A method for generating an output depth map is provided. The method is performed by a computing device. When the computing device executes this method, it receives input image data of a scene, and receives an input depth map of the scene, the input depth map having an input resolution. Using a machine-learning model, the device generates multiple kernels for upsampling the input depth map to produce an output depth map with a higher resolution than the one of the input depth map. These kernels are generated based on the input image data and the input depth map, with each kernel consisting of multiple weights. Subsequently, the computing device applies these kernels to the input depth map to generate the output depth map. Each kernel is applied to a portion of the input depth map to generate a depth value for a pixel of the output depth map.