Depth Value Recovery Using MRF Energy Minimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Time-of-Flight (ToF) depth cameras face challenges in accurately calculating depth values for objects beyond the maximum measurable distance, leading to incorrect or insufficient depth values due to phase wrapping, which affects the precision of depth imaging.
Innovation Solution
The method involves using multiple ToF depth cameras to capture images from different locations and directions, determining the number of mods (NoM) for each pixel to minimize Markov random field (MRF) energy, and updating depth values based on data and discontinuity costs to recover accurate depth values, addressing phase ambiguity and improving depth image consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single ToF depth camera is used to measure depth, then the measurement process is simple, but the measurable distance is limited and depth values become incorrect beyond the maximum measurable distance
Solution Approach 1:
The patent combines multiple ToF depth cameras to capture depth images from different locations and directions. By merging the data from multiple cameras, the system extends the measurable distance range and resolves phase wrapping ambiguities through multi-view consistency, thereby improving depth value accuracy for distant objects.
Solution Approach 2:
The patent introduces an MRF energy minimization framework as an intermediary processing step. This framework uses data costs and discontinuity costs to resolve ambiguities in phase unwrapping by finding the most consistent depth values across multiple views, effectively mediating between the limited range of individual cameras and the need for extended measurement capability.
2Length of stationary object
If phase unwrapping is performed to extend measurable distance, then the measurement range increases, but errors and ambiguities increase without proper constraints
Solution Approach 1:
The patent employs MRF energy minimization with data costs and discontinuity costs as feedback mechanisms. The data cost ensures consistency with observed phase values, while the discontinuity cost penalizes unrealistic depth variations between neighboring pixels. This feedback loop iteratively refines depth estimates to achieve both extended range and high reliability.
Solution Approach 2:
The patent changes the parameter space by introducing NoM (number of mods) as an additional variable to be determined. Instead of directly unwrapping phase, the system jointly optimizes both the depth value and the number of phase wraps, allowing flexible adaptation to different distance ranges while maintaining consistency through the MRF framework.
3Reliability
If multiple depth images from different cameras are processed, then depth consistency improves, but computational complexity increases
Solution Approach 1:
The patent segments the processing task by first determining NoM values for each pixel independently through MRF energy minimization, then using these NoM values to recover depth values. This segmentation of the computational process reduces overall complexity compared to simultaneous optimization of all parameters.
Solution Approach 2:
The patent performs preliminary determination of NoM values before final depth recovery. By pre-determining the number of phase wraps using MRF energy minimization on the first depth image, the system simplifies the subsequent depth recovery process and reduces computational burden during iterative refinement.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enables the recovery of accurate and consistent depth values for pixels across multiple cameras, extending the measurable range and reducing errors caused by phase wrapping, resulting in improved depth image quality.
Implementation Method 1
The ToF depth camera may calculate a distance between the ToF depth camera and an object by measuring a time taken for the IR light from the LED to bounce off the object and return to the sensor
Implementation Method 2
The IR light may be modulated with a frequency f. That is, the IR light may have a frequency f. TOF time-of-flight may be calculated indirectly by measuring an intensity of light returning to the sensor based on two or four phases
Data Source
AI summary
A method and apparatus for processing a depth image determines a number of mods (NoM) for corresponding pixels in a plurality of depth images. The corresponding pixels may represent a same three-dimensional (3D) point. The NoM may be determined to be a value for minimizing a Markov random field (MRF) energy. A depth value for one depth image may be recovered, and a depth value for another depth image may be updated using the recovered depth value.