ToF Depth Map Resolution Enhancement via RGB Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Time-of-Flight (ToF) cameras have low image resolution, limiting their effectiveness in applications such as collision avoidance and navigation for mobile platforms like Unmanned Aerial Vehicles (UAVs) and robots.
Innovation Solution
A method and apparatus that enhance image resolution by combining images from a ToF camera and an RGB camera, calibrating their coordinates, converting images into intensity and grayscale formats, calculating a degradation model, and applying a transformation matrix to improve the resolution of depth maps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a ToF camera is used for depth perception and collision avoidance, then the camera achieves high speed image capturing and depth perception capability, but the image resolution remains low which limits its effectiveness
Solution Approach 1:
The patent combines images from a ToF camera and an RGB camera through coordinate calibration and image fusion. The ToF camera provides depth information while the RGB camera provides high-resolution texture information. By merging these complementary data sources, the system achieves both high-speed depth perception and high image resolution in the fused output image.
Solution Approach 2:
The patent introduces an image fusion algorithm as an intermediary process that takes low-resolution depth maps from the ToF camera and high-resolution images from the RGB camera, then produces a fused image with enhanced resolution. This intermediary processing step resolves the contradiction by transforming data from two different sources into a unified high-quality output.
2Measurement precision
If the ToF camera resolution is increased to improve depth recognition, then the image resolution improves, but the device complexity and cost increase
Solution Approach 1:
The patent divides the imaging function into two separate cameras with specialized roles: the ToF camera captures depth information at lower resolution, while the RGB camera captures high-resolution texture information. This segmentation allows each camera to be optimized for its specific function rather than requiring a single complex high-resolution depth camera.
Solution Approach 2:
The patent merges the output from two relatively simple cameras (low-res ToF and high-res RGB) to achieve the equivalent performance of a complex high-resolution depth camera. This approach reduces device complexity by using two specialized components rather than one highly complex component.
3Measurement precision
If images from multiple cameras are combined and processed through calibration and degradation modeling, then the final depth map resolution is enhanced, but the processing time and computational complexity increase
Solution Approach 1:
The patent performs coordinate calibration and establishes degradation models in advance during an initialization phase. By pre-computing the transformation relationships between camera coordinate systems and characterizing the degradation patterns, the system reduces the computational burden during real-time operation, thus minimizing processing time delays.
Solution Approach 2:
The patent replaces complex real-time iterative optimization algorithms with pre-computed degradation models and transformation matrices. Instead of performing heavy computational operations during real-time image fusion, the system uses predetermined mathematical models to efficiently enhance resolution, significantly reducing processing time.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The solution provides higher resolution depth maps, enhancing the effectiveness of collision avoidance and navigation systems by improving the image resolution of ToF cameras.
Implementation Method 1
A Time-of-Flight (or 'ToF') camera is a range imaging camera based on measuring a Time-of-Flight of a light signal emitting from a camera toward a scene. Light reflected back from the scene has a delayed phase relative to the emitted light signal. The ToF camera relies on measuring the phase shift of the reflected light relative to the emitted light to calculate a distance between the camera and the scene to achieve environmental depth perception.
Data Source
AI summary
An imaging apparatus includes first and second imaging devices configured to capture first and second images of a scene, respectively. The first and second images include multiple first image blocks relative to a first coordinate system and multiple second image blocks relative to a second coordinate system, respectively. The apparatus further includes a processor configured to calibrate one or more first image blocks and one or more corresponding second image blocks using the first and second coordinate systems, convert each calibrated first image block to an intensity image and a first depth map, convert each calibrated second image block to a grayscale image, and generate a second depth map associated with the second image by enhancing a resolution of the first depth map for each calibrated first image block based on calculating a relationship between the intensity image and the grayscale image for each calibrated first and second image blocks.