ToF Depth Camera Fusing Sparse Data with RGB for Dense Imaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current terminal devices, such as mobile phones, cannot accurately acquire depth information, limiting their usage scenarios due to the high cost and complexity of sensors like lidar, necessitating a lower-cost solution for depth image acquisition.
Innovation Solution
An apparatus and method that utilize a speckle array emitted to an object, captured by a CMOS photodiode-based image sensor, and processed to generate a sparse depth image, which is then fused with an RGB image using a pre-trained image fusion model to produce a dense depth image, reducing costs and enhancing precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If lidar sensor is used to obtain depth image, then depth information accuracy is improved, but device cost increases
Solution Approach 1:
The patent replaces expensive lidar sensors with a low-cost depth camera based on IToF technology. The depth camera uses inexpensive components including a speckle light source, image sensor, and processing unit, making it suitable for mass production in consumer electronics while maintaining adequate depth measurement capabilities for applications like background blur and 3D reconstruction
Solution Approach 2:
The patent creates a cost-effective copy of depth sensing capability by implementing IToF technology in a depth camera that mimics the functional output of lidar. The system captures depth information through light propagation time measurement, producing depth images that serve the same purpose as lidar data but at a fraction of the cost, enabling widespread adoption in mobile devices
2Ease of manufacture
If sparse depth image is used to reduce cost, then device cost is reduced, but image resolution deteriorates
Solution Approach 1:
The patent performs preliminary alignment of the sparse depth image with the high-resolution RGB image before fusion. The processing unit aligns the depth image to the RGB image based on spatial correspondence, ensuring that depth information from the low-resolution depth camera is properly registered with the high-resolution color image, preparing the data for effective fusion that will enhance the final depth map resolution
Solution Approach 2:
The patent merges the sparse depth image with the high-resolution RGB image through image fusion. The processing unit combines the depth information from the low-cost depth camera with the detailed spatial information from the RGB image, producing a dense depth image that achieves high resolution without requiring an expensive high-resolution depth sensor. This fusion integrates the advantages of both image sources to overcome the resolution limitation of the sparse depth input
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces the cost of depth image acquisition, improves image fusion precision, and enriches the usage scenarios of terminal devices by providing accurate depth information, making it suitable for low-cost devices like mobile phones.
Implementation Method 1
the photodiode is configured to receive the speckle array reflected by the object, and generate a corresponding photocurrent signal based on the speckle array, a current intensity indicated by the photocurrent signal is positively correlated with a light intensity of light beam irradiation received by the photodiode
Implementation Method 2
acquiring a sparse depth image at a resolution of p using the depth camera based on IToF principle
Data Source
AI summary
Provided are an apparatus for acquiring a depth image, a method for fusing depth images, and a terminal device. The apparatus for acquiring a depth image includes an emitting module, a receiving module, and a processing unit. The emitting module is configured to emit a speckle array to an object, where the speckle array includes p mutually spaced apart speckles. The receiving module includes an image sensor. The processing unit is configured to receive the pixel signal and generate a sparse depth image based on the pixel signal, align an RGB image at a resolution of a*b with the sparse depth image, and fuse the aligned sparse depth image with the RGB image using a pre-trained image fusion model to obtain a dense depth image at a resolution of a*b.