Correcting 3D Image Shadows Using ToF and RGB Sensor Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
When using an image capturing apparatus with both a ToF sensor and an RGB sensor, disparities in the visible light and infrared light images can lead to shadows being generated in different regions, resulting in unnatural dents in the 3D image generated by combining these images.
Innovation Solution
An image processing apparatus that includes a first image acquisition unit for acquiring visible light images, a second image acquisition unit for acquiring infrared light images from a ToF sensor, a difference extraction unit to detect differences between the two images, a shadow region detection unit to identify shadow regions where shadows are generated differently, and a correction unit to correct the distance information of pixels in these shadow regions based on neighboring pixel information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If visible light image and infrared light image are captured separately by different sensors, then both types of images can be obtained, but disparities and shadows are generated at different regions causing unnatural dents in combined 3D image
Solution Approach 1:
The patent applies preliminary action by detecting shadow regions in the infrared image before combining images with visible light image. The shadow region detection unit identifies areas where infrared shadows differ from visible light shadows, and the correction unit pre-corrects depth values in these regions using depth information from surrounding areas, preventing unnatural dents in the final 3D image.
2Adaptability or versatility
If ToF sensor and RGB sensor are positioned at different locations, then both sensors can function independently, but disparity is generated between images
Solution Approach 1:
The patent uses the shadow region detection and correction mechanism as an intermediary to bridge the disparity caused by different sensor positions. By detecting shadow regions and correcting depth values based on surrounding pixel information, the system compensates for the positional mismatch between ToF and RGB sensors, achieving accurate alignment in the combined 3D image.
3Measurement precision
If infrared light is emitted for distance measurement, then depth information can be obtained, but shadows are generated that do not appear in visible light image
Solution Approach 1:
The patent converts the harmful shadow artifacts generated by infrared light emission into useful information. The shadow region detection unit identifies these artifacts by comparing infrared and visible light images, and the correction unit uses the location and characteristics of these shadow regions to appropriately adjust depth values, transforming the problematic shadows into a basis for improving measurement accuracy.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The solution effectively corrects the depth values of pixels in shadow regions, reducing unnatural dents in the 3D image and improving the accuracy and naturalness of the combined image.
Implementation Method 1
a ToF sensor that uses the ToF method emits distance measuring light based on infrared light toward a subject, and receives the distance measuring light reflected by the subject, by an image capturing element for infrared light. The ToF sensor is capable of calculating the distance between the subject and the image capturing apparatus by detecting a time difference from light emission to light reception on a per-pixel basis.
Data Source
AI summary
An image processing apparatus according to the present disclosure includes a first image acquisition unit configured to acquire a first image captured by a first camera that detects light having a first wavelength, a second image acquisition unit configured to acquire a second image captured by a second camera that measures a distance by detecting light having a second wavelength, a difference extraction unit configured to extract a difference between a visible light image and an infrared light image, a shadow region detection unit configured to detect a shadow region based on the difference, and a correction unit configured to correct distance information of a pixel in the shadow region measured by the second camera, based on the distance information of a pixel in a neighborhood of the shadow region.