Dark Flash Normal Camera for Poor-Light Image Relighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for relighting poorly-lit images struggle to accurately estimate surface normals and albedo maps, especially in mobile photography, as they are not well-suited for mobile photography situations where ground truth reflectance maps are often unavailable.
Innovation Solution
Utilizing a combination of RGB and NIR images captured from the same perspective, a prediction engine is trained to estimate surface normals and reflectance maps, incorporating techniques like semantic segmentation, stereo depth, and photometric shading cues to generate robust albedo and surface normal maps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional methods are used to estimate surface normals and albedo maps in poor lighting conditions, then the process is simpler, but the quality and accuracy of the estimated maps deteriorate
Solution Approach 1:
The patent combines RGB and NIR imaging systems into a unified calibration and processing framework. The NIR images are captured simultaneously with RGB images, and both are processed together through a joint calibration procedure that estimates surface normals and albedo maps by leveraging the complementary information from both spectral bands, thereby improving estimation accuracy under poor lighting conditions.
Solution Approach 2:
The patent introduces NIR illumination as an intermediary light source that provides additional information about surface properties. The NIR images serve as a mediator that helps disambiguate between surface reflectance properties and illumination conditions, enabling more accurate estimation of surface normals and albedo maps when visible light is insufficient.
2Measurement precision
If a single NIR image is used with RGB image, then the imaging system remains simple, but the accuracy of relighting under arbitrary visible lighting deteriorates
Solution Approach 1:
The patent changes the spectral parameter by incorporating NIR illumination to probe surface properties. By capturing images in the NIR spectrum and combining them with visible spectrum RGB images, the system gains additional information about surface reflectance characteristics that is independent of visible lighting conditions, thereby improving relighting accuracy under arbitrary visible lighting.
Solution Approach 2:
The patent adds the NIR spectral dimension to the traditional RGB imaging system. This additional dimensional information allows the system to separate surface reflectance properties from illumination effects more effectively, providing robust relighting estimation even when visible lighting conditions are arbitrary or poor.
3Reliability
If ground truth reflectance maps are required for training, then the training process is more accurate, but the applicability to mobile photography deteriorates
Solution Approach 1:
The patent implements a self-service training approach where the system uses its own multi-spectral (RGB+NIR) images to train the prediction engine. By leveraging the complementary information from both spectral bands, the system can self-calibrate and learn surface properties without requiring external ground truth reflectance maps, making it directly applicable to mobile photography scenarios where such ground truth data is unavailable.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach transforms the problem of generating surface normals and albedo maps under poor lighting conditions into a tractable one, providing high-quality image relighting even without ground truth data, leveraging the availability of NIR illumination in modern mobile devices.
Implementation Method 1
a single dark flash image captured under controlled NIR lighting
Implementation Method 2
each of the plurality of NIR images being captured with a NIR illumination source
Data Source
AI summary
Techniques of estimating surface normals and reflectance from poorly-lit images includes using, in addition to an RGB image of a subject of a set of subjects, an image illuminated with near-infrared (NIR) radiation to determine albedo and surface normal maps for performing an image relighting, the image being captured with the NIR radiation from essentially the same perspective from which the RGB image was captured. In some implementations, a prediction engine takes as input a single RGB image and a single NIR image and estimates surface normals and reflectance from the subject.