3D Field Estimation From Low-Gloss Multi-View Images
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for estimating a three-dimensional field using neural radiance fields (NeRF) struggle with glossy surfaces, leading to errors in surface positioning and artifacts like indentations or floaters due to varying pixel values from different viewing angles.
Innovation Solution
An image processing apparatus reduces the gloss component of captured images using background images to generate low gloss images, which are then used for learning a three-dimensional field model, followed by two stages of neural network training to accurately represent the object's shape and color.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multi-viewpoint images of glossy objects are used for NeRF learning, then three-dimensional field estimation can be performed, but surface positioning accuracy deteriorates due to varying pixel values from light reflection
Solution Approach 1:
The patent segments the captured images into different components: gloss components (extracted through image processing) and non-gloss components (diffuse reflection and background). By separating these components, the learning process can focus on the stable non-gloss components for accurate surface positioning, while the gloss components are handled separately or excluded from the primary surface estimation learning.
Solution Approach 2:
The patent introduces an intermediary processing step that extracts and removes the gloss component from captured images before feeding them to the NeRF learning process. This intermediary step (gloss removal processing) acts as a mediator between the raw captured images and the learning model, providing cleaned input that enables accurate surface positioning without the interference of specular reflection variations.
2Ease of manufacture
If standard NeRF learning is applied to glossy surfaces, then three-dimensional field estimation is achieved, but artifacts such as indentations and floaters appear in the estimated field
Solution Approach 1:
The patent performs preliminary processing of the input images before the NeRF learning process by removing gloss components from the captured images. This preliminary action ensures that the learning model receives pre-processed images with consistent pixel values, preventing the formation of artifacts like indentations and floaters in the three-dimensional field estimation while maintaining the simplicity of the overall process.
3Productivity
If pixel values from multiple viewpoints are used directly in learning, then learning speed is maintained, but transmittance estimation errors occur in sampling points
Solution Approach 1:
The patent extracts and removes the problematic gloss component from the captured images before inputting them to the learning model. By taking out the gloss component that causes pixel value variations, the learning process can proceed at normal speed using the remaining non-gloss components, which provide stable and consistent pixel values for accurate transmittance estimation in sampling points.
Data Source
AI summary
Even in a case where an object has a gloss characteristic on a surface, a three-dimensional field relating to the object is estimated with high accuracy. An image processing apparatus according to the present disclosure obtains a plurality of captured images which is obtained by capturing an object from each of a plurality of image capturing viewpoints and a camera parameter corresponding to image capturing from each of the plurality of image capturing viewpoints, generates a plurality of low gloss images in which a gloss component of the object is reduced based on the plurality of captured images; and performs learning of a learning model indicating a three-dimensional field relating to the object by using the plurality of low gloss images and the plurality of captured images and the camera parameter.