3D Reconstruction via Neural Network Normal Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computer vision systems face challenges in generating accurate 3D reconstructions of objects due to global illumination effects like cast-shadows and self-reflections, especially for specular surfaces, as rendering these effects is a slow process, limiting the generation of training data.
Innovation Solution
A computer vision method using a neural network architecture that processes per-pixel illumination information from compensated photometric stereo images, allowing for fast training data generation while learning global illumination effects and real-world imperfections, enabling 3D reconstruction in both near-field and far-field scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If global illumination effects are rendered to train the neural network, then the accuracy of 3D reconstruction is improved, but the training process becomes extremely slow
Solution Approach 1:
The patent segments the illumination model into two parts: pre-computed global illumination components (ambient occlusion, soft shadows) that can be generated offline and stored, and real-time lighting calculations that use these pre-computed components. This allows the neural network to be trained on high-quality rendered images with global illumination effects without requiring slow real-time rendering during inference.
Solution Approach 2:
The patent performs preliminary computation of global illumination effects during the training phase by pre-rendering images with accurate global illumination and storing them as training data. This preliminary action allows the neural network to learn from high-quality examples without requiring slow rendering during actual 3D reconstruction operations.
2Adaptability or versatility
If photometric stereo images are used for 3D reconstruction, then the method can handle specular surfaces, but global illumination effects like cast-shadows and self-reflections degrade the reconstruction accuracy
Solution Approach 1:
The patent converts the harmful effect of global illumination artifacts (cast-shadows, self-reflections) into a benefit by training the neural network to recognize and compensate for these effects. The network learns to distinguish between lighting effects caused by global illumination and actual surface geometry, thereby improving reconstruction accuracy on specular surfaces despite the presence of these artifacts.
Solution Approach 2:
The patent changes the parameter space by incorporating global illumination effects into the training data rather than trying to eliminate them. The neural network is trained on rendered images that include realistic global illumination effects, allowing it to learn appropriate compensation strategies for these effects during actual reconstruction.
3Measurement precision
If a traditional rendering approach is used to generate training data, then accurate global illumination effects can be captured, but the process is too slow for practical training
Solution Approach 1:
The patent segments the rendering pipeline into offline pre-computation of global illumination components and online application during training. Complex global illumination effects are pre-computed and stored as lookup tables or pre-rendered images, eliminating the need for slow real-time rendering during the training process while maintaining accuracy.
Solution Approach 2:
The patent creates simplified copies or approximations of complex global illumination effects that can be applied quickly during training. Instead of performing full physical ray-tracing for each training image, the system uses pre-computed illumination maps and transfer functions that replicate the visual effects of global illumination much faster.
Data Source
AI summary
A computer vision method for generating a three dimensional reconstruction of an object, the method comprising:receiving a set of photometric stereo images of the object, the set of photometric stereo images comprising a plurality of images using illumination from different directions using one or more light sources;using a trained neural network to generate a normal map of the object; andproducing a 3D reconstruction of said object from said normal map,wherein using said trained neural network comprises converting said set of photometric stereo images to an input form suitable for an input layer of said neural network, wherein said input form comprises, for each pixel, a representation of the different lighting directions and their corresponding intensities which have been obtained from photometric stereo images to which a compensation has been applied, the compensation being determined from an estimate of the distance between the lighting source and a point on the object to which the pixel corresponds.