3D Object Learning Model Without Background Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional photogrammetry techniques require background removal or masking during training due to background interference in images captured from multiple viewpoints, which complicates the estimation of three-dimensional object models using machine learning models like NeRF and NeuS.
Innovation Solution
A machine learning model is trained with constraints that set densities at background points and virtual viewpoints to predetermined constants, allowing direct training without background removal, using a surface function defined by a multi-layer perceptron and density functions to generate virtual images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a neural network is trained using a general-purpose GPU, then the training can be performed with widely available hardware, but the training time becomes excessively long due to insufficient processing power
Solution Approach 1:
The patent segments the neural network training workload into multiple processing streams that can be executed in parallel. The variable update computation is divided into first computation units that process different parameter groups simultaneously, enabling the system to achieve high processing throughput using distributed general-purpose GPUs rather than requiring a single complex specialized processor
Solution Approach 2:
The patent employs general-purpose GPUs that can be programmed to perform specialized neural network training operations. By using programmable hardware that can be adapted through software (the variable update computation program), the system achieves specialized processing capability without requiring dedicated hardware design, thus maintaining hardware availability while improving training efficiency
2Measurement precision
If detailed gradient information is computed for all parameters, then the optimization precision is improved, but the computational burden and training time increase significantly
Solution Approach 1:
The patent computes gradient information selectively rather than for all parameters simultaneously. The variable update computation updates parameters in groups across multiple processing stages, computing detailed gradient information only for the current batch of parameters being updated. This partial computation approach maintains optimization precision for active parameters while avoiding the excessive computational burden of computing and storing all gradients at once
3Productivity
If the batch size is increased to improve training efficiency, then more parameters can be updated per iteration, but the memory requirements and computational complexity increase
Solution Approach 1:
The patent segments the parameter update process into multiple batches, where each batch processes a subset of parameters. The variable update computation divides parameters into groups (first parameter groups) and processes them sequentially through multiple iterations. This segmentation allows efficient use of memory by loading only the necessary parameter subsets into GPU memory at each step, rather than requiring all parameters to fit in memory simultaneously
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
Viewpoint image DB 120 stores a plurality of viewpoints and images obtained by capturing images of an object from the plurality of viewpoints in association with each other. Data acquisition means 111 reads out viewpoint image DB 120 and supplies it to generation means 112. Generation means 112 uses machine learning model 121 in an initial state or in the middle of training, to generate a virtual image that can be obtained when an image of a virtual object is captured from a virtual viewpoint. Training means 113 compares the images stored in viewpoint image DB 120 with the virtual images generated by generation means 112, and updates machine learning model 121 that is constrained so that the densities at the background point and the virtual viewpoint are each a predetermined constant. The present invention eliminates the need to remove the background from each image when training a machine learning model that uses images obtained by capturing an object from a plurality of viewpoints as training data to estimate a three-dimensional model of the object in a virtual space.