Neural Radiance Field Ray Sampling for Object Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image processing techniques using Neural Radiance Fields (NeRF) face issues with object estimation failure for small objects and artifacts like 'floater' fog around objects due to uneven sampling of rays in captured images, particularly when the occupied area ratio of the object is small or large.
Innovation Solution
An image processing apparatus that controls the number of learning rays based on object areas in captured images, using a hardware processor and memory to set learning ray groups, ensuring accurate representation of three-dimensional fields without artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If random sampling of rays is used in minibatch learning, then memory usage is reduced and stepwise learning is enabled, but object estimation fails when the occupied area ratio of the object is small
Solution Approach 1:
The patent applies local quality by differentiating the sampling strategy based on spatial location. Instead of uniform random sampling, the system identifies regions with small occupied area ratios and applies focused sampling to those specific areas. This ensures that rays corresponding to small objects or regions are prioritized for learning, thereby improving object estimation accuracy without sacrificing the efficiency benefits of minibatch learning.
2Productivity
If random sampling of rays is used in minibatch learning, then memory usage is reduced, but artifacts such as floater fog occur around images corresponding to objects with large occupied area ratio
Solution Approach 1:
The patent addresses artifact generation by applying localized sampling correction. The system identifies regions with large occupied area ratios and adjusts the sampling distribution in those specific areas to ensure adequate ray coverage. This prevents the floater fog artifact by ensuring that rays corresponding to large objects are properly sampled, while maintaining the overall efficiency of minibatch learning.
3Measurement precision
If all rays are learned at one time (batch learning), then learning accuracy is improved, but memory usage increases significantly
Solution Approach 1:
The patent applies segmentation by dividing the complete set of rays into multiple minibatches for sequential learning. Instead of loading all rays into memory simultaneously, the system processes rays in smaller batches, significantly reducing memory requirements. The segmentation is further optimized by strategically dividing rays based on occupied area ratios, ensuring that each batch contains representative samples that maintain learning accuracy while minimizing memory usage.
Data Source
AI summary
A three-dimensional field corresponding to an object, which is used in a case where a virtual viewpoint image is generated, is estimated with high accuracy. The image processing apparatus according to the present disclosure obtains data of a plurality of captured images obtained by image capturing from a plurality of viewpoints, obtains an object area corresponding to a representation of an object in each of the plurality of captured images, sets a learning ray group that is used for learning of information relating to a three-dimensional field of an image capturing space that is an image capturing target from the plurality of viewpoints, and which corresponds to pixels of each of the plurality of captured images based on the obtained object area, and performs learning of information relating to the three-dimensional field based on the set learning ray group.