Complement Viewpoint Imaging for NeRF Fidelity Gaps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional Neural Radiance Fields (NeRF) struggle with impaired reproduction fidelity of virtual viewpoint images when capturing conditions require angles that are difficult for image capturing apparatuses, such as looking up at an object from below, leading to incomplete training data and inaccurate virtual viewpoint representations.
Innovation Solution
Generate complement viewpoint images from positions not covered by existing cameras and incorporate these images into the training process to enhance the three-dimensional field model, ensuring comprehensive data coverage and improved training accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If image capturing is performed only from available positions, then captured images can be obtained, but reproduction fidelity of virtual viewpoint images is impaired
Solution Approach 1:
The patent generates complement viewpoint images by creating virtual camera positions that copy the role of physical cameras in underrepresented viewpoints. These virtual cameras simulate what would be captured from difficult positions, providing training data for viewpoints that cannot be physically accessed, thereby improving both reproduction fidelity and viewpoint coverage.
Solution Approach 2:
The patent introduces a new dimension to the training data by generating images from complement viewpoints that are symmetric or complementary to the original viewpoints. This expands the training space beyond physical camera positions into a virtual dimension, enabling the neural network to learn representations for viewpoints that would otherwise be inaccessible.
2Reliability
If complement viewpoint images are generated and added to training data, then reproduction fidelity is improved, but device complexity increases
Solution Approach 1:
The patent performs preliminary generation of complement viewpoint images before the main training process. By pre-computing these virtual images and preparing them as training data, the system avoids the need for complex real-time generation during inference, thereby managing processing complexity while improving reproduction fidelity.
Solution Approach 2:
The patent uses generated complement viewpoint images as an intermediary that bridges the gap between available captured images and the desired comprehensive viewpoint coverage. These intermediary images serve as training data that mediates between the limited physical capture data and the requirements for high-fidelity reproduction across all viewpoints.
Data Source
AI summary
An image processing apparatus: obtains a plurality of captured images obtained by image capturing from a plurality of positions, and a plurality of camera parameters on a plurality of viewpoints corresponding to the plurality of positions; generates a camera parameter on a complement viewpoint that is different from the plurality of viewpoints; obtains shape data of an object estimated based on the obtained plurality of camera parameters and the obtained plurality of captured images; generates a complement viewpoint image based on the shape data and the generated camera parameter; and generates information on a three-dimensional field corresponding to a space that is at least part of an image capturing space subjected to image capturing from the plurality of positions, the information being generated based on the obtained plurality of camera parameters, the obtained plurality of captured images, the generated camera parameter, and the generated complement viewpoint image.