Volumetric Neural Rendering Texture Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional 3D reconstruction methods, such as mesh-based approaches, struggle to synthesize highly realistic images for complex objects and do not allow for easy editing of the object's appearance, as they entangle geometry and appearance in a black box volume-encoding neural network.
Innovation Solution
A volumetric neural rendering system using multiple neural networks to disentangle geometry from appearance, generating a continuous 3D model and a continuous 2D texture map, allowing for intuitive appearance editing by modifying the texture map.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If volume-based neural rendering methods are used to achieve high visual quality, then rendering realism is improved, but ease of appearance editing deteriorates because geometry and appearance are entangled in a black box network
Solution Approach 1:
The patent segments the unified neural rendering network into separate components: a geometry network that extracts 3D surface points and a texture network that generates 2D texture maps. This segmentation allows independent manipulation of appearance while preserving the photorealistic rendering capabilities of the original volume-based method.
Solution Approach 2:
The patent extracts the appearance information from the black box neural network by introducing a texture mapping module that projects 3D surface points to 2D texture coordinates. This extraction enables appearance editing by allowing users to modify the separated 2D texture map without affecting the underlying geometry or rendering quality.
2Ease of operation
If mesh-based reconstruction methods are used to enable easy appearance editing, then ease of operation is improved, but rendering realism deteriorates for complex objects
Solution Approach 1:
The patent replaces the traditional mesh-based mechanical representation with a neural network-based continuous representation. The geometry network generates 3D surface points and the texture network produces continuous 2D texture maps, combining the realism of neural rendering with the editability of texture-based approaches without requiring explicit mesh structures.
3Device complexity
If a single black box neural network is used for volumetric rendering, then device complexity is reduced, but adaptability deteriorates because geometry and appearance are entangled
Solution Approach 1:
The patent segments the single neural network into multiple specialized sub-networks: a geometry network for 3D surface extraction and a texture network for 2D texture map generation. This segmentation increases adaptability by allowing independent manipulation of appearance while maintaining relatively simple individual network structures.
Solution Approach 2:
The patent introduces 2D texture coordinates as an intermediary representation between the 3D geometry network and the final rendered output. This intermediary layer enables flexible appearance editing while keeping the underlying neural network structures relatively simple and modular.
Data Source
AI summary
Embodiments are disclosed for neural texture mapping. In some embodiments, a method of neural texture mapping includes obtaining a plurality of images of an object, determining volumetric representation of a scene of the object using a first neural network, mapping 3D points of the scene to a 2D texture space using a second neural network, and determining radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a second neural network to generate a 3D appearance representation of the object.