Single Neural Network for Multi-Scene 3D Reconstruction

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current scene reconstruction techniques require training a different neural network from scratch for each scene, leading to excessive processing resources and memory needs for generating multiple 3D representations.

Innovation Solution

A single trained neural network is used to generate 3D representations for multiple scenes by mapping RGBD images to surface representations and aggregating them in 3D space, reducing the need for multiple neural networks and parameter sets.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a different neural network is trained from scratch for each scene, then the 3D representation can be generated for each scene, but the processing resources and memory requirements become prohibitive

Engineering Contradiction:

Improveability to generate 3D representations for multiple scenesVSAvoidprocessing resources and memory requirements

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The patent trains a single neural network model that can generate 3D representations for multiple different scenes. Instead of training separate neural networks for each scene, the model is trained once on a dataset and then reused for generating 3D representations across various scenes, reducing computational resources and memory requirements while maintaining versatility

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Manufacturing precision

If a complex neural network with vast number of learnable parameters is used, then the 3D representation can be generated effectively, but the processing resources and memory required increase significantly

Engineering Contradiction:

Improvequality of 3D representationVSAvoidnumber of learnable parameters

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent divides the neural network into separate components: a geometry encoder that processes depth images and a texture encoder that processes RGB images. These encoders generate feature representations that are then combined to produce the final 3D representation. This segmentation allows the model to maintain effectiveness while reducing the complexity of any single component

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20240161383A1Techniques for reconstructing different three-dimensional scenes using the same trained machine learning model

Publication Date: 2024.05.16 NVIDIA CORP

AI summary

In various embodiments, a scene reconstruction model generates three-dimensional (3D) representations of scenes. The scene reconstruction model maps a first red, blue, green, and depth (RGBD) image associated with both a first scene and a first viewpoint to a first surface representation of at least a first portion of the first scene. The scene reconstruction model maps a second RGBD image associated with both the first scene and a second viewpoint to a second surface representation of at least a second portion of the first scene. The scene reconstruction model aggregates at least the first surface representation and the second surface representation in a 3D space to generate a first fused surface representation of the first scene. The scene reconstruction model maps the first fused surface representation of the first scene to a 3D representation of the first scene.