3D Surface Completion for Missing Geometry in Scene Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing 3D reconstruction technologies often result in incomplete or missing surfaces and objects due to insufficient observations, leading to challenges in creating immersive experiences and accurate geometry, which can cause disconnects in virtual environments and hinder collision avoidance.
Innovation Solution
An incremental surface completion method using a machine learning system, such as a sparse generative neural network, identifies regions requiring refinement and applies bounding regions to these areas, reducing resource usage and enabling real-time refinement on various devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If traditional 3D reconstruction methods are used, then the reconstruction process can be performed with existing technology, but the resulting 3D reconstruction contains missing surfaces and incomplete geometry
Solution Approach 1:
The patent segments the 3D reconstruction process into two distinct phases: (1) initial reconstruction using traditional methods to capture observable surfaces, and (2) incremental completion using machine learning to fill missing regions. This segmentation allows each phase to specialize - traditional methods handle well-observed areas while ML handles incomplete areas, resolving the contradiction between completeness and complexity.
Solution Approach 2:
The patent performs preliminary 3D reconstruction using traditional computer vision techniques before applying machine learning completion. This preliminary action establishes a baseline geometry that the ML system then refines by filling gaps, allowing the system to leverage both traditional reliability and ML completeness without requiring either approach to work alone.
2Manufacturing precision
If machine learning systems are used to complete missing surfaces, then geometry completeness improves, but computational resource consumption increases
Solution Approach 1:
The patent applies machine learning completion selectively only to identified missing surface regions rather than processing the entire 3D scene. By localizing the ML application to specific incomplete areas, the system achieves high surface completion accuracy where needed while minimizing computational resource consumption in already-complete regions.
Solution Approach 2:
The patent implements incremental surface completion that processes only the necessary portions of the scene - specifically, regions identified as having missing geometry. This partial action approach avoids the excessive computational cost of processing entire scenes while still achieving the required level of completeness for immersive experiences.
3Manufacturing precision
If comprehensive scanning is performed to capture all surfaces, then geometry completeness improves, but scanning time and productivity decrease
Solution Approach 1:
The patent performs a quick preliminary scan to capture observable surfaces, then uses machine learning to complete missing geometry without requiring a slow, comprehensive rescan. This preliminary scanning approach maintains high productivity while the subsequent ML completion step achieves comprehensive scene coverage that would otherwise require much longer scanning time.
Solution Approach 2:
The patent introduces machine learning as an intermediary that bridges the gap between quick incomplete scans and comprehensive complete reconstructions. Instead of requiring slow comprehensive scanning, the ML intermediary generates plausible missing geometry based on observed surfaces, thereby achieving complete scene coverage while maintaining high scanning productivity.
4Speed
If real-time processing is implemented on mobile devices, then processing speed improves, but the ability to handle complex completion tasks deteriorates
Solution Approach 1:
The patent processes only missing surface regions through the machine learning completion pipeline rather than processing entire high-resolution scenes. This local processing approach enables real-time performance on mobile devices with limited computational resources while maintaining completion quality sufficient for immersive experiences.
Solution Approach 2:
The patent implements a partial processing strategy where the ML system focuses computational effort only on regions with missing geometry rather than processing all scene elements. This approach achieves adequate completion quality for real-time applications on mobile devices without the excessive computational load that would be required for full-scene high-fidelity processing.
Data Source
AI summary
Systems and techniques are described herein for reconstructing a three-dimensional (3D) scene of an environment. For example, a computing device can obtain a 3D reconstruction of the 3D scene, wherein the 3D reconstruction of the 3D scene includes reconstructions of geometry information of objects included in one or more images of the 3D scene, determine one or more regions of the 3D reconstruction of the 3D scene that include missing geometry information of the objects included in the one or more images of the 3D scene, determine, using a machine learning system, geometry information for the one or more regions of the 3D reconstruction of the 3D scene, and refine the 3D reconstruction of the 3D scene using the geometry information determined for the one or more regions of the 3D reconstruction of the 3D scene.