3D Scene Reconstruction from 2D Images for VR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Two-dimensional images fail to provide a full three-dimensional experience, limiting the ability to relive the sensation of a captured scene, and carrying bulky 3D cameras is impractical for everyday use.
Innovation Solution
A method to generate a three-dimensional image from a set of two-dimensional input images taken from different vantage points, using sparse and dense reconstruction representations, depth testing, and multi-layered geometric mesh rendering to create a immersive 3D scene.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If two-dimensional images are used to capture scenes, then the ease of operation is improved, but the immersion experience deteriorates
Solution Approach 1:
The patent transforms two-dimensional images into a three-dimensional representation by generating depth maps and constructing a multi-layered geometric mesh. This dimensionality change allows the system to preserve spatial information and provide immersive 3D experiences while maintaining the simplicity of using standard 2D cameras for capture.
2Loss of information
If three-dimensional cameras are used to capture scenes, then the three-dimensional experience is improved, but the device complexity deteriorates
Solution Approach 1:
Instead of using complex 3D cameras, the patent creates a computational copy of the 3D scene by processing multiple 2D images through sparse and dense reconstruction algorithms. This generates a multi-layered geometric mesh that replicates the three-dimensional structure, allowing standard cameras to achieve 3D capture capabilities through software processing.
3Manufacturing precision
If multiple processing steps are applied to generate 3D reconstruction, then the three-dimensional quality is improved, but the processing time deteriorates
Solution Approach 1:
The patent divides the complex 3D reconstruction process into distinct segments: sparse reconstruction to establish initial 3D points, dense reconstruction to generate detailed depth maps, and mesh generation to create the final geometric model. This segmentation allows each stage to be optimized independently and processed efficiently, balancing quality with processing time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
To enable better sharing and preservation of immersive experiences, a graphics system reconstructs a three-dimensional scene from a set of images of the scene taken from different vantage points. The system processes each image to extract depth information therefrom and then stitches the images (both color and depth information) into a multi-layered panorama that includes at least front and back surface layers. The front and back surface layers are then merged to remove redundancies and create connections between neighboring pixels that are likely to represent the same object, while removing connections between neighboring pixels that are not. The resulting layered panorama with depth information can be rendered using a virtual reality (VR) system, a mobile device, or other computing and display platforms using standard rendering techniques, to enable three-dimensional viewing of the scene.