Mobile Surfel Mapping for Real-Time Indoor 3D Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for creating accurate indoor space representations require expensive hardware or resource-intensive algorithms, necessitating significant user training and are not user-friendly.
Innovation Solution
Utilizing lightweight neural networks on mobile devices to estimate depth and semantic information from video data, employing surfel representations to fuse depth information into a 3D model, and generating real-time virtual representations of indoor spaces.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If automated measurement solutions use expensive specialized hardware like active depth sensors, then measurement precision is improved, but device complexity and cost increase
Solution Approach 1:
The patent uses a camera (standard imaging device) to capture 2D images and employs computational algorithms to generate depth information, effectively creating a computational copy of depth data without requiring physical depth sensors. This allows depth estimation using only standard camera hardware combined with neural network processing.
Solution Approach 2:
The patent replaces mechanical/optical depth sensing systems (like active depth sensors) with a computational system consisting of a camera and neural network algorithms. This substitution uses software-based depth estimation instead of hardware-based active sensing, reducing device complexity while maintaining measurement capability.
2Measurement precision
If resource-intensive algorithms are used for depth estimation, then measurement precision is improved, but processing speed and energy efficiency deteriorate
Solution Approach 1:
The patent modifies the parameters of the neural network by using pre-trained models with fixed weights and biases that are optimized for mobile devices. The network architecture is designed with specific layer configurations and activation functions that balance accuracy and computational efficiency, allowing real-time processing on resource-constrained devices.
Solution Approach 2:
The neural network is pre-trained offline on large datasets before deployment on mobile devices. This preliminary training action transfers learned knowledge to the mobile device, enabling the device to perform depth estimation without requiring intensive real-time training computations, thus improving processing speed while maintaining accuracy.
3Loss of information
If comprehensive semantic segmentation is performed to identify all location components, then information completeness is improved, but processing time and computational resources increase
Solution Approach 1:
The patent divides the semantic segmentation task into multiple independent classification heads within the neural network, each targeting specific object categories (e.g., furniture, appliances, structural elements). This segmentation of the classification task allows parallel processing of different object types, reducing overall processing time while maintaining comprehensive semantic coverage.
Data Source
AI summary
Resource-efficient systems, methods, and software to estimate the geometry of indoor space(s) (e.g., a location) and render a model of the indoor space(s) to a user in real time natively on mobile smartphones is described. Lightweight neural networks are used to estimate depths for each frame in received video data of the location, and a surfel representation is used to fuse location depth information into a geometric representation. Detecting when a user has entered a new room based on the geometric representation in real time is also described, and the resulting ability to create full floor scans where each room is fused together and labeled during a scan.