Incremental 3D Semantic Mapping via 2D Plane Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in creating and updating dense, 3-D semantic maps suitable for embedded implementation, particularly in the context of visual-inertial simultaneous localization and mapping (VI-SLAM) systems.
Innovation Solution
A computer-implemented method for incremental dense mapping with semantics in 3-D, which involves obtaining a 2-D image and its associated depth map, projecting segments into the image, performing geometric and semantic segmentation, and updating the 3-D map with semantic class labels and segment IDs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If dense 3-D semantic mapping is implemented with high accuracy and scalability, then mapping precision and versatility improve, but computational complexity and processing requirements increase
Solution Approach 1:
The patent segments the 3-D map into multiple 2-D image planes at different depths, processing each plane independently. This divides the complex 3-D semantic mapping problem into smaller 2-D segmentation tasks that can be handled by standard embedded processors, reducing overall computational complexity while maintaining mapping precision through multi-plane processing
Solution Approach 2:
The patent transforms the 3-D mapping problem into a series of 2-D problems by creating image planes at different depths. This dimensional reduction allows the use of computationally efficient 2-D segmentation algorithms on embedded devices while still achieving accurate 3-D semantic mapping through aggregation of results across multiple planes
2Adaptability or versatility
If semantic class labels and instance segmentation are added to dense 3-D maps, then mapping versatility and application capability improve, but data storage requirements and processing time increase
Solution Approach 1:
The patent segments the 3-D space into multiple 2-D image planes and processes semantic segmentation independently on each plane. This segmentation approach reduces data volume by avoiding redundant storage of semantic information across multiple representations, while maintaining versatility through per-plane semantic labels and instance IDs that can be aggregated to form comprehensive 3-D semantic maps
Solution Approach 2:
The patent creates 2-D image plane representations as copies of 3-D spatial information at different depths. These copies contain semantic labels and instance identifiers that can be stored efficiently and aggregated, reducing the need to store full 3-D semantic data structures while maintaining application capability through the copied representations
3Productivity
If incremental updating of 3-D semantic maps is performed in real-time, then system responsiveness and productivity improve, but computational resource consumption increases
Solution Approach 1:
The patent segments the 3-D map into multiple 2-D image planes that can be updated independently. This allows the system to process only the portions of the map that have changed in each plane, rather than reprocessing the entire 3-D map, thereby improving update speed and reducing energy consumption while maintaining real-time responsiveness
Solution Approach 2:
The patent performs partial updates on specific 2-D image planes only when and where changes are detected, rather than performing complete updates of the entire 3-D semantic map. This partial action approach improves productivity by updating only necessary portions while reducing computational energy consumption by avoiding unnecessary processing of unchanged regions
Data Source
Figure 1A~1B
Figure 1C
Figure 2
AI summary
A computer-implemented method for incremental dense mapping with semantics in 3-D, is provided. The method comprises obtaining a 2-D image of the 3-D environment, along with an associated depth map. Pose information is obtained, which relates the viewpoint of the 2-D image and depth map to the reference frame of an existing 3-D semantic map. In some examples, segments are projected into the 2-D image and associated with pixels, to generate a class label image containing semantic class labels and a segment ID image containing segment IDs. In some examples, the existing 3-D map is sampled at the 3-D points corresponding to the pixels in the depth map, to retrieve a class label image containing semantic class labels and a segment ID image containing segment IDs. Geometric segmentation of the depth map is performed, taking into account the segment ID image extracted from the existing 3-D map. The 2-D image is also segmented, taking into account the class label image extracted from the existing 3-D map. The segment IDs and semantic class labels in the 3-D map are updated based on the results of the geometric segmentation and the segmentation of the 2-D image.