3D Semantic Map Editing for Mobile Robot Camera Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a lack of large-scale datasets tailored to the camera viewpoint of mobile robots for cleaning floors, limiting the effectiveness of deep neural networks in computer vision tasks for these robots.
Innovation Solution
A system and method for generating new annotated training data using a semantic map network that includes a first semantic map module for real-time dense 3D reconstruction and object detection on an edge device, and a second semantic map module for scene completion, object editing, and object arrangement on a computing system, utilizing machine learning models to enhance background and foreground scenes, and generating virtual camera data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are used for computer vision tasks on mobile robots, then the robot's vision capabilities are improved, but the lack of large-scale tailored datasets limits training effectiveness
Solution Approach 1:
The patent creates virtual copies of real-world environments through 3D semantic maps and renders synthetic training images from these digital replicas. This copying approach generates unlimited tailored training data without requiring physical robots to collect real-world data, directly resolving the data scarcity problem while maintaining viewpoint consistency for floor-cleaning robots
Solution Approach 2:
The patent transitions from 2D images to 3D semantic maps, adding depth information and spatial structure to the data representation. This dimensional enhancement allows for more comprehensive scene understanding and generates richer training data that captures environmental geometry, directly addressing the limitation of conventional 2D image datasets
2Loss of information
If 3D semantic maps are generated with complete depth data, then the completeness of environmental information is improved, but the complexity of data processing increases
Solution Approach 1:
The patent segments the environment into distinct semantic components (foreground objects, background elements, occluded regions) and processes each segment separately through specialized neural network branches. This segmentation strategy manages processing complexity by dividing the challenging task of complete 3D reconstruction into manageable sub-tasks, while still achieving comprehensive depth data coverage
Solution Approach 2:
The patent introduces an intermediary scene completion module that fills missing depth information in occluded regions by learning from visible parts and contextual cues. This intermediary processing step recovers lost information without requiring direct sensor measurement of every point, balancing completeness with processing feasibility
Data Source
AI summary
A computer-implemented method and system relate to computer vision. A first semantic map of an environment is three-dimensional (3D). A foreground scene and a background scene are generated individually using the semantic data of the first semantic map. The foreground scene contains foreground components of the first semantic map. The background scene contains background components of the first semantic map. A machine learning model generates an enhanced background view by completing incomplete regions of the background components. Input data is received to modify the background components, the foreground components, or both. A second semantic map is generated in 3D using the enhanced background view, the foreground components, and the input data. The second semantic map is 3D. Virtual camera data is generated using the second semantic map. The virtual camera data includes at least new image data and corresponding new depth data.