Training Image Generation for Segmenting Densely Stacked Objects
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models for object segmentation in robotic systems require large amounts of training data and human intervention, and struggle with densely stacked objects, leading to inefficiencies and potential stack collapse during manipulation.
Innovation Solution
A method and system for generating labelled training images using depth and visual difference masks to autonomously manipulate stackable objects, enabling automated data collection without human intervention, and handling densely stacked scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a large amount of training data is acquired through human intervention, then the machine learning model's object segmentation performance is improved, but the time consumption and operational complexity increase significantly
Solution Approach 1:
The system uses robotic manipulators to automatically manipulate objects and generate training images without human intervention. The depth camera and visual camera capture images during autonomous object manipulation tasks, enabling the system to self-generate training data while maintaining segmentation performance
Solution Approach 2:
The patent replaces manual human operations with automated robotic manipulators controlled by a controller. The robotic system autonomously performs object manipulation tasks while depth and visual cameras capture training images, substituting mechanical human labor with automated robotic systems
2Ease of operation
If objects are scattered on a flat borderless surface, then object manipulation is simplified, but the system cannot handle densely stacked objects in realistic scenarios
Solution Approach 1:
The system dynamically adapts to different object configurations by using depth cameras to perceive the actual spatial arrangement of objects. The robotic manipulator adjusts its manipulation strategy based on real-time depth information, enabling it to handle both scattered and densely stacked objects effectively
Solution Approach 2:
The depth camera acts as an intermediary between the robotic manipulator and the objects, providing three-dimensional spatial information that enables the robot to understand and manipulate densely stacked objects. The depth information mediates the interaction between the robot and complex object arrangements
3Reliability
If pre-trained object segmentation models are used, then initial hypothesis formation is improved, but the models fail to generalize to unseen objects and densely stacked configurations
Solution Approach 1:
The system performs preliminary data collection by capturing depth and visual images during autonomous object manipulation tasks. These pre-collected training images specific to the target objects and configurations are used to fine-tune pre-trained models, enabling them to generalize to unseen objects and densely stacked configurations
Solution Approach 2:
The patent fine-tunes pre-trained segmentation models by changing their parameters through training on custom-collected depth and visual images. This parameter adjustment enables the models to adapt to specific object types and densely stacked configurations, improving generalization capability
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method (10) for generating labelled training images (ΔD, ΔRGB) characterising automatic robotic or manual manipulation of a plurality of stackable objects (24) in a workspace (22), comprising: - obtaining (11) a first training image subset obtained at a first time index comprising a depth map (D1) and a visual image (RGB1) of a plurality of stackable objects (24) in a stacking region (22a) of a workspace (22) optionally comprising a robotic manipulator (26); - obtaining (12) a second training image subset obtained at a second time index comprising a depth map (D2) and a visual image (RGB2) of the stacking region (22a) in the workspace (22), wherein the second training image subset characterizes a changed spatial state of the stacking region (22a); - computing (13) a depth difference mask (ΔD1) based on the depth maps of the first and second training image subsets; - computing (14) a visual difference mask (ΔRGB1) based on the visual images of the first and second training image subsets; - generating (15) an annotated segmentation mask (M) using the depth difference mask (ΔD1) and/or the visual difference mask (ΔRGB1), wherein the annotated segmentation mask localizes a stackable object (24) based on the spatial state of the stacking region at the first time index, before the spatial state was changed by automatic robotic or manual manipulation of the at least one stackable object in the workspace (22).