Robotic Picking with Reference-Image Segmentation in Clutter
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional robotic picking operations face challenges in accurately segmenting items from bulk containers due to heavy clutter, varied item shapes, ambiguous definitions, and limited availability of task-specific training data, leading to picking defects and item damage.
Innovation Solution
The robotic system performs instance segmentation using item attributes and a reference image, employing a new segmentation network structure that fuses features and semantic cues from both images, and uses item attributes to refine segmentation outputs, reducing the need for extensive training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional instance segmentation methods are used to identify items from bulk containers, then the robotic system can perform automated picking, but segmentation accuracy deteriorates due to heavy clutter, varied item shapes, and ambiguous target definitions
Solution Approach 1:
The patent introduces an intermediary reference image that contains explicit target definitions and item attributes. This reference image acts as a mediator between the bulk container image and the segmentation process, providing ground truth information about item locations, shapes, and characteristics. The segmentation network uses this intermediary reference to accurately identify and segment target items even in heavily cluttered environments where conventional methods fail.
2Measurement precision
If supervised instance segmentation algorithms are trained with task-specific data, then segmentation accuracy improves, but the need for extensive training data increases operational complexity and time
Solution Approach 1:
The patent performs preliminary action by pre-defining target definitions and item attributes in a reference image before the actual segmentation task. This reference image is prepared in advance and contains all necessary information about target items including their locations, shapes, and characteristics. By having this preliminary reference information ready, the system eliminates the need for extensive task-specific training data preparation, as the reference image provides the necessary guidance directly during segmentation.
3Productivity
If the robotic system picks items from heavily cluttered bulk containers, then productivity increases, but item damage increases due to inaccurate segmentation and improper grasping
Solution Approach 1:
The patent implements feedback by using the reference image with known item attributes and target definitions to guide and verify the segmentation process. The segmentation network continuously references the ground truth information in the reference image, allowing it to adjust its segmentation decisions in real-time. This feedback mechanism ensures accurate identification of target items and their boundaries, enabling the robotic system to select appropriate grasping points and forces, thereby preventing item damage while maintaining high picking throughput.
Data Source
AI summary
Systems and techniques for performing robotic picking of items based on a reference image and item attributes are described. In an example technique, an image of a picking environment having multiple items is obtained. A reference image of a first item is obtained, based on an item ID associated with the first item. An instance segmentation operation is performed on the image of the picking environment to determine an instance of the first item within the image of the picking environment, based on the reference image and the image of the picking environment. A grasp plan for a robotic arm end-of-arm tool is generated based on segmentation information output from the instance segmentation operation and the item attributes. The end-of-arm tool is controlled to pick an item from the multiple items according to the grasp plan.