Transparent Bin Picking Using RGB Segmentation for Unreliable Depth Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing robot systems fail to accurately identify transparent objects in a bin using depth map images due to light propagation, leading to unreliable point cloud representations and ineffective object recognition for pick and place operations.
Innovation Solution
A system and method employing a deep learning mask R-CNN (convolutional neural network) for image segmentation, which generates a segmentation image by assigning labels to pixels in RGB images, allowing the robot to identify and pick up transparent objects without relying on accurate depth map images, using a 3D camera and a robot controller that performs feature extraction, region proposal, binary segmentation, and grasp pose calculation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth map images are used to identify transparent objects, then distance measurements can be obtained, but the depth map becomes unreliable due to light propagation through transparent objects
Solution Approach 1:
The patent applies image segmentation to divide the RGB image into distinct regions corresponding to different objects. By segmenting the image at the pixel level and labeling regions, the system can identify transparent objects based on color and texture information from the RGB image rather than relying on unreliable depth data. This segmentation approach allows the robot to locate and identify transparent objects without being affected by light propagation issues in depth maps.
2Difficulty of detecting and measuring
If model-based point cloud analysis with CAD templates is used, then object identification can be performed, but transparent objects cannot be properly identified due to ineffective point cloud representation
Solution Approach 1:
The patent replaces the mechanical/optical point cloud-based detection system with a deep learning-based image segmentation system. Instead of using 3D point clouds that fail to represent transparent objects, the system uses 2D RGB image segmentation with convolutional neural networks to identify transparent objects based on their visual features in the image plane. This substitution of detection methodology eliminates the fundamental limitation of point cloud analysis for transparent objects.
3Productivity
If traditional bin picking methods are used, then general objects can be picked up, but transparent objects cannot be properly identified for picking
Solution Approach 1:
The patent changes the detection parameters from 3D point cloud coordinates to 2D image pixel labels. By transforming the problem from three-dimensional point cloud analysis to two-dimensional image segmentation, the system can effectively detect transparent objects using color, texture, and spatial information from the RGB image. This parameter transformation allows the bin picking system to handle both opaque and transparent objects reliably.
Data Source
AI summary
A system and method identifying an object, such as a transparent object, to be picked up by a robot from a bin of objects. The method includes obtaining a 2D red-green-blue (RGB) color image and a 2D depth map image of the objects using a 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the objects. The method generates a segmentation image of the objects using a deep learning mask R-CNN (convolutional neural network) that performs an image segmentation process that extracts features from the RGB image and assigns a label to the pixels so that objects in the segmentation image have the same label. The method then identifies a location for picking up the object using the segmentation image and the depth map image.