Bin Picking Rotation Compensation for Transparent Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing robot systems fail to accurately identify and pick up transparent objects from a bin due to unreliable depth map images generated by 3D cameras, as light propagates through transparent objects, preventing effective representation and reliable distance measurement.
Innovation Solution
A system and method using a convolutional neural network for image segmentation that extracts features from RGB images, assigns labels to pixels, determines object orientation, and identifies the center pixel and orientation of objects, allowing the robot to pick up transparent objects without relying on accurate depth maps, and optionally rotates the objects to a desired orientation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a 3D camera is used to generate depth map images for object identification, then distance measurements can be obtained, but transparent objects cannot be properly identified because light propagates through them without effective reflection
Solution Approach 1:
The patent introduces an intermediary computational model (neural network) that processes RGB image data to infer object properties without relying on direct depth measurements from transparent objects. The neural network acts as a mediator between the camera input and object identification output, enabling reliable detection of transparent objects through learned features from color images rather than failed depth measurements.
Solution Approach 2:
The patent replaces the mechanical/optical depth measurement system (which fails for transparent objects) with a computational image processing system using neural networks. Instead of relying on physical light reflection and depth mapping, the system substitutes a learned computational model that processes RGB images to identify transparent objects, effectively replacing the failed physical measurement approach with a successful computational approach.
2Ease of operation
If model-free point cloud analysis is used to identify objects, then clustering can be performed, but transparent objects are not effectively represented in the point cloud
Solution Approach 1:
The patent replaces the mechanical point cloud analysis system with a computational neural network-based image processing system. Instead of using geometric point cloud clustering that fails for transparent objects, the system substitutes a neural network that processes RGB image data, achieving reliable detection of transparent objects through learned visual features rather than geometric point analysis.
Solution Approach 2:
The patent changes the fundamental parameters used for object identification from geometric point cloud data to neural network-processed RGB image features. This parameter change allows the system to detect transparent objects by learning from color, texture, and pattern information in images rather than relying on geometric depth data that transparent objects do not provide.
3Manufacturing precision
If model-based point cloud analysis with CAD templates is used, then template matching can be performed, but the template search in point cloud is ineffective for transparent objects
Solution Approach 1:
The patent replaces the mechanical template matching system operating on point clouds with a computational neural network system operating on RGB images. Instead of searching for CAD templates in failed point cloud representations of transparent objects, the system substitutes a neural network that directly processes image data to identify transparent objects, achieving both precision and reliability.
Data Source
AI summary
A system and method for identifying an object to be picked up by a robot. The method includes obtaining a 2D red-green-blue (RGB) color image and a 2D depth map image of the objects using a 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the objects. The method generates a segmentation image of the objects using a deep learning convolutional neural network that performs an image segmentation process that extracts features from the RGB image, assigns a label to the pixels so that objects in the segmentation image have the same label and rotates the object using the orientation of the object in the segmented image. The method then identifies a location for picking up the object using the segmentation image and the depth map image and rotates the object when it is picked up.