Robotic Bin Picking With 2D-3D Mask Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotic picking systems struggle to accurately identify objects scattered within bins due to issues like color and intensity similarities between objects and bins, leading to false data and pick failures, especially when using 2D camera systems, and the high costs and computational complexity of 3D camera systems.
Innovation Solution
The system combines 2D and 3D data to differentiate between objects and bins, using a binary intermediate mask model to align and process data, reducing noise and filling missing data, enabling real-time object identification with less expensive hardware and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a 2D camera system is used to differentiate between objects and bins, then the system cost is reduced, but the system fails to accurately differentiate when objects and bins have the same color and/or intensity
Solution Approach 1:
The patent introduces a third dimension (depth) to the imaging system by combining 2D color/intensity data with 3D depth data. This allows the system to differentiate objects from bins based on depth information when color and intensity are insufficient, resolving the contradiction between cost and accuracy.
Solution Approach 2:
The patent merges 2D camera data (color, intensity) with 3D camera data (depth, distance) into a unified analysis framework. By combining these complementary data sources, the system achieves accurate object differentiation without relying solely on expensive 3D hardware or failing 2D systems.
2Measurement precision
If a 3D camera system is used to measure depth data and differentiate between objects and bins, then differentiation accuracy is improved, but hardware cost and computational complexity increase significantly
Solution Approach 1:
The patent processes 3D depth data selectively and partially rather than fully. It uses depth information specifically for differentiation when needed, while relying on 2D data for other aspects of object characterization, reducing the overall computational burden of full 3D processing.
Solution Approach 2:
The patent introduces an intermediary processing layer that translates and integrates 3D depth data with 2D image data. This intermediary framework enables the use of 3D information without requiring full 3D processing capabilities, reducing hardware and computational requirements while maintaining accuracy.
3Productivity
If 3D data processing is performed in real-time for high-speed industrial applications, then processing speed is improved, but computational resources and costs increase significantly
Solution Approach 1:
The patent segments the data processing into distinct stages: 2D image acquisition and processing, 3D depth data acquisition and processing, and integrated analysis. This segmentation allows each stage to be optimized independently, enabling real-time processing without requiring excessive computational resources for unified processing.
Data Source
AI summary
In an aspect of the disclosure, a method, a computer-readable medium, and an apparatus are provided. The apparatus may generate a binary intermediate mask model from 2D image data and 3D data obtained from a background image and a component image. Because the intermediate mask model is binary, erosion and dilation (e.g., 2D data processing) may be used to remove noise from the intermediate mask model. The augmented intermediate mask model with any erroneous and/or missing data corrected, may be applied to the second aligned model. The pixels in the second aligned model that correspond to pixels in the augmented intermediate mask model that are assigned a value of 1 may be identified as foreground pixels. Then, the vision system may generate foreground object data using the 2D image data and the 3D data in the foreground pixels from the second aligned model.