Mix-Size Box Depalletizing Using CPU Image Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing robot pick and place systems face inefficiencies when trying to identify and pick up boxes of varying sizes from a stack, as they require pre-inputting dimensions and often struggle with distinguishing separate boxes from a single large box, especially when using CPUs for deep learning neural networks which are computationally slow compared to GPUs.
Innovation Solution
A system and method using a simplified mask R-CNN executable by a CPU to perform image segmentation, predicting the center area of boxes and employing morphological processes to refine masks and identify picking locations, allowing for accurate box identification without pre-inputting dimensions and using traditional edge detection techniques to enhance edge prediction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a deep learning neural network is used for image segmentation and box identification, then measurement precision and reliability are improved, but processing time increases significantly when using a CPU
Solution Approach 1:
The patent segments the box identification process into two distinct stages: (1) a simplified neural network stage that runs on CPU to identify box centers and generate initial masks, and (2) a traditional image processing stage that refines edge detection. This segmentation allows the system to leverage the speed of CPU for the majority of processing while using traditional algorithms only for the critical edge refinement step, thereby reducing overall processing time while maintaining identification accuracy.
Solution Approach 2:
The patent extracts the computationally intensive deep learning components (full mask R-CNN) from the system and replaces them with a simplified neural network that achieves sufficient accuracy for center detection. By taking out the excessive computational requirements and keeping only the essential neural network functionality, the system achieves real-time performance on CPU while maintaining adequate measurement precision.
2Adaptability or versatility
If boxes of varying sizes are placed on the pallet, then adaptability is improved, but device complexity increases for identifying and distinguishing individual boxes
Solution Approach 1:
The patent implements a dynamic box identification system that adapts to varying box sizes through a two-stage process. The simplified neural network dynamically adjusts to different box configurations and sizes, generating initial masks that are then refined by traditional image processing algorithms. This dynamic approach allows the system to handle mix-size boxes effectively without requiring pre-programmed size parameters or complex configuration, thereby managing device complexity while maintaining high adaptability.
3Ease of operation
If traditional projection templates are used for box identification, then ease of operation is maintained, but measurement precision deteriorates when boxes have different sizes
Solution Approach 1:
The patent changes the fundamental parameters of the identification system by replacing fixed-size projection templates with a simplified neural network that can dynamically adapt to various box sizes. The neural network processes images to identify box centers and generate masks that automatically adjust to different dimensions. This parameter change maintains ease of operation (no manual template configuration needed) while dramatically improving measurement precision for mix-size boxes.
Data Source
AI summary
A system and method for identifying a box to be picked up by a robot from a stack of boxes. The method includes obtaining a 2D red-green-blue (RGB) color image of the boxes and a 2D depth map image of the boxes using a 3D camera. The method employs an image segmentation process that uses a simplified mask R-CNN executable by a central processing unit (CPU) to predict which pixels in the RGB image are associated with each box, where the pixels associated with each box are assigned a unique label that combine to define a mask for the box. The method then identifies a location for picking up the box using the segmentation image.