Depalletizing Box Segmentation Using 3D RGB-Depth Imaging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robot systems are inefficient in determining the size of boxes to be picked up during real-time operations, especially when boxes of different sizes are stacked or placed side-by-side, as they require pre-inputting dimensions and lack the ability to distinguish between separate boxes and a single large box.
Innovation Solution
A system and method using a 3D camera to obtain RGB and depth map images, performing image segmentation with deep learning techniques to assign labels to pixels, allowing the robot to identify the location and size of each box without prior input, employing a segmentation process that extracts features and combines them to predict which pixels belong to which box.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If pre-inputting box dimensions is used, then the robot can perform pick and place operations, but the system cannot handle boxes of different sizes efficiently
Solution Approach 1:
The system performs self-measurement of box dimensions using a 3D camera and image segmentation algorithm. The robot automatically determines the width, length, and height of each box by processing images and depth maps, eliminating the need for manual dimension input and enabling handling of variable box sizes without human intervention.
Solution Approach 2:
The patent replaces manual measurement and input processes with an automated optical measurement system. A 3D camera captures RGB and depth map images, and an image segmentation algorithm automatically extracts box dimensions, substituting the mechanical/manual dimension input process with an automated vision-based measurement system.
2Measurement precision
If traditional template matching is used, then the system can identify box locations, but it cannot distinguish between separate boxes and a single large box when placed side-by-side
Solution Approach 1:
The patent applies image segmentation to divide the image into distinct regions corresponding to individual boxes. By segmenting the depth map and RGB image, the system can identify separate boxes placed side-by-side at the same height, distinguishing them as individual objects rather than a single large box, thereby improving measurement precision.
Solution Approach 2:
The patent utilizes the depth dimension from 3D camera images to distinguish between separate boxes and a single large box. By analyzing depth values in the depth map image, the system can detect depth variations that indicate separate boxes at different depths, adding a dimensional cue that resolves ambiguity in 2D image analysis.
3Productivity
If manual dimension input is required, then the system can operate, but real-time determination of box size is not possible
Solution Approach 1:
The system performs preliminary measurement and identification of box dimensions before the pick and place operation. The 3D camera captures images and the image segmentation algorithm determines box dimensions in advance, allowing the robot to plan and execute the pick operation with accurate dimension information already available, improving operational efficiency.
Solution Approach 2:
The patent replaces manual dimension input operations with automated image-based measurement. The system uses a 3D camera to capture images and an image segmentation algorithm to automatically determine box dimensions, eliminating the need for manual measurement and input operations, thereby improving ease of operation and enabling real-time dimension determination.
Data Source
AI summary
A system and method for identifying a box to be picked up by a robot from a stack of boxes. The method includes obtaining a 2D red-green-blue (RGB) color image of the boxes and a 2D depth map image of the boxes using a 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the boxes. The method generates a segmentation image of the boxes by performing an image segmentation process that extracts features from the RGB image and the depth map image, combines the extracted features in the images and assigns a label to the pixels in a features image so that each box in the segmentation image has the same label. The method then identifies a location for picking up the box using the segmentation image.