Synthetic 3D Training Data for Robotic Picking Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robotic gripper systems face challenges in precisely identifying and handling products that are touching or overlapping on a conveyor belt, leading to inefficiencies and potential damage, particularly due to unpredictable shifts and variability in product size, shape, and weight.
Innovation Solution
Generating synthetic training data using three-dimensional scans to create models that simulate various conditions, allowing for the generation of diverse training images with manipulations such as distortion, rotation, and addition of distractors, which are used to train machine learning models to improve object recognition and grasping accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If real product data is collected for training machine learning models, then training data quality improves, but data collection time and complexity increase
Solution Approach 1:
The patent creates synthetic copies of product images through 3D modeling and rendering instead of collecting real photographs. A 3D model of the product is generated, then multiple synthetic images are rendered from different angles, lighting conditions, and positions to create a comprehensive training dataset without physical data collection
Solution Approach 2:
The system performs preliminary 3D scanning and modeling of the product before training is needed. This advance preparation creates a reusable digital twin that can generate unlimited training images on-demand, eliminating the need for repeated real-world data collection
2Device complexity
If the robotic system uses traditional vision systems to identify products on conveyor, then system complexity remains manageable, but accuracy in handling touching or overlapping products deteriorates
Solution Approach 1:
The system transitions from 2D image analysis to 3D spatial understanding by creating and processing three-dimensional models of products. This dimensional enhancement allows the machine learning model to distinguish overlapping and touching products by analyzing their 3D positions, depths, and spatial relationships rather than relying solely on 2D projections
Solution Approach 2:
The system changes the parameter space for product identification by incorporating multiple 3D parameters (x, y, z coordinates, rotation angles, scale) into the training data. This enriched parameter set enables the model to differentiate products that appear identical in 2D but have distinct 3D characteristics
3Ease of operation
If the robotic gripper applies consistent gripping force, then system control simplicity is maintained, but adaptability to different product sizes and weights deteriorates
Solution Approach 1:
The system performs preliminary identification and characterization of each product's size, shape, and weight properties using 3D scanning and machine learning analysis before the gripping action occurs. This advance knowledge allows the controller to pre-calculate the optimal gripping force for each specific product, eliminating the need for complex real-time force adjustment mechanisms
Solution Approach 2:
The system uses feedback from the machine learning model about product characteristics (size, shape, weight estimates) to dynamically adjust gripping parameters. The model analyzes product features and provides feedback signals that automatically modify gripper force, position, and orientation to match the specific product being handled
Data Source
AI summary
Exemplary embodiments relate to techniques for generating large amounts of machine learning training data for a robotic pick-and-place station. One or more three-dimensional scans of a product may be acquired. The three-dimensional scans may be used to generate images of the product in different orientations, and/or scenes including multiple such products may be generated. The three-dimensional scan and/or the generated images may be manipulated to generate variations. One or more distractors may be applied to approximate a pick-and-place environment that will operate to pick objects similar to the training object.