Synthetic Training Data Generation for Unknown Object Grasping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computer vision methods lack efficient generation of training data for identifying unknown objects in unstructured scenes, particularly for robotic grasping, as they require complex geometric scans or CAD models, and struggle with transparent and reflective objects.
Innovation Solution
A method for generating synthetic training data by creating geometrically accurate object meshes using RGB and depth images, adding keypoints, and determining material properties, allowing for the training of object detectors without manual input, and enabling the identification of graspable faces and robotic grasping points.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If complex geometric scans or CAD models are used to generate training data, then manufacturing precision of object models is improved, but device complexity and time consumption increase significantly
Solution Approach 1:
The patent uses photogrammetry to create accurate 3D copies of objects from multiple 2D images taken by standard cameras. Instead of requiring complex geometric scanners, the system captures photographs from multiple angles and automatically reconstructs the 3D mesh, producing geometrically accurate object models suitable for training data generation without specialized scanning equipment.
Solution Approach 2:
The patent replaces mechanical scanning systems with optical photography and computational algorithms. By substituting physical geometric scanners with camera-based photogrammetry, the system achieves comparable or superior geometric accuracy while eliminating complex mechanical equipment and reducing operational complexity.
2Manufacturing precision
If complex geometric scans or CAD models are used to generate training data, then manufacturing precision of object models is improved, but loss of time increases due to manual input requirements
Solution Approach 1:
The patent implements an automated pipeline where the system performs photogrammetry, mesh generation, material property estimation, and training data creation without human intervention. The workflow automatically processes captured images through multiple computational stages, generating complete training datasets including 3D meshes, material properties, and synthetic images solely through algorithmic operations.
Solution Approach 2:
The patent performs preliminary actions by automatically capturing images from multiple angles and pre-processing them into structured 3D models before training data generation. The system prepares complete object representations including geometry and material properties in advance, creating ready-to-use training datasets that eliminate subsequent manual processing steps.
3Reliability
If traditional methods are used for object detection training, then reliability of object identification is improved for known objects, but adaptability to unknown and transparent objects deteriorates
Solution Approach 1:
The patent creates a universal training data generation system that handles diverse object types including opaque, transparent, and reflective objects through a single photogrammetry-based workflow. The system generates comprehensive training data that captures various material properties and lighting conditions, enabling object detectors to reliably identify both known and unknown objects across different categories and materials.
Solution Approach 2:
The patent estimates material properties including transparency and reflectivity parameters from captured images, then uses these parameters to generate synthetic training images that preserve realistic optical characteristics. By incorporating accurate material property parameters into the training data, the system enables detectors to learn from transparent and reflective objects with the same reliability as opaque objects.
4Ease of manufacture
If photogrammetry and automatic mesh generation are used, then ease of manufacture of training data is improved, but measurement precision of material properties may deteriorate
Solution Approach 1:
The patent incorporates feedback mechanisms where the system iteratively refines material property estimates by comparing synthesized images against captured reference images. The workflow adjusts material parameters such as transparency and reflectivity based on how well the generated images match the original photographs, progressively improving measurement precision while maintaining automated operation.
Data Source
AI summary
A method for generating training data can include: determining a set of images; determining a set of masks based on the images; determining a first mesh based on the set of masks; optionally determining a refined mesh by recomputing the first mesh; optionally determining one or more faces of the refined mesh; optionally adding one or more keypoints to the refined mesh; optionally determining a material property set for the object; optionally generating a full object mesh; determining one or more scenes; optionally determining training data based on the one or more scenes; optionally training one or more object detectors using the training data; and detecting one or more objects using the trained object detector.