Robot Object Modeling With Multi-View Vision for Unfamiliar Items
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robots face challenges in detecting and estimating the pose of objects in their environment when these objects do not match existing 3D models, leading to recognition failures.
Innovation Solution
A method where a robot captures vision sensor data from multiple angles to generate a machine learning model, specifically a CNN, which includes rendering images of the object at various poses and environments to train the model for detection and pose estimation, enabling the robot to recognize and interact with unfamiliar objects.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a robot uses existing 3D models to detect objects, then detection accuracy is improved for known objects, but the robot cannot recognize new or unfamiliar objects
Solution Approach 1:
The system performs preliminary actions by capturing vision sensor data from multiple vantages before the robot encounters the object in operation. This advance data collection enables the generation of a 3D model and trained machine learning model that can be immediately applied when the object is detected, resolving the contradiction between having accurate pre-existing models and being able to recognize new objects.
Solution Approach 2:
The system creates a copy of the unfamiliar object by generating a 3D model from vision sensor data captured from multiple viewpoints. This digital replica serves as a template for training a machine learning model, enabling the robot to recognize and detect the object without requiring physical interaction or prior knowledge of the object's identity.
2Measurement precision
If a robot captures vision sensor data from multiple vantages to generate a model, then the model accuracy is improved, but the time and computational resources required increase
Solution Approach 1:
The system applies partial action by capturing vision sensor data from a selected number of key vantages rather than attempting to capture all possible angles. This approach generates sufficient data to create an accurate 3D model and train an effective machine learning model, achieving good detection performance while minimizing the time and computational resources required.
3Reliability
If a machine learning model is trained with rendered images from multiple poses and environments, then detection robustness is improved, but the training complexity and computational cost increase
Solution Approach 1:
The system uses rendering to create synthetic copies of the object in various poses and environmental conditions. These rendered images serve as training data for the machine learning model, allowing the model to learn robust detection capabilities across diverse scenarios without requiring actual physical examples of the object in every possible situation, thereby reducing training complexity while improving robustness.
Data Source
Figure 1
Figure 2A~2D
Figure 3A
AI summary
Methods and apparatus related to generating a model for an object encountered by a robot in its environment, where the object is one that the robot is unable to recognize utilizing existing models associated with the robot. The model is generated based on vision sensor data that captures the object from multiple vantages and that is captured by a vision sensor associated with the robot, such as a vision sensor coupled to the robot. The model may be provided for use by the robot in detecting the object and/or for use in estimating the pose of the object.