Robot Grasp Pose Generation via Human Pixel Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Robots face challenges in autonomously determining appropriate grasp poses for objects, with existing fully-autonomous methods failing to generate grasp candidates for some objects and human-in-the-loop approaches being time-consuming and computationally demanding.
Innovation Solution
A human-in-the-loop technique that reduces the time and computational resources required, where a user selects pixels in an image to determine 3D points on an object's surface, fitting a local plane and using the plane's normal to define a grasp approach vector for a grasping end effector's pose, ensuring collision-free grasping.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If fully-autonomous approaches are used to generate grasp candidates, then robot independence is improved, but grasp generation reliability deteriorates due to failure to generate candidates for some objects
Solution Approach 1:
The system segments the grasp generation process into two distinct phases: (1) autonomous generation of candidate grasp poses by the robot system, and (2) human validation and selection of feasible grasps. This segmentation allows the robot to independently generate multiple candidates while relying on human expertise to filter and select the most appropriate grasp, thereby maintaining automation while improving reliability.
Solution Approach 2:
The system implements feedback by presenting generated grasp candidates to human operators for validation. Human feedback on which grasps are feasible or infeasible is used to refine and improve the autonomous grasp generation algorithm over time, creating a learning loop that enhances both reliability and automation capability.
2Manufacturing precision
If human-in-the-loop approaches with full 3D representation manipulation are used, then grasp pose specification accuracy is improved, but human time consumption and computational resource usage increase
Solution Approach 1:
Instead of requiring humans to fully specify the complete 6-DOF grasp pose, the system only requires partial input from the user (e.g., selecting objects or providing rough constraints). The autonomous system then generates multiple candidate poses that satisfy these partial constraints, reducing the time and effort required from humans while maintaining accuracy through automated optimization.
Solution Approach 2:
The system uses simplified 2D images or point cloud representations as copies of the full 3D scene, allowing users to interact with less computationally intensive data structures. These simplified representations are then mapped back to full 3D grasp poses, reducing computational resources required while maintaining specification accuracy.
3Adaptability or versatility
If full 3D representations of end effector and object are used in human-in-the-loop approaches, then grasp specification completeness is improved, but computational resource requirements and data availability requirements increase
Solution Approach 1:
The system transforms the problem from 3D space to 2D image space for user interaction, then maps solutions back to 3D. By working in 2D for the human-in-the-loop portion, computational requirements are reduced while still achieving complete 3D grasp specification through the dimensionality transformation and projection processes.
Data Source
AI summary
Generating a grasp pose for grasping of an object by an end effector of a robot. An image that captures at least a portion of the object is provided to a user via a user interface output device of a computing device. The user may select one or more pixels in the image via a user interface input device of the computing device. The selected pixel(s) are utilized to select one or more particular 3D points that correspond to a surface of the object in the robot's environment. A grasp pose is determined based on the particular 3D points. For example, a local plane may be fit based on the particular 3D point(s) and a grasp pose determined based on a normal of the local plane. Control commands can be provided to cause the grasping end effector to be adjusted to the grasp pose, after which a grasp is attempted.