Robot Grasp Control Using Virtual Hand and RGB-D Point Clouds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotic grasping methods lack generalization capabilities and require extensive prior information or annotations, making them impractical for less structured environments, and data collection for training is labor-intensive and difficult to optimize.
Innovation Solution
A method involving a learning phase where a sequence of RGB-D images of a human hand grasping an object is used to create a virtual hand model aligned with the object's point cloud, followed by a grasping phase where the robot matches this model with the object in a scene to determine grasping controls, without requiring full object reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If model-based grasping methods are used to estimate 6D pose with high accuracy, then measurement precision is improved, but data collection becomes labor-intensive and device complexity increases
Solution Approach 1:
The patent uses a virtual hand model as a simplified copy of the real hand to interact with 3D models of objects. This virtual model captures the essential grasping mechanics without requiring complex real-world data collection, thereby maintaining measurement precision while reducing device complexity and annotation burden.
Solution Approach 2:
The patent transforms the complex 6D pose estimation problem into a more manageable parameter space by using pre-defined virtual hand models with fixed degrees of freedom. This parameter transformation allows accurate grasping prediction without requiring extensive annotated training data for full 6D pose estimation.
2Manufacturing precision
If traditional grasping methods with detailed prior information are used, then grasping precision is improved, but adaptability to new objects deteriorates
Solution Approach 1:
The virtual hand model serves as a universal grasping mechanism that can be applied to multiple object types without requiring object-specific calibration. The model's generic structure allows it to adapt to various object geometries while maintaining grasping precision, thus achieving both precision and versatility.
Solution Approach 2:
The patent performs preliminary alignment between the virtual hand model and the object's 3D model before actual grasping execution. This pre-alignment step ensures precise grasping positioning while the underlying virtual model remains object-agnostic, enabling generalization to unseen objects without sacrificing grasping accuracy.
3Measurement precision
If full object reconstruction is performed to enable accurate grasping, then measurement precision is improved, but loss of time increases
Solution Approach 1:
The patent performs partial reconstruction by aligning the virtual hand model only with the relevant portions of the object geometry needed for grasping, rather than reconstructing the entire object in full detail. This partial action approach maintains sufficient measurement precision for grasping while significantly reducing the time required compared to complete object reconstruction.
Data Source
Figure 1~2
Figure 3
AI summary
A method for controlling a grasping robot comprising: a learning phase including: a. obtaining a sequence (SEQ) of RGB-D images showing a hand holding an object, b. performing (S01) a segmentation of the hand and of the object, c. forming (S02), a 3D volume of the hand and a 3D volume of the object, d. extracting a point cloud of the hand from the 3D volume of the hand and a point cloud of the object from the 3D volume of the object e. positioning (S03) a virtual hand model, f. aligning the virtual hand model with the point cloud of the hand to obtain a training output; and a grasping phase including: g. acquiring a RGB-D image, h. determining a point cloud of the object, i. matching the object point clouds, j. determining a grasping control to be used to grasp, by the grasping robot, the object.