Dual-Arm Robot Teaching from Natural Two-Hand Pose Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for teaching industrial robots, especially dual arm systems, are unintuitive, time-consuming, and costly, particularly when requiring precise hand motion detection for operations involving two hands, as they often rely on restrictive hand positioning or full-body image analysis, limiting natural hand movement and increasing the risk of misidentification.
Innovation Solution
A method utilizing a first neural network to identify and crop images of both hands from a camera, followed by a second neural network for 3D pose detection, converting hand pose data into robot gripper pose data, allowing dual arm robots to perform operations without artificial hand positioning constraints, and refining motions using edge detection for precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional teach pendant or motion capture systems are used for robot teaching, then the robot can be programmed to perform operations, but the process becomes time-consuming and costly
Solution Approach 1:
The patent replaces traditional mechanical teaching methods (teach pendants) and expensive motion capture systems with a vision-based system using neural networks. The first neural network detects hand identities and the second neural network detects hand poses from camera images, automatically converting human demonstrations into robot teaching data without manual intervention or specialized hardware.
Solution Approach 2:
The system captures and processes visual copies (camera images) of human hand movements to create robot teaching data. Instead of requiring physical interaction with the robot or specialized motion capture equipment, the system copies the visual appearance and motion patterns of hands from standard camera images and translates them into robot control commands.
2Reliability
If full-body image analysis with anthropomorphic analysis is used to identify left and right hands, then hand identity can be determined, but additional computational steps and separate camera images are required
Solution Approach 1:
The patent segments the hand detection task into two specialized neural networks: the first neural network specifically identifies hand identities (left or right), and the second neural network detects hand poses. This segmentation allows each network to be optimized for its specific function, improving accuracy while maintaining computational efficiency compared to full-body anthropomorphic analysis.
Solution Approach 2:
The first neural network serves multiple functions by simultaneously detecting both hand identity and providing input for pose detection. The system uses a single camera image for both identification and pose detection tasks, eliminating the need for separate camera images or full-body analysis while maintaining reliability.
3Ease of operation
If techniques requiring hands to maintain relative positions or remain within positional boundaries are used, then hand identification becomes easier, but natural hand movements are constrained and misidentification risk increases
Solution Approach 1:
The patent implements a dynamic hand identification system where the first neural network identifies hands based on their visual characteristics and spatial relationships in each frame without imposing fixed positional constraints. The system adapts to natural hand movements and crossings, maintaining ease of identification while preserving full movement freedom for the demonstrator.
4Manufacturing precision
If traditional teach pendant methods are used for dual arm robot teaching, then robot operations can be programmed, but the process becomes even more difficult and time-consuming
Solution Approach 1:
The system captures visual copies of dual hand demonstrations and automatically translates them into dual arm robot teaching data. The first and second neural networks process the camera images to identify both hands and their poses, then convert this information into coordinated control commands for two robot arms, dramatically improving teaching efficiency while maintaining operation accuracy.
Data Source
AI summary
A method for dual arm robot teaching from dual hand detection in human demonstration. A camera image of the demonstrator's hands and workpieces is provided to a first neural network which determines the identity of the left and right hand from the image, and also provides cropped sub-images of the identified hands. The cropped sub-images are provided to a second neural network which detects the poses of both the left and right hand from the images. The dual hand pose data for an entire operation is converted to robot gripper pose data and used for teaching two robot arms to perform the operation on the workpieces, where each hand's motion is assigned to one robot arm. Edge detection from camera images may be used to refine robot motions in order to improve part localization for tasks requiring precision, such as inserting a part into an aperture.