Robot Programming from Human Demonstration Using Vision and Force Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for teaching industrial robots to perform pick and place operations are unintuitive, time-consuming, and error-prone, especially for non-expert operators, and existing systems face challenges with accurate detection due to visual occlusion and difficulty in deciphering hand velocity transitions.
Innovation Solution
A method using force and vision sensors to detect the human demonstrator's hand and workpiece positions and state transitions, generating robot programming commands that simplify the teaching process by capturing motions and state change logic, and refining motions to remove extraneous hand movements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a motion capture system with multiple cameras and marker dots is used to teach robot programming, then the accuracy of recording positions and orientations is improved, but the cost and setup complexity increase significantly
Solution Approach 1:
The patent extracts and eliminates the complex motion capture system infrastructure (multiple cameras, marker dots, synchronization equipment) by replacing it with a simplified vision system using standard cameras and computer vision algorithms to detect hand and workpiece positions without requiring specialized hardware infrastructure
Solution Approach 2:
The patent creates a virtual model of the work cell and objects based on images captured by standard cameras, generating coordinate information through image processing rather than direct physical measurement with specialized equipment, thus copying the essential spatial relationships without the complex measurement infrastructure
2Device complexity
If visual tracking alone is used to detect hand and workpiece positions, then the system simplicity is improved, but the accuracy deteriorates due to visual occlusion and difficulty in deciphering hand velocity transitions
Solution Approach 1:
The patent merges multiple detection approaches by combining vision sensor data with force sensor data, where the force sensor detects contact events and state transitions that are difficult to determine from vision alone, compensating for visual occlusion and improving overall detection accuracy
Solution Approach 2:
The patent implements feedback mechanisms where force sensor information about contact forces and state transitions is continuously fed back to refine the interpretation of vision data, allowing the system to correctly identify hand velocity transitions and contact events even when visual information is ambiguous or occluded
3Device complexity
If a teach pendant is used to instruct robot movements incrementally, then the robot programming can be performed with simple equipment, but the time required for programming increases significantly
Solution Approach 1:
The patent uses vision and force sensors to automatically capture and record the human operator's natural hand movements and actions as a model, then copies this demonstrated behavior into robot program commands, eliminating the need for slow incremental teaching while maintaining equipment simplicity
Solution Approach 2:
The patent replaces the mechanical incremental teaching process (manual movement and recording) with an automated sensing and recognition system that captures motion data through vision and force sensors, then processes this data to generate robot commands automatically, substituting manual mechanical teaching with automated sensor-based programming
4Measurement precision
If existing systems use special gloves with sensors to determine hand actions, then the hand motion detection accuracy is improved, but the ease of operation deteriorates due to the special equipment requirement
Solution Approach 1:
The patent enables the human operator to perform demonstration without any special equipment or accessories, using natural hand movements that are captured by external vision and force sensors, thus maintaining ease of operation while achieving accurate hand motion detection through non-contact sensing methods
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The method enables faster, more intuitive, and accurate robot programming by accurately capturing human demonstrations, reducing errors and complexity, and improving the reliability of pick and place operations.
Implementation Method 1
a vision sensor to detect position and pose of the human's hand and optionally a workpiece during teaching of an operation
Implementation Method 2
The force sensor, located either beneath the workpiece or on a tool, is used to detect force information
Data Source
AI summary
A method for teaching a robot to perform an operation based on human demonstration using force and vision sensors. The method includes a vision sensor to detect position and pose of both the human's hand and optionally a workpiece during teaching of an operation such as pick, move and place. The force sensor, located either beneath the workpiece or on a tool, is used to detect force information. Data from the vision and force sensors, along with other optional inputs, are used to teach both motions and state change logic for the operation being taught. Several techniques are disclosed for determining state change logic, such as the transition from approaching to grasping. Techniques for improving motion programming to remove extraneous motions by the hand are also disclosed. Robot programming commands are then generated from the hand position and orientation data, along with the state transitions.