AR-Guided Robot Assembly Learning Without External Sensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for programming robotic systems, such as programming by demonstration (PbD), face challenges when dealing with large, heavy, fragile, or dangerous objects, as they require extensive computational resources and can be tiring or dangerous for operators, especially in complex physical environments.
Innovation Solution
An augmented reality (AR) system is used to enable a human operator to demonstrate assembly tasks using both physical and virtual objects, where the AR device tracks the operator's 3D motion and object poses, allowing the robotic system to learn the task without the need for external sensors in the physical environment, thereby reducing computational and operational burdens.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Object-affected harmful factors
If programming by demonstration is used in a complete virtual environment using VR devices, then operator safety is improved and computational resources are reduced, but the complexity of recreating complex physical environments in VR increases significantly
Solution Approach 1:
The patent introduces an AR device as an intermediary between the physical environment and the robotic system. The AR device captures real-world sensor data and overlays virtual representations, allowing the operator to interact with a simplified virtual model while the system processes real physical environment data. This mediator approach maintains operator safety through virtual interaction while avoiding the complexity of fully recreating physical environments in VR.
Solution Approach 2:
The patent creates a partial virtual copy of physical objects and environments through AR overlays rather than complete VR recreation. Virtual representations of objects are superimposed on the real physical environment, allowing the robotic system to learn from demonstrated actions on virtual copies while operating in the actual physical space. This reduces the computational burden compared to full VR environment recreation.
2Measurement precision
If multiple sensors are disposed in the physical environment to capture sequential configurations, then measurement precision is improved, but device complexity and computational resources increase
Solution Approach 1:
The AR device serves multiple functions: it captures sensor data from the physical environment, renders virtual overlays, tracks operator actions, and transmits data to the robotic system. By consolidating these functions into a single multi-functional device rather than deploying multiple specialized sensors throughout the environment, the system achieves comprehensive measurement precision while reducing overall device complexity.
Solution Approach 2:
The AR device leverages the operator's own device (smartphone, tablet, or AR glasses) to perform sensing and data capture functions. The operator's device already possesses cameras, sensors, and processing capabilities that can be utilized for capturing configuration data, eliminating the need for dedicated external sensor systems and reducing device complexity while maintaining measurement precision.
3Manufacturing precision
If operators manually move robot components through sequential configurations, then the robot learns accurate task demonstrations, but operator fatigue and risk increase when objects are large, heavy, or dangerous
Solution Approach 1:
The operator interacts with virtual copies of robot components and objects displayed through the AR device rather than physically manipulating actual heavy or dangerous objects. The virtual representations allow the operator to demonstrate complete task sequences with full motion ranges without physical strain or safety risks, while the system captures accurate configuration data for robotic learning.
Solution Approach 2:
The AR device acts as an intermediary that translates the operator's virtual manipulation actions into accurate robotic configuration data. The operator moves virtual objects through the AR interface, and the system maps these actions to the corresponding physical robot component configurations, maintaining demonstration accuracy while eliminating the need for direct physical interaction with heavy or dangerous objects.
Data Source
AI summary
A method for programming a robotic system by demonstration is described. In one aspect, the method includes displaying a first virtual object in a display of an augmented reality (AR) device, the first virtual object corresponding to a first physical object in a physical environment of the AR device, tracking, using the AR device, a manipulation of the first virtual object by a user of the AR device, identifying an initial state and a final state of the first virtual object based on the tracking, the initial state corresponding to an initial pose of the first virtual object, the final state corresponding to a final pose of the first virtual object, and programming by demonstration a robotic system using the tracking of the manipulation of the first virtual object, the first initial state of the first virtual object, and the final state of the first virtual object.