AR Motion Authoring for Adaptive Human-Robot Task Synchronization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing human-robot collaborative systems face inefficiencies and ambiguities in spatially and temporally coordinated tasks due to reliance on explicit communications and require adaptive methods for dynamic interactions, especially in ad-hoc tasks where traditional motion capture systems are cumbersome.
Innovation Solution
An augmented reality (AR) system that records and displays human motions, allowing users to intuitively author and visualize collaborative tasks, enabling real-time inference and feedback for robots to adapt and synchronize with human actions through a graphical user interface and sensor data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional motion capture systems (body-suit or external camera) are used to record human motions, then motion recording accuracy is improved, but device complexity and ease of operation deteriorate due to heavy dependencies and offline capture requirements
Solution Approach 1:
The patent replaces traditional mechanical motion capture systems (body-suits, external cameras) with an augmented reality headset-based system that uses onboard sensors and computer vision algorithms to track human motion, eliminating heavy hardware dependencies while maintaining tracking accuracy
Solution Approach 2:
The system creates a virtual representation (digital twin) of the human operator within the AR environment, copying the operator's motions and actions into the virtual space to enable real-time interaction and task demonstration without requiring physical motion capture equipment
2Loss of information
If explicit communication modalities (speech and gestures) are used for human-robot collaboration, then communication clarity is improved, but efficiency and coordination accuracy worsen due to ambiguities in spatially and temporally coordinated tasks
Solution Approach 1:
The system captures and replicates the human operator's actual physical actions and motions in the virtual environment, creating an accurate behavioral model that eliminates the ambiguities inherent in speech and gesture-based communication by directly observing and copying real human behavior
Solution Approach 2:
The AR system provides real-time visual feedback to the operator showing their captured actions and the robot's responses, enabling continuous adjustment and synchronization of human-robot coordination to improve collaboration efficiency
3Ease of operation
If programming by demonstration is used to generate task plans, then ease of operation is improved, but adaptability to dynamic interactions worsens when robots operate in isolation from human context
Solution Approach 1:
The AR system serves multiple functions simultaneously: it records human motions, tracks actions in real-time, generates task plans, and provides visual feedback, enabling a single integrated system that both simplifies programming and enhances adaptability to dynamic human-robot interactions
Solution Approach 2:
The augmented reality environment acts as an intermediary between the human operator and the robot, mediating the transfer of intent and action information while providing the contextual understanding needed for adaptive collaboration in dynamic situations
4Ease of operation
If ad-hoc task demonstrations are performed with users' bodies, then ease of operation is improved, but measurement precision worsens without proper motion capture infrastructure
Solution Approach 1:
The patent substitutes complex motion capture infrastructure with augmented reality headset sensors and computer vision processing, enabling accurate motion tracking during natural ad-hoc demonstrations without requiring specialized capture equipment or controlled environments
Data Source
AI summary
A system and method for authoring and performing Human-Robot-Collaborative (HRC) tasks is disclosed. The system and method adopt an embodied authoring approach in Augmented Reality (AR), for spatially editing the actions and programming the robots through demonstrative role-playing. The system and method utilize an intuitive workflow that externalizes user's authoring as demonstrative and editable AR ghost, allowing for spatially situated visual referencing, realistic animated simulation, and collaborative action guidance. The system and method utilize a dynamic time warping (DTW) based collaboration model which takes the real-time captured motion as inputs, maps it to the previously authored human actions, and outputs the corresponding robot actions to achieve adaptive collaboration.