AR Robot Teleoperation With Virtual Surrogates for Safer Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current human-robot interaction systems, particularly in teleoperation and robot intent communication, face challenges in effectively coordinating human and robot activities in shared environments, leading to safety issues and inefficiencies due to limited situational awareness and poor communication of robot intentions.
Innovation Solution
The use of augmented reality (AR) technologies to enhance human-robot interaction by providing intuitive and visual feedback through virtual surrogates, augmented environments, and user interfaces, allowing users to monitor robot states and intentions directly within the operating environment, reducing the need for context switching and improving situational awareness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional teleoperation interfaces are used, then robot control is achieved, but operator situational awareness and task performance deteriorate due to context switching between multiple displays
Solution Approach 1:
The patent merges robot camera feeds, sensor data, robot state information, and environmental data into a single augmented reality display that overlays virtual information onto the physical environment. This consolidation eliminates the need for operators to switch between multiple displays, improving situational awareness while maintaining comprehensive robot monitoring capabilities.
Solution Approach 2:
The system introduces a virtual surrogate as an intermediary representation of the robot's state and intentions. This virtual avatar serves as a mediator between the physical robot and the operator, providing intuitive visual feedback about robot status, sensor detections, and planned actions without requiring direct inspection of multiple technical displays.
2Productivity
If autonomous robot navigation is implemented, then robot mobility and task efficiency improve, but safety and coordination deteriorate due to unpredictable robot behavior
Solution Approach 1:
The system displays the robot's planned future actions and navigation paths in advance through the virtual surrogate before the robot executes them. This preliminary visualization allows human operators and nearby workers to anticipate robot movements and adjust their behavior accordingly, maintaining safety while enabling autonomous operation.
Solution Approach 2:
The virtual surrogate provides continuous feedback about the robot's intended actions, current state, and sensor perceptions. This real-time information flow enables dynamic coordination between autonomous robot behavior and human activities, allowing operators to monitor and intervene when necessary while maintaining efficient autonomous operation.
3Loss of information
If multiple information displays are provided to the operator, then comprehensive robot monitoring is achieved, but operator cognitive load and distraction increase
Solution Approach 1:
The system transitions from two-dimensional displays to three-dimensional augmented reality visualization, overlaying virtual information directly onto the physical environment. Robot state indicators, sensor detections, and navigation paths are positioned in spatial context rather than requiring operators to mentally integrate information from separate flat screens, reducing cognitive load while maintaining information completeness.
Solution Approach 2:
The augmented reality display presents different types of information at appropriate locations in the visual field. Critical safety information appears in prominent positions, while less critical data is distributed throughout the environment. This spatial organization of information quality reduces the mental effort required to prioritize and process multiple data streams.
Data Source
AI summary
Systems and methods for human-robot communication are proved. More particularly, some embodiments use augmented reality to facilitate communication of robot intention and teleoperation in human-robot cooperative environments. Various embodiments of the present technology provide a middleware that integrates augmented reality (AR) with novel teleoperation interfaces to increase operation effectiveness, support the user in conducting concurrent work, and decrease stress. Various embodiments provide predictive graphical interfaces such that a teleoperator controls a virtual robot surrogate, rather than directly operating the robot itself, providing the user with foresight regarding where the physical robot will end up and how it will get there. In accordance with various embodiments a user may select between two AR interfaces using such a surrogate: one focused on real-time control and one inspired by waypoint delegation.