Haptic Mixed-Reality Robot Control for Adaptive Teleoperation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current robotic control methods, such as teleoperation and full automation, are limited in their ability to adapt to varied and dynamic environments, requiring either intense human operator involvement or pre-programmed artificial intelligence that struggles with complexity and variability.
Innovation Solution
A haptic-enabled mixed reality system that allows human operators to control robots remotely by visualizing a 3D scene and providing high-level guidance through indirect and intermittent inputs, using RGBD sensors and haptic interfaces to interact with virtualized environments, enabling real-time adaptation and precise control.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If teleoperation mode is used to precisely control every robot movement in real-time, then control precision is improved, but operator burden and complexity increase significantly
Solution Approach 1:
The control task is segmented into two levels: high-level task-oriented instructions from the human operator and low-level movement execution from the robot's pre-programmed AI. This segmentation allows the operator to focus on strategic decisions while the robot handles precise movement execution, resolving the contradiction between control precision and operator burden.
Solution Approach 2:
A virtual assistant agent acts as an intermediary between the human operator and the robot control system. The virtual assistant translates high-level human instructions into detailed movement commands, enabling the operator to maintain control precision without bearing the full burden of real-time movement control.
2Ease of operation
If full automation mode is used with pre-programmed AI control, then operator burden is reduced, but adaptability to varied environments deteriorates
Solution Approach 1:
The control mode dynamically adjusts between full automation and human intervention based on environmental complexity and task requirements. The system transitions from pre-programmed AI control in routine scenarios to human-guided control in varied environments, maintaining both ease of operation and adaptability.
Solution Approach 2:
The system continuously monitors environmental feedback and task progress to determine when human intervention is needed. When the virtual assistant detects situations beyond pre-programmed capabilities, it seamlessly transitions to human operator guidance, ensuring adaptability while maintaining ease of operation through automated routine tasks.
3Adaptability or versatility
If mixed-initiative control mode is implemented with virtual assistant, then adaptability is improved, but system complexity increases
Solution Approach 1:
The virtual assistant serves as an intermediary layer that manages the complexity of mixed-initiative control. It handles the integration between human instructions and robot execution, translating high-level commands into coordinated actions while managing sensor data and environmental modeling, thereby improving adaptability without exposing the full system complexity to the operator.
Solution Approach 2:
The system creates a virtual copy or representation of the physical environment through sensor fusion and mapping. This virtual model allows the virtual assistant to simulate and plan actions before execution, improving adaptability to varied environments while keeping the actual control system complexity manageable through virtual experimentation.
4Loss of information
If haptic feedback is added to the interface, then operator awareness and control are improved, but device complexity and cost increase
Solution Approach 1:
The haptic feedback system is segmented to provide specific tactile information for different interaction scenarios: surface texture feedback for object recognition, resistance feedback for forceful interactions, and vibration feedback for system status. This targeted segmentation improves operator awareness without requiring a fully complex haptic interface for all situations.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A virtual pointer object (118) is displayed within a virtualized environment corresponding to a physical environment currently surrounding a robot (102). The virtualized environment is generated and updated in accordance with streaming environment data received from sensors (106) collocated with the robot (102). First user input is detected via a haptic-enabled input device (114) that causes the virtual pointer object (118) to move along a movement path in the virtualized environment, where the movement path is constrained by simulated surfaces in the virtualized environment. Haptic feedback is generated via the haptic-enabled input device (114) in accordance with simulated material and/or structural characteristics of the movement path. The virtualized environment is modified at the locations of marking inputs along the movement path and affects path planning for the robot (102) within the physical environment.