Virtual reality and digital twin-based robot teleoperation method and system

Abstract

The present application specifically relates to the technical field of human-machine collaborative intelligent manufacturing and assembly, and provides a virtual reality and digital twin-based robot teleoperation method and system. The solution comprises: starting up an industrial robot and a wearable device; establishing a bidirectional communication connection between a virtual environment of the wearable device and the industrial robot, and rendering a virtual robot of the wearable device in the virtual environment; and inputting a target pose operation instruction into the virtual robot, and in response to receiving a target motion trajectory planned by the wearable device according to the target pose operation instruction, controlling the industrial robot to perform an operation task according to the motion trajectory. The solution enables information interaction between an operator and a robot, thereby achieving immersive real-time robot teleoperation by means of instant and accurate operation commands and on-site feedback, and significantly improving teleoperation accuracy for complex tasks and flexible manipulation tasks.

Description

A method and system for teleoperating robots based on virtual reality and digital twins Technical Field

[0001] This application relates to the field of human-machine collaborative intelligent manufacturing assembly technology, and in particular to a robot teleoperation method and system based on virtual reality and digital twins. Background Technology

[0002] Industrial robots, as a crucial supporting technology for modern manufacturing, are widely used in various production processes, such as machining, welding, and material handling. However, with the increasing complexity of production scenarios and the growing flexibility of tasks, robots cannot autonomously complete all tasks in all situations, leading to the development of robot teleoperation technology.

[0003] Currently, an increasing number of researchers are attempting to incorporate virtual reality (VR) and digital twins into human-computer interaction and robot teleoperation. VR technology can provide operators with an immersive, remote, and real-world environment, and can be combined with multimodal interaction methods, enabling operators to obtain information about the robot, operating platform, and its surrounding environment in a more intuitive way, enhancing their perception and cognitive abilities regarding the environment. Digital twin technology can use digital technology to describe and model the characteristics, behaviors, formation processes, and performance of physical entities. However, motion planning and execution in human-computer interaction and robot teleoperation based on VR and digital twins require operators to possess professional cognitive and operational skills. Otherwise, operators will struggle to obtain environmental information from the remote location, their visual feedback will hinder their ability to build an effective environmental understanding, and they will find it difficult to intuitively operate the robot and its end effectors to perform complex and precise operations, reducing the accuracy of teleoperation.

[0004] Therefore, there is an urgent need for a robot teleoperation method in the field of robot teleoperation in manufacturing scenarios that can provide low cognitive burden and low learning cost, so as to improve the teleoperation accuracy of complex and flexible tasks. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this application is to provide a robot teleoperation method and system based on virtual reality and digital twin, which aims to solve the problem of low teleoperation accuracy for complex and flexible tasks in the prior art.

[0006] To achieve the above objectives, the first aspect of this application provides a robot teleoperation method based on virtual reality and digital twins, comprising:

[0007] Start up industrial robots and wearable devices;

[0008] A bidirectional communication connection is established between the virtual environment of the wearable device and the industrial robot, and the virtual robot of the wearable device is rendered in the virtual environment;

[0009] The target pose operation command is input into the virtual robot. In response to receiving the target motion trajectory planned by the wearable device according to the target pose operation command, the industrial robot is controlled to perform the operation task according to the target motion trajectory.

[0010] Optionally, establishing a bidirectional communication connection between the virtual environment of the wearable device and the industrial robot includes:

[0011] Set the communication address of the communication port of the industrial robot;

[0012] Configure the communication port of the virtual environment based on the communication address;

[0013] Based on the communication port, a two-way communication connection is established between the virtual environment of the wearable device and the industrial robot.

[0014] Optionally, the process of generating the target motion trajectory includes:

[0015] Based on the target pose operation command, the spatial target pose of the virtual robot is determined;

[0016] Based on the spatial target pose, the wearable device uses a preset path planning algorithm to generate the target's motion trajectory.

[0017] Optionally, the step of generating the target motion trajectory based on the spatial target pose using the wearable device with a preset path planning algorithm includes:

[0018] Real-time collection of the industrial robot's operating environment and motion information;

[0019] The working environment information and the motion information are transmitted back and rendered into the virtual environment to update the rendering effect of the virtual robot;

[0020] Based on the rendering effect, the wearable device is controlled to generate the target motion trajectory using a path planning algorithm.

[0021] Optionally, based on the rendering effect, controlling the wearable device to generate the target motion trajectory using a path planning algorithm includes:

[0022] Based on the rendering effect, the virtual robot is aligned with the industrial robot to obtain the current motion posture of the virtual robot;

[0023] Based on the current motion posture of the virtual robot, the wearable device is controlled to generate the target motion trajectory using a path planning algorithm.

[0024] Optionally, the industrial robot includes a robotic arm and a robotic hand, and controlling the industrial robot to perform the task according to the target motion trajectory includes:

[0025] According to the motion trajectory, the handle controller of the industrial robot adjusts the end effector position of the robotic arm, and the data glove controller of the industrial robot adjusts the opening and closing state of the robotic hand's fingers.

[0026] Based on the end effector pose of the robotic arm and the opening / closing state of the fingers, the system performs a task. A second aspect of this application provides a robot teleoperation system based on virtual reality and digital twins, the system comprising:

[0027] An initialization module is used to start industrial robots and wearable devices;

[0028] A two-way communication establishment module is used to establish a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and to render the virtual robot of the wearable device in the virtual environment;

[0029] The task execution module is used to input the target pose operation command into the virtual robot, and in response to receiving the target motion trajectory planned by the wearable device according to the target pose operation command, control the industrial robot to perform the task according to the target motion trajectory.

[0030] Optionally, it also includes a field vision and safety feedback module, which is used to control the field vision and safety feedback module of the industrial robot to collect the working environment information and motion information of the industrial robot in real time; and to control the field vision and safety feedback module to transmit the working environment information and motion information back and render them into the virtual environment.

[0031] A third aspect of this application provides a smart terminal, the smart terminal including a memory, a processor, and a robot teleoperation program based on virtual reality and digital twin stored in the memory and executable on the processor, wherein when the robot teleoperation program based on virtual reality and digital twin is executed by the processor, it implements any one of the steps of the above-described robot teleoperation method based on virtual reality and digital twin.

[0032] A fourth aspect of this application provides a computer-readable storage medium storing a robot teleoperation program based on virtual reality and digital twins, wherein the robot teleoperation program based on virtual reality and digital twins, when executed by a processor, implements any of the steps of the above-described robot teleoperation method based on virtual reality and digital twins.

[0033] Compared with existing technologies, the beneficial effects of this solution are as follows:

[0034] This application establishes a two-way communication connection between a wearable device's virtual environment and an industrial robot, rendering the virtual robot within the virtual environment. Virtual reality devices provide an input method that aligns with human operational intuition, enabling operators to quickly learn to send immediate operational commands, reducing learning costs and improving operational efficiency. By utilizing the target motion trajectory planned by the wearable device according to the target pose operation commands, the industrial robot is controlled to execute tasks along the trajectory. This allows for information interaction between the operator and the robot, achieving immersive real-time robot teleoperation through immediate and accurate operational commands and on-site feedback, significantly improving the teleoperation accuracy for complex and flexible tasks. Furthermore, this application employs a modular approach to decouple the operation and execution ends, allowing each module to meet the customized needs of actual tasks and satisfy various complex flexible production requirements. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 is a summary flowchart of the robot teleoperation method based on virtual reality and digital twin of this application;

[0037] Figure 2 is a detailed flowchart of the robot teleoperation method based on virtual reality and digital twin of this application;

[0038] Figure 3 is a block diagram of the robot teleoperation system module and data interaction based on virtual reality and digital twin of this application;

[0039] Figure 4 is a schematic diagram of the robot teleoperation system module based on virtual reality and digital twin of this application;

[0040] Figure 5 is a schematic diagram of the intelligent terminal structure of this application. Detailed Implementation

[0041] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0042] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0043] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0044] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0045] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0047] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0048] In the field of human-machine collaborative intelligent manufacturing assembly within production scenarios, as the flexibility of production scenarios and tasks increases, traditional operation methods based on displays and button inputs continuously increase the cognitive and operational burden on operators. Under traditional robot teleoperation modes, operators struggle to obtain environmental information from the remote site, and visual feedback hinders their ability to build an effective understanding of the environment. Furthermore, operators find it difficult to intuitively manipulate the robot and its end effectors to perform complex and precise operations, reducing the efficiency of teleoperation.

[0049] It is evident that the current field of robot teleoperation in manufacturing scenarios lacks a robot teleoperation system that offers low cognitive burden and low learning cost. Based on this, this paper proposes a robot teleoperation method based on virtual reality and digital twins. It primarily uses virtual reality technology to provide operators with an immersive remote environment, enabling them to obtain information about the robot, the operating platform, and its surrounding environment in a more intuitive way, enhancing their perception and cognition of the environment. By combining virtual reality technology with multimodal data provided by data glove controllers and ergonomic handles, the operating logic becomes more intuitive, reducing the operator's cognitive burden and learning cost, and achieving safe, natural, and efficient real-time two-way communication. Therefore, for operators, this solution, aided by virtual reality technology, enhances their cognitive ability regarding the production environment, reducing their learning cost and cognitive burden. For industrial robots, it enables real-time reception of operator action commands and execution of corresponding operations, significantly improving the teleoperation accuracy for complex and flexible tasks, and providing comprehensive feedback to operators through multimodal data.

[0050] This application provides a robot teleoperation method based on virtual reality and digital twins, deployed on electronic devices such as computers and servers, and applied to robot manufacturing scenarios, specifically addressing robot teleoperation assisted by virtual reality and digital twins. Specifically, as shown in Figures 1-3, the steps of this embodiment include:

[0051] Step S100: Start the industrial robot and wearable device;

[0052] Specifically, in this embodiment, the operator controls the on-site robot (i.e., the industrial robot) by wearing a wearable device. First, the industrial robot control module needs to be activated, loading the initial control parameters of the control system. The initial position is then calibrated using sensors to ensure the robotic arm is ready to work. Monitoring equipment such as on-site cameras and infrared detectors are also activated to provide initialization information and communication ports for subsequent bidirectional communication between the wearable device and the industrial robot. Next, the wearable device is activated, along with its virtual environment module. A virtual model of the physical equipment and work area is loaded and rendered into the wearable device to construct and render a virtual environment, providing virtual environment data and communication ports for subsequent bidirectional communication.

[0053] This embodiment does not specifically limit the types of industrial robots and wearable devices. Industrial robots can be collaborative robots, handling robots, welding robots, etc., and wearable devices can be smart glasses, smartwatches, smart clothing, etc., worn on one or more parts of the body. For example, a wearable device can be a composite wearable device consisting of a virtual reality head-mounted display device and a virtual reality wearable input device, so as to immerse the user in the surrounding environment through the virtual reality head-mounted display device and accurately simulate virtual actions and adjust virtual action postures through the virtual reality wearable input device to control the movement of industrial robots.

[0054] Furthermore, to ensure the safety of production operations, an on-site vision and safety feedback module is installed in the industrial robot. When the on-site vision and safety feedback module is activated, it collects information about the on-site environment to monitor potential dangerous actions such as collisions and squeezing that may occur during the operation. If a potential dangerous situation is detected, early warning information is promptly fed back so that the operator can adjust the control actions of the industrial robot in a timely manner.

[0055] Step S200: Establish a bidirectional communication connection between the virtual environment of the wearable device and the industrial robot, and render the virtual robot of the wearable device in the virtual environment;

[0056] Specifically, before establishing a bidirectional communication connection between the wearable device's virtual environment and the industrial robot, compatible software interfaces, including communication protocols, data formats, and instruction sets, are designed for both the wearable device and the industrial robot to ensure they can correctly understand each other's data and instructions. Then, a hardware connection is established between the wearable device and the industrial robot via wireless communication (such as Wi-Fi or Bluetooth) or wired communication to facilitate data exchange and instruction transmission. Based on this hardware connection and software interface, bidirectional communication between the wearable device and the industrial robot can be achieved, enabling them to send and receive data, instructions, and status information. For example, the wearable device can send instructions to the industrial robot, requesting it to perform specific tasks; simultaneously, the industrial robot can also provide feedback on task progress and error information to the wearable device.

[0057] Then, the virtual robot from the wearable device is rendered in the virtual environment. Specifically, this involves: First, using technologies such as Virtual Reality (VR), Augmented Reality (AR), or Mixed Reality (MR), a virtual environment is constructed, including various virtual objects, scenes, and interactive elements. This provides operators with an immersive, remote, and present environment, enabling them to obtain information about the robot, the operating platform, and its surroundings in a more intuitive way, enhancing their perception and cognitive abilities regarding the environment. Next, within the virtual environment, a corresponding virtual robot is designed based on the characteristics and requirements of the wearable device, ensuring that the virtual robot meets its interactivity, operability, and visualization requirements within the virtual environment. The virtual robot is essentially a digitized industrial robot, possessing similar functions and appearance. Finally, a graphics rendering engine (such as Unity or Unreal Engine) is used to render the virtual robot into the virtual environment, converting the virtual robot's 3D model into an image that can be displayed on the screen, ensuring that the virtual robot has a realistic appearance and dynamic effects within the virtual environment.

[0058] In one implementation, step S200, establishing a bidirectional communication connection between the virtual environment of the wearable device and the industrial robot, includes:

[0059] Step S210: Set the communication address of the communication port of the industrial robot;

[0060] Specifically, determine the communication port of the industrial robot. This port can be a network interface (such as an Ethernet interface or Wi-Fi module) or a serial communication interface (such as RS-232, RS-485, or USB). Then, assign a unique communication address to the industrial robot's communication port; this can be an IP address (for the network interface) or a serial port address (for the serial communication interface). Configure the robotic arm control and safety parameters through the industrial robot's controller, and set the robot's initial posture. Start the industrial robot's network communication port, set the communication address, and wait for the virtual environment's communication port to connect.

[0061] Step S220: Configure the communication port of the virtual environment based on the communication address;

[0062] Specifically, in the virtual environment of the wearable device, configure a communication port corresponding to the industrial robot's communication port. If the industrial robot uses a network interface, a network interface also needs to be configured in the virtual environment, and the same subnet and gateway should be set up to ensure that both are on the same network. If the industrial robot uses a serial communication interface, a virtual serial port (such as a software-simulated serial port) may be needed in the virtual environment to communicate with the industrial robot. After configuring the communication port, start the network communication module of the wearable device and match the communication address of the virtual environment's communication port with the communication port of the industrial robot's host computer.

[0063] Step S230: Based on the communication port, establish a bidirectional communication connection between the virtual environment of the wearable device and the industrial robot.

[0064] Specifically, the communication port of the wearable device and the host computer communication port of the industrial robot are connected. A two-way communication connection is established between the virtual environment of the wearable device and the industrial robot through the same communication protocol and data format. In other words, the real-time synchronization of information interaction is achieved through the digital twin communication mechanism, which lays a good foundation for data and information exchange during the subsequent execution of work tasks.

[0065] In this embodiment, according to the type and address of the industrial robot's communication port, the corresponding communication port and address are configured in the virtual robot, which can easily and quickly establish a two-way communication relationship between the virtual and real robots, and improve the timeliness of data and information transmission between them.

[0066] Step S300: Input the target pose operation command into the virtual robot, and in response to receiving the target motion trajectory planned by the wearable device according to the target pose operation command, control the industrial robot to perform the work task according to the target motion trajectory.

[0067] Specifically, operators generate target pose operation commands through gestures, buttons, voice, and other methods, and input these commands into the virtual environment to control the virtual robot. These target pose operation commands include, but are not limited to, the target position (coordinates) and orientation (direction, angle, etc.) that the end effector (such as a gripper or welding torch) of the industrial robot needs to achieve. It is evident that virtual reality devices can provide input methods that align with human operational intuition, enabling operators to learn to send immediate operation commands in a short time, reducing learning costs while improving operational efficiency.

[0068] After receiving the target pose operation command, the virtual robot uses a path planning algorithm to plan the target motion trajectory from its current position to the target position. This trajectory consists of a series of points or path segments, each containing information such as the robot's position, velocity, and acceleration. The target motion trajectory is then transmitted to a wearable device. Upon receiving the trajectory, the industrial robot executes it using its control system and actuators. During the execution of the task within the effective human-robot working area, the industrial robot combines the acquired target motion trajectory information, real-time pose information, and pre-labeled robot joints—multimodal input information—within a deep reinforcement learning algorithm to generate the end effector's trajectory. An inverse kinematics solver then calculates the required joint space combination sequence for the industrial robot. The process of calculating the required joint space combination sequence using the inverse kinematics solver includes: establishing an industrial robot model, defining the target position and pose sequence, calculating the joint angles corresponding to each joint point using inverse kinematics principles, and generating the joint space combination sequence to ensure the industrial robot can accurately execute the target motion trajectory. Meanwhile, mapping the joint space combination sequence to the joint space of the virtual robot through the network communication module allows the operator to adjust the motion trajectory in real time based on the feedback of the joint space combination sequence.

[0069] In this embodiment, a novel robot teleoperation mode is designed to fully combine the operator's flexibility and adaptability with the robot's reliability and accuracy. This allows the operator to remotely control the industrial robot and its end effector in an area far from the danger zone. This not only enables the industrial robot to perform more complex operations more flexibly, thereby significantly improving the teleoperation accuracy and efficiency of complex and flexible tasks, but also ensures the safety of the operator.

[0070] Step S320, which involves generating the target motion trajectory based on the spatial target pose using the wearable device and a preset path planning algorithm, includes:

[0071] Step S321: Collect the working environment information and motion information of the industrial robot in real time;

[0072] Specifically, the operator controls the industrial robot's on-site vision and safety feedback module, which collects real-time information about the robot's operating environment and motion. This module includes an on-site vision unit and a safety feedback unit. The on-site vision unit uses high-precision cameras and other vision sensors to capture various images and videos of the robot's operation, and performs rapid and accurate analysis to provide intuitive visual feedback to the operator. The images and videos include the robot's own motion status, such as position, speed, and posture, as well as obstacles and objects in the operating environment. The safety feedback unit uses multiple built-in sensors to monitor the interaction between the industrial robot and its environment in real time, ensuring the robot's safety and reliability during operation. These sensors include force sensors, temperature sensors, and pressure sensors. Upon detecting potential safety hazards, such as collision risks with obstacles, overheating, or overpressure, the module immediately triggers a safety alarm and warns the operator through various means, including sound, light, and electricity. Simultaneously, the module can automatically adjust the robot's motion parameters or perform an emergency stop operation according to preset safety strategies to prevent accidents. It is evident that the on-site vision and safety feedback module provides operators with comprehensive and accurate data support by collecting and analyzing the industrial robot's operating environment and motion information in real time. This not only improves the robot's operating efficiency and accuracy but also greatly enhances the safety and reliability of the operation process.

[0073] Step S322: The working environment information and the motion information are transmitted back and rendered to the virtual environment to update the rendering effect of the virtual robot;

[0074] Specifically, the operator controls the on-site vision and safety feedback module to transmit work environment information and motion information back to the virtual environment. The wearable device performs operations such as coordinate transformation, scale adjustment, and physical attribute matching on the received work environment information and motion information, dynamically adjusting the rendering effect on the virtual environment and virtual robot, and feeding back the real-time rendering results and analysis results in the virtual environment to the operator. This allows the operator to more intuitively understand the dynamic changes in the working environment and motion state of the industrial robot, thereby making more accurate and efficient decisions.

[0075] Step S323: Based on the rendering effect, control the wearable device to generate the target motion trajectory using a path planning algorithm.

[0076] Specifically, based on the rendering effect, the virtual robot is aligned with the industrial robot. This means synchronizing the collected industrial robot information with the virtual robot's state to ensure consistency in time and space, thereby obtaining the virtual robot's current motion posture, including its position, orientation, and joint angles. If the virtual robot and the industrial robot are out of sync, both are restarted, and the alignment operation is performed.

[0077] Based on the current motion posture of the virtual robot, considering the kinematic constraints (such as joint limitations and speed limits) and dynamic characteristics (such as inertia and friction) of the wearable device, as well as obstacles and obstacle avoidance strategies in the working environment, the wearable device is controlled to generate a target motion trajectory using a path planning algorithm. This generated target motion trajectory is then converted into a sequence of instructions that the wearable device can understand and execute. It is easy to understand that during the execution of the task, the device's status and environmental changes are continuously monitored, and the trajectory is corrected or replanned as necessary.

[0078] In this embodiment, by collecting real-time information on the working environment and motion of the industrial robot, comprehensive and accurate data support is provided to the operator, ensuring that the robot can perform tasks safely and efficiently. By aligning the virtual robot with the industrial robot, the accuracy and effectiveness of the planned target motion trajectory can be improved.

[0079] In one embodiment, if the industrial robot includes a robotic arm and a robotic hand, step S300, controlling the industrial robot to perform the task according to the target motion trajectory, includes:

[0080] Step S330: According to the motion trajectory, control the handle controller of the industrial robot to adjust the end pose of the robotic arm, and control the data glove controller of the industrial robot to adjust the opening and closing state of the robotic hand's fingers;

[0081] Specifically, the wearable device in this embodiment includes a head-mounted virtual reality device and a wearable device. The head-mounted virtual reality device is used for immersive experience of the surrounding environment of the workspace, while the wearable device is used for accurately simulating virtual movements and adjusting virtual movement postures, as well as executing input operations for movement commands. The wearable device includes a robotic arm and a five-finger robotic hand. The robotic arm's end-effector posture can be changed by pressing buttons on the controller handle, including adjusting the wrist direction and movement speed. The opening and closing of the five fingers of the robotic hand can be changed by the data glove controller, including adjusting the opening and closing state of the fingers. In actual operation, the data glove controller and controller handle need to be activated and connected to the virtual reality engine beforehand, and their corresponding models need to be rendered on the head-mounted display device. The robot receives instructions from the operator and attempts to move to the target position.

[0082] Step S340: Execute the task based on the end-effector pose of the robotic arm and the opening / closing state of the fingers.

[0083] Specifically, the robot arm is controlled to move towards the target pose along the target motion trajectory; the robot arm motor settings are initialized, and the target motion trajectory is transmitted to the robot arm motors to drive the robot arm motors so that each robotic finger moves to its designated position. This enables the robot to perform real-time operations with six degrees of freedom in the workspace using a data glove controller and a handle controller to complete various flexible operations. At the same time, the vision and safety feedback module captures the robot's movement and on-site environmental information in real time and transmits it back to the virtual environment to be rendered as dynamic images for operator monitoring; the virtual engine renders the input controller model in real time so that the operator can confirm their input operations; the infrared detector continuously detects whether the robot has entered a restricted area. If the robot arm moves into a restricted area, the safety detection module issues a stop command, the robot will interrupt its movement and return to the initial position, and repeat steps S330 and S340 until the industrial robot's robot arm and five-fingered robotic hand have all moved to the target pose, stop moving, and wait for new operation commands in a loop.

[0084] In this embodiment, the accuracy of performing tasks along the target motion trajectory is improved by the mutual coordination of the head-mounted virtual reality device and the wearable device; by setting up a robotic arm and a five-fingered robotic hand for the wearable device, the flexibility and precision of the movements can be effectively improved.

[0085] Furthermore, to address the issue of translating operator commands into actual robot operations, step S330 involves motion planning for the robotic arm and five-finger manipulator (i.e., the end effector). Specifically, this includes: First, transforming the target spatial coordinates input by the operator via the handle controller into target points within the industrial robot coordinate system. Next, reading the spatial attitude quaternion data from each inertial sensor of the data glove controller at a set frequency, and calculating the angles formed by the fingertip sensors and wrist sensors using the angle calculation module, converting these angles into the corresponding manipulator finger posture input format. Finally, through the established bidirectional communication relationship, sending this data from the operator's terminal to the on-site robot host computer and inputting it into the robot and end effector drives.

[0086] In summary, the beneficial effects of the method described in this application include:

[0087] This application enables real-time acquisition of operator input commands, robot status, and work environment information, facilitating real-time information exchange between the operator and the robot. This allows for immersive real-time remote robot operation through immediate and accurate commands and on-site feedback. Virtual reality devices provide an input method that aligns with human intuition, enabling operators to learn to send immediate commands quickly, reducing learning costs and improving operational efficiency. The industrial robot and the virtual robot achieve real-time synchronization through a digital twin communication mechanism, while visual information is transmitted back to the operator in real-time via visual sensors, ensuring immediate feedback to the operator during operation.

[0088] The system proposed in this application is modular, thus possessing high adaptability and scalability. The system places the operator in a safe position while the robot operates within the on-site work environment, allowing for flexible changes to the on-site operating environment while ensuring the necessary environmental requirements for robot operation are met. Because this application employs a modular approach to decouple the operating and execution ends, each module can be customized according to actual task requirements, such as replacing the robotic arm's end effector or changing the input method of the operator's wearable device. Furthermore, functional modules can be added to the existing system based on task needs to meet various complex flexible production requirements.

[0089] As shown in Figure 4, corresponding to the above-mentioned robot teleoperation method based on virtual reality and digital twins, this application embodiment also provides a robot teleoperation system based on virtual reality and digital twins. The robot teleoperation system based on virtual reality and digital twins includes:

[0090] Initialization module 410 is used to start industrial robots and wearable devices;

[0091] The bidirectional communication establishment module 420 is used to establish a bidirectional communication connection between the virtual environment of the wearable device and the industrial robot, and to render the virtual robot of the wearable device in the virtual environment;

[0092] The task execution module 430 is used to input the target pose operation command into the virtual robot, and in response to receiving the target motion trajectory planned by the wearable device according to the target pose operation command, control the industrial robot to perform the task according to the target motion trajectory.

[0093] Furthermore, the above system also includes a field vision and safety feedback module, which is used to control the field vision and safety feedback module of the industrial robot to collect the working environment information and motion information of the industrial robot in real time; and to control the field vision and safety feedback module to transmit the working environment information and motion information back and render them into the virtual environment.

[0094] Specifically, in this embodiment, the specific functions of the robot teleoperation system based on virtual reality and digital twin can also be referred to the corresponding description in the robot teleoperation method based on virtual reality and digital twin, which will not be repeated here.

[0095] Based on the above embodiments, this application also provides a smart terminal, the principle block diagram of which is shown in Figure 5. The smart terminal includes a processor, a memory, a network interface, and a display screen connected via a system bus. The processor of the smart terminal provides computing and control capabilities. The memory of the smart terminal includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a robot teleoperation program based on virtual reality and digital twins. The internal memory provides an environment for the operation of the operating system and the robot teleoperation program based on virtual reality and digital twins stored in the non-volatile storage medium. The network interface of the smart terminal is used for communication with external terminals via a network connection. When the robot teleoperation program based on virtual reality and digital twins is executed by the processor, it implements the steps of any of the above-described robot teleoperation methods based on virtual reality and digital twins. The display screen of the smart terminal can be a liquid crystal display screen or an e-ink display screen.

[0096] Those skilled in the art will understand that the principle block diagram shown in Figure 5 is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on the smart terminal to which the solution of this application is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0097] In one embodiment, a smart terminal is provided, the smart terminal including a memory, a processor, and a robot teleoperation program based on virtual reality and digital twin stored in the memory and executable on the processor. When the robot teleoperation program based on virtual reality and digital twin is executed by the processor, it implements the steps of any robot teleoperation method based on virtual reality and digital twin provided in the embodiments of this application.

[0098] This application also provides a computer-readable storage medium storing a robot teleoperation program based on virtual reality and digital twins. When the robot teleoperation program based on virtual reality and digital twins is executed by a processor, it implements the steps of any of the robot teleoperation methods based on virtual reality and digital twins provided in this application.

[0099] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0101] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0103] In the embodiments provided in this application, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of the modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0104] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions are not in essence a departure from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for teleoperating a robot based on virtual reality and digital twin, characterized in that, Includes the following steps: Start up industrial robots and wearable devices; A bidirectional communication connection is established between the virtual environment of the wearable device and the industrial robot, and the virtual robot of the wearable device is rendered in the virtual environment; The target pose operation command is input into the virtual robot. In response to receiving the target motion trajectory planned by the wearable device according to the target pose operation command, the industrial robot is controlled to perform the operation task according to the target motion trajectory.

2. The robot teleoperation method based on virtual reality and digital twin according to claim 1, characterized in that, The establishment of a bidirectional communication connection between the virtual environment of the wearable device and the industrial robot includes: Set the communication address of the communication port of the industrial robot; Configure the communication port of the virtual environment based on the communication address; Based on the communication port, a two-way communication connection is established between the virtual environment of the wearable device and the industrial robot.

3. The robot teleoperation method based on virtual reality and digital twins according to claim 1, characterized in that, The process of generating the target motion trajectory includes: Based on the target pose operation command, the spatial target pose of the virtual robot is determined; Based on the spatial target pose, the wearable device uses a preset path planning algorithm to generate the target's motion trajectory.

4. The robot teleoperation method based on virtual reality and digital twin according to claim 3, characterized in that, The step of generating the target motion trajectory based on the spatial target pose using the wearable device and a preset path planning algorithm includes: Real-time collection of the industrial robot's operating environment and motion information; The working environment information and the motion information are transmitted back and rendered into the virtual environment to update the rendering effect of the virtual robot; Based on the rendering effect, the wearable device is controlled to generate the target motion trajectory using a path planning algorithm.

5. The robot teleoperation method based on virtual reality and digital twin according to claim 4, characterized in that, Based on the rendering effect, controlling the wearable device to generate the target motion trajectory using a path planning algorithm includes: Based on the rendering effect, the virtual robot is aligned with the industrial robot to obtain the current motion posture of the virtual robot; Based on the current motion posture of the virtual robot, the wearable device is controlled to generate the target motion trajectory using a path planning algorithm.

6. The robot teleoperation method based on virtual reality and digital twin according to claim 1, characterized in that, The industrial robot includes a robotic arm and a robotic hand. Controlling the industrial robot to perform tasks according to the target motion trajectory includes: According to the motion trajectory, the handle controller of the industrial robot adjusts the end effector position of the robotic arm, and the data glove controller of the industrial robot adjusts the opening and closing state of the robotic hand's fingers. The task is performed based on the end-effector pose and the opening / closing state of the fingers.

7. A robot teleoperation system based on virtual reality and digital twin, characterized in that, The system includes: An initialization module is used to start industrial robots and wearable devices; A two-way communication establishment module is used to establish a two-way communication connection between the virtual environment of the wearable device and the industrial robot, and to render the virtual robot of the wearable device in the virtual environment; The task execution module is used to input the target pose operation command into the virtual robot, and in response to receiving the target motion trajectory planned by the wearable device according to the target pose operation command, control the industrial robot to perform the task according to the target motion trajectory.

8. The robot teleoperation system based on virtual reality and digital twin according to claim 7, characterized in that, It also includes a field vision and safety feedback module, which controls the field vision and safety feedback module of the industrial robot to collect the working environment information and motion information of the industrial robot in real time; and controls the field vision and safety feedback module to transmit the working environment information and motion information back and render them into the virtual environment.

9. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a robot teleoperation program based on virtual reality and digital twin stored in the memory and executable on the processor. When the robot teleoperation program based on virtual reality and digital twin is executed by the processor, it implements the steps of the robot teleoperation method based on virtual reality and digital twin as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a robot teleoperation program based on virtual reality and digital twins, which, when executed by a processor, implements the steps of the robot teleoperation method based on virtual reality and digital twins as described in any one of claims 1-6.

Images