Robot control system and robot control method

Abstract

The present disclosure relates to a robot control system and a robot control method, which, when a robot operator remotely controls a robot to generate a map for the robot to autonomously navigate in the field, enable the robot to move according to an operating command and even with a simple operating command (operation). The robot control method may comprise the steps of: receiving sensor data from a robot; receiving, from an operator, a target point command regarding a target point to which the robot is to move; generating, on the basis of the sensor data and the target point command, a waypoint for the robot to move to the target point; generating a movement path for moving to the waypoint; and transmitting, to the robot, a driving command according to the movement path.

Description

Robot control system and robot control method

[0001] The present disclosure relates to a robot control system and a robot control method.

[0002] A robot refers to a machine or mechanical device that has a human-like appearance and function, is programmable, and automatically performs a complex series of actions.

[0003] The conventional robot industry has mainly focused on industrial robots, but recently, due to advancements in robot technology and public interest in using robots, the number of robots performing various functions is increasing.

[0004] In particular, mobile robots refer to robots capable of changing their position and moving or driving using their own power, and one of the biggest challenges for such mobile robots is their driving capability. To reach a desired destination, mobile robots must be able to independently perceive their surroundings, determine their current location, and formulate a driving path plan based on this information.

[0005] In particular, as autonomous driving technology that makes decisions and operates on its own without human intervention has been integrated into robots, robots have reached a level where they can drive without human control.

[0006] For a robot to be installed at a site (or service area) and operate autonomously, a map for autonomous driving must be generated and loaded onto the robot. To this end, during installation, a robot operator may need to visit the site in person and control the robot to move within the site to acquire the map.

[0007] However, having a robot operator visit the site every time a robot is installed can result in significant labor and time costs.

[0008] To reduce costs and time, instead of the robot operator visiting the site in person, one may consider having the operator remotely access the robot and remotely control it to move around the site and acquire the map.

[0009] However, remotely controlling a robot to move around the site without accidents while viewing camera footage transmitted by the robot on a limited-sized monitor requires a high level of remote control proficiency from the robot operator. Even a minor error in remote control can cause a major accident involving the robot and / or the site.

[0010] The present disclosure is proposed to solve the aforementioned problems and aims to provide a robot control system and a robot control method that enable a robot to move within a site according to an operating command with simple operating commands (or operations) when a robot operator remotely controls a robot, for example, to generate a map for the robot to drive autonomously in a site.

[0011] To achieve the above objective, the present disclosure may provide a robot control system comprising a robot, a robot control server, and a terminal, wherein the robot control server receives sensor data from the robot and receives a target point command regarding a target point to which the robot must move from the terminal, generates a waypoint for the robot to move to the target point based on the sensor data and the target point command, generates a movement path for moving to the waypoint, and controls the robot to transmit a driving command according to the movement path.

[0012] The above sensor data may include LiDAR sensor data sensed by the robot.

[0013] The robot control server can repeatedly generate new waypoints from the time the robot arrives at the waypoint until it reaches the target point.

[0014] The robot control server can generate the waypoint by selecting a candidate waypoint area considering the position of the robot, performing a convolution operation on the candidate waypoint area with a 2D convolution filter, and reflecting weights according to the distance and direction of the target point in the result of the convolution operation.

[0015] The robot control server can output the drivable area and drivable area of ​​the robot based on the sensor data to the terminal.

[0016] The above target point command is a voice command of an operator input into the terminal, and the robot control server can infer the target point command by converting the voice command into Speech-to-Text (STT) and performing prompt engineering based on a Large Language Model (LLM) on the STT-converted voice command.

[0017] The robot control server can determine at least one action according to the target point command, establish a local plan for driving the robot based on the at least one action, and control the robot to transmit the local plan as a driving command.

[0018] The robot control server can build a first map for the robot based on sensor data received from the robot driving according to the driving command, and control the installation of the first map on the robot.

[0019] The robot control server can build a second map for the robot based on sensor data received from the robot driving according to the first map, and control the installation of the second map on the robot.

[0020] The robot control server can control the coordinate transformation of the second map based on the feature points of the first map and the feature points of the second map, and the installation of the coordinate-transformed second map on the robot.

[0021] In addition, to achieve the above objective, the present disclosure may provide a robot control method comprising the steps of: receiving sensor data from a robot; receiving a target point command from an operator regarding a target point to which the robot is to move; generating a waypoint for the robot to move to the target point based on the sensor data and the target point command; generating a movement path for moving to the waypoint; and transmitting a driving command according to the movement path to the robot.

[0022] The effects of the robot control system and robot control method according to the present disclosure are described as follows.

[0023] According to at least one of the various aspects of the present disclosure, there is an advantage that, when a robot operator controls a robot remotely, the robot can move around the site in accordance with the operating command with a simple operating command (or operation).

[0024] In other words, according to the present disclosure, a remote assisted driving mode is proposed that enables easier remote control by utilizing sensor information received from a robot. The robot transmits sensor data to a remote control server, and an operator performs remote control of the robot through the remote control server, while minimizing the operator's operation. Therefore, the impact related to the operator's proficiency can be minimized. Furthermore, support for multi-robots can also be achieved with minimal remote driving intervention.

[0025] In addition, according to at least one of the various aspects of the present disclosure, there is an advantage in that a map reflecting changes in the site can be generated based on a map mounted on the robot even while the robot is autonomously driving in the site, and the generated map can be newly mounted on the robot.

[0026] FIG. 1 is a block diagram showing the configuration of a robot according to one aspect of the present disclosure.

[0027] FIG. 2 is a block diagram showing the configuration of a robot control system according to one aspect of the present disclosure.

[0028] FIG. 3 is a flowchart showing the process of generating an autonomous driving map for a robot and mounting it on a robot according to one aspect of the present disclosure.

[0029] FIG. 4 is an example of sensor data that can be displayed on an operator terminal for remote intervention by a robot operator according to one aspect of the present disclosure.

[0030] FIG. 5 is a flowchart for field driving control of a robot through remote intervention by an operator according to one aspect of the present disclosure.

[0031] FIG. 6 is a conceptual diagram of a driving path setting for a robot according to one aspect of the present disclosure.

[0032] FIG. 7 is a flowchart of a driving path setting for a robot according to one aspect of the present disclosure.

[0033] FIG. 8 is an example of waypoint generation for robot driving control according to one aspect of the present disclosure.

[0034] FIGS. 9 to 12 are examples of target point commands for driving a robot according to one aspect of the present disclosure.

[0035] FIGS. 13 to 15 are flowcharts of the execution of an action of a robot according to one aspect of the present disclosure.

[0036] FIG. 16 is a flowchart for field driving control of a robot through remote intervention by an operator according to one aspect of the present disclosure.

[0037] FIG. 17 is an example of a robot control system according to one aspect of the present disclosure.

[0038] FIGS. 18 and 19 are examples of execution screens of a robot control application according to one aspect of the present disclosure.

[0039] FIG. 20 is a flowchart showing how an autonomous driving map for a robot is updated and mounted on a robot according to one aspect of the present disclosure.

[0040] FIG. 21 is an example of an execution screen of a robot control application according to one aspect of the present disclosure.

[0041] FIG. 22 is a flowchart for coordinate transformation of an autonomous driving map for a robot according to one aspect of the present disclosure.

[0042] FIG. 23 and FIG. 22 are examples of coordinate transformation of an autonomous driving map for a robot according to one aspect of the present disclosure.

[0043] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components, regardless of drawing symbols, are assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "bu" and "bu" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles.

[0044] In addition, when describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art may obscure the essence of the embodiments disclosed in this specification, such detailed description is omitted. Furthermore, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification, and the technical concept disclosed in this specification is not limited by the attached drawings; it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0045] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. When a component is described as being "connected" or "joined" to another component, it should be understood that it may be directly connected or joined to that other component, or that there may be other components in between.

[0046] On the other hand, when it is stated that one component is "directly connected" or "directly coupled" to another component, it should be understood that there are no other components in between.

[0047] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0048] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0049] FIG. 1 is a block diagram showing the configuration of a robot according to one aspect of the present disclosure.

[0050] Referring to FIG. 1, a robot (100) according to one aspect of the present disclosure may include a sensor (110), a microphone (120), an obstacle detection unit (130), a wireless communication unit (140), a memory (150), a motion detection sensor (160), a driving drive unit (170), and a processor (190). Since the components illustrated in FIG. 1 are not all essential for implementing the robot, the robot described herein may have more or fewer components than those listed above.

[0051] The sensor (110) can acquire sensor data about the surroundings of the robot (100). The sensor (110) may further include one or more of a depth sensor (111) or an RGB sensor (113).

[0052] The depth sensor (111) can detect light irradiated from a light-emitting part (not shown) that is reflected back from an object. The depth sensor (111) can measure the distance to the object based on the time difference of detecting the returned light, the amount of returned light, etc. Based on the measured distance between objects, the depth sensor (111) can obtain two-dimensional image information or three-dimensional image information about the area around the vacuum cleaner (100).

[0053] The RGB sensor (113) can acquire color image information of objects around the vacuum cleaner (100). The color image information may be a captured image of an object. The RGB sensor (113) may be named an RGB camera.

[0054] The sensor (110) may further include a front camera, a rear camera, a side camera, an odometry sensor, and a LiDAR sensor, although not shown in FIG. 1. The odometry sensor is a sensor designed to measure the distance traveled by a robot, which can be calculated by multiplying the number of wheel rotations by the tire circumference, and can provide wheel speed values ​​to various robots, including ABS. The LiDAR sensor is a sensor capable of emitting millions of laser pulses per second and analyzing the reflected light to create an image, and can be used for navigation or distance measurement.

[0055] The microphone (120) can receive the user's voice. The received user's voice can have its intent information analyzed through a voice server (not shown), etc. At this time, the user's voice may be a voice for controlling the robot (100).

[0056] The obstacle detection unit (130) may include an ultrasonic sensor, an infrared sensor, a laser sensor, etc. For example, the obstacle detection unit (130) may irradiate a laser light onto a cleaning area and extract a pattern of the reflected laser light. The obstacle detection unit (130) may detect an obstacle based on the position and pattern of the extracted laser light. However, if a depth sensor (110) is used to detect an obstacle, the configuration of the obstacle detection unit (130) may be omitted.

[0057] The wireless communication unit (140) may include at least one of a mobile communication module, a wireless internet module, and a short-range communication module.

[0058] A mobile communication module transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network built according to technical standards or communication methods for mobile communication (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G, etc.).

[0059] A wireless internet module refers to a module for wireless internet access, which may be embedded in or externally mounted on a robot (100). The wireless internet module is configured to transmit and receive wireless signals in a communication network according to wireless internet technologies.

[0060] Wireless Internet technologies include, for example, Wireless LAN (WLAN), Wireless-Fidelity (Wi-Fi), Wireless Fidelity (Wi-Fi) Direct, Digital Living Network Alliance (DLNA), Wireless Broadband (WiBro), World Interoperability for Microwave Access (WiMAX), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), Long Term Evolution (LTE), These include LTE-A (Long Term Evolution-Advanced) and 5G.

[0061] The short-range communication module is for short-range communication and can support short-range communication by using at least one of the following technologies: Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus).

[0062] The memory (150) can store a SLAM map created through a Simultaneous Localization and Mapping (SLAM) algorithm.

[0063] The movement detection sensor (160) can detect the movement of the robot (100). Specifically, the movement detection sensor (160) can detect that the robot (100) is lifted and moved by a user. The movement detection sensor (160) may include one or more of a floor detection sensor (161) and a gyroscope sensor (163).

[0064] The floor detection sensor (161) can detect whether the robot (100) has been moved by a user by using infrared rays. A detailed explanation of this will be provided later.

[0065] The gyro sensor (163) can measure the angular velocity of the robot (100) for each of the x-axis, y-axis, and z-axis. The gyro sensor (163) can detect movement by the user of the robot (100) by using the amount of change in angular velocity for each axis.

[0066] The movement detection sensor (160) may also include a wheel sensor, a cliff sensor, etc., and can detect movement by the user of the robot (100) using the same.

[0067] The driving drive unit (170) can move the robot (100) in a specific direction or by a specific distance. The driving drive unit (170) may include a left wheel drive unit (171) that drives the left wheel of the robot (100) and a right wheel drive unit (173) that drives the right wheel.

[0068] The left wheel drive unit (171) may include a motor for driving the left wheel, and the right wheel drive unit (173) may include a motor for driving the right wheel.

[0069] In FIG. 1, the driving drive unit (170) is described as including a left wheel drive unit (171) and a right wheel drive unit (173), but it is not limited thereto, and if there is only one wheel, only one drive unit may be provided.

[0070] The processor (190) can control the overall operation of the robot (100).

[0071] To explain the artificial intelligence utilized in this disclosure, artificial intelligence refers to the field of researching artificial intelligence or methodologies capable of creating it, and machine learning refers to the field of researching methodologies for defining and solving various problems addressed in the field of artificial intelligence. Machine learning is also defined as an algorithm that improves the performance of a task through continuous experience with that task.

[0072] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to any model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses. An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0073] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.

[0074] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters, on the other hand, refer to parameters that must be set prior to training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0075] The objective of training an artificial neural network can be viewed as determining model parameters that minimize the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process of an artificial neural network.

[0076] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0077] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result) that the neural network must infer when the training data is input. Unsupervised learning refers to a method of training an artificial neural network without labels provided for the training data. Reinforcement learning refers to a learning method in which an agent defined within an environment is trained to select an action or sequence of actions that maximizes the cumulative reward in each state.

[0078] Machine learning implemented using a Deep Neural Network (DNN) that includes multiple hidden layers among artificial neural networks is also called Deep Learning, and Deep Learning is a part of Machine Learning. Hereinafter, Machine Learning is used in a sense that includes Deep Learning.

[0079] Object detection models using machine learning include the single-stage YOLO (You Only Look Once) model and the two-stage Faster R-CNN (Regions with Convolutional Neural Networks) model.

[0080] The YOLO (You Only Look Once) model is a model that can predict objects present in an image and their locations by looking at the image only once.

[0081] The YOLO (You Only Look Once) model divides the original image into grids of equal size. Then, for each grid, it predicts the number of bounding boxes specified in a predefined form centered on the grid center, and calculates confidence based on this.

[0082] Subsequently, whether the image contains an object or consists solely of a background is included, and a location with high object confidence is selected so that the object category can be identified.

[0083] The Faster R-CNN (Regions with Convolutional Neural Networks) model is a model that can detect objects faster than the RCNN model and the Fast RCNN model.

[0084] This explains the Faster R-CNN (Regions with Convolutional Neural Networks) model in detail.

[0085] First, feature maps are extracted from the image using a Convolutional Neural Network (CNN) model. Based on the extracted feature maps, multiple Regions of Interest (RoIs) are identified. RoI pooling is performed for each region of interest.

[0086] RoI pooling is a process of setting a grid to fit a predetermined size of H x W for a feature map onto which a region of interest is projected, and extracting the largest value for each cell contained in the grid to extract a feature map with a size of H x W.

[0087] A feature vector is extracted from a feature map with size H x W, and identification information of an object can be obtained from the feature vector.

[0088] Hereinafter, with reference to FIG. 2, the configuration of a robot control system according to one aspect of the present disclosure will be described. FIG. 2 is a block diagram showing the configuration of a robot control system according to one aspect of the present disclosure.

[0089] Referring to FIG. 2, a robot control system (10) according to one aspect of the present disclosure may include a robot (100), a robot control server (200), and an operator terminal (300).

[0090] The robot (100) is as described above with reference to FIG. 1, so a detailed description will be omitted for the sake of brevity of the present disclosure.

[0091] The robot control server (200) is a server for remotely controlling the robot (100) and can be implemented, for example, as a cloud server.

[0092] The robot control server (200) may include a wireless communication unit (240), memory (250), and a processor (290). The remote control server (200) described in this specification may have more or fewer components than those listed above.

[0093] The wireless communication unit (240) may include at least one of a mobile communication module, a wireless internet module, and a short-range communication module. Since the mobile communication module, the wireless internet module, and the short-range communication module are as described above, a detailed description will be omitted for the sake of brevity of the present disclosure.

[0094] The robot control server (200) can communicate with the wireless communication unit (140) of the robot (100) through the wireless communication unit (240).

[0095] The memory (250) stores data that supports various functions of the robot control server (200). The memory (250) can store a number of application programs (or applications) running on the robot control server (200), data for the operation of the robot control server (200), and commands. The application program can be stored in the memory (250), installed on the robot control device (200), and driven by the processor (290) to perform the operation (or function) of the robot control server (200).

[0096] The memory (250) can store a map database (DB) (251) for storing maps created for the robot (100).

[0097] The processor (290) can control the overall operation of the robot control server (200). The processor (290) can run applications stored in memory (250).

[0098] The operator terminal (300) is a terminal (300) for a robot operator to connect to a robot control server (200) and remotely control a robot (100), and may include a laptop computer, a desktop computer, a tablet, and a smartphone.

[0099] The operator terminal (300) may include a user input unit (320), a wireless communication unit (340), a memory (350), a display (360), and a processor (390). The operator terminal (300) described in this specification may have more or fewer components than those listed above.

[0100] The user input unit (320) is for receiving information (or commands) from a user (or operator), and when information is input through the user input unit (320), the processor (390) can control the operation of the operator terminal (300) to correspond to the input information. The user input unit (320) may include mechanical input means (or mechanical keys, for example, buttons, dome switches, jog wheels, jog switches, etc. located on the front, rear, or side of the operator terminal (300)) and touch input means (for example, a touchscreen).

[0101] The user input unit (320) may further include a microphone (321). The microphone (321) processes an external acoustic signal into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application running) being performed on the operator terminal (300).

[0102] The wireless communication unit (340) may include at least one of a mobile communication module, a wireless internet module, and a short-range communication module. Since the mobile communication module, the wireless internet module, and the short-range communication module are as described above, a detailed description will be omitted for the sake of brevity of the present disclosure.

[0103] The operator terminal (300) can communicate with the wireless communication unit (240) of the robot control server (200) through the wireless communication unit (340).

[0104] The memory (350) stores data that supports various functions of the operator terminal (300). The memory (350) can store a number of application programs (or applications) running on the operator terminal (300), data for the operation of the robot control server (200), and commands. An application program can be stored in the memory (350), installed on the operator terminal (300), and driven by the processor (390) to perform the operation (or function) of the operator terminal (300).

[0105] The display (360) displays (outputs) information processed at the operator terminal (300). For example, the display unit (360) can display execution screen information of an application running on the operator terminal (300), or UI (User Interface) and GUI (Graphic User Interface) information based on such execution screen information.

[0106] The processor (390) can control the overall operation of the operator terminal (300). The processor (390) can run applications stored in memory (350).

[0107] Hereinafter, with reference to FIG. 3, a process will be described in which a robot (100) according to one aspect of the present disclosure is first installed at a site, and a map for autonomous driving at the site is generated and installed (or mounted) on the robot (100). FIG. 3 is a flowchart showing the sequence in which a map for autonomous driving for a robot is generated and mounted on the robot according to one aspect of the present disclosure.

[0108] First, the robot (100) can be delivered from the factory to the site and deployed [S301]. The robot (100) delivered to the site may not yet have a map for autonomous driving installed for the site.

[0109] When the robot (100) is powered on at the site, communication can be automatically established between the robot (100) and the robot control server (200) [S303]. Communication between the robot (100) and the robot control server (200) can be performed, for example, in a P2P (Peer to Peer) manner.

[0110] A robot operator can access a robot control server (200) through a terminal (300) and request to build (or construct) a map for autonomous driving for the robot (100) [S305].

[0111] The robot control server (200) can build a map for the robot (100) based on sensor data received from the robot (100) [S307]. The building of the map will be explained again later.

[0112] When map building is complete, the robot control server (200) can output the completed map to the terminal (300) [S309].

[0113] The operator can check the completed map output through the terminal (300), set at least one Point of Interest (POI) on the map, and request that the map of the robot control server (200) be applied to the robot (100) [S311]. The POI may refer to a place that can be frequently set as a target point when the robot (100) drives (or patrols) in the field.

[0114] Then, the robot control server (200) can store the map and the POI set therein in the map DB (251) in conjunction with the robot (100) [S313].

[0115] And, the robot control server (200) can download the map and set POI to the robot (100) [S315].

[0116] Then, the robot (100) can install the downloaded map, set the POI on the map according to the downloaded POI, and start autonomous driving [S317].

[0117] In order for the robot control server (200) to build a map, it may be necessary for the robot (100) to move to the site to acquire data about the entire site through the sensor (110) and to receive the acquired data (hereinafter referred to as sensor data) from the robot (100).

[0118] In order to acquire sensor data for the entire site suitable for building a map, it is necessary for the robot (100) to drive around various parts of the site. For the robot (100) to drive around the site, remote intervention by a robot operator through a robot control server (200) may be required. Hereinafter, with reference to FIG. 4, sensor data that can be displayed on an operator terminal (300) for remote intervention by a robot operator will be described. FIG. 4 is an example of sensor data that can be displayed on an operator terminal for remote intervention by a robot operator according to one aspect of the present disclosure.

[0119] Let us assume that a robot (100) is deployed in a field (FA) as illustrated in (4-1) of FIG. 4. Then, the robot (100) can transmit sensor data for an area (SA) in the field (FA) that can be covered by a sensor (110) and an obstacle detection unit (130) to a robot control server (200). Then, the robot control server (200) can transmit the sensor data received from the robot (100) to an operator terminal (300) so that it is displayed on the display (360) of the operator terminal (300).

[0120] As illustrated in (4-2) of FIG. 4, sensor data displayed on the display (360) of the operator terminal (300) can be divided into a drivable area (A) sensed as an area where the robot can drive and a non-drivable area (B) sensed as an area where the robot cannot drive. Additionally, the robot (100) can be further displayed in the sensor data displayed on the display (360) of the operator terminal (300).

[0121] The operator can remotely intervene for the field driving of the robot (100) while viewing sensor data displayed on the display (360).

[0122] Hereinafter, field driving control of a robot through remote intervention by an operator will be described with reference to FIG. 5. FIG. 5 is a flowchart of field driving control of a robot through remote intervention by an operator according to one aspect of the present disclosure.

[0123] A robot control server (200) can receive the current state of the robot and sensor data measured from the robot (100) [S501]. At this time, the robot control server (200) can receive relative position data through the current odometry sensor and sensor data through the LiDAR sensor from the robot (100). The sensor data may further include at least one of the front camera image, rear camera image, and side camera image of the robot (100). The robot (100) can transmit the current state of the robot (100) and the sensor data measured from the robot (100) to the robot control server with minimal delay (e.g., 50ms) via wireless communication.

[0124] The robot control server (200) can calculate the drivable area and drivable area of ​​the robot (100) based on sensor data and output it to the operator terminal (300) [S503].

[0125] The robot control server (200) can receive an input corresponding to the operator's target point among the output robot's drivable areas from the operator terminal (300) [S505].

[0126] The robot control server (200) can calculate at least one of the optimal linear velocity and angular velocity based on the current state of the robot (100) and sensor data, and predict the path of the robot (100) to move to a target point [S507]. That is, the robot control server (200) can calculate the optimal movement path based on the sensor data received from the robot (100) and the target point.

[0127] The robot control server (200) can use the calculated movement speed (at least one of linear speed and angular speed) and the predicted movement path for remote driving control [S509].

[0128] The robot control server (200) can transmit driving commands to the robot (100) according to the movement speed and movement path [S511].

[0129] The robot (100) can perform driving according to driving commands received from the robot control server (200) [S513].

[0130] The above steps can be repeated for field driving of the robot (100) for map building.

[0131] Hereinafter, with reference to FIGS. 6 and FIGS. 7, a driving path setting of a robot according to one aspect of the present disclosure will be described. FIGS. 6 is a conceptual diagram of a driving path setting of a robot according to one aspect of the present disclosure. FIGS. 7 is a flowchart of a driving path setting of a robot according to one aspect of the present disclosure.

[0132] As illustrated in FIG. 6, a robot (100) can be deployed at a site (FA). The site (FA) may include a drivable area (A) and a non-drivable area (B) of the robot.

[0133] The robot (100) can transmit sensor data for an area (SA) that can be covered by the sensor (110) and the obstacle detection unit (130) in the field (FA) to the robot control server (200). Then, the robot control server (200) can transmit the sensor data received from the robot (100) to the operator terminal (300) so that it is displayed on the display (360) of the operator terminal (300).

[0134] Sensor data output on the display (360) of the operator terminal (300) can be displayed by distinguishing between a drivable area (A) that is sensed as an area where the robot (100) can drive and a non-drivable area (B) that is sensed as an area where the robot (100) cannot drive.

[0135] The operator may set a point in the drivable area (A) displayed on the display (360) of the terminal (300) as a target point (GP). However, in this case, the operator may have to repeatedly set the target point (GP) until the robot reaches the destination. That is, if the number of times the operator sets the target point (GP) until the robot reaches the destination is reduced, the operator can easily intervene remotely in the robot's driving. This will be explained further with reference to FIG. 7.

[0136] The operator can input a command (hereinafter referred to as a target point command) for the robot's target point (GP) into the terminal (300) while viewing the drivable area (A) and the non-drivable area (B) displayed on the display (360) of the terminal (300). The target point command can be input through the user input section (320) of the operator terminal (300). In particular, the target point command can be input as the operator's voice (or speech) through the microphone (321) of the user input section (320). The target point (GP) of the target point command is not limited to within the drivable area (A) currently displayed on the display (360). For example, the target point command may not include specific location information for the target point, such as "forward" (or "go forward"). The target point command can be transmitted to the robot control server (200) through the operator terminal (300) [S701].

[0137] The robot control server (200) receives a target point command and can generate a target point corresponding to the target point command [S703]. The robot control server (200) can generate a target point from the target point command even if the target point command does not include specific location information for the target point. The target point can be generated based on sensor information of the sensor (110) (e.g., LiDAR sensor) and the target point command. The area covered by the sensor (110) may be wider than the area (SA) covered by both the sensor (110) and the obstacle detection unit (130).

[0138] For example, if the target point command is "advance," the robot (100) can calculate a target position to which it can advance as much as possible by considering the drivable area (A) and the drivable area (B) at its current location in the field to best conform to the target point command, and the calculated target position can be created as a target point. Referring to FIG. 6, the target position to which the robot (100) can advance as much as possible can be created as a target point (GP). Just because the target point command is "advance," the target point does not necessarily have to be located in front of the robot (100) on a vertical line. As long as it conforms to the target point command "advance," the target point may be located in front of the robot (100) on an oblique line within a predetermined angle.

[0139] The generated target point (GP) is exemplified as a point outside the driving area (A) currently displayed on the display (360) of the operator terminal (300). For example, the target point (GP) may be a point located at the 1 o'clock direction, at least 15m ahead of the robot (100).

[0140] Next, the robot control server (200) can generate a waypoint (VP) that the robot (100) must pass through to move to a target point (GP) by considering the drivable area (A) and the non-drivable area (B) at the current location of the field (FA) [S705]. The waypoint (VP) may be a point within the drivable area (A) (or the area (SA) that can be covered by both the sensor (110) and the obstacle detection unit (130)) currently displayed on the display (360) of the operator terminal (300). For example, the waypoint (VP) may be a point located within 2m of the robot (100). The waypoint (VP) may be generated with further consideration of obstacles such as objects, people, or other robots located within the drivable area (A).

[0141] Next, the robot control server (200) can perform robot driving control according to FIG. 5 [S707]. If the target point (GP) is not located within the driving area (A) currently displayed on the display (360) of the operator terminal (300) in the display (360), the robot control server (200) can remotely control the driving of the robot (100) by replacing the target point with a waypoint in the robot driving control according to FIG. 5. If, as the robot drives, the target point (GP) comes to be located within the driving area (A) currently displayed on the display (360) of the operator terminal (300), the robot control server (200) can perform the robot driving control according to FIG. 5 as is.

[0142] If the robot (100) has arrived at the target point, the robot control server (200) can repeat steps S701 to Fig. 707 until the robot (100) reaches the destination.

[0143] If the robot (100) has not reached the target point, the robot control server (200) can repeat steps S705 and S707.

[0144] Hereinafter, with reference to FIG. 8, we will look more specifically at how the robot control server (200) generates a waypoint (VP). FIG. 8 is an example of waypoint generation for robot driving control according to one aspect of the present disclosure.

[0145] The robot control server (200) can select a candidate waypoint area (CA) by considering the current position of the robot (100). The candidate waypoint area (CA) may be a predetermined area within the area (SA) that can be covered by the sensor (110) and the obstacle detection unit (130). In FIG. 8, the candidate waypoint area (CA) is exemplified as an area spaced a predetermined distance forward from the robot (100).

[0146] The robot control server (200) can perform convolution operations with a 2D convolution filter on a waypoint candidate region (CA). The waypoint candidate region (CA) can be represented in the form of a matrix of, for example, 3 points x 80 points. The 2D convolution filter can be a specific kernel (NxN (a window filter of size N x N).

[0147] The robot control server (200) can reflect weights according to the target point command (target point distance and direction) in the convolution operation result.

[0148] Since the location with the maximum value of the weighted convolution operation result represents the location with the lowest collision probability of the robot (100), the robot control server (200) can generate the location with the maximum value as a waypoint.

[0149] Hereinafter, with reference to FIGS. 9 to 12, examples of target point commands will be described. FIGS. 9 to 12 are examples of target point commands for driving a robot according to one aspect of the present disclosure.

[0150] First, with reference to FIG. 9, I will explain the “go forward” target point command. The “go forward” target point command may include a “start” action, a “move” action, and a “stop with condition” action.

[0151] That is, in response to the target point command "go forward," the robot control server (200) can remotely control the robot (100) so that the robot (100) starts driving according to the "start" action, executes forward movement according to the "move" action, and stops driving when the execution condition of the target point command is satisfied according to the "stop with condition" action.

[0152] Referring to FIG. 10, I will describe the "Go to the diagonal right corridor" target point command. The "Go to the diagonal right corridor" target point command may include a "start" action, a "move right diagonal" action, a "move" action, and a "stop with condition" action.

[0153] That is, in response to the target point command "Go to the diagonal right corridor," the robot control server (200) can remotely control the robot (100) so that the robot (100) starts driving according to the "start" action, executes moving in the right diagonal direction according to the "move right diagonal" action, executes moving forward according to the "move" action, and stops driving when the execution condition of the target point command is satisfied according to the "stop with condition" action.

[0154] When executing movement in the right diagonal direction, the robot (100) can be remotely controlled to move to the target point by passing through the waypoint in the rightmost drivable area (A) among several candidate waypoints.

[0155] The "Go to the diagonal left corridor" target point command is identical except that it includes the "move left diagonal" action instead of the "move right diagonal" action, so I will omit a detailed explanation.

[0156] Referring to FIG. 11, I will describe the “go back” target point command. The “go back” target point command may include a “start” action, a “rotate 180 degree” action, a “move” action, and a “stop with condition” action.

[0157] That is, in response to the target point command "go back," the robot (100) can start driving according to the "start" action, rotate 180 degrees backward according to the "rotate 180 degree" action, move forward according to the "move" action, and stop driving when the execution condition of the target point command is satisfied according to the "stop with condition" action. The robot control server (200) can remotely control the robot (100).

[0158] Referring to FIG. 12, I will describe the "Go to right corridor" target point command. The "Go to right corridor" target point command may include a "start" action, a "rotate CW 90 degree" action, a "move" action, and a "stop with condition" action.

[0159] That is, in response to the target point command "Go to the right corridor," the robot (100) can be remotely controlled by a robot control server (200) to start driving according to the "start" action, perform a 90-degree clockwise rotation according to the "rotate CW 90 degree" action, perform a forward movement according to the "move" action, and stop driving when the execution condition of the target point command is satisfied according to the "stop with condition" action.

[0160] The "Go to the left corridor" target point command is identical except that it includes the "rotate CCW 90 degree" action instead of the "rotate CW 90 degree" action, so I will omit a detailed explanation.

[0161] Hereinafter, with reference to FIGS. 13 to 15, the execution of some of the actions described above will be described as examples of actions. FIGS. 13 to 15 are flowcharts of the execution of actions of a robot according to one aspect of the present disclosure.

[0162] First, referring to Fig. 13, we will look at the "move" action.

[0163] When the robot control server (200) initiates the execution of a "move" action, it can create a waypoint within a predetermined distance (e.g., 2m) in front of the robot (100) for the "move" action [S1301].

[0164] The robot control server (200) can remotely control the robot (100) to perform driving toward a waypoint [S1303].

[0165] If the robot (100) reaches a waypoint (or within a predetermined range from the waypoint) after driving, another waypoint can be created to reach the target point [S1305, S1301].

[0166] However, if the robot (100) has not reached a waypoint after driving, the robot control server (200) determines whether a predetermined time has elapsed since the start of driving, and if a predetermined time has elapsed, the robot (100) may terminate the "move" action [S1307].

[0167] If a predetermined amount of time has not elapsed since the start of driving, the robot control server (200) determines whether the robot (100) has reached a target point, and if it has reached a target point, it can terminate the "move" action of the robot (100) [S1309].

[0168] If the robot (100) has not reached the target point, the robot control server (200) can control the robot (100) to continue driving toward the waypoint [S1303].

[0169] Below, with reference to FIG. 14, we will examine the "move right diagonal" action.

[0170] The robot (100) may be in motion according to a "move" action. This is as described in FIG. 13.

[0171] The robot control server (200) can determine whether there is a free space on the right side where the robot (100) can drive within a predetermined distance (e.g., 2m) in front of the robot (100) while the robot (100) is driving [S1401].

[0172] If there is no free space to drive on the right, the robot (100) can continue driving according to the “move” action.

[0173] However, if there is a free space to drive on the right, the robot control server (200) can control the robot (100) to drive while creating a waypoint for the robot (100) to move to the right [S1403].

[0174] If the robot (100) has entered the right passageway, the robot control server (200) can control the robot (100) to drive according to a "move" action.

[0175] If the robot (100) has not entered the right passage, the robot control server (200) determines whether a predetermined time has elapsed since the start of step S1403, and if a predetermined time has elapsed, the robot (100) can terminate the "move right diagonal" action [S1407].

[0176] If a predetermined amount of time has not elapsed since the start of step S1403, the robot control server (200) can remotely control the robot (100) to continue performing step S1403 [S1407, S1403].

[0177] Below, with reference to FIG. 15, we will examine the "rotate" action.

[0178] The robot control server (200) can transmit an angular velocity command for rotation to the robot (100) [S1501]. The angular velocity command may include a command regarding a target rotation angle and a target rotation speed.

[0179] The robot (100) can perform rotation according to an angular velocity command [S1503]. The robot (100) may also perform rotation after a predetermined waiting time (e.g., 0.5 seconds) has elapsed from the time of receiving the angular velocity command.

[0180] The robot control server (200) can determine whether the robot (100) has rotated to reach a target rotation angle [S1505].

[0181] If the robot (100) reaches a target rotation angle, the robot control server (200) can control the robot (100) to drive according to a "move" action.

[0182] If the robot (100) has not reached the target rotation angle, it is determined whether a predetermined time has elapsed since the start of step S1503, and if a predetermined time has elapsed, the “rotate” action of the robot (100) can be terminated [S1507].

[0183] If a predetermined amount of time has not elapsed since the start of step S1503, the robot control server (200) can remotely control the robot (100) to continue performing step S1503 [S1507, S1503].

[0184] Hereinafter, with reference to FIGS. 16 and 17, field driving control of a robot based on remote intervention by an operator according to an operator's voice input into an operator terminal (300) will be described. FIG. 16 is a flowchart of field driving control of a robot through remote intervention by an operator according to one aspect of the present disclosure. FIG. 17 is an example of a robot control system according to one aspect of the present disclosure.

[0185] A robot control server (200) can receive the current state of the robot and sensor data measured from the robot (100) [S1601]. At this time, the robot control server (200) can receive relative position data through the current odometry sensor and sensor data through the lidar sensor from the robot (100). The sensor data may further include at least one of the front camera image, rear camera image, and side camera image of the robot (100). The robot (100) can transmit the current state of the robot (100) and the sensor data measured from the robot (100) to the robot control server with minimal delay (e.g., 50ms) via wireless communication.

[0186] The robot control server (200) can calculate the drivable area and drivable area of ​​the robot (100) based on sensor data and output it to the operator terminal (300) [S1603].

[0187] The robot control server (200) can receive voice commands for the robot's target point (GP) that the operator speaks into the microphone (321) of the terminal (300) from the terminal (300) [S1605].

[0188] The robot control server (200) can convert the operator's voice commands into STT (Speech-to-Text) [S1607].

[0189] The robot control server (200) can infer target point commands by performing prompt engineering based on LLM (Large Language Model) on STT-converted voice commands [S1609, S1611].

[0190] The robot control server (200) can determine at least one action based on the inferred target point command [S1613].

[0191] The robot control server (200) can establish a local plan for robot driving based on at least one determined action [S1615]. The local plan may include a movement speed and a movement path for executing an inferred target point command.

[0192] The robot control server (200) can transmit a local plan to the robot (100) [S1617].

[0193] The robot (100) can perform driving according to a local plan received from the robot control server (200) [S1619].

[0194] The above steps can be repeated for field driving of the robot (100) for map building.

[0195] As illustrated in FIG. 17, an operator can remotely control the driving of the robot (100) through a terminal (300) and a robot control server (200). The operator can remotely control the driving of the robot (100) while viewing the execution screen of a robot control application running on the terminal (300) to connect to the robot control server (200).

[0196] With reference to FIGS. 18 and 19, the execution screen of a robot control application displayed on an operator terminal (300) will be described with further reference. FIGS. 18 and 19 are examples of the execution screen of a robot control application according to one aspect of the present disclosure.

[0197] FIG. 18 illustrates the execution screen (1000) of a robot control application when the map generation menu (1001) of the robot control application is being executed.

[0198] The application execution screen (1000) may include a front camera image (1010) and a rear camera image (1020) captured by the robot (100). If necessary, the application execution screen (1000) may further display a side camera image, or may display it instead of at least one of the front camera image (1010) and the rear camera image (1020).

[0199] The application execution screen (1000) may further include an utterance command input area (1100) for an operator (or user). The operator may start utterance by selecting the voice input start menu (1110) of the utterance input area (1100) and end utterance by selecting the voice input stop menu (1120) of the utterance input area (1100). A target point command (1130) corresponding to the utterance may be output from the utterance command input area (1100). The operator may check whether the target point command (1130) output from the utterance command input area (1100) is the intended target point command.

[0200] The application execution screen (1000) may further include a driving area (1200) of the robot (100). In the driving area (1200), the driving area (A) and the driving area (B) described in (4-2) of FIG. 4 may be displayed.

[0201] In the driving area (1200), at least one predefined target point command may be displayed as a corresponding menu. The at least one predefined target point command menu may include a “move left_corridor” menu (1201), a “move diagonal left_corridor” menu (1201), a “move forward” menu (1203), a “move backward” menu (1204), a “move diagonal right_corridor” menu (1205), and a “move right_corridor” menu (1206). The operator may input the target point command by, for example, touching or selecting with a mouse, the target point command menu instead of speaking.

[0202] Additionally, a virtual joystick menu (1220) may be displayed in the driving area (1200). The operator can remotely control the driving of the robot (100) by manipulating the virtual joystick menu (1220) via touch or mouse instead of inputting a target point command.

[0203] At least one target point command menu and virtual joystick menu (1220) do not necessarily have to be displayed in the driving area (1200), and may, of course, be displayed elsewhere on the application execution screen (1000).

[0204] The application execution screen (1000) may further include a map output area (1300) for outputting a map being built (or created) in real time while the robot (100) is driving. The operator can intuitively understand the progress of the map building while looking at the map output area (1300).

[0205] FIG. 19 illustrates the execution screen (1000) of a robot control application when the navigation menu (1002) of the robot control application is being executed.

[0206] The application execution screen (1000) may include a front camera image (1010) and a rear camera image (1020) captured by the robot (100). If necessary, the application execution screen (1000) may further display a side camera image, or may display it instead of at least one of the front camera image (1010) and the rear camera image (1020).

[0207] The application execution screen (1000) may further include a map output area (1400) that outputs a map of completed building for the robot (100). The operator can check whether the map output in the map output area (1400) is suitable for the site where the robot (100) is currently operating.

[0208] The robot (100) does not necessarily have to be operated at only one site. Since the robot (100) can be operated at multiple sites in rotation as needed, multiple maps corresponding to each site can be built. The multiple maps that have been built may be stored in the map DB (251) of the robot control server (200). Therefore, a map selected by the operator from among the multiple maps that have been built can be displayed in the map output area (1400).

[0209] The application execution screen (1000) may further include a map installation area (1500) for installing or updating a map of the robot (100). By selecting an execution button in the map installation area (1500), the operator can remotely control the robot (100) so that a completed building map being output in the map output area (1400) is installed or updated on the robot (100).

[0210] In this way, while the robot (100) is autonomously driving at the site according to the map installed (or updated) on the robot (100), the driving area and the driving area may change according to changes in the layout of the site. The autonomous driving map installed on the robot (100) for the autonomous driving of the robot (100) will be referred to as a reference map. That is, even while the robot (100) is autonomously driving at the site according to the reference map, the autonomous driving map of the robot (100) needs to be updated periodically or at the request of the operator.

[0211] Hereinafter, with reference to FIG. 20, we will examine in more detail how a new map is built and the newly built map is installed (or updated) on the robot (100) while the robot (100) is autonomously driving through the site according to a reference map. FIG. 20 is a flowchart showing how an autonomous driving map for a robot is updated and mounted on the robot according to one aspect of the present disclosure.

[0212] As described above, a robot (100) equipped with a reference map generated as described above drives autonomously according to the reference map, and a robot control server (200) can receive the current state of the robot and sensor data measured from the robot from the robot (100) that is driving autonomously [S2001].

[0213] During autonomous driving of the robot (100), the robot control server (200) may initiate map building [S2003]. The robot control server (200) may periodically and automatically initiate map building, or it may initiate map building in response to a map building start command from an operator received through an operator terminal (300).

[0214] The robot control server (200) can acquire a new map by performing map building based on the current state of the robot (100) that is autonomously driven and sensor data measured from the robot [S2005, S2007].

[0215] The robot control server (200) can detect feature points of a new map based on a reference map and can transform the coordinates of the new map based on the detected feature points [S2009, S2011]. The robot control server (200) can ensure that POIs set in the reference map are also set in the new map. The transformation of the coordinates of the new map will be explained again later.

[0216] The robot control server (200) can output the coordinate-transformed new map to the operator terminal (300) [S2013].

[0217] The operator may approve a new map output from the terminal (300), change and / or delete existing POIs in the new map, and, if necessary, set new POIs in the new map [S2015].

[0218] The robot control server (200) can store the new map and the corresponding POI in the map DB (251) in conjunction with the robot (100) [S2017].

[0219] And, the robot control server (200) can download the new map and the POI set therein to the robot (100) [S2019].

[0220] Then, the robot (100) can install the downloaded new map, set POIs on the new map according to the downloaded POIs, and start autonomous driving [S2021].

[0221] Hereinafter, with reference to FIG. 21, a description will be made with further reference to the execution screen of a robot control application displayed on an operator terminal (300) when a new map is built while the robot (100) is autonomously driving through the site according to a reference map. FIG. 21 is an example of an execution screen of a robot control application according to one aspect of the present disclosure.

[0222] The application execution screen (1000) may include a front camera image (1010) and a rear camera image (1020) captured by the robot (100). If necessary, the application execution screen (1000) may further display a side camera image, or may display it instead of at least one of the front camera image (1010) and the rear camera image (1020).

[0223] The application execution screen (1000) may further include a map building command area (1610) for an operator to input a map building start command during autonomous driving of the robot (100). The operator can input a map building start command by selecting a map update start menu (1610) of the map building command area (1610).

[0224] The application execution screen (1000) may further include a reference map output area (1700) that outputs a reference map used for autonomous driving of the robot at the site, and a new map output area (1800) that outputs a new map being built while the robot is autonomous driving at the site. It can be seen that the XY coordinates of the reference map output in the reference map output area (1700) are different from the XY coordinates of the new map in the new map output area (1800). This is because the map is generated based on images according to the autonomous driving of the robot (100). Therefore, the new map that has been completed requires coordinate transformation to conform to the reference map.

[0225] The application execution screen (1000) may further include a POI setting area (1900). The operator can change and / or delete existing POIs through menu operations in the POI setting area (1900), and may also register new POIs as needed.

[0226] Hereinafter, with reference to FIGS. 22 and FIGS. 23, the coordinate transformation of the novel map described above will be explained. FIG. 22 is a flowchart for coordinate transformation of an autonomous driving map for a robot according to one aspect of the present disclosure. FIG. 23 is an example of coordinate transformation of an autonomous driving map for a robot according to one aspect of the present disclosure.

[0227] The robot control server (200) can detect feature points of an existing map and feature points of a new map, as illustrated in (23-1) and (23-2) of FIG. 23 [S2201, S2202]. Feature points may refer to characteristic shapes of a map. (23-1) of FIG. 23 illustrates feature points (2310) of a reference map, and (23-2) of FIG. 23 illustrates feature points (2320) of a new map.

[0228] The robot control server (200) can match the feature points (2310) of the reference map with the feature points (2320) of the new map [S2205].

[0229] The robot control server (200) can calculate a coordinate transformation formula to transform the coordinates of a new map to match the coordinates of a reference map based on mutually matched feature points [S2207].

[0230] The robot control server (200) can convert the coordinates of the new map based on the calculated coordinate conversion formula as shown in (23-3) of FIG. 23 [S2209].

[0231] The foregoing invention may be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and also include implementations in the form of carrier waves (e.g., transmission over the Internet). Accordingly, the above detailed description should not be interpreted restrictively in all respects and should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

1. A robot control system comprising a robot, a robot control server, and a terminal, The above robot control server is, Receive sensor data from the above robot, and Receives a target point command regarding the target point to which the robot must move from the terminal, and Based on the sensor data and the target point command, a waypoint is generated for the robot to move to the target point, and Generate a movement path to move to the above waypoint, and A robot control system that controls the transmission of driving commands according to the above movement path to the robot.

2. In Paragraph 1, A robot control system characterized by the above sensor data including LiDAR sensor data sensed by the robot.

3. In Paragraph 1, The above robot control server is, A robot control system characterized by repeatedly generating new waypoints from the time the robot arrives at the waypoint until it reaches the target point.

4. In Paragraph 1, The above robot control server is, A robot control system characterized by selecting a candidate waypoint area considering the position of the robot, performing a convolution operation on the candidate waypoint area with a 2D convolution filter, and generating the waypoint by reflecting weights according to the distance and direction of the target point in the result of the convolution operation.

5. In Paragraph 1, The above robot control server is, A robot control system characterized by outputting the drivable area and drivable area of ​​the robot based on the sensor data to the terminal.

6. In Paragraph 1, The above target point command is an operator's voice command input into the terminal, and The above robot control server is, Convert the above voice command into STT (Speech-to-Text), and A robot control system characterized by inferring the target point command by performing prompt engineering based on a Large Language Model (LM) on the STT-converted voice command.

7. In Paragraph 1, The above robot control server is, Determine at least one action according to the above target point command, and Establish a local plan for driving the robot based on at least one of the above actions, and A robot control system characterized by controlling the transmission of the local plan to the robot as the driving command.

8. In Paragraph 1, The above robot control server is, A first map for the robot is built based on the sensor data received from the robot driving according to the above driving command, and A robot control system characterized by controlling the installation of a first map on the robot.

9. In Paragraph 8, The above robot control server is, A second map for the robot is built based on the sensor data received from the robot driving according to the first map, and A robot control system characterized by controlling the installation of a second map on the robot.

10. In Paragraph 9, The above robot control server is, Based on the feature points of the first map and the feature points of the second map, the second map is coordinate-transformed, and A robot control system characterized by controlling the installation of the coordinate-transformed second map on the robot.

11. A step of receiving sensor data from the robot; A step of receiving a target point command from an operator regarding a target point to which the robot must move; A step of generating a waypoint for the robot to move to the target point based on the sensor data and the target point command; A step of generating a movement path to move to the above waypoint; and A robot control method comprising the step of transmitting a driving command according to the above movement path to the robot.

12. In Paragraph 11, A robot control method characterized in that the sensor data includes lidar sensor data sensed by the robot.

13. In Paragraph 11, A robot control method characterized by including the step of repeatedly generating new waypoints from the time the robot arrives at the waypoint until it reaches the target point.

14. In Paragraph 11, A robot control method characterized by including the step of: selecting a candidate waypoint area considering the position of the robot, performing a convolution operation on the candidate waypoint area with a 2D convolution filter, and generating the waypoint by reflecting weights according to the distance and direction of the target point in the result of the convolution operation.

15. In Paragraph 11, A robot control method characterized by including the step of outputting the drivable area and drivable area of ​​the robot based on the sensor data to a terminal.

16. In Paragraph 11, The above target point command is a voice command of the above operator, and A step of converting the above voice command into STT (Speech-to-Text); and, A robot control method characterized by including the step of inferring the target point command by performing prompt engineering based on a Large Language Model (LM) on the STT-converted voice command.

17. In Paragraph 11, A step of determining at least one action according to the above target point command; A step of establishing a local plan for driving the robot based on at least one action; and, A robot control method characterized by including the step of transmitting the local plan to the robot as the driving command.

18. In Paragraph 11, A step of building a first map for the robot based on sensor data received from the robot driving according to the driving command; and A robot control method characterized by including the step of installing a first map on the robot.

19. In Paragraph 18, A step of building a second map for the robot based on sensor data received from the robot driving according to the first map; and A robot control method characterized by including the step of installing a second map on the robot.

20. In Paragraph 19, A step of coordinate transforming the second map based on feature points of the first map and feature points of the second map; and A robot control method characterized by including the step of installing the coordinate-transformed second map on the robot.

Images