Artificial intelligence apparatus, operation method thereof, and recording medium

Abstract

An artificial intelligence apparatus according to an embodiment of the present disclosure may comprise one or more processors that: acquire a command and an image related to an object and a target space; partition the target space into a plurality of subspaces, which are wider than the object, on the basis of the command and the image; set any one subspace from among the plurality of partitioned subspaces as a destination to determine a movement path of the object; determine the possibility of collision with an obstacle on the movement path; and if it is determined that there is the possibility of collision with the obstacle, set another subspace from among the plurality of subspaces as the destination to set a new movement path, and then re-determining the possibility of collision.

Description

Artificial intelligence device, its method of operation, and recording medium

[0001] The present invention relates to an artificial intelligence device, and more specifically, to an artificial intelligence device capable of controlling the operation of a robot.

[0002] The development of artificial intelligence (AI) robots has progressed rapidly, and innovative changes are expected to continue in the future. The advancement of AI robots is having a significant impact across technology, society, and the economy.

[0003] AI robots are equipped with two arms, and controlling objects with dual-armed AI robots is very complex and difficult. Users want AI robots to move similarly to humans.

[0004] However, while AI robots can move objects as instructed, they find it difficult to perform actions like avoiding obstacles or intelligently placing objects down, unlike humans. In other words, even if an AI robot collides with another object while moving it, it cannot easily detect such an event.

[0005] In addition, AI robots have difficulty recognizing spaces where they can place objects while avoiding obstacles (especially when moving multiple objects), and there is a problem in that they cannot intelligently plan in advance with humans how to place objects.

[0006] The purpose of the present disclosure is to enable the device to mimic actions such as avoiding obstacles or intelligently placing objects when moving, similar to a human.

[0007] The purpose of the present disclosure is to enable the movement path of an object to be modified by detecting a collision with another object in advance while the object is being moved.

[0008] The purpose of the present disclosure is to enable the recognition of a space where a drop-off can be made while avoiding obstacles, and to intelligently plan a movement path.

[0009] An artificial intelligence device according to one embodiment of the present disclosure may include one or more processors that acquire commands and images related to an object and a target space, divide the target space into a plurality of subspaces larger than the object based on the commands and images, determine a movement path of the object by setting one of the divided subspaces as a destination, determine a possibility of collision with an obstacle on the movement path, and if it is determined that a possibility of collision with the obstacle exists, set another of the plurality of subspaces as the destination to establish a new movement path, and then re-evaluate the possibility of collision.

[0010] A method of operation of an artificial intelligence device according to one embodiment of the present disclosure may include: acquiring commands and images related to an object and a target space; dividing the target space into a plurality of subspaces that are larger than the object based on the commands and images; determining a movement path of the object by setting one of the divided subspaces as a destination; determining a possibility of collision with an obstacle on the movement path; and, if it is determined that a possibility of collision with an obstacle exists, setting another of the plurality of subspaces as the destination to establish a new movement path, and then re-determining the possibility of collision.

[0011] A computer-readable recording medium having a program for performing a method of operation of an artificial intelligence device according to one embodiment of the present disclosure, wherein the method of operation may include: acquiring a command and an image related to an object and a target space; dividing the target space into a plurality of subspaces that are larger than the object based on the command and the image; determining a movement path of the object by setting one of the divided subspaces as a destination; determining a possibility of collision with an obstacle on the movement path; and, if it is determined that a possibility of collision with an obstacle exists, setting another subspace among the plurality of subspaces as the destination to establish a new movement path, and then re-determining the possibility of collision.

[0012] According to an embodiment of the present disclosure, an artificial intelligence robot can move an object by avoiding obstacles. Accordingly, the artificial intelligence robot can operate safely even in an environment where unexpected obstacles or people are present.

[0013] In addition, according to an embodiment of the present disclosure, damage to an artificial intelligence robot or surrounding equipment due to a collision with an obstacle can be prevented.

[0014] FIG. 1 is a block diagram for explaining the components of an artificial intelligence device according to one embodiment of the present disclosure.

[0015] FIG. 2 is a drawing for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0016] FIGS. 3a and 3b are drawings for explaining the configuration of an artificial intelligence robot according to one embodiment of the present disclosure.

[0017] FIG. 4 is a flowchart for explaining the operation method of an artificial intelligence device according to one embodiment of the present disclosure.

[0018] FIG. 5 is a drawing illustrating an example of moving an object to a partial space in a target space when no obstacles exist, according to one embodiment of the present disclosure.

[0019] FIGS. 6a to 6d are drawings illustrating an example of moving an object to a partial space in a target space when an obstacle is present, according to an embodiment of the present disclosure.

[0020] FIGS. 7a and 7b are drawings illustrating a method for detecting whether an obstacle exists on a path of movement between an object and a destination according to an embodiment of the present disclosure.

[0021] FIGS. 8A and 8B are drawings illustrating the operation of an artificial intelligence robot that avoids an obstacle when an obstacle exists on the path of an object according to an embodiment of the present disclosure.

[0022] FIGS. 9 to 11 are drawings illustrating a method of moving a plurality of objects to a target space according to an embodiment of the present disclosure.

[0023] Artificial intelligence refers to the field of researching artificial intelligence or the methodologies to create it, while machine learning refers to the field of researching methodologies to define the various problems addressed in the field of artificial intelligence and to solve them.

[0024] Machine learning is also defined as an algorithm that improves the performance of a task through consistent experience.

[0025] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to a model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses.

[0026] An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0027] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.

[0028] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters refer to parameters that must be set before training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0029] The objective of training an artificial neural network can be viewed as determining model parameters that minimize the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process of an artificial neural network.

[0030] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0031] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result value) that the artificial neural network must infer when training data is input into it.

[0032] Unsupervised learning can refer to a method of training an artificial neural network without being given labels for the training data.

[0033] Reinforcement learning can refer to a learning method that trains an agent defined within an environment to select an action or sequence of actions that maximizes the cumulative reward in each state.

[0034] Machine learning implemented as a deep neural network (DNN) containing multiple hidden layers among artificial neural networks is also called deep learning, and deep learning is a part of machine learning.

[0035] In the following, machine learning is used to include deep learning.

[0036] FIG. 1 is a block diagram for explaining the components of an artificial intelligence device according to one embodiment of the present disclosure.

[0037] The artificial intelligence device (100) can be implemented as a fixed device or a mobile device, such as a TV, projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia player), navigation, tablet PC, wearable device, set-top box (STB), DMB receiver, radio, washing machine, refrigerator, desktop computer, digital signage, robot, vehicle, etc.

[0038] Referring to FIG. 1, the artificial intelligence device (100) may include a communication interface (110), an input interface (120), a learning processor (130), a sensor (140), an output interface (150), a memory (170), and a processor (180).

[0039] The communication interface (110) can transmit and receive data with external devices, such as other artificial intelligence devices or AI servers (200), using wired or wireless communication technology. For example, the communication interface (110) can transmit and receive sensor information, user input, learning models, control signals, etc., with external devices.

[0040] The communication technologies used by the communication interface (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0041] The input interface (120) can acquire various types of data.

[0042] The input interface (120) may include a camera (121) for capturing images, a microphone (122) for receiving audio signals, and a user input interface (123) for receiving information from a user.

[0043] A camera (121) or a microphone (122) can be treated as a sensor, and a signal obtained from the camera (121) or the microphone (122) can be named as sensing data or sensor information.

[0044] The input interface (120) can obtain input data to be used when obtaining an output using training data and a training model for model training. The input interface (120) may also obtain unprocessed input data, in which case the processor (180) or the learning processor (130) can extract input features as a preprocessing step for the input data.

[0045] The camera (121) processes image frames, such as still images or video, obtained by an image sensor in video call mode or shooting mode. The processed image frames may be displayed on a display (151) or stored in memory (170).

[0046] The microphone (122) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application running) being performed by the artificial intelligence device (100). Meanwhile, various noise removal algorithms can be applied to the microphone (122) to remove noise generated during the process of receiving external acoustic signals.

[0047] The user input interface (123) is for receiving information from a user, and when information is input through the user input interface (123), the processor (180) can control the operation of the artificial intelligence device (100) to correspond to the input information.

[0048] The user input interface (123) may include mechanical input means (or mechanical keys, such as buttons, dome switches, jog wheels, jog switches, etc. located on the front / rear or side of the artificial intelligence device (100)) and touch input means.

[0049] As an example, a touch input means may consist of a virtual key, soft key, or visual key displayed on a touchscreen through software processing, or a touch key placed on a part other than the touchscreen.

[0050] The learning processor (130) can train a model composed of an artificial neural network using training data. The trained artificial neural network can be called a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for judgment to perform some action.

[0051] The learning processor (130) can perform AI processing together with the learning processor (240) of the AI ​​server (200).

[0052] The learning processor (130) may include memory integrated into or implemented in the artificial intelligence device (100). The learning processor (130) may also be implemented using memory (170), external memory directly coupled to the artificial intelligence device (100), or memory maintained in an external device.

[0053] The sensor (140) can acquire at least one of internal information of the artificial intelligence device (100), surrounding environment information of the artificial intelligence device (100), and user information using various sensors.

[0054] The sensor (140) may include one or more of a proximity sensor, an illuminance sensor, an accelerometer, a magnetic sensor, a gyroscope, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar sensor, and a radar sensor.

[0055] The output interface (150) can generate output related to visual, auditory, or tactile senses.

[0056] The output interface (150) may include a display (151) for outputting an image, an audio output interface (152) for outputting audio, a haptic device (153) for outputting tactile information, and a light output interface (154) for outputting light.

[0057] The display (151) displays (outputs) information processed by the artificial intelligence device (100). For example, the display (151) can display information on the execution screen of an application running on the artificial intelligence device (100), or UI (User Interface) and GUI (Graphic User Interface) information based on such execution screen information.

[0058] The display (151) can be implemented as a touch screen by forming a layered structure with the touch sensor or by being formed as an integral unit. The touch screen functions as a user input interface (123) that provides an input interface between the artificial intelligence device (100) and the user, and at the same time, can provide an output interface between the artificial intelligence device (100) and the user.

[0059] The audio output interface (152) can output audio data received from the communication interface (110) or stored in the memory (170) in call signal reception, call mode or recording mode, voice recognition mode, broadcast reception mode, etc.

[0060] The audio output interface (152) may include at least one of a receiver, a speaker, and a buzzer.

[0061] The haptic device (153) generates various tactile effects that the user can feel. A typical example of the tactile effect generated by the haptic device (153) can be vibration.

[0062] The light output interface (154) outputs a signal to indicate the occurrence of an event using the light of the light source of the artificial intelligence device (100). Examples of events occurring in the artificial intelligence device (100) may include receiving a message, receiving a call signal, a missed call, an alarm, a schedule notification, receiving an email, receiving information through an application, etc.

[0063] The memory (170) can store data that supports various functions of the artificial intelligence device (100). For example, the memory (170) can store input data, training data, training models, training history, etc. obtained from the input interface (120).

[0064] The processor (180) can determine at least one executable operation of the artificial intelligence device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm.

[0065] The processor (180) can control the components of the artificial intelligence device (100) to perform a determined operation.

[0066] To this end, the processor (180) can request, search, receive, or utilize data from the learning processor (130) or memory (170), and can control the components of the artificial intelligence device (100) to execute a predicted operation or a preferred operation among the at least one executable operation.

[0067] If the processor (180) requires the connection of an external device to perform a determined operation, it can generate a control signal to control the external device and transmit the generated control signal to the external device.

[0068] The processor (180) can obtain intent information regarding user input and determine the user's requirements based on the obtained intent information.

[0069] The processor (180) can obtain intent information corresponding to user input by using at least one of a Speech To Text (STT) engine for converting voice input into a string or a Natural Language Processing (NLP) engine for obtaining intent information of natural language.

[0070] At least one of the STT engine or NLP engine may be composed of an artificial neural network, at least a portion of which is trained according to a machine learning algorithm. Additionally, at least one of the STT engine or NLP engine may be trained by a learning processor (130), trained by a learning processor (240) of an AI server (200), or trained by distributed processing thereof.

[0071] The processor (180) can collect history information, including the operation details of the artificial intelligence device (100) or user feedback regarding the operation, and store it in memory (170) or a learning processor (130), or transmit it to an external device such as an AI server (200). The collected history information can be used to update the learning model.

[0072] The processor (180) can control at least some of the components of the artificial intelligence device (100) to run an application stored in memory (170).

[0073] The processor (180) can operate two or more of the components included in the artificial intelligence device (100) in combination with each other to operate the application.

[0074] FIG. 2 is a drawing for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0075] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network.

[0076] The AI ​​server (200) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network. The AI ​​server (200) may be included as part of the configuration of the artificial intelligence device (100) to perform at least some of the AI ​​processing together.

[0077] The AI ​​server (200) may include a communication interface (210), memory (230), a learning processor (240), and a processor (260).

[0078] The communication interface (210) can transmit and receive data with an external device, such as an artificial intelligence device (100).

[0079] The memory (230) may include a model memory (231). The model memory (231) may store a model (or artificial neural network, 231a) that is being learned or has been learned through the learning processor (240).

[0080] The learning processor (240) can train the artificial neural network (231a) using training data. The training model may be used while mounted on the AI ​​server (200) of the artificial neural network, or it may be used while mounted on an external device such as an artificial intelligence device (100).

[0081] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0082] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0083] FIGS. 3a and 3b are drawings for explaining the configuration of an artificial intelligence robot according to one embodiment of the present disclosure.

[0084] Referring to FIG. 3a, an artificial intelligence robot (300) according to one embodiment of the present disclosure may include a communication circuit (310), a first arm (320), a second arm (330), a motor (340), a camera (350), a memory (370), and a hardware processor (390).

[0085] The communication circuit (310) can transmit and receive data with external devices, such as an artificial intelligence device (100) or an AI server (200), using wired or wireless communication technology. For example, the communication interface (110) can transmit and receive sensor information, user input, learning models, control signals, etc., with external devices.

[0086] The communication technologies used by the communication circuit (310) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0087] The first arm (320) may represent the left arm of the artificial intelligence robot (300). The first arm (320) may be referred to as the first gripper, left arm, or lead arm.

[0088] The second arm (330) may represent the right arm of the artificial intelligence robot (300). The second arm (330) may be referred to as a second gripper, right arm, or follow arm.

[0089] The motor (340) can control the operation of the first arm (320) and the second arm (330). The motor (340) may be provided in one or more ways.

[0090] The motor (340) may include a first sub-motor for controlling the operation of the first arm (320) and a second sub-motor for controlling the operation of the second arm (330).

[0091] The camera (350) can acquire an image or video obtained by an image sensor.

[0092] The memory (370) can store multiple action plan templates. The memory (370) can further store multiple embedding vectors corresponding to the multiple action plan templates.

[0093] The hardware processor (390) can control the overall operation of the artificial intelligence robot (300). The hardware processor (390) may be provided in multiple units.

[0094] The hardware processor (390) can perform at least some of the operations performed by the processor (180) of the artificial intelligence device (100) described later.

[0095] The hardware processor (390) can control the operation of the first arm (320) and the second arm (330) according to robot control commands received from the artificial intelligence device (100).

[0096] A communication circuit (310), a motor (340), a camera (350), a memory (370), and a hardware processor (390) may be provided in the main body (not shown) of the artificial intelligence robot (300). A first arm (320) and a second arm (330) may be connected to the main body.

[0097] Referring to FIG. 3b, an artificial intelligence robot (300-1) according to another embodiment of the present disclosure may include an artificial intelligence device (100), a first arm (320), a second arm (330), and a motor (340).

[0098] That is, the artificial intelligence device (100) of FIG. 1 can be mounted on an artificial intelligence robot (300-1). The artificial intelligence device (100) and the motor (340) can be provided on the body of the artificial intelligence robot (300-1). The first arm (320) and the second arm (330) can be connected to the body.

[0099] FIG. 4 is a flowchart for explaining the operation method of an artificial intelligence device according to one embodiment of the present disclosure.

[0100] In the following description, the artificial intelligence device (100) may be any one of a terminal such as a computer, a laptop, or a smartphone. The processor (180) may be provided in multiple units.

[0101] The processor (180) of the artificial intelligence device (100) can acquire command and sensing information related to an object and a target space (S401).

[0102] The command may be a command to move an object to a target space (or a specific location) using the arm of the artificial intelligence robot (300). The command may be either a text instruction or a voice instruction.

[0103] In one embodiment, the processor (180) can receive text instructions through the user input interface (123).

[0104] In another embodiment, the processor (180) can receive voice instructions through the microphone (122).

[0105] In one embodiment, the sensing information may include one or more of an image, the distance between an object and an artificial intelligence robot (300) obtained from a distance measuring sensor, or the temperature measured through a temperature sensor. The sensing information may be referred to as observation information.

[0106] The image can be acquired by either a camera (121) equipped in an artificial intelligence device (100) or a camera (350) equipped in an artificial intelligence robot (300).

[0107] The image may be an initial image taken in correspondence with the position where the head of the artificial intelligence robot (300) is looking.

[0108] The processor (180) can divide the target space into multiple sub-spaces larger than the area of ​​the object based on command and sensing information (S403).

[0109] The processor (180) can identify a target space and an object based on an image included in the sensing information. The target space may also be a space formed by an object, such as a tray, or an object.

[0110] The processor (180) can identify a target space, objects, and obstacles from an image through an object detection model trained based on artificial intelligence. The object detection model can output identification information of objects included in the image, coordinate information of the area containing the objects, identification information of the target space, coordinate information of the target space, identification information of obstacles, and coordinate information of obstacles. The coordinate information may include coordinate values ​​of the area. The object detection model may be stored in memory (170).

[0111] Object detection models using machine learning include the single-stage YOLO (You Only Look Once) model and the two-stage Faster R-CNN (Regions with Convolutional Neural Networks) model.

[0112] The YOLO (You Only Look Once) model can predict objects present in an image and their locations all at once.

[0113] The YOLO (You Only Look Once) model divides the original image into grids of equal size. Then, for each grid, it predicts the number of bounding boxes specified in a predefined form centered on the grid center, and calculates confidence based on this.

[0114] After that, whether the image contains an object or only a background is included, and a location with high object reliability is selected so that the object category (object identification information) can be identified. The processor (180) can calculate the area of ​​the object based on the coordinates of the bounding box output by the YOLO model.

[0115] The Faster R-CNN (Regions with Convolutional Neural Networks) model is a model that can detect objects faster than the RCNN model and the Fast RCNN model.

[0116] This explains the Faster R-CNN (Regions with Convolutional Neural Networks) model in detail.

[0117] First, feature maps are extracted from the image using a Convolutional Neural Network (CNN) model. Based on the extracted feature maps, multiple Regions of Interest (RoIs) are identified. RoI pooling is performed for each region of interest.

[0118] RoI pooling is a process of setting a grid to fit a predetermined size of H x W for a feature map onto which a region of interest is projected, and extracting the largest value for each cell contained in the grid to extract a feature map with a size of H x W.

[0119] A feature vector is extracted from a feature map having size H x W, and identification information of an object can be obtained from the feature vector.

[0120] Faster R-CNN can output object identification information and coordinate information of the region of interest containing the object.

[0121] The processor (180) can calculate the area of ​​the object and the area of ​​the target space based on the coordinate information of the bounding box of the identified object or the area of ​​interest of the object.

[0122] The processor (180) can divide the target space into multiple subspaces larger than the area of ​​the object based on the calculated areas. The area of ​​each subspace may be larger than the area of ​​the object.

[0123] The area of ​​each of the multiple subspaces may be the same or different from each other.

[0124] The processor (180) can determine one of the divided multiple partial spaces as a destination and set a movement path for an object to move to the determined destination (S405).

[0125] The destination can be a point in the subspace. The destination can be the center point of the subspace. The destination can be a point in the subspace where the object can be placed normally.

[0126] The processor (180) may determine a space among a plurality of subspaces where no obstacles exist as a destination. The processor (180) may recognize objects other than the objects included in the command and the destination space as obstacles. If an obstacle exists in a subspace, the processor (180) may exclude that subspace from being a candidate for a destination.

[0127] The processor (180) can assign priorities to multiple subspaces where no obstacles exist and determine a destination according to the priorities. The priority can be determined by at least one of the area of ​​the subspace or the distance between an object and the subspace.

[0128] In one example, a higher priority can be assigned as the area of ​​the subspace increases. In another example, a higher priority can be assigned as the distance between the object and the subspace decreases.

[0129] In another embodiment, the processor (180) may determine the largest area among a plurality of subspaces where no obstacles exist as the destination. The processor (180) may assign priority to each of the plurality of subspaces according to the size of the area of ​​each of the plurality of subspaces where no obstacles exist.

[0130] In another embodiment, the processor (180) can measure the distance between each of a plurality of subspaces where no objects or obstacles exist, and can set the subspace corresponding to the shortest distance among the measured distances as the destination. The processor (180) can assign the highest priority to the subspace corresponding to the shortest distance among the measured distances.

[0131] In another embodiment, the processor (180) may set the partial space with the largest area among the multiple partial spaces where no obstacles exist and corresponding to the shortest distance among the measured distances as the highest priority destination.

[0132] The processor (180) can set a movement path for moving an object to a determined subspace among a plurality of subspaces. The movement path may be a path where the first arm (320) or the second arm (330) of the artificial intelligence robot (300) carries the object to a destination.

[0133] The movement path can follow either a straight line or a curve.

[0134] The processor (180) can determine whether there is a possibility of collision on the object's path of movement (S407).

[0135] The processor (180) can determine whether there is a possibility of collision due to an obstacle on the path of movement between the location of the object and the location of the determined destination.

[0136] The processor (180) can determine the possibility of a collision based on whether the line connecting the position of the object and the position of the destination intersects a two-dimensional polygon corresponding to an obstacle. This will be described later.

[0137] If the processor (180) determines that there is a possibility of collision, it can determine a new path to move to the determined destination by deciding another part space among the multiple part spaces as the destination (S409).

[0138] When the processor (180) determines that there is a possibility of collision with an obstacle when moving an object along a set movement path, it may switch the destination from the existing first subspace to a new second subspace. The processor (180) may determine the destination from the highest priority first subspace to the next priority second subspace. The processor (180) may set a new movement path connecting the location of the determined second subspace and the object.

[0139] Meanwhile, if the processor (180) determines that there is no possibility of collision on the new movement path of the object, it can control both arms of the artificial intelligence robot (300) to move the object according to the set movement path (S411).

[0140] The processor (180) can transmit a control command to the artificial intelligence robot (300) to move an object according to a new movement path set through the communication interface (110). The control command may include information about the object's movement path and location information of the destination.

[0141] FIG. 5 is a drawing illustrating an example of moving an object to a partial space in a target space when no obstacles exist, according to one embodiment of the present disclosure.

[0142] Referring to FIG. 5, the target space (500) can be divided into first to third partial spaces (510, 520, 530).

[0143] The artificial intelligence device (100) can determine any one of the plurality of subspaces (510, 520, 530) as the destination of an object (or target object, 501).

[0144] For example, the artificial intelligence device (100) can determine the second subspace (520), which is the most central of the plurality of subspaces (510, 520, 530), as the destination.

[0145] The artificial intelligence device (100) can determine any one of a plurality of subspaces (510, 520, 530) as the destination of the object (501) based on priority.

[0146] As another example, the artificial intelligence device (100) can determine the largest area of ​​the multiple subspaces (510, 520, 530), the subspace (520), as the destination of the object (501).

[0147] As another example, the artificial intelligence device (100) can determine the third subspace (530) closest to the object among the plurality of subspaces (510, 520, 530) as the destination of the object (501).

[0148] The artificial intelligence device (100) can set a movement path (540) to move the object (501) to the destination as the destination of the object (501) is determined.

[0149] FIGS. 6a to 6d are drawings illustrating an example of moving an object to a partial space in a target space when an obstacle is present, according to an embodiment of the present disclosure.

[0150] First, FIG. 6a is described. In FIG. 6a, it is assumed that the destination of the object (501) is determined to be a third subspace (530). The artificial intelligence device (100) can set a third movement path (613) for moving the object (501) to the third subspace (530). The artificial intelligence device (100) can determine whether there is a possibility of collision with an obstacle (601) on the third movement path (613).

[0151] If the artificial intelligence device (100) determines that there is a possibility of collision due to an obstacle (601) on the third movement path (613), it may change the destination from the third partial space (530) to the first partial space (510) or the second partial space (520). The artificial intelligence device (100) may determine the space with higher priority among the first partial space (510) and the second partial space (520) as the destination.

[0152] If the artificial intelligence device (100) determines the destination as the first partial space (510), it can change the travel path from the third travel path (613) to the first travel path (611). If the artificial intelligence device (100) determines the destination as the second partial space (520), it can change the travel path from the third travel path (613) to the second travel path (612).

[0153] Next, Fig. 6b will be explained.

[0154] The artificial intelligence device (100) can divide the target space (620) into a plurality of sub-spaces (621 to 624). The artificial intelligence device (100) can determine the destination of the object (501) to one of the sub-spaces (622, 623) among the plurality of sub-spaces (621 to 624) where there is no possibility of collision with the obstacle (601). The artificial intelligence device (100) can determine one of the sub-spaces (622, 623) as the destination according to priority. The artificial intelligence device (100) can control the artificial intelligence robot (300) to move the object (501) along a movement path (622a) corresponding to the sub-space (622) or a movement path (623a) corresponding to the sub-space (623).

[0155] Next, Fig. 6c is explained.

[0156] The artificial intelligence device (100) can divide the target space (630) into multiple sub-spaces (631, 632) where there is no possibility of collision with an obstacle (601). The artificial intelligence device (100) can determine any one of the sub-spaces (631, 632) as a destination according to priority. The artificial intelligence device (100) can control the artificial intelligence robot (300) to move an object (501) along a movement path (631a) corresponding to the sub-space (631) or a movement path (632a) corresponding to the sub-space (632).

[0157] Next, Fig. 6d will be explained.

[0158] The artificial intelligence device (100) can divide the target space (640) into a plurality of sub-spaces (641, 642). The artificial intelligence device (100) can determine the destination of the object (501) in the sub-space (642) among the plurality of sub-spaces (641, 642) where there is no possibility of collision with an obstacle (601). The artificial intelligence device (100) can control the artificial intelligence robot (300) to move the object (501) along a movement path (642a) corresponding to the sub-space (642).

[0159] As such, according to an embodiment of the present disclosure, the artificial intelligence robot (300) can move an object by avoiding obstacles. Accordingly, the artificial intelligence robot (300) can operate safely even in an environment where there are unexpected obstacles or people. In addition, damage to the artificial intelligence robot (300) or surrounding equipment due to collision with obstacles can be prevented.

[0160] In addition, a robot with an obstacle avoidance function according to an embodiment of the present disclosure can plan an optimized path and select the most efficient path to a destination, thereby reducing working time.

[0161] FIGS. 7a and 7b are drawings illustrating a method for detecting whether an obstacle exists on a path of movement between an object and a destination according to an embodiment of the present disclosure.

[0162] Referring to FIG. 7a, if there is a first vector (710) connecting the first point (701) and the second point (702) and a second vector (720) connecting the third point (703) and the fourth point (704), whether there is an intersection point between the two vectors (710, 720) can be determined based on the determinant.

[0163] The determinant can be obtained as shown in the following [Equation 1].

[0164] [Mathematical Formula 1]

[0165]

[0166] The artificial intelligence device (100) can determine that there is no intersection point between two vectors (710, 720) when there is no determinant.

[0167] The artificial intelligence device (100) can determine that the direction of the first vector (710) and the direction of the second vector (720) are in agreement (or parallel).

[0168] If there is no intersection point between the two vectors (710, 720), the following relationship [Equation 2] may hold in the determinant.

[0169] [Mathematical Formula 2]

[0170]

[0171] The artificial intelligence device (100) can detect a collision with an obstacle based on whether a line connecting each corner of a polygon corresponding to the obstacle, the initial position of the object, and the destination of the object intersects.

[0172] Referring to FIG. 7b, a third vector (713) connecting the initial position (O) of the target object and the destination (P) of the target object is shown. The third vector (713) may represent the movement path of the target object. The outer object (700) having a rectangular shape may have four vertices (p1, p2, p3, p4).

[0173] The artificial intelligence device (100) can determine that two vectors intersect if there is a determinant between the third vector (713) and the fourth vector (731) connecting P3 and P1. The artificial intelligence device (100) can determine that two vectors intersect if there is a determinant between the third vector (713) and the fifth vector (732) connecting P4 and P2.

[0174] The artificial intelligence device (100) can determine that an external object (700) is an obstacle if the distance (d) between the first intersection point (C1) between the third vector (713) and the fourth vector (731) and the second intersection point (C2) between the third vector (713) and the fifth vector (732) is a real value greater than 0.

[0175] The distance (d) between the first intersection (C1) and the second intersection (C2) can be obtained by the following [Equation 3].

[0176] [Mathematical Formula 3]

[0177]

[0178] α may be the angle formed between the vector (713) connecting the initial position (O) and destination (P) of the object and the vector (733) connecting P2 and P4.

[0179] Cos(α) can be obtained through the following [Equation 4].

[0180] [Mathematical Formula 4]

[0181]

[0182] In this way, according to an embodiment of the present disclosure, obstacles present on a movement path can be detected based on the distance between each corner of a polygon and the intersection points with a line corresponding to the movement path of an object.

[0183] FIGS. 8A and 8B are drawings illustrating the operation of an artificial intelligence robot that avoids an obstacle when an obstacle exists on the path of an object according to an embodiment of the present disclosure.

[0184] Referring to FIG. 8a, in the case of the prior art, when the second arm (330) of the artificial intelligence robot (300) moves the target object (800) along the movement path (801) to the destination (804) of the target space (803), there is a possibility of collision due to the presence of an obstacle (805).

[0185] FIG. 8b shows that when the obstacle avoidance technology according to the present disclosure is applied, the artificial intelligence device (100) can obtain a modified movement path (806) by modifying the existing movement path (801). According to an embodiment of the present disclosure, when the second arm (330) of the artificial intelligence robot (300) moves the target object (800) to the destination (807) of the target space (803) along the modified movement path (806), the obstacle (805) can be avoided.

[0186] FIGS. 9 to 11 are drawings illustrating a method of moving a plurality of objects to a target space according to an embodiment of the present disclosure.

[0187] Referring to FIG. 9, the artificial intelligence device (100) below may be any one of a terminal such as a computer, a laptop, or a smartphone. The processor (180) may be provided in multiple numbers.

[0188] The processor (180) of the artificial intelligence device (100) can acquire commands and sensing information related to a plurality of objects and a target space (S901).

[0189] The command may be a command to move multiple objects to a target space (or a specific location) using the arm of the artificial intelligence robot (300). The command may be either a text instruction or a voice instruction.

[0190] Based on command and sensing information, the processor (180) can divide the target space into multiple subspaces that are larger than the area of ​​the object and greater than the number of objects (S903).

[0191] The processor (180) can identify a target space and multiple objects based on an image included in the sensing information. The target space may also be a space formed by a single object, such as a tray, or an object.

[0192] The processor (180) can identify a target space and multiple objects from an image through an object detection model.

[0193] The processor (180) can determine the movement path for each of the multiple objects (S905).

[0194] The processor (180) can determine a movement path for moving each of a plurality of objects to each of a plurality of subspaces. The processor (180) can determine the movement path of each object using a collision prediction pre-planning algorithm. The collision prediction pre-planning algorithm may be an algorithm that plans the destination of the object and the order of movement of the object so as not to cause a collision with other objects or obstacles when moving each object.

[0195] The collision prediction pre-planning algorithm can be either Dijkstra's Algorithm or the A* algorithm.

[0196] Dijkstra's Algorithm is an algorithm that finds the shortest path from one point to another, and it can be an algorithm that plans a movement path without collisions with obstacles based on a priority queue.

[0197] The A* algorithm can be an algorithm that plans a movement path using a heuristic function.

[0198] The processor (180) can determine whether there is a possibility of collision on each movement path (S907).

[0199] If the processor (180) determines that there is a possibility of collision on each movement path, it can modify the movement path (S909).

[0200] The processor (180) can modify at least one of the movement paths if there is a possibility of collision with an obstacle on any one of the movement paths.

[0201] When the processor (180) determines that there is no possibility of collision on each movement path, it can control the two arms of the artificial intelligence robot (300) so that multiple objects move according to the determined movement paths (S911).

[0202] Referring to FIG. 10, a target space (1000) and a plurality of objects (1001 to 1004) are illustrated. An artificial intelligence device (100) can identify the target space (1000) and the plurality of objects (1001 to 1004) based on an image included in the sensing information. The image may be an image captured by the artificial intelligence device (100) or an artificial intelligence robot (300).

[0203] The artificial intelligence device (100) can receive a command to move a plurality of objects (1001 to 1004) onto a target space (1000).

[0204] The artificial intelligence device (100) can divide the target space (1000) into a plurality of sub-spaces (1010 to 1040) based on command and sensing information.

[0205] The artificial intelligence device (100) can obtain multiple movement paths corresponding to each of the multiple objects (1001 to 1004) so ​​that no collision occurs between each object.

[0206] First, the artificial intelligence device (100) can control the arm of the artificial intelligence robot (300) to move the first object (1001) to the first partial space (1010) along the first movement path (1001a).

[0207] After that, the artificial intelligence device (100) can control the arm of the artificial intelligence robot (300) to move the second object (1002) to the second partial space (1020) along the second movement path (1002a).

[0208] After that, the artificial intelligence device (100) can control the arm of the artificial intelligence robot (300) to move the third object (1003) to the third partial space (1030) along the third movement path (1003a).

[0209] After that, the artificial intelligence device (100) can control the arm of the artificial intelligence robot (300) to move the fourth object (1004) to the fourth subspace (1040) along the fourth movement path (1004a).

[0210] When the fourth object (1004) is moved to the fourth subspace (1040) before the second object (1002), there is a risk of collision with the fourth object (1004) when the second object (1002) moves. The artificial intelligence device (100) can set movement paths (1001a, 1002a, 1003a, 1004a) through a collision planning prior algorithm so that no collision occurs between the objects (1001 to 1004).

[0211] Referring to FIG. 11, a target space (1100) and a plurality of objects (1101 to 1003) are illustrated. An artificial intelligence device (100) can identify the target space (1100) and the plurality of objects (1101 to 1103) based on an image included in the sensing information. The image may be an image captured by the artificial intelligence device (100) or an artificial intelligence robot (300).

[0212] The artificial intelligence device (100) can receive a command to move a plurality of objects (1101 to 1103) onto a target space (1100).

[0213] The artificial intelligence device (100) can divide the target space (1100) into a plurality of sub-spaces (1110 to 1130) based on command and sensing information.

[0214] The artificial intelligence device (100) can obtain multiple movement paths corresponding to each of the multiple objects (1101 to 1103) so that no collision occurs between each object.

[0215] First, the artificial intelligence device (100) can control the arm of the artificial intelligence robot (300) to move the first object (1001) to the first partial space (1110) along the first movement path (1101a).

[0216] After that, the artificial intelligence device (100) can control the arm of the artificial intelligence robot (300) to move the second object (1102) to the second partial space (1120) along the second movement path (1102a).

[0217] After that, the artificial intelligence device (100) can control the arm of the artificial intelligence robot (300) to move the third object (1103) to the third partial space (1130) along the third movement path (1103a).

[0218] When the second object (1102) is moved to the second subspace (1120) before the first object (1101), there is a risk of collision with the second object (1102) when the first object (1101) moves. The artificial intelligence device (100) can set movement paths (1101a, 1102a, 1103a) through a collision planning prior algorithm so that no collision occurs between the objects (1101 to 1103).

[0219] An artificial intelligence device (100) according to one embodiment of the present disclosure may include one or more processors (180) that acquire commands and images related to an object and a target space, divide the target space into a plurality of subspaces larger than the object based on the commands and images, determine a movement path of the object by setting one of the divided subspaces as a destination, determine a possibility of collision with an obstacle on the movement path, and if it is determined that a possibility of collision with the obstacle exists, set another subspace among the plurality of subspaces as the destination to establish a new movement path, and then re-determine the possibility of collision.

[0220] The above one or more processors (180) may set the other partial space as the final destination of the object if there is no possibility of collision on the new movement path.

[0221] The above one or more processors (180) can control the operation of the artificial intelligence robot to move the object to the final destination according to the new movement path.

[0222] The above one or more processors (180) can assign priority to each of the plurality of partial spaces, and the priority can be determined according to at least one of the area of ​​the partial space or the distance between the object and the partial space.

[0223] One or more processors (180) can determine the likelihood of collision with the obstacle based on whether a line from the initial position of the object toward the destination intersects two corners of a polygon corresponding to the obstacle.

[0224] One or more processors may calculate the distance between the line and two intersection points between the two corners, and if the calculated distance is greater than 0, determine that the obstacle exists on the movement path.

[0225] In the case where the command moves multiple objects to the target space, the number of the multiple subspaces may be greater than the number of objects.

[0226] The above one or more processors (180) can generate multiple movement paths for moving each of the plurality of objects to each of the respective partial spaces, and control the artificial intelligence robot so that the plurality of objects are moved according to the generated multiple movement paths.

[0227] The artificial intelligence device (100) may further include a memory (170) that stores an object detection model learned based on an artificial neural network, and the one or more processors (180) can identify the object, the target space, and the obstacle from the image through the object detection model.

[0228] The artificial intelligence device (100) may further include a communication interface (110), and the one or more processors (180) may transmit a control command to the artificial intelligence robot to move the object along the new movement path through the communication interface.

[0229] The above-described disclosure can be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include a Hard Disk Drive (HDD), a Solid State Disk (SSD), a Silicon Disk Drive (SDD), ROM, RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. Additionally, the computer may include a processor (180) of an artificial intelligence device.

Claims

1. In an artificial intelligence device, One or more processors comprising: acquiring commands and images related to an object and a destination space; dividing the destination space into a plurality of subspaces larger than the object based on the commands and images; determining the movement path of the object by setting one of the divided subspaces as a destination; determining the possibility of collision with an obstacle on the movement path; and, if it is determined that there is a possibility of collision with the obstacle, setting another of the plurality of subspaces as the destination to establish a new movement path, and then re-evaluating the possibility of collision. Artificial intelligence device.

2. In Paragraph 1, The above one or more processors If there is no possibility of collision on the new movement path, the other partial space is set as the final destination of the object. Artificial intelligence device.

3. In Paragraph 2, The above one or more processors Controlling the movement of an artificial intelligence robot to move the object to the final destination according to the new movement path. Artificial intelligence device.

4. In Paragraph 1, The above one or more processors A priority is assigned to each of the plurality of subspaces, and the priority is determined according to at least one of the area of ​​the subspace or the distance between the object and the subspace. Artificial intelligence device.

5. In Paragraph 1, The above one or more processors Determining the possibility of collision with the obstacle based on whether a line from the initial position of the object toward the destination intersects two corners of a polygon corresponding to the obstacle. Artificial intelligence device.

6. In Paragraph 5, The above one or more processors Calculate the distance between the above line and the two intersection points between the two corners, and if the calculated distance is greater than 0, determine that the obstacle exists on the movement path. Artificial intelligence device.

7. In Paragraph 1, The above one or more processors If the above command moves multiple objects to the above target space, the number of the multiple subspaces is greater than the number of the objects Artificial intelligence device.

8. In Paragraph 7, The above one or more processors Generating multiple movement paths that move each of the above multiple objects to each of the above partial spaces, and Controlling an artificial intelligence robot so that the plurality of objects move according to the generated plurality of movement paths Artificial intelligence device.

9. In Paragraph 1, It further includes memory for storing an object detection model trained based on an artificial neural network, and The above one or more processors Identifying the object, the target space, and the obstacle from the image through the object detection model Artificial intelligence device.

10. In Paragraph 1, Includes additional communication interfaces, The above one or more processors Transmitting a control command to an artificial intelligence robot to move the object along the new movement path through the communication interface above. Artificial intelligence device.

11. In a method of operating an artificial intelligence device, A step of acquiring commands and images related to the object and the target space; A step of dividing the target space into multiple subspaces wider than the object based on the above command and image; A step of determining the movement path of the object by setting one of the multiple divided partial spaces as a destination; A step of determining the possibility of collision with an obstacle on the above movement path; and If it is determined that there is a possibility of collision with the above obstacle, the method includes the step of setting another part space among the plurality of part spaces as the destination to establish a new movement path, and then re-evaluating the possibility of collision. Method of operation of an artificial intelligence device.

12. In Paragraph 11, If there is no possibility of collision on the new movement path, the method further includes the step of setting the other partial space as the final destination of the object. Method of operation of an artificial intelligence device.

13. In Paragraph 12, The method further includes the step of controlling the movement of an artificial intelligence robot to move the object to the final destination according to the new movement path. Method of operation of an artificial intelligence device.

14. In Paragraph 11, The method further includes the step of assigning priority to each of the plurality of subspaces, wherein the priority is determined according to at least one of the area of ​​the subspace or the distance between the object and the subspace. Method of operation of an artificial intelligence device.

15. In Paragraph 11, The step of determining the above collision possibility is A step of determining the possibility of collision with the obstacle based on whether a line from the initial position of the object toward the destination intersects two corners of a polygon corresponding to the obstacle. Method of operation of an artificial intelligence device.

16. In Paragraph 15, The step of determining the above collision possibility is The method includes the step of calculating the distance between the line and the two intersection points between the two corners, and determining that the obstacle exists on the movement path if the calculated distance is greater than 0. Method of operation of an artificial intelligence device.

17. In Paragraph 11, If the above command moves multiple objects to the above target space, the number of the multiple subspaces is greater than the number of the objects Method of operation of an artificial intelligence device.

18. In Paragraph 17, A step of generating a plurality of movement paths for moving each of the plurality of objects to each of the respective partial spaces; and The method further includes the step of controlling an artificial intelligence robot so that the plurality of objects move according to the generated plurality of movement paths. Method of operation of an artificial intelligence device.

19. In Paragraph 11, The method further includes the step of identifying the object, the target space, and the obstacle from the image through an object detection model trained based on an artificial neural network. Method of operation of an artificial intelligence device.

20. A computer-readable recording medium having a program recorded thereon for performing a method of operating an artificial intelligence device, The above method of operation A step of acquiring commands and images related to the object and the target space; A step of dividing the target space into multiple subspaces wider than the object based on the above command and image; A step of determining the movement path of the object by setting one of the multiple divided partial spaces as a destination; A step of determining the possibility of collision with an obstacle on the above movement path; and A recording medium comprising the step of, when it is determined that there is a possibility of collision with the above obstacle, setting another part space among the plurality of part spaces as the destination to establish a new movement path, and then re-evaluating the possibility of collision.

Images