Artificial intelligence device, method for operating same, and recording medium

Abstract

An artificial intelligence device according to one embodiment of the present disclosure may comprise: a memory that stores a plurality of action plan templates; and one or more processors that obtain a command and an initial image, obtain one of the plurality of action plan templates on the basis of the obtained command and initial image, generate a robot control command from the command, the initial image, and the obtained action plan template, and control both arms of a robot using the generated robot control command.

Description

Artificial intelligence device, its method of operation, and recording medium

[0001] The present invention relates to an artificial intelligence device, and more specifically, to an artificial intelligence device capable of controlling the operation of a robot.

[0002] The development of artificial intelligence (AI) robots has progressed rapidly, and innovative changes are expected to continue in the future. The advancement of AI robots is having a significant impact across technology, society, and the economy.

[0003] AI robots are equipped with two arms, and controlling objects with dual-armed AI robots is very complex and difficult. Users want AI robots to move similarly to humans.

[0004] However, to achieve human-like two-arm movements for every object in the world, an enormous amount of data is required for each object.

[0005] In addition, if the motion sequence of each arm is not defined, there is a problem where the robot has difficulty performing actual movements.

[0006] The purpose of the present disclosure is to establish an action plan for two arms so that an artificial intelligence robot can follow a movement similar to how a human acted.

[0007] The purpose of the present disclosure is to enable an artificial intelligence robot to accurately perform sequential actions based on an action plan template generated based on a human demonstration video.

[0008] An artificial intelligence device according to one embodiment of the present disclosure may include a memory for storing a plurality of action plan templates and one or more processors for acquiring a command and an initial image, acquiring one of the plurality of action plan templates based on the acquired command and the initial image, generating a robot control command from the command, the initial image and the acquired action plan template, and controlling two arms of a robot through the generated robot control command.

[0009] A method of operation of an artificial intelligence device according to one embodiment of the present disclosure may include the steps of storing a plurality of action plan templates, acquiring a command and an initial image, acquiring one of the plurality of action plan templates based on the acquired command and the initial image, generating a robot control command from the command, the initial image and the acquired action plan template, and controlling both arms of a robot through the generated robot control command.

[0010] In a computer-readable recording medium having a program recorded thereon for performing a method of operation of an artificial intelligence device according to one embodiment of the present disclosure, the method of operation

[0011] The method may include the steps of storing a plurality of action plan templates, acquiring a command and an initial image, acquiring one of the plurality of action plan templates based on the acquired command and initial image, generating a robot control command from the command, the initial image and the acquired action plan template, and controlling both arms of the robot through the generated robot control command.

[0012] According to an embodiment of the present disclosure, an artificial intelligence robot can be operated in a manner similar to how a human acts. Accordingly, tasks that would otherwise be performed by a human can be carried out by the artificial intelligence robot, thereby significantly improving task automation, productivity, and stability.

[0013] According to an embodiment of the present disclosure, the accuracy of task execution can be significantly improved as the operation of the artificial intelligence robot is controlled in a manner similar to that performed by a human.

[0014] According to an embodiment of the present disclosure, automatic control of a robot can be easily achieved with only simple commands and an initial image.

[0015] FIG. 1 is a block diagram for explaining the components of an artificial intelligence device according to one embodiment of the present disclosure.

[0016] FIG. 2 is a drawing for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0017] FIG. 3 is a drawing for explaining the configuration of an artificial intelligence robot according to one embodiment of the present disclosure.

[0018] FIG. 4 is a flowchart for explaining the operation method of an artificial intelligence device according to one embodiment of the present disclosure.

[0019] FIGS. 5 to 8 are drawings for explaining a method of constructing an action plan template according to one embodiment of the present disclosure.

[0020] FIGS. 9 and FIGS. 10 are drawings illustrating the process of obtaining a modified action plan template corresponding to an instruction and an initial image according to an embodiment of the present disclosure.

[0021] FIGS. 11a to 11g are drawings illustrating the process of an artificial intelligence robot performing a task according to robot control commands and action trajectory information.

[0022] Artificial intelligence refers to the field of researching artificial intelligence or the methodologies to create it, while machine learning refers to the field of researching methodologies to define the various problems addressed in the field of artificial intelligence and to solve them.

[0023] Machine learning is also defined as an algorithm that improves the performance of a task through consistent experience.

[0024] An Artificial Neural Network (ANN) is a model used in machine learning that can refer to a model capable of problem-solving, composed of artificial neurons (nodes) that form a network through the connection of synapses.

[0025] An artificial neural network can be defined by connection patterns between neurons in different layers, a learning process that updates model parameters, and an activation function that generates output values.

[0026] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and the artificial neural network may include synapses connecting the neurons. In an artificial neural network, each neuron may output a function value of an activation function for input signals, weights, and biases input through the synapses.

[0027] Model parameters refer to parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters refer to parameters that must be set before training in a machine learning algorithm, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0028] The objective of training an artificial neural network can be viewed as determining model parameters that minimize the loss function. The loss function can be used as an indicator to determine optimal model parameters during the training process of an artificial neural network.

[0029] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0030] Supervised learning refers to a method of training an artificial neural network with labels provided for the training data; a label can refer to the correct answer (or result value) that the artificial neural network must infer when training data is input into it.

[0031] Unsupervised learning can refer to a method of training an artificial neural network without being given labels for the training data.

[0032] Reinforcement learning can refer to a learning method that trains an agent defined within an environment to select an action or sequence of actions that maximizes the cumulative reward in each state.

[0033] Machine learning implemented as a deep neural network (DNN) containing multiple hidden layers among artificial neural networks is also called deep learning, and deep learning is a part of machine learning.

[0034] In the following, machine learning is used to include deep learning.

[0035] FIG. 1 is a block diagram for explaining the components of an artificial intelligence device according to one embodiment of the present disclosure.

[0036] The artificial intelligence device (100) can be implemented as a fixed device or a mobile device, such as a TV, projector, mobile phone, smartphone, desktop computer, laptop, digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia player), navigation, tablet PC, wearable device, set-top box (STB), DMB receiver, radio, washing machine, refrigerator, desktop computer, digital signage, robot, vehicle, etc.

[0037] Referring to FIG. 1, the artificial intelligence device (100) may include a communication interface (110), an input interface (120), a learning processor (130), a sensor (140), an output interface (150), a memory (170), and a processor (180).

[0038] The communication interface (110) can transmit and receive data with external devices, such as other artificial intelligence devices or AI servers (200), using wired or wireless communication technology. For example, the communication interface (110) can transmit and receive sensor information, user input, learning models, control signals, etc., with external devices.

[0039] The communication technologies used by the communication interface (110) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0040] The input interface (120) can acquire various types of data.

[0041] The input interface (120) may include a camera (121) for capturing images, a microphone (122) for receiving audio signals, and a user input interface (123) for receiving information from a user.

[0042] A camera (121) or a microphone (122) can be treated as a sensor, and a signal obtained from the camera (121) or the microphone (122) can be named as sensing data or sensor information.

[0043] The input interface (120) can obtain input data to be used when obtaining an output using training data and a training model for model training. The input interface (120) may also obtain unprocessed input data, in which case the processor (180) or the learning processor (130) can extract input features as a preprocessing step for the input data.

[0044] The camera (121) processes image frames, such as still images or video, obtained by an image sensor in video call mode or shooting mode. The processed image frames may be displayed on a display (151) or stored in memory (170).

[0045] The microphone (122) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application running) being performed by the artificial intelligence device (100). Meanwhile, various noise removal algorithms can be applied to the microphone (122) to remove noise generated during the process of receiving external acoustic signals.

[0046] The user input interface (123) is for receiving information from a user, and when information is input through the user input interface (123), the processor (180) can control the operation of the artificial intelligence device (100) to correspond to the input information.

[0047] The user input interface (123) may include mechanical input means (or mechanical keys, such as buttons, dome switches, jog wheels, jog switches, etc. located on the front / rear or side of the artificial intelligence device (100)) and touch input means.

[0048] As an example, a touch input means may consist of a virtual key, soft key, or visual key displayed on a touchscreen through software processing, or a touch key placed on a part other than the touchscreen.

[0049] The learning processor (130) can train a model composed of an artificial neural network using training data. The trained artificial neural network can be called a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for judgment to perform some action.

[0050] The learning processor (130) can perform AI processing together with the learning processor (240) of the AI ​​server (200).

[0051] The learning processor (130) may include memory integrated into or implemented in the artificial intelligence device (100). The learning processor (130) may also be implemented using memory (170), external memory directly coupled to the artificial intelligence device (100), or memory maintained in an external device.

[0052] The sensor (140) can acquire at least one of internal information of the artificial intelligence device (100), surrounding environment information of the artificial intelligence device (100), and user information using various sensors.

[0053] The sensor (140) may include one or more of a proximity sensor, an illuminance sensor, an accelerometer, a magnetic sensor, a gyroscope, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, a lidar sensor, and a radar sensor.

[0054] The output interface (150) can generate output related to visual, auditory, or tactile senses.

[0055] The output interface (150) may include a display (151) for outputting an image, an audio output interface (152) for outputting audio, a haptic device (153) for outputting tactile information, and a light output interface (154) for outputting light.

[0056] The display (151) displays (outputs) information processed by the artificial intelligence device (100). For example, the display (151) can display information on the execution screen of an application running on the artificial intelligence device (100), or UI (User Interface) and GUI (Graphic User Interface) information based on such execution screen information.

[0057] The display (151) can be implemented as a touch screen by forming a layered structure with the touch sensor or by being formed as an integral unit. The touch screen functions as a user input interface (123) that provides an input interface between the artificial intelligence device (100) and the user, and at the same time, can provide an output interface between the artificial intelligence device (100) and the user.

[0058] The audio output interface (152) can output audio data received from the communication interface (110) or stored in the memory (170) in call signal reception, call mode or recording mode, voice recognition mode, broadcast reception mode, etc.

[0059] The audio output interface (152) may include at least one of a receiver, a speaker, and a buzzer.

[0060] The haptic device (153) generates various tactile effects that the user can feel. A typical example of the tactile effect generated by the haptic device (153) can be vibration.

[0061] The light output interface (154) outputs a signal to indicate the occurrence of an event using the light of the light source of the artificial intelligence device (100). Examples of events occurring in the artificial intelligence device (100) may include receiving a message, receiving a call signal, a missed call, an alarm, a schedule notification, receiving an email, receiving information through an application, etc.

[0062] The memory (170) can store data that supports various functions of the artificial intelligence device (100). For example, the memory (170) can store input data, training data, training models, training history, etc. obtained from the input interface (120).

[0063] The processor (180) can determine at least one executable operation of the artificial intelligence device (100) based on information determined or generated using a data analysis algorithm or a machine learning algorithm.

[0064] The processor (180) can control the components of the artificial intelligence device (100) to perform a determined operation.

[0065] To this end, the processor (180) can request, search, receive, or utilize data from the learning processor (130) or memory (170), and can control the components of the artificial intelligence device (100) to execute a predicted operation or a preferred operation among the at least one executable operation.

[0066] If the processor (180) requires the connection of an external device to perform a determined operation, it can generate a control signal to control the external device and transmit the generated control signal to the external device.

[0067] The processor (180) can obtain intent information regarding user input and determine the user's requirements based on the obtained intent information.

[0068] The processor (180) can obtain intent information corresponding to user input by using at least one of a Speech To Text (STT) engine for converting voice input into a string or a Natural Language Processing (NLP) engine for obtaining intent information of natural language.

[0069] At least one of the STT engine or NLP engine may be composed of an artificial neural network, at least a portion of which is trained according to a machine learning algorithm. Additionally, at least one of the STT engine or NLP engine may be trained by a learning processor (130), trained by a learning processor (240) of an AI server (200), or trained by distributed processing thereof.

[0070] The processor (180) can collect history information, including the operation details of the artificial intelligence device (100) or user feedback regarding the operation, and store it in memory (170) or a learning processor (130), or transmit it to an external device such as an AI server (200). The collected history information can be used to update the learning model.

[0071] The processor (180) can control at least some of the components of the artificial intelligence device (100) to run an application stored in memory (170).

[0072] The processor (180) can operate two or more of the components included in the artificial intelligence device (100) in combination with each other to operate the application.

[0073] FIG. 2 is a drawing for explaining the configuration of an artificial intelligence server according to one embodiment of the present disclosure.

[0074] Referring to FIG. 2, the AI ​​server (200) may refer to a device that trains an artificial neural network using a machine learning algorithm or uses a trained artificial neural network.

[0075] The AI ​​server (200) may be composed of multiple servers to perform distributed processing and may be defined as a 5G network. The AI ​​server (200) may be included as part of the configuration of the artificial intelligence device (100) to perform at least some of the AI ​​processing together.

[0076] The AI ​​server (200) may include a communication interface (210), memory (230), a learning processor (240), and a processor (260).

[0077] The communication interface (210) can transmit and receive data with an external device, such as an artificial intelligence device (100).

[0078] The memory (230) may include a model memory (231). The model memory (231) may store a model (or artificial neural network, 231a) that is being learned or has been learned through the learning processor (240).

[0079] The learning processor (240) can train the artificial neural network (231a) using training data. The training model may be used while mounted on the AI ​​server (200) of the artificial neural network, or it may be used while mounted on an external device such as an artificial intelligence device (100).

[0080] The learning model may be implemented in hardware, software, or a combination of hardware and software. If part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in memory (230).

[0081] The processor (260) can use a learning model to infer a result value for new input data and generate a response or control command based on the inferred result value.

[0082] FIG. 3 is a drawing for explaining the configuration of an artificial intelligence robot according to one embodiment of the present disclosure.

[0083] Referring to FIG. 3, the artificial intelligence robot (300) may include a communication circuit (310), a first arm (320), a second arm (330), a motor (340), a camera (350), a memory (370), and a hardware processor (390).

[0084] The communication circuit (310) can transmit and receive data with external devices, such as an artificial intelligence device (100) or an AI server (200), using wired or wireless communication technology. For example, the communication interface (110) can transmit and receive sensor information, user input, learning models, control signals, etc., with external devices.

[0085] The communication technologies used by the communication circuit (310) include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc.

[0086] The first arm (320) may represent the left arm of the artificial intelligence robot (300). The first arm (320) may be referred to as the first gripper, left arm, or lead arm.

[0087] The second arm (330) may represent the right arm of the artificial intelligence robot (300). The second arm (330) may be referred to as a second gripper, right arm, or follow arm.

[0088] The motor (340) can control the operation of the first arm (320) and the second arm (330). The motor (340) may be provided in one or more ways.

[0089] The motor (340) may include a first sub-motor for controlling the operation of the first arm (320) and a second sub-motor for controlling the operation of the second arm (330).

[0090] The camera (350) can acquire an image or video obtained by an image sensor.

[0091] The memory (370) can store multiple action plan templates. The memory (370) can further store multiple embedding vectors corresponding to the multiple action plan templates.

[0092] The hardware processor (390) can control the overall operation of the artificial intelligence robot (300). The hardware processor (390) may be provided in multiple units.

[0093] The hardware processor (390) can perform at least some of the operations performed by the processor (180) of the artificial intelligence device (100) described later.

[0094] The hardware processor (390) can control the operation of the first arm (320) and the second arm (330) according to robot control commands received from the artificial intelligence device (100).

[0095] A communication circuit (310), a motor (340), a camera (350), a memory (370), and a hardware processor (390) may be provided in the main body (not shown) of the artificial intelligence robot (300). A first arm (320) and a second arm (330) may be connected to the main body.

[0096] FIG. 4 is a flowchart for explaining the operation method of an artificial intelligence device according to one embodiment of the present disclosure.

[0097] In the following description, the artificial intelligence device (100) may be any one of a terminal such as a computer, a laptop, or a smartphone. The processor (180) may be provided in multiple units.

[0098] The processor (180) of the artificial intelligence device (100) can acquire command and sensing information (S401).

[0099] In one embodiment, the command may be a command that directs the operation of the artificial intelligence robot (300). In particular, the command may be a command that directs the operation of the first arm (320) and the second arm (330) equipped in the artificial intelligence robot (300).

[0100] In one embodiment, the command may be either a text instruction or a voice instruction.

[0101] In one embodiment, the sensing information may include one or more of an image, the distance between an object and an artificial intelligence robot (300) obtained from a distance measuring sensor, or the temperature measured through a temperature sensor. The sensing information may be referred to as observation information.

[0102] The image can be acquired by either a camera (121) equipped in an artificial intelligence device (100) or a camera (350) equipped in an artificial intelligence robot (300).

[0103] The image may be an initial image taken in correspondence with the position where the head of the artificial intelligence robot (300) is looking.

[0104] The processor (180) of the artificial intelligence device (100) can obtain an action plan template based on the acquired command and sensing information (S403).

[0105] The processor (180) can call an action plan template that matches the acquired command and sensing information among a plurality of action plan templates.

[0106] The memory (170) of the artificial intelligence device (100) may store multiple action plan templates. The memory (170) may be referred to as a database.

[0107] The processor (180) can generate an action plan template based on instructions and a video of a person demonstrating the action plan.

[0108] The following describes how to build an action plan template.

[0109] FIGS. 5 to 8 are drawings for explaining a method of constructing an action plan template according to one embodiment of the present disclosure.

[0110] Referring to FIG. 5, the processor (180) of the artificial intelligence device (100) can generate a first action plan based on text instructions (S501).

[0111] The processor (180) can obtain a text instruction through the user input interface (123). The text instruction (601) is as shown in FIG. 6.<put cups and red can on the green tray> It can be a text-type directive such as

[0112] The processor (180) can generate a text-based first action plan (603) based on text instructions (601) and an initial sampling image (602). The initial sampling image (602) may be sampled from a human demonstration video (604). The initial sampling image (602) may be an image sampled at an early point in time of the human demonstration video (604).

[0113] The first action plan (603) may be a plan that specifically describes the text instruction (601). The first action plan (603) is, as illustrated in FIG. 6,<Pick white cup and put it in the green tray, Pick red can and put it in the green tray> It can be a plan of a text type like this.

[0114] The first action plan (603) can be referred to as a directive-based Sub-goal Planning.

[0115] The processor (180) can obtain a first action plan (603) from a text instruction (601) and an initial sampling image (602) using a Large Language Model (LLM) stored in memory (170).

[0116] The LLM may be an artificial intelligence-based model that generates response results from text or text and images. The processor (180) may generate a prompt requesting an action plan from a text instruction (601) and an initial sampling image (602), and may input the generated prompt into the LLM. The LLM may output a first action plan (603) in response to the prompt.

[0117] In another embodiment, the LLM may be stored in the memory (230) of the AI ​​server (200). The artificial intelligence device (100) may transmit a text instruction (601) and an initial sampling image (602) to the AI ​​server (200) and receive a first action plan (603) from the AI ​​server (200).

[0118] The processor (180) of the artificial intelligence device (100) can generate a second action plan based on a human demonstration video (S503).

[0119] The human demonstration video (604) may be a video containing an act of a person directly demonstrating an action in response to a command or instruction. For example, the human demonstration video (604) may be a video containing an act of a person directly moving a specific object to a specific location. The human demonstration video (604) may be filmed in advance and stored in memory (170).

[0120] The processor (180) can perform image sampling from a human demonstration video (604). The processor (180) can obtain an action description (605) from each of the plurality of sampled images sampled from the human demonstration video (604). The action description can describe the sampled images.

[0121] The action description (605) may include the arm used for the sampling image, the object held by the arm used, the location where the object is placed, and the skill used. The action description (605)<Used hand, Pick object, Place Position, Skill> It can have a format like this.

[0122] The processor (180) can obtain multiple action descriptions corresponding to each of the multiple sampled images. If the number of sampled images is t, the number of action descriptions may also be t.

[0123] The processor (180) can acquire multiple action descriptions as a second action plan (606). The second action plan (606) may be referred to as a video-based Sub-goal Planning.

[0124] Meanwhile, the skill included in the action description (605) may represent a unit action of the artificial intelligence robot (300). The skill may be any one of PICK, PLACE, HOLD, and IDLE, but this is merely an example.

[0125] FIGS. 7A and 7B are drawings illustrating robot skill information representing unit actions of an artificial intelligence robot according to an embodiment of the present disclosure.

[0126] The first robot skill information (700) may include a skill item (710), a marker color item (720), and a robot action item (730) of the artificial intelligence robot (300).

[0127] The skill item (710) may be an item indicating the skill type of the artificial intelligence robot (300).

[0128] Skills can be classified into four types. Referring to Fig. 7a, the PICK skill may represent the act of opening the gripper at position A and then closing the gripper. The PLACE skill may represent the act of opening the closed gripper at position B. The HOLD skill may represent the act of maintaining the closed state of the gripper at position C. The IDLE skill may represent the act of maintaining the open state of the gripper at position D.

[0129] The marker color item (720) may indicate the color marked on each skill. The PICK skill may be marked in green, the PPLACE skill may be marked in pink, the HOLD skill may be marked in red, and the IDLE skill may be marked in blue.

[0130] The robot action item (730) can represent the movement of the gripper corresponding to each skill.

[0131] The second robot skill information (740) may include a specific description of each skill. The second robot skill information (740) may be included in the first robot skill information (700). The second robot skill information (740) may be included in the robot action item (730) of the first robot skill information (700).

[0132] Again, Figure 4 is explained.

[0133] The processor (180) of the artificial intelligence device (100) can generate an action plan template based on a first action plan and a second action plan, and can store the generated action plan template in memory (170) (S505).

[0134] The processor (180) can generate an action plan template based on the first action plan (603), the second action plan (606), and robot skill information (700, 740).

[0135] The processor (180) can generate a prompt including a first action plan (603), a second action plan (606), and robot skill information (700, 740), and the generated prompt can be input into an LLM model. The processor (180) can obtain an action plan template corresponding to the prompt from the LLM. The prompt may be a command to generate an action plan using the first action plan (603), the second action plan (606), and robot skill information (700, 740).

[0136] Referring to FIG. 8, a behavior plan template (800) generated based on a text-based first behavior plan (603), a video-based behavior plan (606), and second robot skill information (740) is shown.

[0137] The action plan template (800) may include a dual arm plan (810). The action plan template (800) may further include a text directive (601) which is the basis for the creation of the action plan template (800), an initial sampling image (602) which is the basis for the creation of the action plan template (800), and an object list (820).

[0138] The action plan template (800) may include only the dual arm plan (810). Memory (170) may store the action plan template (800) by matching it to the text instructions (601), initial images (6020), and object lists (820).

[0139] The dual arm plan (810) may include one or more skill types or skill targets that each of the first arm (320) and the second arm (330) must perform sequentially. The dual arm plan (810) may include multiple dual arm sub-plans.

[0140] For example, the first dual arm sub-plan (811) may be a plan in which the first arm (320) performs a PICK skill on a pot lid and the second arm (330) performs an IDLE skill. The second dual arm sub-plan (812) may be a plan in which the first arm (320) performs a HOLD skill and the second arm (330) performs a PICK skill on an onion.

[0141] The object list (820) may include the names of objects identified from the initial sampling image (602).

[0142] The processor (180) can generate multiple action plan templates in this manner and can store the generated multiple action plan templates in memory (170).

[0143] The processor (180) can convert at least one of the text instructions (601) or the initial sampling image (602) into an embedding vector and can store the converted embedding vector in memory (170) by matching it with an action plan template (800). The converted embedding vector can be used to call the action plan template.

[0144] Again, Figure 4 is explained.

[0145] The processor (180) can extract a behavior plan template that matches the command and sensing information among a plurality of behavior plan templates stored in memory (170). The processor (180) can convert the command and sensing information into an embedding vector. The processor (180) can determine the embedding vector most similar to the converted embedding vector by comparing the converted embedding vector with the embedding vectors stored in memory (170). The processor (180) can obtain a behavior plan template that matches the determined embedding vector.

[0146] The processor (180) of the artificial intelligence device (100) can generate a robot control command based on commands, sensing information and an acquired action plan template (S405), and can control the operation of the first arm (320) and the second arm (330) of the artificial intelligence robot (300) according to the generated robot control command (S407).

[0147] The processor (180) can modify the action plan template obtained based on command and sensing information, and can obtain the modified action plan template as a robot control command. The modified action plan template may include skill types and targets of skills that the first arm (320) and the second arm (330) each must sequentially perform.

[0148] The processor (180) can transmit robot control commands to the artificial intelligence robot (300) through the communication interface (110). The artificial intelligence robot (300) can control the motor (340) so that the first arm (320) and the second arm (330) operate according to the received robot control commands.

[0149] In this way, according to an embodiment of the present disclosure, a dual-arm robot can mimic a motion similar to a human's motion according to a motion sequence for two arms.

[0150] FIGS. 9 and FIGS. 10 are drawings illustrating the process of obtaining a modified action plan template corresponding to an instruction and an initial image according to an embodiment of the present disclosure.

[0151] Referring to FIG. 9, the artificial intelligence device (100)<Pick ingredients on the tray and put them in the pot> You can obtain a text instruction (901) and an initial image (902) that is.

[0152] The text instruction (901) may be an instruction typed through the user input interface (123). The text instruction (901) may be an instruction in which voice obtained through the microphone (122) is converted into text.

[0153] The initial image (902) may be an image obtained through the camera (121) of the artificial intelligence device (100) or the camera (350) of the artificial intelligence robot (300). The initial image (902) may be an image taken before the operation of the artificial intelligence robot (300) begins. In FIG. 9, the initial image (902) may include a pot with a lid, a carrot, and broccoli.

[0154] An Embodied Cognition Agent (ECA, 910) can call an action plan template (930) from a template database (920) based on a text instruction (901) and an initial image (902). The ECA (910) may be hardware or software for calling the action plan template (930) of an artificial intelligence robot (300).

[0155] If the ECA (910) is hardware, the ECA (910) may be included in the processor (180).

[0156] If ECA (910) is software, ECA (910) can be executed by instructions from the processor (180).

[0157] The template database (920) may be included in memory (170) or provided separately. The template database (920) may store a plurality of action plan templates. The template database (920) may store a plurality of action plan templates and an embedding vector corresponding to each of the plurality of action plan templates.

[0158] ECA (910) can convert text instructions (901) and initial images (902) into embedding vectors. ECA (910) can compare the converted embedding vectors with multiple embedding vectors stored in a template database (920) and extract the embedding vector most similar to the converted embedding vector among the multiple embedding vectors.

[0159] The ECA (910) can call an action plan template (930) that is matched with the extracted embedding vector. The action plan template (930) may include instructions (931), an object list (932), and a dual arm plan (933).

[0160] The instruction (931) may be a text instruction entered by the user.

[0161] The object list (932) may include information about objects extracted from the initial image (902). The object list (932) may include a pot with a lid, a yellow tray, a carrot, an onion, broccoli, and a green bell pepper.

[0162] The dual arm plan (933) may include skill types and skill targets that each of the first arm (320) and the second arm (330) must perform sequentially.

[0163] ECA (910) can obtain a modified action plan template (940) by modifying the action plan template (930) based on the extracted action plan template (930), initial image (902), and second robot skill information (740).

[0164] ECA (910) can compare the objects included in the object list (932) of the action plan template (930) with the objects included in the initial image (902). As a result of the comparison, ECA (910) can remove the remaining objects from the object list (932) excluding the objects included in the initial image (902). ECA (910) can modify the dual arm plan (933) to include only the skills for the objects included in the initial image (902), and can obtain the modified dual arm plan as the modified action plan template (940).

[0165] The ECA (910) can obtain a modified action plan template (940) as a robot control command and transmit the obtained robot control command to the artificial intelligence robot (300). The artificial intelligence robot (300) can sequentially control the operation of the first arm (320) and the second arm (330) according to the modified dual arm plan included in the robot control command.

[0166] According to another embodiment of the present disclosure, the artificial intelligence device (100) can transmit action trajectory information to the artificial intelligence robot (300) in addition to robot control commands.

[0167] The action trajectory information may include information regarding the movement trajectories of the first arm (320) and the second arm (330) according to the modified dual arm plan. The action trajectory information may be information indicating the action trajectories that the first arm (320) and the second arm (330) must perform according to the modified dual arm plan.

[0168] The action trajectory information may include the location of skill execution for each of the first arm (320) and the second arm (330). The location of skill execution may include one or more of the location of an object included in the initial image (902) or the location where each arm must be moved.

[0169] The processor (180) can identify multiple objects from the initial image (902) and obtain the location (or coordinates) of each identified object.

[0170] FIGS. 11a to 11g are drawings illustrating the process of an artificial intelligence robot performing a task according to robot control commands and action trajectory information.

[0171] In FIGS. 11a to 11g, it is assumed that an artificial intelligence robot (300) is controlled based on a modified action plan template (940) obtained according to the embodiment of FIGS. 9 and 10.

[0172] The robot control command may include a modified action plan template (940) and a message requesting control of the artificial intelligence robot (300) according to the modified action plan template (940).

[0173] The modified action plan template (940) may include a plurality of dual-arm sub-plans (941 to 947). The plurality of dual-arm sub-plans (941 to 947) may be actions that the artificial intelligence robot (300) must perform sequentially.

[0174] Referring to FIGS. 11a through 11g, a first image (1110) and a second image (1120) are illustrated. An artificial intelligence device (100) can receive the first image (1110) and the second image (1120) from an artificial intelligence robot (300) and display them on a display (151). The first image (1110) and the second image (1120) may be images displayed by a display provided in the artificial intelligence robot (300).

[0175] The first image (1110) may be an image from the perspective of the first arm (320) of the artificial intelligence robot (300), and the second image (1120) may be an image from the perspective of the second arm (330) of the artificial intelligence robot (300).

[0176] Referring to FIG. 11a, the first dual arm sub-plan (941) may be a plan in which the first arm (320) performs a PICK skill on the pot lid (1111) and the second arm (330) performs an IDLE skill.

[0177] The artificial intelligence device (100) can display a green marker (1131) on the pot lid (1111) of the first image (1110) based on the position of the pot lid (1111) included in the action trajectory information and the skill type of the first arm (320) included in the first dual arm sub-plan (941). The green marker (1131) may be a marker that identifies an object (or the position of the object) to which the PICK skill of the first arm (320) is to be applied.

[0178] The artificial intelligence robot (300) can recognize the green marker (1131) and can perform a PICK skill on the position corresponding to the green marker (1131) through the first arm (320).

[0179] The artificial intelligence device (100) can display a blue marker (1132) on the second arm (330) of the second image (1120) based on the position of the second arm (330) included in the action trajectory information and the skill type of the second arm (330) included in the first dual arm sub-plan (941). The blue marker (1132) may be a marker that identifies the location to apply the IDLE skill of the second arm (330). The IDLE skill may be an action that maintains the open state of the second arm (330).

[0180] The artificial intelligence robot (300) can recognize the blue marker (1132) and can perform an IDLE skill on the location corresponding to the blue marker (1132) through the second arm (330).

[0181] Referring to FIG. 11b, the second dual arm sub-plan (942) may be a plan in which the first arm (320) performs a HOLD skill and the second arm (330) performs a PICK skill for broccoli (1112).

[0182] The artificial intelligence device (100) can display a red marker (1133) on the first arm (320) of the first image (1110) based on the position of the first arm (320) included in the action trajectory information and the skill type of the first arm (320) included in the second dual arm sub-plan (942). The red marker (1133) may be a marker that identifies the position to apply the HOLD skill of the first arm (320).

[0183] The artificial intelligence robot (300) can recognize the red marker (1133) and perform a HOLD skill at a position corresponding to the red marker (1133) through the first arm (320).

[0184] The artificial intelligence device (100) can display a green marker (1131) on the broccoli (1112) in the second image (1120) based on the location of the broccoli (1112) included in the action trajectory information and the skill type of the second arm (330) included in the second dual arm sub-plan (942). The green marker (1131) may be a marker that identifies the location of the object to which the PICK skill of the second arm (330) is to be applied.

[0185] The artificial intelligence robot (300) can recognize the green marker (1131) and can perform a PICK skill on the position corresponding to the green marker (1131) through the second arm (330).

[0186] Referring to FIG. 11c, the third dual arm sub-plan (943) may be a plan in which the first arm (320) performs a HOLD skill and the second arm (330) performs a PLACE skill for the pot (1113).

[0187] The artificial intelligence device (100) can display a red marker (1133) on the first arm (320) of the first image (1110) based on the position of the first arm (320) included in the action trajectory information and the skill type of the first arm (320) included in the third dual arm sub-plan (943). The red marker (1133) may be a marker that identifies the position to apply the HOLD skill of the first arm (320).

[0188] The artificial intelligence robot (300) can recognize the red marker (1133) and perform a HOLD skill at a position corresponding to the red marker (1133) through the first arm (320).

[0189] The artificial intelligence device (100) can display a pink marker (1134) on the pot (1113) of the second image (1120) based on the location of the pot (1113) included in the action trajectory information and the skill type of the second arm (330) included in the third dual arm sub-plan (943). The pink marker (1134) may be a marker that identifies the location of the object to which the PLACE skill of the second arm (330) is to be applied.

[0190] The artificial intelligence robot (300) can recognize the pink marker (1134) and can perform a PLACE skill on the position corresponding to the pink marker (1134) through the second arm (330).

[0191] Referring to FIG. 11d, the fourth dual arm sub-plan (944) may be a plan in which the first arm (320) performs a HOLD skill and the second arm (330) performs a PICK skill for the carrot (1114).

[0192] The artificial intelligence device (100) can display a red marker (1133) on the first arm (320) of the first image (1110) based on the position of the first arm (320) included in the action trajectory information and the skill type of the first arm (320) included in the fourth dual arm sub-plan (944). The red marker (1133) may be a marker that identifies the position to apply the HOLD skill of the first arm (320).

[0193] The artificial intelligence robot (300) can recognize the red marker (1133) and perform a HOLD skill at a position corresponding to the red marker (1133) through the first arm (320).

[0194] The artificial intelligence device (100) can display a green marker (1131) on the carrot (1113) of the second image (1120) based on the location of the carrot (1114) included in the action trajectory information and the skill type of the second arm (330) included in the fourth dual arm sub-plan (944).

[0195] The artificial intelligence robot (300) can recognize the green marker (1131) and can perform a PICK skill on the position corresponding to the green marker (1131) through the second arm (330).

[0196] Referring to FIG. 11e, the fifth dual arm sub-plan (945) may be a plan in which the first arm (320) performs a HOLD skill and the second arm (330) performs a PLACE skill for the pot (1113).

[0197] The artificial intelligence device (100) can display a red marker (1133) on the first arm (320) of the first image (1110) based on the position of the first arm (320) included in the action trajectory information and the skill type of the first arm (320) included in the fifth dual arm sub-plan (945). The red marker (1133) may be a marker that identifies the position to apply the HOLD skill of the first arm (320).

[0198] The artificial intelligence robot (300) can recognize the red marker (1133) and perform a HOLD skill at a position corresponding to the red marker (1133) through the first arm (320).

[0199] The artificial intelligence device (100) can display a pink marker (1134) on the pot (1113) of the second image (1120) based on the location of the pot (1113) included in the action trajectory information and the skill type of the second arm (330) included in the fifth dual arm sub-plan (945). The pink marker (1134) may be a marker that identifies the location of the object to which the PLACE skill of the second arm (330) is to be applied.

[0200] The artificial intelligence robot (300) can recognize the pink marker (1134) and can perform a PLACE skill on the position corresponding to the pink marker (1134) through the second arm (330).

[0201] Referring to FIG. 11f, the sixth dual arm sub-plan (946) may be a plan in which the first arm (320) performs a PICK skill on the pot lid (1111) and the second arm (330) performs an IDLE skill.

[0202] The artificial intelligence device (100) can display a pink marker (1134) on the pot (1113) of the first image (1110) based on the position of the pot lid (1111) included in the action trajectory information and the skill type of the first arm (320) included in the sixth dual arm sub-plan (946).

[0203] The artificial intelligence robot (300) can recognize the pink marker (1134) and can perform a PLACE skill on the position corresponding to the pink marker (1134) through the first arm (320).

[0204] The artificial intelligence device (100) can display a blue marker (1132) on the second arm (330) of the second image (1120) based on the position of the second arm (330) included in the action trajectory information and the skill type of the second arm (330) included in the sixth dual arm sub-plan (946). The blue marker (1132) may be a marker that identifies the location to apply the IDLE skill of the second arm (330). The IDLE skill may be an action that maintains the open state of the second arm (330).

[0205] The artificial intelligence robot (300) can recognize the blue marker (1132) and can perform an IDLE skill on the location corresponding to the blue marker (1132) through the second arm (330).

[0206] Referring to Fig. 11g, the seventh dual arm sub-plan (947) may be a plan in which the first arm (320) performs an IDLE skill and the second arm (330) performs an IDLE skill.

[0207] The artificial intelligence device (100) can display a blue marker (1132) on the first image (1110) based on the position of the first arm (330) included in the action trajectory information and the skill type of the first arm (320) included in the seventh dual arm sub-plan (947).

[0208] The artificial intelligence robot (300) can recognize a blue marker (1132) and can perform an IDLE skill on a location corresponding to the blue marker (1132) through the first arm (320).

[0209] The artificial intelligence device (100) can display a blue marker (1132) on the second image (1120) based on the position of the second arm (330) included in the action trajectory information and the skill type of the second arm (330) included in the seventh dual arm sub-plan (947). The blue marker (1132) may be a marker that identifies the position to which the IDLE skill of the second arm (330) is applied. The IDLE skill may be an action that maintains the open state of the second arm (330).

[0210] The artificial intelligence robot (300) can recognize the blue marker (1132) and can perform an IDLE skill on the location corresponding to the blue marker (1132) through the second arm (330).

[0211] As such, according to an embodiment of the present disclosure, the artificial intelligence robot (300) can be operated in a manner similar to how a human acts. Accordingly, tasks that a human would otherwise have to perform are performed through the artificial intelligence robot (300), thereby greatly improving the automation, productivity, and stability of the tasks.

[0212] An artificial intelligence device (100) according to one embodiment of the present disclosure may include a memory (170) for storing a plurality of action plan templates and one or more processors (180) for acquiring a command and an initial image, acquiring one of the plurality of action plan templates based on the acquired command and initial image, generating a robot control command from the command, the initial image and the acquired action plan template, and controlling both arms of a robot through the generated robot control command.

[0213] One or more processors (180) can modify the action plan template based on the command and the initial image, and obtain the modified action plan template as the robot control command.

[0214] The above one or more processors (180) can generate a first action plan based on text instructions, generate a second action plan based on a human demonstration video, generate an action plan template based on the first action plan and the second action plan, and store the generated action plan template in the memory.

[0215] One or more processors (180) above can generate the first action plan from the text instructions and the initial sampling images sampled from the human demonstration video using a Large Language Model (LLM).

[0216] The above one or more processors (180) can obtain a plurality of action descriptions from each of a plurality of sampling images sampled from the person demonstration video, and obtain the obtained plurality of action descriptions as the second action plan.

[0217] The above one or more processors (180) generate the action plan template based on the first action plan, the second action plan, and robot skill information, and the robot skill information may include information about skills performed by the first arm and the second arm of the robot.

[0218] The above action plan template may include the text instructions, the initial sampling image, and skill types that each of the first and second arms of the robot must perform.

[0219] The above one or more processors (180) can transmit the robot control command and action trajectory information regarding the movement trajectory of the robot's two arms to the robot.

[0220] The above action trajectory information may include the location of skill execution for each of the first arm and the second arm of the robot.

[0221] The memory (170) stores a plurality of embedding vectors corresponding to each of the plurality of action plan templates, and the one or more processors (180) convert the command and the initial image into embedding vectors, extract the embedding vector most similar to the converted embedding vector among the plurality of embedding vectors, and obtain an action plan template corresponding to the extracted embedding vector.

[0222] The above-described disclosure can be implemented as computer-readable code on a medium on which a program is recorded. A computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include a Hard Disk Drive (HDD), a Solid State Disk (SSD), a Silicon Disk Drive (SDD), ROM, RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. Additionally, the computer may include a processor (180) of an artificial intelligence device.

Claims

1. In an artificial intelligence device, Memory for storing multiple action plan templates; and One or more processors comprising acquiring a command and an initial image, acquiring one of the plurality of action plan templates based on the acquired command and initial image, generating a robot control command from the command, the initial image and the acquired action plan template, and controlling both arms of a robot through the generated robot control command. Artificial intelligence device.

2. In Paragraph 1, The above one or more processors Modifying the action plan template based on the above command and the above initial image, and obtaining the modified action plan template as the robot control command Artificial intelligence device.

3. In Paragraph 1, The above one or more processors Generating a first action plan based on text instructions, generating a second action plan based on a human demonstration video, generating an action plan template based on the first action plan and the second action plan, and storing the generated action plan template in the memory. Artificial intelligence device.

4. In Paragraph 3, The above one or more processors Using a Large Language Model (LLM), the first action plan is generated from the text instructions and initial sampling images sampled from the human demonstration video. Artificial intelligence device.

5. In Paragraph 4, The above one or more processors Obtaining multiple action descriptions from each of the multiple sampling images sampled from the above human demonstration video, and obtaining the obtained multiple action descriptions as the second action plan. Artificial intelligence device.

6. In Paragraph 3, The above one or more processors Based on the above first action plan, the above second action plan, and robot skill information, the above action plan template is generated, and The above robot skill information Information regarding skills performed by the first arm and the second arm of the above-mentioned robot Artificial intelligence device.

7. In Paragraph 4, The above action plan template is including the above text instruction, the above initial sampling image, and the skill type to be performed by each of the first and second arms of the robot. Artificial intelligence device.

8. In Paragraph 1, The above one or more processors Transmitting the above robot control command and action trajectory information regarding the movement trajectories of both arms of the robot to the robot Artificial intelligence device.

9. In Paragraph 8, The above action trajectory information is including the position of skill execution for each of the first and second arms of the above-mentioned robot Artificial intelligence device.

10. In Paragraph 1, The above memory is A plurality of embedding vectors corresponding to each of the above plurality of action plan templates are stored, and The above one or more processors Converting the above command and the above initial image into embedding vectors, extracting the embedding vector most similar to the converted embedding vector among the plurality of embedding vectors, and obtaining an action plan template corresponding to the extracted embedding vector. Artificial intelligence device.

11. In a method of operating an artificial intelligence device, A step of storing multiple action plan templates; Step of acquiring commands and initial images; A step of obtaining one of the plurality of action plan templates based on the obtained command and initial image; A step of generating a robot control command from the above command, the above initial image, and the acquired action plan template; and including the step of controlling both arms of the robot through the generated robot control command. Method of operation of an artificial intelligence device.

12. In Paragraph 11, The step of generating the above robot control command The method includes the step of modifying the action plan template based on the above command and the above initial image, and obtaining the modified action plan template as the robot control command. Method of operation of an artificial intelligence device.

13. In Paragraph 11, The above-mentioned storage step Step of generating a first action plan based on text instructions, Step of generating a second action plan based on a human demonstration video, A step of generating the action plan template based on the first action plan and the second action plan and Includes the step of saving the generated action plan template Method of operation of an artificial intelligence device.

14. In Paragraph 13, The step of generating the above-mentioned first action plan The method includes the step of generating the first action plan from the text instructions and the initial sampling images sampled from the human demonstration video using a Large Language Model (LLM). Method of operation of an artificial intelligence device.

15. In Paragraph 14, The step of generating the above second action plan A step of obtaining a plurality of action descriptions from each of a plurality of sampling images sampled from the above human demonstration video, and A step of acquiring a plurality of acquired action descriptions into the second action plan Method of operation of an artificial intelligence device.

16. In Paragraph 13, The above-mentioned storage step The method includes the step of generating the action plan template based on the first action plan, the second action plan, and robot skill information. The above robot skill information Information regarding skills performed by the first arm and the second arm of the above-mentioned robot Method of operation of an artificial intelligence device.

17. In Paragraph 14, The above action plan template is including the above text instruction, the above initial sampling image, and the skill type to be performed by each of the first and second arms of the robot. Method of operation of an artificial intelligence device.

18. In Paragraph 11, The method further includes the step of transmitting the above robot control command and action trajectory information regarding the movement trajectories of both arms of the robot to the robot, The above action trajectory information is including the position of skill execution for each of the first and second arms of the above-mentioned robot Method of operation of an artificial intelligence device.

19. In Paragraph 11, A step of storing a plurality of embedding vectors corresponding to each of the plurality of action plan templates; A step of converting the above command and the above initial image into an embedding vector Extracting the embedding vector most similar to the transformed embedding vector among the plurality of embedding vectors above, and obtaining an action plan template corresponding to the extracted embedding vector. Method of operation of an artificial intelligence device.

20. A computer-readable recording medium having a program recorded thereon for performing a method of operating an artificial intelligence device, The above method of operation Step of storing multiple action plan templates, Step to acquire commands and initial images, A step of obtaining one of the plurality of action plan templates based on the obtained command and initial image, A step of generating a robot control command from the above command, the above initial image, and the acquired action plan template, and including the step of controlling both arms of the robot through the generated robot control command. Recording media.

Images