Learning device, control device, learning method, control method, learning program, and control program

Abstract

A learning device (100A) for learning information for causing an operating entity to execute a task is provided with a first acquisition unit (133) and a training unit (134). The first acquisition unit (133) acquires status information indicating a status of a task target, which is the target of the task, and index information indicating a character string in which a status change predicted from the status is defined as task contents. The training unit (134) uses training data for a generative model, the training data including the index information and the status information associated with each other, to train an estimation model that estimates, from input information acquired from the operating entity, target index information corresponding to the task contents, among the index information, for achieving the task.

Description

Learning device, control device, learning method, control method, learning program, and control program

[0001] The present invention relates to a learning device, a control device, a learning method, a control method, a learning program, and a control program.

[0002] In recent years, multimodal large-scale language models (MLMs) have become known as machine learning models that can handle multiple types of data (modalities), such as images, audio, and video, in addition to text. Furthermore, technologies that use MLMs to control robot behavior (also called actions) and autonomous driving are known.

[0003] Wenlong Huang, 4 others, "VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models", November 2023, [Retrieved November 19, 2020], Internet <URL: https: / / arxiv.org / pdf / 2307.05973>

[0004] The learning device according to the present invention is a learning device for learning information to cause an operating body to perform a task, and comprises: a first acquisition unit that acquires situation information indicating the situation of a task target which is the target of the task, and index information that points to a string of characters that defines a change in the situation predicted from the situation as the task content; and a learning unit that uses learning data which associates the index information and the situation information to a generation model, and learns an estimation model which estimates from the index information the target index information corresponding to the task content for realizing the task, from input information acquired from the operating body.

[0005] Furthermore, the control device according to the present invention is a control device that performs control to cause an operating body to execute a task, and comprises: a second acquisition unit that acquires sensor information from the operating body; an estimation unit that estimates the target index information corresponding to the sensor information based on the output result obtained by inputting the sensor information to a trained estimation model which has been trained to estimate from the input information the target index information corresponding to the task content for realizing the task from the index information, based on learning data which associates the status information indicating the status of a task target that is the target of the task and index information which points to a string that defines a change in the status predicted from the status as the task content; and a control unit that outputs control information corresponding to the estimated target index information to the operating body, thereby controlling it to execute the task in accordance with the control information.

[0006] Figure 1 is a diagram illustrating tokenization in robot control. Figure 2 is a diagram showing an example of MM-LLM configuration. Figure 3 is a diagram showing one embodiment of information processing according to the proposed technology. Figure 4 is a diagram showing another example of information processing according to the embodiment. Figure 5 is a diagram showing an example of system configuration according to the embodiment. Figure 6 is a diagram showing an example of the configuration of a learning device according to the embodiment. Figure 7 is a conceptual diagram showing the assignment process in the embodiment. Figure 8 is a diagram showing an example of the configuration of a control device according to the embodiment. Figure 9 is a sequence diagram showing the procedure of conversion processing by the learning device. Figure 10 is a sequence diagram showing the procedure of assignment processing by the learning device. Figure 11 is a sequence diagram showing the procedure of estimation processing by the control device. Figure 12 is a diagram showing an example of a conversion table prepared for each type of communication delay. Figure 13 is a diagram showing an example of information processing for robot control based on the type of communication delay. Figure 14 is a hardware configuration diagram showing an example of a computer that realizes the functions of a server device according to the embodiment.

[0007] Embodiments of the present invention will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant explanations will be omitted.

[0008] The one or more embodiments (including examples, modifications, and applications) described below can each be implemented independently. On the other hand, at least some of the embodiments described below may be implemented in appropriate combination with at least some of the other embodiments. These embodiments may contain novel features that differ from each other. Therefore, these embodiments may contribute to solving different objectives or problems and may produce different effects.

[0009] [1. Introduction] As an example of a multimodal large-scale language model, the VLA (Vision-Language-Action) model is known. Furthermore, technologies are known that use multimodal large-scale language models to control robot actions (also called behaviors or movements) and autonomous driving. In such technologies, images of the environment surrounding the robot and text indicating human instructions to the robot are input into the VLA model, and robot actions corresponding to the environment surrounding the robot and the human instructions are generated. Note that a robot acting in response to its surrounding environment and human instructions can be rephrased as a robot performing a task according to human instructions.

[0010] (Tokenization) Here, using Figure 1, we will explain tokenization in robot control using a multimodal large-scale language model (hereinafter abbreviated as "MM-LLM").

[0011] Figure 1 illustrates tokenization in robot control. Figure 1 shows a scenario in which a robot RB (an example of a moving object) is controlled. Figure 1(a) conceptually illustrates tokenization, while Figure 1(b) shows a concrete example of tokenization.

[0012] First, let's explain Figure 1(a). In order to control the robot RB, the MM-LLM must be trained to generate the actions of the robot RB. Normally, as a preprocessing step for learning, LLM performs tokenization, which is the process of dividing text into tokens (the smallest units of elements in text, such as words, punctuation marks, and symbols). However, in order to train the MM-LLM to generate actions for the robot RB, it is necessary to represent the actions of the robot RB as a string that the LLM can understand, i.e., a token (hereinafter referred to as "token Tk"). Specifically, it is necessary to describe the actions of the robot RB as token Tk, which is a string that can be processed by a natural language tokenizer.

[0013] For example, to control the robot RB, commands are used to change the position and rotation of the robot RB, and token Tk can be described based on these commands. Figure 1(a) shows an example in which the components of the position change amount ΔT are represented by "token Tk11" for Δposition X, which is the position change amount in the X axis, "token Tk12" for Δposition Y, which is the position change amount in the Y axis, and "token Tk13" for Δposition Z, which is the position change amount in the Z axis.

[0014] Furthermore, Figure 1(a) shows an example in which the components of the rotational change amount ΔR are represented by "token Tk21" for Δrotation X, which is the rotational change amount in the X-axis, "token Tk22" for Δrotation Y, which is the rotational change amount in the Y-axis, and "token Tk23" for Δrotation Z, which is the rotational change amount in the Z-axis.

[0015] In this state, the tokenizer converts each token Tk into a token ID (token number). As shown in Figure 1(a), the tokenizer generates a sequence of token IDs (token numbers) by converting each token Tk. The token IDs are then converted into vectors that represent the position change amount ΔT or the rotation change amount ΔR, and these vectors are used as training data.

[0016] Next, Figure 1(b) will be explained. In Figure 1(b), one training image IM １We will use this to explain a concrete example of tokenization. (Training image IM) １ This is an example of situational information that indicates the status of the task target, which is the target of the task performed by the robot RB. In the example in Figure 1(b), the training image IM １ (Image IM) ｎ An example of this may include annotation information AN. Annotation information AN may include text TX that indicates the expected change in the situation based on the situation of the task target. １ (Text TX ｎ (An example of this may be included.) Text TX １ This indicates something like "a person in black clothing moving from position coordinates P1 (X1, Y1, Z1) to position coordinates P2 (X2, Y2, Z2)," and explains the predicted change in situation (will move to position coordinates P2) based on the situation of the task subject (the person in black clothing) (is at position coordinates P1).

[0017] Here, position coordinates P1 (X1, Y1, Z1) and position coordinates P2 (X2, Y2, Z2) can be the basis for commands to change the position and rotation of the robot RB. Therefore, as shown in Figure 1(b), based on position coordinates P1 and P2, text TX １ This is converted into tokens Tk11, Tk12, Tk13, Tk21, Tk22, and Tk23.

[0018] Token Tk11 may be a string representing Δposition X based on position coordinates P1 and P2. Token Tk12 may be a string representing Δposition Y based on position coordinates P1 and P2. Token Tk13 may be a string representing Δposition Z based on position coordinates P1 and P2.

[0019] Token Tk21 may be a string representing a Δ rotation X based on position coordinates P1 and P2. Token Tk22 may be a string representing a Δ rotation Y based on position coordinates P1 and P2. Token Tk23 may be a string representing a Δ rotation Z based on position coordinates P1 and P2.

[0020] In such a state, the tokenizer converts each token Tk into a token ID (token number). FIG. 1(b) shows an example in which the tokenizer converts token Tk11 to token ID "100", token Tk12 to token ID "101", and token Tk13 to token ID "110". Further, FIG. 1(b) shows an example in which the tokenizer converts token Tk21 to token ID "127", token Tk22 to token ID "217", and token Tk23 to token ID "241".

[0021] Further, the sequence of token IDs such as "100 101 110 127 217 241" is composed of 6 tokens (6 tokens in total).

[0022] (Training of MM-LLM) For example, in MM-LLM, text TX ｎ and teacher image IM ｎ the sequences of token IDs obtained by converting each of them are trained such that paired sequences are brought closer to each other within the same space (semantic space or vector space).

[0023] For example, the teacher image IM ｎ text TX included as annotation information AN in ｎ is converted into a token ID via a text encoder, and the token ID is converted into a vector T ｎ . As described with reference to FIG. 1(b), a teacher image IM １ text TX included as annotation information AN in １ is converted into a token ID via a text encoder, and the token ID obtained by the conversion is further converted into a vector T １ .

[0024] Further, the teacher image IM ｎ is converted into a vector I via an image encoder ｎ . For example, the teacher image IM １ is converted into a vector I via an image encoder １ . For example, the teacher image IM １The image contains a person in black clothing, and the portion of the image extracted using bounding boxes is annotated with "person in black clothing," thus indicating that this "partial image" and "person in black clothing" are a pair. Therefore, in MM-LLM, for example, according to an optimization algorithm, a vector I corresponding to the "partial image" is generated. １ And, vector T corresponding to "person in black suit" １ Learning is performed based on the loss function to bring the two values ​​as close as possible.

[0025] Also, the training image IM １ In this case, the partial image from which the part of the person in black clothing is extracted is annotated with the predicted change in situation, "moving from position coordinate P1 to position coordinate P2," so it is found that this "partial image" and "moving from position coordinate P1 to position coordinate P2" are a pair. Therefore, in MM-LLM, following the same optimization algorithm, vector I corresponding to the "partial image" is generated. １ And the vector T corresponding to "movement from position coordinate P1 to position coordinate P2" １ Learning is performed based on the loss function to bring the two values ​​as close as possible.

[0026] (Problems and Solutions) Next, the problems related to the proposed technology of this application will be explained using the example in Figure 2. Figure 2 is a diagram showing an example configuration of MM-LLM. According to the example in Figure 2, MM-LLM (for example, Llama3) may consist of an image encoder, projection, and LLM. The image encoder extracts image features, and for example, CLIP can be used. However, since the image features themselves cannot be input to the LLM, they are adjusted for the LLM by projection and then input to the LLM.

[0027] In the learning process, for example, in the first stage, only projection may be learned with the image encoder and LLM fixed, and in the second stage, both projection and LLM may be learned with the image encoder fixed.

[0028] In the example in Figure 2, the training image IM ｎ and text TX ｎThe projection is learned to adjust to the characteristics of the model. Also, in MM-LLM training, question-and-answer pairs are required as training data. For example, a question such as "What <task> should the robot perform to track a person?" is generated, and the text TX ｎ The LLM may be trained to predict the <task> based on training data in which the behavioral descriptions provided by the LLM are the answers.

[0029] Figure 2 shows a scenario where, after the user inputs instruction information (prompt) CM (text) such as "Please tell me what actions the robot should take based on the image in order to track the person in black," the MM-LLM outputs control information (for example, a speed command value) to accomplish the task of tracking the person in black.

[0030] In this example, the inference process involves the detection image IM detected by the robot RB as sensor information. ｘ The instruction information CM may be input to MM-LLM. Figure 2 shows that MM-LLM is input to the detected image IM ｘ An example is shown in which inference is performed based on the instruction information CM, and the sequence of token IDs "100 101 110" is output. The token IDs are converted into control information CF by detokenization, and the control information CF is passed to the robot RB. The robot RB then takes action according to the control information CF, and as a result, the task of "tracking a person in black clothing" is achieved.

[0031] As explained above, in robot control using MM-LLM, discretized actions such as Δposition X, Δposition Y, and Δposition Z are used, and by converting each action into a string (token Tk), it becomes possible to learn robot data. However, when using such discretized information, a sequence of token IDs is obtained, so in inference, autoregression is performed as many times as there are token IDs (i.e., the number of tokens), and there is a problem that the computational complexity (computation time) increases in proportion to the number of token IDs.

[0032] In the example in Figure 2, MM-LLM predicts token ID "100" in the first inference, token ID "101" in the second inference, and token ID "110" in the third inference. In other words, MM-LLM performs three rounds of autoregression in serial computation. In the case of six tokens shown in Figure 1(b), six rounds of autoregression are performed, so the computational cost in serial computation becomes even greater.

[0033] Such increased computational complexity can be disadvantageous in robot control. As will be explained in detail in Figure 4, in the proposed technology, the MM-LLM is mounted on an external device (server device 100) that is communicatively connected to the robot RB. The external device then inputs sensor information acquired from the robot RB via wireless communication to the MM-LLM on its own side, and responds to the robot RB via wireless communication with control information CF based on the estimation results output by the MM-LLM. In such a system, there is a problem that whether or not the robot RB can perform the task accurately is affected by the communication delay. Taking the task of "tracking a person in black" as an example, the greater the communication delay between the external device and the robot RB, the more likely the robot RB is to lose track of the person in black.

[0034] In other words, in robot control using MM-LLM, if the sequence of token IDs corresponding to the discretized actions is used directly as training data, the inference time will increase in proportion to the number of token IDs (token count). Since the inference time becomes the communication delay time, there is a problem that the longer the inference time, the lower the accuracy of the task.

[0035] The proposed technology of this application has been developed in view of the above-mentioned problems, and aims to generate an MM-LLM trained on training data by providing training data that eliminates the need for the MM-LLM to perform autoregressive calculations multiple times. Specifically, the proposed technology of this application is characterized by replacing the sequence of token IDs with index information idx during training, thereby controlling the output of the sequence of token IDs to be one token (performing autoregressive calculations only once).

[0036] By using MM-LLM trained based on index information idx, communication delay can be reduced, which is expected to solve the problem of robot RB performing tasks with high accuracy.

[0037] [2. Specific Examples of the Proposed Technology] Figure 3 is a diagram showing one embodiment of information processing related to the proposed technology. Figure 3 shows an example of information processing according to the embodiment. Figure 3 also shows a scene in which the information processing according to the embodiment is executed by a server device 100, and the server device 100 may consist of a learning device 100A and a control device 100B. Although Figure 3 shows an example in which the learning device 100A and the control device 100B are integrated as a server device 100, the learning device 100A and the control device 100B may be implemented as separate devices.

[0038] Furthermore, the upper part of Figure 3 shows an example of information processing in the learning phase, and the lower part of Figure 3 shows an example of information processing in the inference phase. Information processing in the learning phase may be performed by the learning device 100A, and information processing in the inference phase may be performed by the control device 100B.

[0039] First, let's explain the learning phase. Figure 3 shows the learning phase in which training data LDA is generated from the original data ODA, and this training data LDA is input to the pre-trained MM-LLM1. Each of the original data ODA is a text TX ｎ and teacher image IM ｎ It may consist of the following, and as an example of such a configuration, Figure 3 shows the training image IM １ One source data ODA containing is shown in the foreground. Also, as shown in the example in Figure 3, the training image IM １ Text TX included as annotation information AN １ The following shows an example where the token ID has been converted to a token ID (TkID), and the sequence of token IDs "100 101 110" (3 tokens) is obtained through the conversion. Below is the training image IM １ The information processing according to the embodiment will be explained using this example, but other source data ODA will be processed in the same manner.

[0040] Returning to the explanation, the learning device 100A uses the teacher image IM １ By comparing the corresponding token ID sequence "100 101 110" with the conversion table TB (step S11), the token ID "100 101 110" is converted into index information idx, which is represented by one token (step S12). The conversion table TB may be prepared in advance, and characters such as 0-9 and A-z may be used as candidates for the index information idx.

[0041] Figure 3 shows the learning device 100A, which displays a teacher image IM. １ An example is shown in which the sequence of token IDs corresponding to the tokens is converted into a single character "0" as index information idx. Also, although not shown, the learning device 100A uses a teacher image IM ２ In some cases, the sequence of token IDs corresponding to the tokens is converted into a single character "1" as index information idx. Another example is the learning device 100A, which uses a training image IM. ３ In some cases, the sequence of token IDs corresponding to a single character, "2", is converted into an index information idx.

[0042] In this state, the learning device 100A uses index information idx and teacher image IM ｎ By associating the two, training data LDA is generated, and the training data LDA is input to MM-LLM1 (step S13). For example, the learning device 100A associates the index information idx "0" with the training image IM １ By associating these, a single dataset is generated in the training data LDA.

[0043] Then, the learning device 100A learns an estimation model based on the learning data LDA and MM-LLM1 to estimate the target index information idx corresponding to the task content for realizing the task from the index information idx, from predetermined input information (step S14). Specifically, the learning device 100A learns the detection image IM detected by the robot RB as sensor information. ｘWhen an example of input information obtained from the operating body is input, MM-LLM1 is trained to output target index information idx, which is the index information idx corresponding to the task content to be instructed to the robot RB. As a result, the learning device 100A can generate MM-LLM2 (an example of an estimated model) as the trained MM-LLM1. The learning device 100A may also deploy MM-LLM2 to the control device 100B.

[0044] Next, the inference phase will be explained. In the inference phase shown in Figure 3, the target index information idx is estimated using MM-LLM2, and robot control is performed based on this target index information idx. The control device 100B receives the detection image IM detected by the robot RB as sensor information. ｘ These can be input sequentially, and one detection image IM ｘ An example is given in which the target index information idx is estimated based on the detection image IM acquired sequentially by the robot RB. ｘ You may estimate the target index information idx for each case.

[0045] For example, the control device 100B receives the detected image IM from the robot RB. ｘ Upon receiving the received detection image IM ｘ By inputting this into MM-LLM2 (step S21), inference is performed to obtain target index information idx corresponding to the task content (step S22).

[0046] MM-LLM2 outputs index information idx as an estimation result from inference (step S23). Figure 3 shows an example where MM-LLM2 outputs a single character "0" represented by one token as the target index information idx.

[0047] Therefore, based on the estimation result, the control device 100B processes the detected image IM input this time. ｘ The control device 100B may estimate the target index information idx corresponding to the detected image IM input this time. ｘFor the task targets included in the list, it is acceptable to estimate "0" as the target index information idx, which serves as the basis for the command instructing how the robot RB should operate to accomplish the task.

[0048] Then, the control device 100B compares the target index information idx "0" with the conversion table TB (step S24), and converts the target index information idx into a sequence of token IDs (step S25). Figure 3 shows an example in which the sequence of token IDs "100 101 110" (3 tokens) is obtained by the conversion.

[0049] The control device 100B may detokenize each token ID included in the sequence, converting each token ID into control information CF, which is actual coordinate information, and pass the control information CF to the robot RB. The robot RB takes action according to the control information CF, and as a result, a task such as "tracking a person in black clothing" is realized.

[0050] Figure 3 shows an example in which the server device 100 performs information processing according to the embodiment based on the sequence of token IDs. Specifically, it shows an example in which the learning device 100A converts the sequence of token IDs into index information idx and learns this index information idx as learning data, and an example in which the control device 100B converts the index information idx obtained as the output result of the AI ​​model back into the sequence of token IDs.

[0051] Thus, Figure 3 shows an example in which a sequence of token IDs obtained by converting the coordinate information CD is used. However, the server device 100 may use the coordinate information CD itself without converting it to a sequence of token IDs. Note that the coordinate information CD referred to here is the text TX included in the annotation information AN. ｎ This may be the coordinate portion that can be used as a command in robot control, and as an example, it may be the rotation angle information (x, y, yaw) that indicates the posture (rotation) of the robot RB. This point will be explained using Figure 4.

[0052] Figure 4 shows another example of information processing according to the embodiment. Figure 4 shows an example of information processing when coordinate information CD is used. It is the same as the example in Figure 3 except that the coordinate information CD is converted to index information idx. For this reason, Figure 4 will be explained focusing on the conversion between coordinate information CD and index information idx.

[0053] In the example in Figure 4, the training image IM １ One source data ODA containing the training image IM is shown in the foreground. １ Text TX included as annotation information AN １ Next, an example is shown in which the coordinate information CD is given as "1,0,0". In this example, in step S11, the learning device 100A compares the coordinate information CD "1,0,0" with the conversion table TB, and in step S12, converts the coordinate information CD "1,0,0" into index information idx.

[0054] Here, according to the tokenizer, the coordinate value "1,0,0" may be represented by at least 3 tokens, but the learning device 100A uses the training image IM １ The coordinate information CD "1,0,0" corresponding to this is converted into index information idx, which is represented by one token. Figure 4 shows the learning device 100A and the teacher image IM １ An example is shown in which the coordinate information CD corresponding to the coordinates is converted to the single character "1" as index information idx. Note that the conversion table TB may be prepared in advance, but in order to realize this conversion, the sequence information is replaced with rotation angle information (x, y, yaw) compared to the conversion table TB shown in Figure 3.

[0055] In step S13, the learning device 100A receives index information idx and teacher image IM ｎ By associating the two, training data LDA is generated, and the training data LDA is input to MM-LLM1. For example, the learning device 100A associates the index information idx "1" with the training image IM １ By associating these, a single dataset is generated in the training data LDA.

[0056] Furthermore, Figure 4 shows an example in which MM-LLM2 outputs the character "1" represented by one token as the target index information idx. In this example, the control device 100B processes the detected image IM input this time. ｘ For the task targets included in the list, it is acceptable to estimate "1" as the target index information idx, which serves as the basis for the command instructing how the robot RB should operate to accomplish the task.

[0057] Then, in step S24, the control device 100B compares the target index information idx "1" with the conversion table TB, and in step S25, converts the target index information idx into coordinate information CD. Figure 4 shows an example in which coordinate information CD "1,0,0" is obtained through the conversion.

[0058] The control device 100B may pass the coordinate information CD obtained by the transformation to the robot RB as control information CF. The robot RB takes action according to the control information CF, and as a result, a task such as "tracking a person in black clothing" is realized.

[0059] Up to this point, the information processing according to the embodiment has been described. According to the information processing according to the embodiment, the estimation result by the AI ​​model can be controlled to output in one token, so that MM-LLM does not have to perform autoregressive calculations multiple times. For this reason, according to the information processing according to the embodiment, communication delay can be reduced, so that the robot RB can perform tasks with high accuracy.

[0060] Furthermore, this proposed method can improve communication quality and increase operational efficiency through the use of AI, and will serve as an innovative technological foundation in the telecommunications business, thereby contributing to the achievement of Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0061] [3. System Configuration] From here, we will describe the system including the server device 100 that performs information processing according to the above embodiment. First, we will explain vRAN and AI-RAN.

[0062] (About vRAN) In the communication network to which the mobile communication terminals (User Equipment, UE) that we use every day are connected, i.e., the Radio Access Network (RAN), a vast number of radio base stations (gNBs) are operated to provide communication services over a wide area. The main roles of gNBs include transmitting and receiving radio waves for wireless communication with UEs and processing radio signals. Because gNBs require high throughput and very strict latency requirements, dedicated hardware optimized for gNBs has traditionally been used.

[0063] On the other hand, server virtualization and cloud computing have established mechanisms that allow users to access the necessary computing resources when and where they are needed. Specifically, by abstracting computing resources and separating hardware and software, it has become possible to reduce physical resources and rapidly deploy systems. In communication infrastructure, this implementation method is called NFV (Network Functions Virtualization).

[0064] According to NFV, by implementing network functions that were previously handled by dedicated hardware as software functions, it became possible to run virtualized workloads on inexpensive general-purpose servers. Therefore, vRAN (virtual Radio Access Network) is a virtualization system that implements the gNB (graphic network bounding box) that constitutes RAN, with the gNB's functions configured as software on a general-purpose server. Furthermore, vRAN is a concrete example of a system requiring high throughput and low latency.

[0065] In vRAN, the gNB, which was previously composed of dedicated hardware, is basically divided into three components: CU (Central Unit), DU (Distributed Unit), and RU (Radio Unit). In this example, the RU corresponds to a slave station, and the DU and CU correspond to a master station.

[0066] Therefore, the functional units of the DU and CU, which are responsible for wireless signal processing, are implemented in software on the general-purpose server. The vRAN transmits and receives wireless signals with the UE, and in the information processing according to this embodiment, the robot RB corresponds to the UE.

[0067] Here, we will also explain specific examples of the processing performed by RU, DU, and CU, respectively. For example, in the 5th generation (5G) cellular standard of 3GPP (3rd Generation Partnership Project) (registered trademark), gNB can be divided into L1 layer (physical layer), L2 layer (data link layer), and L3 layer (network layer) as the 5G protocol layer.

[0068] The RU processes RF (radio frequency) and the lower part of the L1 layer (Low-PHY).

[0069] The DU and CU perform radio signal processing. Specifically, the DU handles tasks in the upper part of the L1 layer (High-PHY). These tasks include, for example, channel estimation, equalization, modulation and demodulation, and coding (error correction).

[0070] Furthermore, the DU handles L2 layer tasks. These tasks include, for example, media access control (MAC) tasks (e.g., wireless resource scheduling) and wireless link control (RLC) tasks (e.g., packet segmentation and reconstruction).

[0071] The CU manages PDCP (Packet Data Convergence Protocol), SDAP (Service Data Adaptation Protocol), and RRC (Radio Resource Control) protocol entities.

[0072] (About AI-RAN) AI-RAN is a technology that enables vRAN applications and AI applications to run on the same virtualization platform. On a general-purpose server, it is possible to use not only software that implements RAN control functions, but also other software. For example, the amount of communication (traffic) of wireless communication by UE fluctuates greatly depending on the time of day, so there is surplus resources (computing power) on a general-purpose server during times when there are few users, such as from late night to early morning. The AI-RAN concept utilizes these surplus resources for AI applications.

[0073] One implementation method for AI-RAN involves using a GPU (graphics processing unit) as an accelerator (a function that accelerates application processing) for a general-purpose server. For example, the computing resources of a general-purpose server (a server equipped with a GPU) on which RAN software has been installed can be utilized for AI processing (for example, the computational processing that AI uses to derive conclusions from new data: AI inference).

[0074] (Regarding the system configuration) Based on the explanation of vRAN and AI-RAN, the system configuration including the server device 100 will be described. Figure 5 is a diagram showing an example of the system configuration according to the embodiment. Figure 5 shows an example in which the vRAN application and the AI ​​application are implemented on the same virtualization platform.

[0075] Furthermore, Figure 5 shows a server device 100 as an example of a general-purpose server. According to the server device 100, it is possible to provide a large-capacity, high-performance, and high-quality vRAN at carrier grade on a virtualization platform equipped with GPU (Graphics Processing Unit) computing, and to provide various AI applications such as generative AI on top of it.

[0076] More specifically, the server device 100 has a virtualization infrastructure equipped with computing resources (CR) such as a GPU, and a RAN control function F1 corresponding to a vRAN application and an AI function F2 corresponding to an AI application are implemented on the virtualization infrastructure.

[0077] In the example shown in Figure 5, the RAN control function F1 executes L1 processing, L2 processing, and L3 processing. Furthermore, L1 processing may be implemented by L1 software, and L2 and L3 processing may be implemented by L2 / L3 software.

[0078] Furthermore, AI processing by AI function F2 includes image diagnosis, voice analysis, specialized LLM, confidential LLM, LLM robot, and LLM autonomous driving. For this reason, the information processing according to the embodiment may be executed by AI function F2, which corresponds to an AI application.

[0079] [4. Configuration of the Learning Device] The learning device 100A according to the embodiment will be described using Figure 6. Figure 6 is a diagram showing an example of the configuration of the learning device 100A according to the embodiment. As shown in Figure 6, the learning device 100A has a communication unit 110A, a storage unit 120A, and a control unit 130A.

[0080] (Communication Unit 110A) The communication unit 110A is implemented by, for example, a NIC (Network Interface Card). For example, the communication unit 110A transmits and receives information with the robot RB and the control device 100B.

[0081] (Storage Unit 120A) The storage unit 120A is implemented by, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120A may store, for example, data and programs related to information processing according to the embodiment. Also, as shown in Figure 6, the storage unit 120A may have an AI model storage unit 121, a conversion table storage unit 122, and a training data storage unit 123.

[0082] (Control Unit 130A) The control unit 130A is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., which executes various programs (for example, information processing programs according to the embodiment) stored in the memory device inside the learning device 100A using RAM as the working area. The control unit 130A is also implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0083] As shown in Figure 6, the control unit 130A includes a collection unit 131, a conversion unit 132, a first acquisition unit 133, a learning unit 134, and a transmission unit 135, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130A is not limited to the configuration shown in Figure 6, and other configurations are also possible as long as they perform the information processing described later. Also, the connection relationships of the various processing units in the control unit 130A are not limited to the connection relationships shown in Figure 6, and other connection relationships are also possible.

[0084] (Collection Unit 131) The collection unit 131 collects the original data ODA. For example, the collection unit 131 collects annotated training images IM as the original data ODA. ｎ The collection unit may collect the original ODA data from any external device, or it may collect the original ODA data provided by the user.

[0085] (Conversion Unit 132) The conversion unit 132 uses the conversion table TB to perform information conversion to index information idx. For example, the conversion unit 132 may convert the sequence of token IDs into index information idx, which is represented by one token, by comparing the sequence of token IDs with the conversion table TB. Alternatively, the conversion unit 132 may convert the coordinate information CD, which is the basis of the token Tk before it is converted into a token ID, into index information idx, which is represented by one token, by comparing the coordinate information CD with the conversion table TB.

[0086] (First acquisition unit 133) The first acquisition unit 133 acquires the index information idx obtained by the conversion unit 132. As explained in Figure 1, etc., the index information idx refers to a string that defines the task content as situation information indicating the status of the task target which is the target of the task, and the situation changes predicted from this status.

[0087] (Learning Unit 134) The learning unit 134 uses learning data that associates index information idx with situation information to a generation model (for example, a large-scale language model) to learn an estimation model that estimates the target index information idx corresponding to the task content for realizing the task from the index information idx, based on input information obtained from the operating entity. For example, the learning unit 134 associates index information idx with a training image IM ｎ The learning data LDA, which associates the data with the target index information idx, is used for MM-LLM1, and the detection image IM obtained from the robot RB is used. ｘ Generate MM-LLM2 by estimation.

[0088] For example, the learning unit 134 may learn to output the target index information idx as one token. One method for learning to output the target index information idx as one token is to use a predetermined index information idx in which the string is pre-represented as one token. The predetermined index information idx in which the string is pre-represented as one token may be, for example, any single character from 0 to 9 or A to z. Therefore, the first acquisition unit 133 may acquire a predetermined index information idx in which the string is pre-represented as one token as the index information idx. The learning unit 134 may also use a single character index information idx such as 0 to 9 or A to z and a training image IM ｎ MM-LLM2 may be generated using the training data LDA which associates the two.

[0089] However, matrices such as 0-9 and A-z have limitations in terms of the number of elements, and may not be able to cover all commands for robot control. Therefore, the learning unit 134 may learn to assign index information idx in a one-to-one relationship to vectors (lexical vectors) V mapped to the semantic space, so that the target index information idx corresponding to the input information is output in one token. For example, the learning unit 134 uses the token ID used to calculate the vector V, which identifies the token Tk corresponding to the vector V, to determine the index information idx and the training image IM. ｎ It is permissible to perform the process of assigning the linked elements to vector V. This assignment process will be explained using Figure 7.

[0090] Figure 7 is a conceptual diagram showing the assignment process in an embodiment. The large-scale language model constructs a fixed-dimensional semantic space SP that represents the semantic similarity between images and text. Llama3 (registered trademark), an example of MM-LLM, constructs a 4096-dimensional semantic space SP through pre-training and has a vocabulary size of 128,256 tokens. In other words, Llama3 has 128,256 lexical vectors V in its 4096-dimensional semantic space SP.

[0091] Therefore, in this embodiment, when Llama3 is used as MM-LLM1, the number of index information idx can be expanded to 128,256. Specifically, simply using a matrix such as 0-9, A-z, etc., only allows the use of a small number of index information idx, but by using Llama3 as MM-LLM1, that number can be expanded to 128,256.

[0092] Furthermore, if the vocabulary size is 128,256 tokens, then in the case of a three-dimensional input (for example, a sequence of token IDs consisting of 3 tokens, or coordinate information CD consisting of 3 tokens), it is possible to have a resolution of 50 tokens for each of the three dimensions.

[0093] In the assignment process according to this embodiment, for example, the learning unit 134 may assign index information idx in a one-to-one relationship to a vector V that is mapped to the semantic space SP constructed by MM-LLM1.

[0094] According to the example in Figure 7, for example, the learning unit 134 selects the training image IM from among the vectors V in the semantic space SP. １ and text TX １ A predetermined index information idx may be assigned to a vector V1 that corresponds to its semantic similarity to the given value. Figure 7 shows an example in which the learning unit 134 assigns the index information idx "0" to a vector V1.

[0095] Furthermore, the learning unit 134 selects the training image IM from among the vectors V in the semantic space SP. ２ and text TX ２ A predetermined index information idx may be assigned to a vector V2 that corresponds to its semantic similarity to the given value. Figure 7 shows an example in which the learning unit 134 assigns the index information idx "1" to a vector V2.

[0096] Furthermore, the learning unit 134 selects the training image IM from among the vectors V in the semantic space SP. ３ and text TX ３ A predetermined index information idx may be assigned to a vector V3 that corresponds to its semantic similarity to the given value. Figure 7 shows an example in which the learning unit 134 assigns the index information idx "0" to a vector V3.

[0097] Furthermore, the learning unit 134 does not simply assign index information idx to the vector V mapped to the semantic space SP, but rather uses text TX ｎ and training image IM ｎ Based on the token ID obtained by converting each, the index information idx and the training image IM ｎ And, text TX ｎ You can also assign them while they are linked together.

[0098] For example, the learning unit 134 transmits text via a text encoder. ｎ Vector T transformed from ｎand a teacher image IM via the image encoder ｎ vector I converted from ｎ may be mapped into the semantic space SP based on the relationship with ｎ vector T and vector I ｎ . Then, the learning unit 134 may assign the index information idx, the teacher image IM ｎ and the text TX ｎ in an associated state to the vector V obtained by the mapping. As an example, the vector T converted from the text TX １ １ and the teacher image IM １ vector I converted from １ may be mapped into the semantic space SP based on the relationship with １ vector T and vector I １ . Then, the learning unit 134 may assign the index information idx "0", the teacher image IM １ and the text TX １ in an associated state to the vector V1 obtained by the mapping.

[0099] Note that FIG. 7 shows an example where the learning unit 134 assigns the index information idx for the combination of the teacher image IM ｎ and the text TX ｎ among combinations of teacher images IM １ and texts TX １ including the combination of teacher image IM ２ and text TX ２ , the combination of teacher image IM ３ and text TX ３ , but the index information idx is similarly assigned to other combinations. Accordingly, for example, when MM-LLM1 is Llama3, the learning unit 134 can assign a different index information idx to each of up to 128,256 vectors V. In other words, the learning unit 134 can use 128,256 types of index information idx as the learning data LDA.

[0100] ​(Transmitting Unit 135) Returning to the explanation of Figure 6, the transmitting unit 135 transmits information to the control device 100B. For example, the transmitting unit 135 transmits the MM-LLM2 generated by the learning unit 134 to the control device 100B.

[0101] [5. Configuration of the Control Device] The control device 100B according to the embodiment will be described with reference to Figure 8. Figure 8 is a diagram showing an example of the configuration of the control device 100B according to the embodiment. As shown in Figure 8, the control device 100B has a communication unit 110B, a storage unit 120B, and a control unit 130B.

[0102] (Communication Unit 110B) The communication unit 110B is implemented by, for example, a NIC. For example, the communication unit 110B transmits and receives information with the robot RB and the learning device 100A.

[0103] (Storage Unit 120B) The storage unit 120B is implemented by, for example, a semiconductor memory element such as RAM or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120B may store, for example, data and programs related to information processing according to the embodiment. Also, as shown in Figure 8, the storage unit 120A may have an AI model storage unit 124, a conversion table storage unit 125, and a sensor information storage unit 126.

[0104] (Control Unit 130B) The control unit 130B is implemented by a CPU, MPU, etc., which executes various programs (for example, information processing programs according to the embodiment) stored in the storage device inside the control device 100B using RAM as a working area. The control unit 130B is also implemented by an integrated circuit such as an ASIC or FPGA.

[0105] As shown in Figure 8, the control unit 130B includes a second acquisition unit 136, an estimation unit 137, a conversion unit 138, and an operation control unit 139, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130B is not limited to the configuration shown in Figure 8, and other configurations are also possible as long as they perform the information processing described later. Also, the connection relationships of the various processing units in the control unit 130B are not limited to the connection relationships shown in Figure 8, and other connection relationships are also possible.

[0106] (Second acquisition unit 136) The second acquisition unit 136 acquires sensor information from the moving body. For example, the second acquisition unit 136 acquires the detection image IM detected by the robot RB as sensor information. ｘ Obtain it.

[0107] (Estimation Unit 137) The estimation unit 137 estimates the target index information idx corresponding to the sensor information based on the output result obtained by inputting the sensor information into the trained estimation model. For example, the estimation unit 137 inputs the detected image IM to MM-LLM2. ｘ Based on the output result from MM-LLM2 when the input is given, the detected image IM ｘ The system estimates the target index information idx corresponding to the input. For example, the estimation unit 137 estimates the detection image IM input this time based on the output result. ｘ You may estimate the target index information idx, which serves as the basis for the commands that instruct the robot RB to operate in order to accomplish the task for the task targets included in the list.

[0108] (Conversion Unit 138) The conversion unit 138 uses the conversion table TB to perform information conversion to a sequence of token IDs. For example, the conversion unit 138 may convert the target index information idx into a sequence of token IDs by comparing the target index information idx with the conversion table TB. Alternatively, the conversion unit 138 may convert the target index information idx into coordinate information CD by comparing the target index information idx with the conversion table TB.

[0109] (Motion control unit 139) The motion control unit 139 controls the movement of the moving body based on the conversion result from the conversion unit 138. For example, the motion control unit 139 controls the actions of the robot RB so that the robot RB can perform a task based on the conversion result from the conversion unit 138.

[0110] For example, if the target index information idx is converted into a sequence of token IDs, the operation control unit 139 converts each token ID into control information CF, which is the actual coordinate value, by detokenizing each token ID included in the sequence.

[0111] On the other hand, if the target index information idx is converted to coordinate information CD, the operation control unit 139 may use the coordinate information CD as control information CF.

[0112] The motion control unit 139 then controls the robot RB using the control information CF. For example, the motion control unit 139 outputs the control information CF to the robot RB, thereby controlling the robot RB to take action according to the commands indicated by the control information CF.

[0113] [6. Processing Procedure] Next, the operation procedure of the server device 100 will be explained using Figures 9 to 11. In Figures 9 to 11, the operation procedure of the learning device 100A and the operation procedure of the control device 100B will be explained separately.

[0114] (6-1. Conversion Process) Figure 9 is a sequence diagram showing the procedure of the conversion process by the learning device 100A. The collection unit 131 takes annotated training image IM as the source data ODA. ｎ Collect (step S901).

[0115] The conversion unit 132 converts the teacher image IM ｎ Based on the annotation information AN assigned to it, the action of the robot RB is converted into a token Tk, which is a string that LLM can understand (step S902).

[0116] Furthermore, the conversion unit 132 converts token Tk into token IDs (step S903). As a result, the conversion unit 132 can obtain a sequence of token IDs from the token Tk corresponding to the discretized action.

[0117] Furthermore, the conversion unit 132 converts the sequence of token IDs into index information idx, which is represented by one token, by matching the sequence of token IDs with the conversion table TB (step S904). The first acquisition unit 133 may output the index information idx to the learning unit 134.

[0118] The conversion unit 132 converts the teacher image IM ｎ vector I ｎ Convert to (step S905). Also, the conversion unit 132 converts the teacher image IM ｎText TX included as annotation information AN ｎ vector T ｎ Convert to (step S906).

[0119] (6-2. Assignment Processing) Figure 10 is a sequence diagram showing the procedure for assignment processing by the learning device 100A. The learning unit 134 is vector I ｎ and vector T ｎ Based on the relationship, vector I is added to the semantic space SP constructed by pre-training MM-LLM1. ｎ and vector T ｎ Mapping (step S1001).

[0120] The learning unit 134, in mapping, vector I ｎ and vector T ｎ A vector V can be generated that reduces the loss by learning to bring the pairs closer together. As a result, the learning unit 134 can generate MM-LLM2, which is a further learned version of MM-LLM1.

[0121] Next, the learning unit 134 applies index information idx and training image IM to the vector V obtained by mapping. ｎ And, text TX ｎ Assign them while linking them together (step S1002).

[0122] Furthermore, the transmitting unit 135 transmits MM-LLM2 to the control device 100B (step S1003).

[0123] (6-3. Estimation Processing) Figure 11 is a sequence diagram showing the procedure of estimation processing by the control device 100B. The second acquisition unit 136 acquires the detected image IM detected by the robot RB as sensor information. ｘ It is determined whether or not the detected image IM was acquired (step S1101). The second acquisition unit 136 determines whether or not the detected image IM ｘ While the detection image IM has not been obtained (step S1101; No), ｘ Wait until it can be obtained.

[0124] Meanwhile, the estimation unit 137 detects the image IM ｘIf obtained (step S1101; Yes), the detected image IM is sent to MM-LLM2. ｘ Enter the value (step S1102).

[0125] Then, the estimation unit 137 detects the image IM in MM-LLM2. ｘ Based on the output result from MM-LLM2 when the input is given, the detected image IM ｘ The target index information idx corresponding to this is estimated (step S1103).

[0126] The conversion unit 138 converts the target index information idx into a sequence of token IDs by comparing it with the conversion table TB (step S1104).

[0127] The operation control unit 139 converts each token ID into control information CF, which is the actual coordinate value, by detokenizing each token ID included in the sequence of token IDs (step S1105).

[0128] Then, the motion control unit 139 controls the robot RB using the control information CF (step S1106). For example, the motion control unit 139 outputs the control information CF to the robot RB, thereby controlling the robot RB to take action according to the command indicated by the control information CF.

[0129] [7. Modifications] The information processing according to the embodiment may be carried out in a manner different from the example described above. Therefore, modifications of the information processing according to the embodiment will be described below.

[0130] (7-1. Communication delay countermeasures) The first acquisition unit 133 may acquire index information idx for each predetermined piece of information corresponding to the communication delay that may occur between the robot RB and the server device 100 (learning device 100A). The learning unit 134 may then learn an estimation model to output target index information idx for each piece of predetermined information.

[0131] For example, the first acquisition unit 133 may acquire index information idx for each type of communication delay, which is determined according to the communication delay time that may occur between the robot RB and the server device 100 (learning device 100A). In this case, the learning unit 134 may learn MM-LLM1 to output target index information idx corresponding to each type of communication delay for each type of communication delay. The information processing for communication delay countermeasures will be explained using Figures 12 and 13.

[0132] In Figures 12 and 13, "Small communication delay," "Medium communication delay," and "Large communication delay" are shown as examples of communication delay types, but the number and content of communication delay types are not limited to these examples.

[0133] First, Figure 12 shows an example of a conversion table TB1 prepared for each type of communication delay. Compared to a conversion table TB that does not consider the type of communication delay (Figures 3 and 4), the conversion table TB1 may store index information idx that points to the sequence of each token ID for each type of communication delay. Although not shown in Figure 12, the conversion table TB1 may also store index information idx that points to each coordinate information CD for each type of communication delay.

[0134] Using the example in Figure 3, the conversion unit 132 compares the sequence of token IDs for each type of communication delay with the conversion table TB1 (step S11), and converts each sequence into a single-character index information idx corresponding to each type of communication delay (step S12). According to the example in Figure 3, the conversion unit 132 can convert the token ID sequence "100 101 110" corresponding to "Small communication delay" into a single-character index information idx of "0". The conversion unit 132 can also convert the token ID sequence "200 101 110" corresponding to "Medium communication delay" into a single-character index information idx of "0". Furthermore, the conversion unit 132 can convert the token ID sequence "300 101 110" corresponding to "Large communication delay" into a single-character index information idx of "0".

[0135] In this state, the learning unit 134 receives the index information idx and the teacher image IM ｎ By associating these, learning data LDA is generated, and the learning data LDA is input to MM-LLM1 (step S13). For example, the learning unit 134 associates index information idx "0" corresponding to the combination of communication delay types with the teacher image IM １ By associating these, a single dataset is generated in the training data LDA.

[0136] The learning unit 134 may then train MM-LLM1 to output target index information idx for each type of communication delay (step S14). As a result, the learning unit 134 can generate MM-LLM2 (an example of an estimation model) as the trained MM-LLM1, and deploy MM-LLM2 to the control device 100B.

[0137] Next, Figure 13 shows an example of information processing for robot control based on the type of communication delay. In Figure 13, MM-LLM1 is trained to output target index information idx according to predetermined information for each type of communication delay, and the scene is shown in which robot control is performed using the trained MM-LLM2.

[0138] Figure 13 shows that the robot RB first detects the image IM ｘ An example of uploading to the server device 100 (control device 100B) is shown (step S1301). The robot RB also maintains a timestamp "TM1" indicating the time of upload "TM1".

[0139] The estimation unit 137 of the control device 100B detects the image IM using the second acquisition unit 136. ｘ When acquired, the acquired detection image IM ｘ Input this into MM-LLM2 (step S1302).

[0140] Then, the estimation unit 137 detects the image IM in MM-LLM2. ｘ Based on the output result from MM-LLM2 when the input is given, the detected image IM ｘThe corresponding target index information idx is estimated for each type of communication delay (step S1303). Figure 3 shows an example in which the estimation unit 137 estimates target index information idx "0" for "small communication delay", target index information idx "0" for "medium communication delay", and target index information idx "0" for "large communication delay".

[0141] The conversion unit 138 converts each target index information idx for each type of communication delay into a sequence of token IDs by matching it with the conversion table TB1, and the operation control unit 139 converts the token IDs for each type of communication delay into control information CF by detokenizing each token ID included in the sequence of token IDs (step S1304). Figure 13 shows an example in which the operation control unit 139 obtains control information CF1 corresponding to "communication delay: small", control information CF2 corresponding to "communication delay: medium", and control information CF3 corresponding to "communication delay: large" through conversion.

[0142] The motion control unit 139 then controls the robot RB using the control information CF1 to CF3. For example, the motion control unit 139 outputs the control information CF1 to CF3 to the robot RB, thereby controlling the robot RB to take action according to the command indicated by any of the control information CF1 to CF3.

[0143] Figure 13 shows an example in which the robot RB acquires control information CF1 to CF3 (step S1305). The robot RB also stores a timestamp "TM2" indicating the time "TM2" at which the information was acquired.

[0144] The robot RB determines the type of communication delay based on timestamps TM1 and TM2, and executes an action using the control information CF from control information CF1 to CF3 that corresponds to the determination result (step S1306). For example, the robot RB may calculate the difference between timestamps TM1 and TM2 as the communication delay time, and determine the type corresponding to the communication delay time as the current type of communication delay. For example, the robot RB may determine "communication delay: small" if the communication delay time is less than 1 second, "communication delay: medium" if the communication delay time is between 1 and 3 seconds, and "communication delay: large" if the communication delay time is greater than 3 seconds.

[0145] Therefore, for example, if robot RB determines that the communication delay is small, it may execute an action using control information CF1. If robot RB determines that the communication delay is medium, it may execute an action using control information CF2. If robot RB determines that the communication delay is large, it may execute an action using control information CF3.

[0146] (7-2. Regarding the Generative Model) In the above embodiment, an example of a generative model was shown in which a multimodal large-scale language model (MM-LLM) is used. However, in the above embodiment, a normal large-scale language model (LLM) that is not multimodal can also be used. In this case, the estimation unit 137 uses the detected image IM ｘ By inputting this into the object detection model, the object detection model generates text TX that indicates the situation of the task target (e.g., a person) detected by the model. ｎ The target index information idx may be output by inputting it into the LLM. In this example, the learning unit 134 may perform learning using a pre-trained LLM instead of MM-LLM, and the training data LDA contains the teacher image IM ｎ It is not necessary to use it.

[0147] [8. Hardware Configuration] The server device 100 according to the embodiment may be implemented by a computer 1000 having the configuration shown in Figure 14. Figure 14 is a hardware configuration diagram showing an example of a computer that implements the functions of the server device 100 according to the embodiment. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I / F) 1500, input / output interface (I / F) 1600, and media interface (I / F) 1700.

[0148] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0149] The HDD 1400 stores programs executed by the CPU 1100, and data used by such programs. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0150] The CPU 1100 controls output devices such as displays and input devices such as keyboards via the input / output interface 1600. The CPU 1100 acquires data from input devices via the input / output interface 1600. The CPU 1100 also outputs the generated data to output devices via the input / output interface 1600.

[0151] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0152] For example, when the computer 1000 functions as a server device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded on the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[0153] [9. Others] Furthermore, all or part of the processes described as being performed automatically in each of the above embodiments can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0154] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0155] Furthermore, the above embodiments can be combined as appropriate, provided that the processing content is not contradictory.

[0156] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, including the embodiments described in the section on the present invention.

[0157] 100 Server device 100A Learning device 130A Control unit 131 Collection unit 132 Conversion unit 133 First acquisition unit 134 Learning unit 135 Transmission unit 100B Control device 130B Control unit 136 Second acquisition unit 137 Estimation unit 138 Conversion unit 139 Operation control unit

Claims

1. A learning device for learning information to cause an operating body to perform a task, comprising: a first acquisition unit that acquires situation information indicating the situation of a task target which is the target of the task, and index information that points to a string that defines a situation change predicted from the situation as the task content; and a learning unit that learns an estimation model that uses learning data which associates the index information and the situation information to a generation model, and estimates from the input information acquired from the operating body the target index information that corresponds to the task content for realizing the task among the index information.

2. The learning device according to claim 1, wherein the learning unit learns to output the target index information as one token.

3. The learning device according to claim 2, wherein the first acquisition unit acquires predetermined index information in which the string is represented in advance by one token as the index information, and the learning unit learns the estimation model using learning data which associates the predetermined index information with the situation information.

4. The learning device according to claim 2, wherein the generative model is a large-scale language model, the large-scale language model constructs a multidimensional semantic space through pre-training, and the learning unit learns to output the target index information corresponding to the input information as one token by assigning the index information in a one-to-one relationship to the lexical vectors mapped to the semantic space.

5. The learning device according to claim 4, wherein the learning unit, as the assignment process, assigns the index information and the situation information to the lexical vector based on the token ID used in the calculation of the lexical vector, which identifies the token corresponding to the lexical vector.

6. The learning device according to claim 1, wherein the first acquisition unit acquires the index information defined for each string for each predetermined piece of information corresponding to the communication delay that may occur between the operating body and the learning device, the learning unit learns an estimation model to output the target index information corresponding to the predetermined piece of information for each predetermined piece of information, and the operating body executes the task based on the target index information corresponding to the predetermined piece of information corresponding to the current communication delay, from among the target index information output by the estimation model.

7. The learning device according to claim 6, wherein the predetermined information is a type of communication delay determined according to the time of communication delay that may occur between the operating body and the learning device.

8. The learning device according to claim 1, wherein the task is a task performed by the operating body on the task target, the first acquisition unit acquires image information indicating the status of the task target as status information, and the index information acquires index information indicating coordinate information that defines a change in the situation predicted from the situation as the task content, and the learning unit learns an estimation model that outputs target index information from the index information that corresponds to the task content for realizing the task when image information detected by the operating body as sensor information is input.

9. A control device that performs control for an operating body to execute a task, comprising: a second acquisition unit that acquires sensor information from the operating body; an estimation unit that estimates the target index information corresponding to the sensor information based on the output result obtained by inputting the sensor information to a trained estimation model, which is trained to estimate from the input information the target index information corresponding to the task content for realizing the task from the index information, based on training data that associates the status information of a task target which is the target of the task and the index information that points to a string that defines a change in the status predicted from the status as the task content; and a control unit that controls the operating body to execute the task according to the control information by outputting control information corresponding to the estimated target index information.

10. The control device according to claim 9, wherein the control device has correspondence information relating the candidate string to the candidate target index information, and the control unit compares the estimated target index information with the correspondence information to convert the estimated target index information into the string corresponding to the target index information, and outputs the control information including the string obtained by the conversion to the operating body.

11. A learning method to be performed by a learning device that learns information for an operating entity to perform a task, comprising: a first acquisition step of acquiring situation information indicating the situation of a task target which is the target of the task, and index information indicating a string that defines a situation change predicted from the situation as the task content; and a learning step of learning an estimation model which uses learning data that associates the index information with the situation information to a generation model, and estimates from the index information the target index information corresponding to the task content for realizing the task, from input information acquired from the operating entity.

12. A control method executed by a control device that controls an operating body to perform a task, comprising: a second acquisition step of acquiring sensor information from the operating body; an estimation step of estimating the target index information corresponding to the sensor information based on the output result obtained by inputting the sensor information to a trained estimation model that has been trained to estimate from the input information the target index information corresponding to the task content for realizing the task from the index information, based on training data that associates situation information indicating the situation of a task target which is the target of the task and index information indicating a string that defines a situation change predicted from the situation as the task content; and a control step of controlling the operating body to perform the task according to the control information by outputting control information corresponding to the estimated target index information to the operating body.

13. A learning program to be executed by a learning device that learns information for an operating entity to perform a task, comprising: a first acquisition procedure for acquiring situation information indicating the situation of a task target which is the target of the task, and index information indicating a string that defines a situation change predicted from the situation as the task content; and a learning procedure for learning an estimation model that uses learning data associating the index information and the situation information to a generation model to estimate, from the input information acquired from the operating entity, the target index information from the index information that corresponds to the task content for realizing the task.

14. A control program to be executed by a control device that controls an operating body to perform a task, comprising: a second acquisition procedure for acquiring sensor information from the operating body; an estimation procedure for estimating the target index information corresponding to the sensor information based on the output result obtained by inputting the sensor information to a trained estimation model that has been trained to estimate from the input information the target index information corresponding to the task content for realizing the task from the index information, based on training data that associates situation information indicating the situation of a task target which is the target of the task and index information indicating a string that defines a situation change predicted from the situation as the task content; and a control procedure for controlling the operating body to perform the task according to the control information by outputting control information corresponding to the estimated target index information to the operating body.

Images