Information processing device, information processing method, and program

Abstract

An information processing device according to one aspect of the present disclosure comprises: a front end that receives a query; and a back end that causes each of a plurality of multi-label methods to determine a candidate label of the query, receives the candidate label and a score for the candidate label from each of the plurality of multi-label methods, and determines an assigned label on the basis of the score given by each of the plurality of multi-label methods.

Description

Information processing device, information processing method, and program

[0001] This disclosure relates to an information processing device, an information processing method, and a program. In particular, it relates to a method for determining labels according to input content.

[0002] A multi-label method is known that uses a machine learning model trained with training data showing which labels a query matches, to estimate the degree of matching for each label to a query, and then assigns a label.

[0003] Furthermore, multi-label methods using Large Language Models (LLMs), such as generative AI (Artificial Intelligence) models, are known. An LLM is a language model constructed using a large amount of data and deep learning technology.

[0004] A multi-label method has been proposed in which audio data is labeled using a model trained on each listener, and the label that maximizes the mean of the posterior probability is used as the integrated output (see, for example, Patent Document 1).

[0005] Several methods exist for multi-label classification, and it has been proposed to output an order for at least one classification (see, for example, Patent Document 2).

[0006] International Publication No. 2021 / 166207, Japanese Patent Publication No. 2022-505985

[0007] When using a multi-label method with a machine learning model, a large amount of training data is necessary to obtain highly accurate results; if the training data is insufficient, the accuracy of label determination will be low. On the other hand, when using a multi-label method with a generative AI model, dedicated training data is not used, so there is a risk that the desired label determination may not be possible.

[0008] Therefore, it is necessary to perform appropriate label determination.

[0009] An information processing device in one aspect of the present disclosure includes a front end that receives a query, and a back end that causes each of a plurality of multi-label methods to determine candidate labels for the query, receives the candidate labels and a score for the candidate labels from each of the plurality of multi-label methods, and determines an assigned label based on the score from each of the plurality of multi-label methods.

[0010] An information processing method in one aspect of the present disclosure receives a query, causes each of a plurality of multi-label methods to determine candidate labels for the query, receives the candidate labels and a score for the candidate labels from each of the plurality of multi-label methods, and determines an assigned label based on the score from each of the plurality of multi-label methods.

[0011] A program in one aspect of this disclosure causes a computer to receive a query, determine candidate labels for the query for each of a plurality of multi-label methods, receive the candidate labels and scores for the candidate labels from each of the plurality of multi-label methods, and determine an assigned label based on the scores from each of the plurality of multi-label methods.

[0012] According to this disclosure, appropriate label determination can be performed.

[0013] Example of the overall system configuration Another example of the overall system configuration Example of the information processing device configuration Example of a functional block diagram of the information processing device Another example of a functional block diagram of the information processing device Flowchart sequence diagram executed by the information processing device Yet another example of a functional block diagram of the information processing device Flowchart sequence diagram executed by the information processing device Example of a response document

[0014] The embodiments of this disclosure will be described in detail below with reference to the drawings as appropriate.

[0015] (Embodiment 1) Figure 1A shows an example of the overall configuration of system 100. System 100 can integrate labels assigned by each of multiple multi-labeling methods and assign labels to input information such as queries. System 100 includes an input / output device 110, an information processing device 120, and a multi-labeling device 140.

[0016] The input / output device 110 receives information such as queries from the user to the information processing device 120. It also outputs answers such as labels from the information processing device 120 to the user.

[0017] The information processing device 120 analyzes the input information and outputs it to the multi-label device 140.

[0018] The multi-labeling device 140 has multiple multi-labeling methods, and can assign labels to queries using each of the multiple multi-labeling methods. In Figure 1A, the multi-labeling device 140 includes two multi-labeling methods, but the multi-labeling device 140 may include three or more multi-labeling methods.

[0019] Figure 1B shows another example of the overall configuration of system 100. System 100 includes an input / output device 110, an information processing device 120, a storage device 130, and a multi-label device 140.

[0020] Since the input / output device 110 and the information processing device 120 are the same as those shown in Figure 1A, their explanation will be omitted.

[0021] The storage device 130 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The storage device 130 may reside in the cloud.

[0022] The storage device 130 may store documents. Documents stored in the storage device 130 may be assigned an index. Documents to which the assigned label determined using the multi-label device 140 is assigned as an index are searched at high speed.

[0023] The multi-label device 140 includes a method using LLM 141 and a method using machine learning device 142. The multi-label device 140 may assign labels to queries using the index assigned to the text stored in the storage device 130 as the label. If the multi-label device 140 includes three or more multi-label methods, there may be a machine learning device different from machine learning device 142, or an LLM different from LLM 141.

[0024] LLM141 is a language model built using a large amount of data and deep learning technology. It outputs in response to prompts input to LLM141. A prompt is an instruction given to an AI such as LLM.

[0025] The machine learning device 142 is a model trained using abundant training data (supervised data). The training data consists of "labels assigned to queries".

[0026] In this disclosure, the machine learning device 142 is a machine learning device that has been trained to specifically assign labels to queries, while the LLM 141 is a general LLM and is not specifically trained to assign labels to queries.

[0027] Figure 2 shows an example configuration of the information processing device 120. The information processing device 120 includes a processor 121, an input device 122, and an output device 123. In addition to the devices shown in Figure 2, it may also include a communication device, memory, and storage. Each device, such as the processor 121 and the input device 122, is connected by a bus 124 for communicating information. The bus 124 may be configured using a single bus, or different buses may be configured for each device.

[0028] The processor 121 is composed of a computer including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 121 receives a query from the input / output device 110 via the input device 122 and sends an answer to the query to the input / output device 110. The processor 121 may also retrieve the query by searching the storage device 130.

[0029] The input device 122 may be an input device that receives input information from an external source (e.g., a keyboard, mouse, microphone, switch, button, sensor, etc.), or it may be an interface or communication device that receives data. The input device 122 may be connected to the storage device 130 and other devices not shown.

[0030] The output device 123 may be an output device that outputs user responses to the outside (e.g., a display, speaker, LED lamp, etc.), or it may be an interface or communication device that transmits data. The output device 123 may be connected to the storage device 130 and other devices not shown.

[0031] The input device 122 and the output device 123 may be configured as an integrated unit (for example, a touch panel), or the communication device may be the input device 122 and / or the output device 123.

[0032] Memory is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory may also be called registers, cache, main memory, etc. Memory can store executable programs (program code), software modules, etc., for carrying out a wireless communication method according to one embodiment of this disclosure.

[0033] The storage is a computer-readable recording medium, and may be configured by at least one of, for example, an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disc (for example, a compact disc, a digital versatile disc, a Blu-ray (registered trademark) disc), a smart card, a flash memory (for example, a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. The storage may also be referred to as an auxiliary storage device. The above-described storage medium may be, for example, a database including at least one of a memory and a storage, a server, or other suitable media.

[0034] The communication device is hardware (input / output device) for performing communication between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like.

[0035] Further, the information processing apparatus 120 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array), and part or all of each functional block may be implemented by the hardware. For example, the processor 121 may be implemented using at least one of these pieces of hardware.

[0036] FIG. 3 shows an example of a functional block diagram of the information processing apparatus 120. The information processing apparatus 120 includes a front end 125 and a back end 126.

[0037] The front end 125 receives a query. For example, the front end 125 receives the query from the input / output device 110, but may receive the query from a source other than the input / output device 110. For example, the front end 125 may receive the query as a result of searching the storage device 130. The front end 125 outputs the query received from the input / output device 110 to the back end 126. Also, the front end 125 transmits, to the device that received the query, for example the input / output device 110, a "label assigned to the query (hereinafter referred to as an 'assigned label')" input from the back end 126 as a reply to the query.

[0038] The back end 126 instructs the multi-label device 140 to perform label determination based on the query input from the front end 125. The back end 126 determines the assigned label based on the label received from the multi-label device 140 (hereinafter referred to as a "candidate label"). The back end 126 outputs the assigned label to the front end 125.

[0039] (Embodiment 2) In Embodiment 2, the back end 126 transmits the query received from the front end 125 to each of the multi-label methods, for example, the LLM 141 and the machine learning device 142, and determines the assigned label based on majority voting from the received candidate labels. The back end 126 uses the determined assigned label and RAG (Retrieval-Augmented Generation) to generate a reply to the query. RAG is a technology for searching for desired information and causing a generative AI such as an LLM to generate a reply based on the search results.

[0040] FIG. 4 shows another example of a functional block diagram of the information processing apparatus 120. The information processing apparatus 120 includes a front end 125 and a back end 126. The back end 126 includes an instruction unit 127 and a determination unit 128.

[0041] The instruction unit 127 transmits the query input from the front end to the LLM 141 and the machine learning device 142.

[0042] The decision unit 128 determines the assigned label based on the candidate labels received from the LLM 141 and the machine learning device 142. The decision unit 128 may output the determined assigned label to the front end 125.

[0043] Figure 5 shows a flowchart of the operations performed by the information processing device 120.

[0044] The information processing device 120 determines whether or not it has received a query (step S501). Step S501 is executed by the instruction unit 127. If the information processing device 120 has not received a query (step S501, No), it returns to step S501 and the information processing device 120 waits for the query to be received.

[0045] When the information processing device 120 receives a query (step S501, Yes), it instructs the LLM 141 and the machine learning device 142 to perform a multi-label determination on the query (step S502). Specifically, the information processing device 120 transmits an instruction to the LLM 141 and the machine learning device 142 to perform a multi-label determination on the query. Step S502 is executed by the instruction unit 127.

[0046] The information processing device 120 determines whether or not it has received candidate labels from the LLM 141 and the machine learning device 142 (step S503). Step S503 is performed by the determination unit 128. If the information processing device 120 has not received candidate labels from the LLM 141 and the machine learning device 142 (step S503, No), the process returns to step S503, and the information processing device 120 waits for the reception of candidate labels. Step S503 is performed by the determination unit 128.

[0047] When the information processing device 120 receives candidate labels from the LLM 141 and the machine learning device 142 (step S503, Yes), it determines the assigned label based on the candidate labels received from the LLM 141 and the candidate labels received from the machine learning device 142 (step S504). Step S504 is performed by the determination unit 128.

[0048] The determination unit 128 determines the assigned label by majority vote of candidate labels from each of the multiple multi-label methods received from the multi-label device 140. The determination unit 128 instructs each of the multiple multi-label methods to determine the label, and determines the candidate label determined by a majority of the multi-label methods as the assigned label. If the multi-label device 140 has two multi-label methods, the determination unit 128 determines the assigned label to be the overlapping candidate label from among the candidate labels determined by both multi-label methods.

[0049] The information processing device 120 generates a response text using RAG based on the assigned label determined in step S504 (step S505). RAG may be a RAG using the storage device 130.

[0050] The information processing device 120 transmits the response text generated by RAG as the response to the query (step S506). For example, if the information processing device 120 receives a query from the input / output device 110, it transmits the response text to the input / output device 110.

[0051] Figure 6 shows the sequence diagram.

[0052] The input / output device 110 sends a query to the front end 125 (step S601), and the front end 125 outputs the received query to the back end 126 (step S602). A device other than the input / output device 110 may also send a query to the front end 125.

[0053] The backend 126 sends instructions to the LLM 141 and the machine learning device 142 to perform query label determination (steps S603, S604).

[0054] The LLM 141 and the machine learning device 142 determine the candidate labels and respond (steps S605, S606). The candidate labels may be determined based on a score. The LLM 141 may calculate a score based on a confidence level indicating how confident it is in the determination result, and an inference consistency indicating the similarity to the result of performing Few-Shot Learning. The machine learning device 142 may calculate a score by multiplying the prediction probability by the validation accuracy of the accuracy validation data and the Scaling Factor. The LLM 141 and the machine learning device 142 may determine the three labels with the highest scores as candidate labels. The number of candidate labels to be determined can be arbitrarily decided and does not have to be three.

[0055] The backend 126 determines the assigned label based on the candidate labels received in steps S605 and S606 (step S607). For example, the backend 126 determines the assigned label by majority vote of the candidate labels received from the multi-label device 140. The determination unit 128 determines the candidate labels determined by a majority of the multi-label methods as the assigned labels. Since the multi-label device 140 has two multi-label methods, the determination unit 128 determines the overlapping candidate labels among the candidate labels determined by both multi-label methods as the assigned labels.

[0056] The backend 126 searches the storage device 130 for sentences similar to the query based on the determined assigned label (step S608). The storage device 130 responds to the backend 126 with similar sentences (step S609). The sentences stored in the storage device 130 may have labels assigned as indexes that the multi-label device 140 may assign. Since the sentences stored in the storage device 130 have labels assigned as indexes, the backend 126 can perform a search of the storage device 130 in a short time.

[0057] The backend 126 instructs the LLM 141 to generate a response sentence based on similar sentences (step S610). The LLM 141 transmits the generated response sentence to the input / output device 110 via the backend 126 and frontend 125 (steps S611, S612, S613). The frontend 125 transmits the response sentence to the device that sent the query to the frontend 125.

[0058] The backend 126 may instruct a generation AI other than LLM 141 to generate a response text.

[0059] Steps S608 to S611 are RAGs. The backend 126 can generate response texts in any RAG. Since the index used for searching in the RAG is determined as the label assigned to the query, similar texts to be referenced when generating response texts to the query can be searched at high speed.

[0060] <Modification of Embodiment 2> The LLM 141 and the machine learning device 142 may determine all labels with scores above a threshold as candidate labels, or they may determine the three labels with the highest scores among the labels with scores above a threshold as candidate labels.

[0061] If the decision unit 128 cannot determine the assigned label, for example, if there are no overlapping candidate labels among the candidate labels determined by both multi-label methods, it may instruct the LLM 141 and the machine learning device 142 to increase the number of candidate labels that the LLM 141 and the machine learning device 142 determine, or to lower the score threshold.

[0062] The decision unit 128 may send instructions to the LLM 141 and the machine learning device 142 to change the number of candidate labels to be judged according to the number of assigned labels determined. For example, if the number of assigned labels determined is small, i.e., below the threshold for the number of labels, the decision unit 128 may instruct the LLM 141 and the machine learning device 142 to increase the number of candidate labels to be judged or to lower the score threshold.

[0063] The LLM 141 and the machine learning device 142 may respond with the candidate labels along with the score for each candidate label. The backend 126 may exclude candidate labels with scores below a threshold from being assigned labels.

[0064] The backend 126 may transmit the candidate labels received from the LLM 141 and / or the machine learning device 142 to, for example, the input / output device 110, making the candidate labels available for review by the user. The user may add and / or delete candidate labels, or the user may decide on an assigned label from the candidate labels.

[0065] (Embodiment 3) In Embodiment 3, the backend 126 transmits the query received from the frontend 125 to the LLM 141 and the machine learning device 142, and determines the assigned label based on the scores of the candidate labels received from the LLM 141 and the machine learning device 142. Except for the method of determining the assigned label, it is the same as in Embodiment 2.

[0066] The decision unit 128 receives the candidate labels determined by each of the multiple multi-label methods and the score for each candidate label. The decision unit 128 may calculate a score (hereinafter referred to as the "final score") based on the scores from each of the multiple multi-label methods, for example, LLM 141 and the machine learning device 142, and determine the assigned label based on the final score.

[0067] For example, the final score may be the highest score obtained from each of several multi-label methods. That is, the final score of candidate label X = MAX(X) ＬＬＭ , X ＭＬ ) can also be used to calculate it.

[0068] The determination unit 128 may decide on candidate labels whose final score is equal to or greater than a threshold as assigned labels, or it may decide on a predetermined number of candidate labels as assigned labels in descending order of their final scores.

[0069] The final score may be calculated by other methods.

[0070] For example, the final score may be calculated based on a weighted sum of scores from each of a plurality of multi-label methods. For candidate label X, LLM 141 sets the score as X ＬＬＭ and calculates it, and the machine learning device 142 sets the score as X ＭＬ when calculated, with α and β as weighting coefficients, final score of candidate label X = α × X ＬＬＭ + β × X ＭＬ may be calculated as.

[0071] <Modification 1 of Embodiment 3> The backend 126 may transmit candidate labels received from LLM 141 and / or the machine learning device 142 to, for example, the input / output device 110, so that the candidate labels can be checked by a user. The user may add and / or delete candidate labels, or may determine the label to be assigned.

[0072] <Modification 2 of Embodiment 3> The score calculated by LLM 141 is a confidence level, and the score calculated by the machine learning device 142 is a reliability level. Since LLM 141 and the machine learning device 142 learn different data and also use different calculation methods for confidence level and reliability level, a "deviation" occurs between the score calculated by LLM 141 and the score calculated by the machine learning device 142. Therefore, the backend 126 may eliminate the "deviation" by performing arithmetic processing using a correction value A on the confidence level or the reliability level based on the calculated score or past history.

[0073] For example, the score (confidence level) X of a candidate label assigned by LLM 141 ＬＬＭ has an average value of "50", and the score (reliability level) X of a candidate label assigned by the machine learning device 142 ＭＬ has an average value of "80", the backend 126 may calculate "30 (=80-50)" or "1.6 (80 / 50)" as the correction value A. For example, the backend 126 may calculate X ＬＬＭ may add "30" to X ＭＬ may subtract "30" from X ＬＬＭ may multiply by "1.6" to X ＭＬ may divide by "1.6" from.

[0074] <Modification 3 of Embodiment 3> The score calculated by the machine learning device 142 may be corrected based on the field to be labeled.

[0075] For example, the machine learning device 142 is a learning model that assigns labels to sports videos, and is trained using 10 million training data points. It is then tuned with 10,000 training data points to assign labels to general videos (movies).

[0076] In such cases, it is estimated that labeling sports videos as a genre is highly reliable, whereas labeling general video genres, such as movies, cannot be expected to be highly reliable. Therefore, backend 126 assigns score X ＭＬ If it is a score for a sports video, it may be corrected by performing a calculation using correction value B. For example, backend 126 calculates score X ＭＬ If X is a score for a sports video, ＭＬ You may add "10" to it. Backend 126 is score X ＭＬ If the video is not a sports video, high reliability cannot be expected, so the candidate labels from the machine learning device 142 are not used as assigned labels. ＭＬ You may multiply it by "0". The correction value B may be predetermined by the operations manager, or it may be determined based on a table that associates genres with the correction value B.

[0077] Similarly, for LLM141, the score (confidence level) may be adjusted depending on the query genre.

[0078] (Embodiment 4) In Embodiment 4, the backend 126 selects which of the multiple multilabel methods to query based on the input query, and sends an instruction to the selected multilabel method to perform label determination on the query.

[0079] Figure 7 shows yet another example of a functional block diagram of the information processing device 120. The information processing device 120 has a front end 125 and a back end 126. The back end 126 has an instruction unit 127 and a selection unit 129.

[0080] The selection unit 129 selects which multi-label method to instruct to perform label determination based on the query input from the front-end 125. The selection unit 129 outputs the selected multi-label method to the instruction unit 127.

[0081] The instruction unit 127 sends an instruction to perform a label determination on the query for the multi-label method selected by the selection unit 129. Based on the selection result of the selection unit 129, the instruction unit 127 sends the query input from the front end 125 to the LLM 141 or the machine learning device 142, for example.

[0082] Figure 8 shows a flowchart of the actions performed by the information processing device 120.

[0083] The information processing device 120 determines whether or not it has received a query (step S801). Step S801 is executed by the instruction unit 127. If the information processing device 120 has not received a query (step S801, No), it returns to step S801 and the information processing device 120 waits for the query to be received.

[0084] When the information processing device 120 receives a query (step S801, Yes), it determines which of the multiple multi-label methods is best suited to query the received query (step S802). For example, the information processing device 120 determines whether the conditions for querying the machine learning device 142 are met. For example, the information processing device 120 may determine whether the conditions for querying the machine learning device 142 are met using a rule-based method, or it may determine whether the conditions for querying the machine learning device 142 are met using AI. When determining using a rule-based method, the information processing device 120 determines, for example, that querying the machine learning device 142 is best if the conditions for querying the machine learning device 142 are met, and that querying the LLM 141 is best if the conditions for querying the machine learning device 142 are not met. For example, the information processing device 120 determines that the conditions for querying the machine learning device 142 are met if the received query has a text length of 100 characters or less, the proportion of special characters is 20% or less, and the degree of specialization to a particular field is high. The information processing device 120 may determine which of the three or more multi-label methods is optimal.

[0085] If the information processing device 120 determines that it is optimal to query the machine learning device 142 (step S802, machine learning device), for example, if it determines that the conditions for querying the machine learning device 142 are met, it sends an instruction to the machine learning device 142 to perform multi-label determination on the query (step S803).

[0086] If the information processing device 120 determines that it is optimal to query the LLM 141 (step S802, LLM), for example, if it determines that the conditions for querying the machine learning device 142 are not met, it sends an instruction to the LLM 141 to perform a multi-label determination on the query (step S804).

[0087] The information processing device 120 uses the labels received from the machine learning device 142 or LLM 141 as assigned labels and generates a response text using RAG (step S805). Step S805 is the same as step S506.

[0088] The information processing device 120 transmits the response text generated by RAG as the response to the query (step S806). For example, if the information processing device 120 receives a query from the input / output device 110, it transmits the response text to the input / output device 110. Step S806 is the same as step S507.

[0089] Figure 9 shows the sequence diagram.

[0090] The input / output device 110 sends a query to the front end 125 (step S901), and the front end 125 outputs the received query to the back end 126 (step S902). A device other than the input / output device 110 may also send a query to the front end 125.

[0091] The backend 126 decides which multi-label method to use to determine the labels of the received query (step S903). The backend 126 may decide whether to send an instruction to LLM 141 or to machine learning device 142 by determining whether the conditions for querying the machine learning device 142 are met using a rule-based method, or it may decide whether to send an instruction to LLM 141 or to machine learning device 142 using AI. For example, the backend 126 may decide to query the machine learning device 142 if the received query satisfies all three conditions: (1) the text length is 100 characters or less, (2) the proportion of special characters is 20% or less, and (3) it has a high degree of specialization to a specific field. If any one of the conditions is not met, it may decide to query LLM 141. Alternatively, the AI ​​may determine whether condition (3) is met, and then the decision may be made using a rule-based method.

[0092] The backend 126 sends an instruction to the determined destination to perform a query label determination (step S904) and receives candidate labels (step S905). In Figure 9, the case where an instruction is sent to the LLM 141 is shown with a solid line, and the case where an instruction is sent to the machine learning device 142 is shown with a dotted line. The backend 126 sends an instruction to one of the devices.

[0093] The backend 126 searches the storage device 130 for sentences similar to the query based on the determined assigned label (step S906). Step S907 is the same as step S608.

[0094] The storage device 130 responds to the backend 126 with similar documents (step S907). Step S907 is the same as step S609. Documents stored in the storage device 130 may have labels assigned as indexes that the multi-label device 140 may assign.

[0095] The backend 126 instructs the LLM 141 to generate a response sentence based on similar sentences (step S908). Step S908 is the same as step S610.

[0096] The response text generated by LLM141 is sent to the input / output device 110 via the backend 126 and frontend 125 (steps S909, S910, S911). Steps S909, S910, and S911 are the same as steps S611, S612, and S613. The frontend 125 sends the response text to the device that sent the query to the frontend 125.

[0097] The backend 126 may instruct a generation AI other than LLM 141 to generate a response text.

[0098] Steps S906 to S909 are RAGs. The backend 126 may generate the response text using other RAGs.

[0099] <Modification of Embodiment 4> The LLM 141 and the machine learning device 142 may determine all labels with a score above a threshold as candidate labels, or they may determine the three labels with the highest scores among the labels with a score above a threshold as candidate labels.

[0100] If the selection unit 129 cannot determine which multi-label method is optimal, it may determine that a predetermined multi-label method, such as LLM 141, is optimal.

[0101] The selection unit 129 may have the generating AI determine which multi-label method is optimal.

[0102] (Embodiment 5) Embodiment 5 sends queries sequentially to multiple multi-label methods.

[0103] (Embodiment 5-1) Similar to Embodiment 4, the backend 126 determines which multi-label method to use to determine the label and sends a query to the determined multi-label method (hereinafter referred to as the "first multi-label method").

[0104] A query is sent to another multilabel method (e.g., a second multilabel method) based on the candidate labels determined by the chosen multilabel method. For example, a query may be sent to the second multilabel method if the score of the candidate labels determined by the first multilabel method is below a threshold.

[0105] The threshold for determining whether or not to send a query to the second multi-label method may be determined according to the multi-label method. For example, if the first multi-label method is a label assignment determined by LLM 141, the threshold may be set to "50", and if the first multi-label method is a label assignment by the machine learning device 142, the threshold may be set to "80".

[0106] (Embodiment 5-2) The backend 126 may send a query to the machine learning device 142 and receive the candidate labels determined by the machine learning device 142 along with their scores. If the score of the candidate labels determined by the machine learning device 142 is less than a threshold, the backend 126 may send a query to the LLM 141. The backend 126 may send a query to the LLM 141 if the score of a predetermined number of the candidate labels is less than a threshold.

[0107] The backend 126 may decide on the candidate label determined by the LLM 141 as the assigned label.

[0108] When the backend 126 receives the candidate labels determined by the LLM 141, it may determine the assigned label based on the candidate labels determined by the machine learning device 142 and the candidate labels determined by the LLM 141. The method for determining the assigned label may be the same as in embodiment 2 or 3, for example.

[0109] (Embodiment 5-3) The backend 126 may send a query to the LLM 141 and receive a score along with the candidate label determined by the LLM 141. If the score of the candidate label determined by the LLM 141 is below a threshold, or if the candidate label determined by the LLM 141 is hallucination, the backend 126 may send a query to the machine learning device 142. The backend 126 may delete the candidate label that has been determined to be hallucination.

[0110] The backend 126 may decide on the assigned label to be the candidate label determined by the machine learning device 142.

[0111] When the backend 126 receives candidate labels determined by the machine learning device 142, it may determine an assigned label based on the candidate labels determined by the machine learning device 142 and the candidate labels determined by the LLM 141. The method for determining the assigned label may be the same as in, for example, Embodiment 2 or 3.

[0112] <Modification of Embodiment 5> The backend 126 may correct the score of each candidate label in the same manner as in the modifications 2 and 3 of Embodiment 3.

[0113] When selecting from three or more multi-label methods, you can send a query to one of the multi-label methods, and then, depending on the result, send a query to another multi-label method (the second multi-label method). The second multi-label method may be determined based on the candidate labels assigned by the first multi-label method, or it may be determined in advance.

[0114] <Examples> Examples of Embodiment 2 are described below.

[0115] The user inputs a query into the input / output device 110, for example, "Please tell me what specific tasks are involved in construction notification."

[0116] The information processing device 120 receives the following labels from the LLM 141 for the query: "NOC News (score: 80)", "Common Function (score: 88)", and "Construction Notification Management (score: 92)".

[0117] The information processing device 120 receives the following labels from the machine learning device 142 for the query: "Construction notification management (score: 85)", "Backup machine management (score: 90)", and "NOC news (score: 75)".

[0118] The LLM 141 and the machine learning device 142 may respond with three or more labels in descending order of score, or they may respond with labels that have a score above a threshold.

[0119] The information processing device 120 calculates, for example, the final score of candidate label X = MAX(X ＬＬＭ , X ＭＬ Based on the above, the final scores for each candidate label may be calculated as follows: "NOC News (score: 80)", "Common Function (score: 88)", "Construction Notification Management (score: 92)", and "Backup Machine Management (score: 90)".

[0120] For example, the decision unit 128 may determine that the two labels assigned to the highest final scores are "Construction Notification Management" and "Backup Machine Management".

[0121] Backend 126 generates response text using RAG based on the determined assigned labels. The assigned labels determined for a query can be used as an index in RAG searches.

[0122] Figure 10 shows an example of a response to the query, "Please tell me what specific tasks are involved in construction notification."

[0123] In the embodiments described above, the notation for each component may be replaced with other notations such as "circuitry," "assembly," "device," "unit," or "module."

[0124] Furthermore, this disclosure can be implemented in software, hardware, or software in conjunction with hardware. Each functional block used in the description of the above embodiments may be implemented in part or in whole as an integrated circuit (LSI), and each process described in the above embodiments may be controlled in part or in whole by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may also be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0125] The integrated circuit implementation method is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, a Field Programmable Gate Array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that allows for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0126] Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that can replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possibility.

[0127] This disclosure is useful for an information processing device for multi-class classification of text using LLM.

[0128] 100 System 110 Input / Output Device 120 Information Processing Device 121 Processor 122 Input Device 123 Output Device 124 Bus 125 Front End 126 Back End 127 Instruction Unit 128 Decision Unit 129 Selection Unit 130 Storage Device 140 Multi-Label Device 141 LLM 142 Machine Learning Device

Claims

1. An information processing device comprising: a front-end that receives a query; and a back-end that causes each of a plurality of multi-label methods to determine candidate labels for the query, receives the candidate labels and scores for the candidate labels from each of the plurality of multi-label methods, and determines an assigned label based on the scores from each of the plurality of multi-label methods.

2. The information processing apparatus according to claim 1, wherein the plurality of multi-label methods include a multi-label method using LLM (Large Language Models) and a multi-label method using a machine learning device.

3. The information processing apparatus according to claim 1, wherein the backend determines the assigned label based on the weighted sum of the scores obtained by each of the multiple multi-label methods, or the highest score among the scores obtained by each of the multiple multi-label methods.

4. The information processing apparatus according to claim 1, wherein the backend generates a response sentence to the query using the assigned label.

5. The information processing apparatus according to claim 4, wherein the backend searches for similar documents to which the assigned label is assigned as an index, and generates a response document to the query based on the similar documents.

6. An information processing method that receives a query, has each of a plurality of multi-label methods determine candidate labels for the query, receives the candidate labels and scores for the candidate labels from each of the plurality of multi-label methods, and determines an assigned label based on the scores from each of the plurality of multi-label methods.

7. A program for causing a computer to perform information processing, which involves causing the computer to receive a query, to determine candidate labels for the query for each of a plurality of multi-label methods, to receive the candidate labels and scores for the candidate labels from each of the plurality of multi-label methods, and to determine an assigned label based on the scores from each of the plurality of multi-label methods.

Images