Processing system, processing device, processing method, and program

Abstract

A processing device 10 comprises: a division unit 11 that divides object data A to acquire first divided object data DA1 and second divided object data DA2, and divides related data R to acquire first divided related data DR1 and second divided related data DR2; a calculation unit 12 that calculates the degree of contribution of the first divided related data DR1 to the first divided object data DA1 and the degree of contribution of the second divided related data DR2 to the first divided object data DA1; a selection unit 13 that selects, from among the first divided related data DR1 and the second divided related data DR2, divided related data DR for which a higher degree of contribution to the first divided object data DA1 is calculated; and a display unit 14 that displays the first divided object data DA1 and the selected divided related data DR in association with each other.

Description

Processing system, processing apparatus, processing method, and program

[0001] The present disclosure relates to a processing system, a processing apparatus, a processing method, and a program.

[0002] With the recent progress of large language models (LLMs), the use of LLMs has been considered for various tasks. The output of an LLM may contain errors.

[0003] There is RAG (Retrieval-Augmented Generation) that improves the answer accuracy of an LLM. Non-Patent Document 1 selects and displays the basis data for the answer output by the LLM from the related data obtained from RAG.

[0004] Sato Chihiro, Funaki Kengo, Niiji Naoyuki, Fujimoto Tomoya, "Extraction method of basis information regarding the output of large language models", [online], September 11, 2024, 2024 Institute of Electronics, Information and Communication Engineers Society Conference, [searched on February 12, 2025], Internet <URL: https: / / pub.confit.atlas.jp / ja / event / society2024 / presentation / B-14-06>

[0005] Non-Patent Document 1 displays the basis data for the entire text of the answer output by the LLM. In Non-Patent Document 1, there may be cases where the basis for a predetermined part such as a sentence of the answer cannot be confirmed.

[0006] The present disclosure has been made in view of the above circumstances, and an object of the present disclosure is to provide a technique capable of displaying the basis for a predetermined part of an object.

[0007] A processing system according to one aspect of the present disclosure includes a model that provides target data as answer data to question data, a search unit that searches for related data related to the question data, and a processing unit connected to the model and the search unit, wherein the processing unit includes a splitting unit that splits the target data to obtain first split target data and second split target data, and splits the related data to obtain first split related data and second split related data, a calculation unit that calculates the contribution of the first split related data to the first split target data and the contribution of the second split related data to the first split target data, a selection unit that selects the split related data from the first split related data and the second split related data that has been calculated to have a high contribution to the first split target data, and a display unit that displays the first split target data and the selected split related data in association.

[0008] A processing apparatus according to one aspect of the present disclosure includes: a splitting unit that splits target data to obtain first splitting target data and second splitting target data; a calculation unit that calculates the contribution of the first splitting related data to the first splitting target data and the contribution of the second splitting related data to the first splitting target data; a selection unit that selects from the first splitting related data and the second splitting related data the splitting related data for which a high contribution to the first splitting target data has been calculated; and a display unit that displays the first splitting target data and the selected splitting related data in association.

[0009] A processing method in one aspect of the present disclosure involves a computer dividing target data to obtain first divided target data and second divided target data, dividing related data associated with the target data to obtain first divided related data and second divided related data, calculating the contribution of the first divided related data to the first divided target data and the contribution of the second divided related data to the first divided target data, selecting the divided related data from the first divided related data and the second divided related data that has been calculated to have a high contribution to the first divided target data, and displaying the first divided target data and the selected divided related data in association.

[0010] One aspect of this disclosure is a program that causes a computer to function as the processing device.

[0011] This disclosure provides a technology that can display the basis for a given part of a subject.

[0012] Figure 1 is a diagram illustrating the system configuration of the processing system of the disclosure and the functional blocks of the processing unit. Figure 2 is a diagram illustrating the overview of the processing of the processing system of the disclosure. Figure 3 is a diagram illustrating an example of data to be partitioned. Figure 4 is a diagram illustrating an example of partition-related data. Figure 5 is a diagram illustrating the data structure of contribution data and an example of the data. Figure 6 is a diagram illustrating the data structure of evidence data and an example of the data. Figure 7 is a diagram illustrating the calculation of the token sequence generation probability using LLM. Figure 8 is a diagram illustrating an example of contribution calculation. Figure 9 is a diagram illustrating the overview of contribution calculation. Figure 10 is a flowchart illustrating the processing method. Figure 11 is a diagram illustrating the hardware configuration of the computer used in the processing unit.

[0013] Embodiments of this disclosure will be described below with reference to the drawings. In the drawings, the same parts are denoted by the same reference numerals and their descriptions are omitted.

[0014] (Processing System) The processing system 1 shown in Figure 1 comprises a processing unit 10, a model 20, and a search unit 30. The processing unit 10 is connected to the model 20 and the search unit 30. The model 20 and the search unit 30 may be implemented in the processing unit 10 or in any computer.

[0015] The processing system 1 according to this disclosure displays a predetermined portion of the data to be divided DA of the target data A output by the model 20, by associating it with the portion of the division-related data DR of the related data R output by the search unit 30 that has a high contribution to the predetermined portion of the data to be divided DA.

[0016] Model 20 provides target data A. Model 20 is, for example, an LLM. Model 20 may also provide target data A as response data to question data Q. Target data A is, for example, the data that Model 20 provides to question data Q. Alternatively, Model 20 provides target data A as response data to question data Q and related data R. Target data A is, for example, the data that Model 20 provides to question data Q and related data R.

[0017] The search unit 30 searches for and outputs related data R related to the question data Q input to the model 20 from a database (not shown), for example, the RAG. The related data R is, for example, data retrieved by the search unit 30 in relation to the question data Q.

[0018] Referring to Figure 2, the processing overview of the processing system 1 will be explained. The processing unit 10 acquires target data A and related data R. Target data A and related data R are each text data, and each has multiple sentences or lines. Related data R serves as the basis for target data A.

[0019] The processing unit 10 divides the target data A and related data R into multiple parts. The processing unit 10 outputs the parts of the data to be divided DA and the parts of the related data DR that serve as the basis for those parts, in correspondence.

[0020] In the example shown in Figure 2, the processing unit 10 displays the first data to be divided DA1 and the first related data to be divided DR1 in association. The first related data to be divided DR1 serves as the basis for the first data to be divided DA1. The processing unit 10 displays the second data to be divided DA2 and the second related data to be divided DR2 in association. The second related data to be divided DR2 serves as the basis for the second data to be divided DA2. The processing unit 10 displays the third data to be divided DA3 and the third related data to be divided DR3 in association. The third related data to be divided DR3 serves as the basis for the third data to be divided DA3.

[0021] Processing system 1 can display the rationale for each part of the target.

[0022] (Processing device) As shown in Figure 1, the processing device 10 includes data to be divided DA, division-related data DR, contribution data 21, and basis data 22, as well as the functions of a division unit 11, a calculation unit 12, a selection unit 13, and a display unit 14. Each piece of data is stored in a storage device such as a memory 902 or storage 903. Each function is implemented in the CPU 901.

[0023] The data to be divided, DA, consists of multiple data columns separated from the target data, A. As shown in Figure 3, the data to be divided, DA, comprises three data to be divided, DA1-DA3, etc. Each of the data to be divided, DA1-DA3, etc., separated from the data to be divided, is sometimes referred to as a part of the data to be divided, DA.

[0024] The partitioned related data DR consists of multiple data columns partitioned from the related data R. As shown in Figure 4, the partitioned related data DR comprises three partitioned related data DR1-DR3, etc. Each of the partitioned related data DR1-DR3, etc., partitioned from the partitioned related data DR may be referred to as a part of the partitioned related data DR.

[0025] The contribution data 21 is data that associates the contribution of each part of the partitioning-related data DR with each part of the partitioning-target data DA. The contribution data 21 associates one contribution with each identifier of each part of the partitioning-target data DA and each identifier of each part of the partitioning-related data DR, for example, as shown in Figure 5.

[0026] The supporting data 22 is data that associates each part of the data to be split DA with the part of the related data DR that has the highest contribution. The supporting data 22 may also be dictionary data in which each part of the data to be split DA is used as the key and each part of the related data DR that has the highest contribution is used as the value.

[0027] The splitting unit 11 splits the target data A to obtain the first split target data DA1, the second split target data DA2, the third split target data DA3, etc. The number of data columns in the split target data DA split from the target data A is not limited.

[0028] The splitting unit 11 splits the related data R associated with the target data A to obtain the first split related data DR1, the second split related data DR2, the third split related data DR3, etc. The number of data columns in the split related data DR split from the related data R is not limited.

[0029] The splitting unit 11 divides either the target data A or the related data R into multiple data columns based on criteria such as punctuation, line breaks, and character count. Each data column consists of a single sentence, phrase, line, multiple sentences, a single paragraph, or multiple paragraphs. The splitting unit 11 may divide the target data A and the related data R at the same granularity or at different granularities.

[0030] The calculation unit 12 calculates the contribution of each part of the division-related data DR to each part of the division target data DA. The calculation unit 12 stores the calculated contributions in the contribution data 21.

[0031] Specifically, the calculation unit 12 calculates the contribution of the first division-related data DR1 to the first division target data DA1, the contribution of the second division-related data DR2 to the first division target data DA1, etc. The calculation unit 12 calculates the contribution of the first division-related data DR1 to the second division target data DA2, the contribution of the second division-related data DR2 to the second division target data DA2, etc. The calculation unit 12 calculates the contribution of the first division-related data DR1 to the third division target data DA3, the contribution of the second division-related data DR2 to the third division target data DA3, etc.

[0032] Here, the calculation unit 12 inputs the first splitting-related data DR1 into the model 20 and calculates the contribution from the probability that the first splitting target data DA1 is generated. Similarly, the calculation unit 12 inputs the first splitting-related data DR1 into the model 20 and calculates the contribution from the probability that the second splitting target data DA2 is generated. The calculation unit 12 inputs the first splitting-related data DR1 into the model 20 and calculates the contribution from the probability that the third splitting target data DA3 is generated. The calculation unit 12 also inputs the second splitting-related data DR2 into the model 20 and calculates the respective contributions from the probabilities that the first splitting target data DA1 is generated, the second splitting target data DA2 is generated, and the third splitting target data DA3 is generated.

[0033] The calculation unit 12 may calculate the SHAP value, which is calculated from each generation probability, as the contribution. Alternatively, to reduce the computational burden of calculating the SHAP value, the calculation unit 12 may calculate an approximate value of the SHAP value as the contribution. The SHAP value and the approximate value of the SHAP value are calculated using existing libraries. These existing libraries are publicly available at https: / / shap.readthedocs.io / en / latest / index.html, https: / / shap.readthedocs.io / en / latest / generated / shap.ExactExplainer.html#shap.ExactExplainer, etc.

[0034] The selection unit 13 selects the portion of the division-related data DR that has the highest contribution to each portion of the division-target data DA.

[0035] Specifically, the selection unit 13 selects the splitting-related data from the first splitting-related data DR1 and the second splitting-related data DR2, etc., that has been calculated to have a high contribution to the first splitting target data DA1. The selection unit 13 selects the splitting-related data from the first splitting-related data DR1 and the second splitting-related data DR2, etc., that has been calculated to have a high contribution to the second splitting target data DA2. The selection unit 13 selects the splitting-related data from the first splitting-related data DR1 and the second splitting-related data DR2, etc., that has been calculated to have a high contribution to the third splitting target data DA3.

[0036] The selection criteria used by the selection unit 13 are set in advance. For example, the selection unit 13 may select the part with the highest contribution among the parts of the division-related data DR, select multiple parts in order of highest contribution, or select one or more parts for which a contribution of a predetermined threshold or higher has been calculated.

[0037] The selection unit 13 associates each selected part of the division-related data DR with each part of the division target data DA and outputs the basis data 22.

[0038] The display unit 14 displays each portion of the data DA to be divided in association with the portion of the division-related data DR selected by the selection unit 13.

[0039] The display unit 14 displays, for example, the first data to be divided DA1 in association with the portion of the division-related data DR selected by the selection unit 13. Similarly, the display unit 14 displays the second data to be divided DA2 in association with the portion of the division-related data DR selected by the selection unit 13. The display unit 14 displays the third data to be divided DA3 in association with the portion of the division-related data DR selected by the selection unit 13.

[0040] The display unit 14 may also display the portion of the division-related data DR that corresponds to the portion of the data DA to be divided, as specified by the user, in the basis data 22.

[0041] (Calculation Unit) Referring to FIGS. 7-9, the method for calculating the contribution degree by the calculation unit 12 of the present disclosure will be described.

[0042] Generally, in the internal processing of an LLM, for each token of the input, the generation probability of the next possible token is calculated. By multiplying the generation probabilities for each token, the generation probability of any partial token sequence of the input to the LLM is calculated. Here, a token is a unit such as a word.

[0043] As shown in FIG. 7, the token sequence "The cat is cute, but the dog is not." is input to the LLM. At this time, the case of calculating the generation probability of "but the dog is not.", which is part of the token sequence, will be described. The LLM calculates the probability that each token such as and, but, so, etc. will come after the token "," immediately following the part of the token sequence for which the generation probability is to be calculated. Similarly, the LLM calculates the probability that each token such as the, a, loud, etc. will come after the first token "but" of the token sequence for which the generation probability is to be calculated.

[0044] Thus, for each token of the part of the token sequence for which the generation probability is to be calculated, the generation probability is calculated.

[0045] Here, the generation probability of "but the dog is not." after the token sequence "The cat is cute," is the multiplication of the probabilities enclosed by the rectangle in FIG. 7. Since this generation probability is a very small value, in practice, the logarithm of the multiplication value is used.

[0046] Referring to FIG. 8, the process of calculating the contribution degree from the generation probability described by taking FIG. 7 as an example will be explained. Each row in FIG. 8 indicates calculating the generation probability for the divided target data DA1, DA2, and DA3. Each column in FIG. 8 indicates the condition for calculating the generation probability. Here, the condition for calculating the generation probability is a combination such as the divided related data DR1-DR3, etc., and the target data A.

[0047] The calculation unit 12 generates combinations of each part of the division-related data DR by excluding part or all of the division-related data DR1 - DR3, etc. The calculation unit 12 inputs a prompt including each generated combination and the target data A into the model 20, and calculates the generation probability for the division target data DA1, DA2, and DA3 for each condition. Note that the calculation unit 12 may obtain the generation probability by inputting the prompt into the model 20, or may calculate the generation probability when the prompt is input into the model 20 by means of simulation or the like.

[0048] The calculation unit 12 calculates the SHAP value from the calculated generation probability and records it in the contribution degree data 21.

[0049] Referring to FIG. 9, the outline of the calculation of the contribution degree will be described. The calculation unit 12 uses the prompt template of the model 20 to add only a part of the division-related data DR to the prompt, and further generates a text obtained by combining the division target data DA at the end of the prompt. The calculation unit 12 inputs the generated text into the model 20. The calculation unit 12 outputs the SHAP value calculated from the logarithm of the generation probability obtained by the model 20 as the contribution degree. Here, the prompt template may be prepared along with the model 20, may be defined in advance by the calculation unit 12 or the like, or may be generated from the question data Q.

[0050] The model 20 calculates the generation probability of the words that may come next for each word in the input text. The next word for a part of the input text is determined as a word sequence. The calculation unit 12 obtains the generation probability of the subsequence of the input word sequence by obtaining the logarithm of the multiplication value of the generation probability of each word.

[0051] (Processing method) Referring to FIG. 10, the processing method in the processing system 1 of the present disclosure will be described.

[0052] In step S1, the processing device 10 acquires the target data A as the answer data obtained by inputting the question data Q into the model 20. In step S2, the processing device 10 acquires the related data R as the search result obtained by inputting the question data Q into the search unit 30.

[0053] In step S3, the processing unit 10 divides each of the target data A and related data R into multiple parts. In step S4, the processing unit 10 calculates the contribution between each part of the data to be divided DA and each part of the related data DR.

[0054] In step S5, the processing unit 10 selects the portion of the division-related data DR that has a high contribution, calculated in step S4, for each portion of the data DA to be divided. In step S6, the processing unit 10 displays the portion of the division-related data DR selected in step S5 in association with each portion of the data DA to be divided.

[0055] The processing system 1 of this disclosure can display the basis for each part of the subject. This allows for verification of consistency by comparing each part of the subject with the basis.

[0056] The processing unit 10 described above in this disclosure is, for example, a general-purpose computer system comprising a CPU (Central Processing Unit, processor) 901, memory 902, storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), communication device 904, input device 905, and output device 906. In this computer system, each function of the processing unit 10 is realized when the CPU 901 executes a program loaded onto the memory 902.

[0057] The processing unit 10 may be implemented on a single computer, or on multiple computers. Furthermore, the processing unit 10 may be a virtual machine implemented on a computer.

[0058] The program of the processing unit 10 can be stored on a computer-readable recording medium such as an HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or it can be distributed over a network. A computer-readable recording medium is, for example, a non-transitory recording medium.

[0059] This disclosure is not limited to the embodiments described above, and numerous modifications are possible within the scope of its essence.

[0060] 1 Processing System 10 Processing Unit 11 Splitting Unit 12 Calculation Unit 13 Selection Unit 14 Display Unit 20 Model 21 Contribution Data 22 Basis Data 30 Search Unit 901 CPU 902 Memory 903 Storage 904 Communication Device 905 Input Device 906 Output Device A Target Data DA Splitting Target Data DR Splitting Related Data Q Question Data R Related Data

Claims

1. A processing system comprising: a model that provides target data as answer data to question data; a search unit that searches for related data related to the question data; and a processing unit connected to the model and the search unit, wherein the processing unit comprises: a splitting unit that splits the target data to obtain first split target data and second split target data, and splits the related data to obtain first split related data and second split related data; a calculation unit that calculates the contribution of the first split related data to the first split target data and the contribution of the second split related data to the first split target data; a selection unit that selects the split related data from the first split related data and the second split related data for which a high contribution to the first split target data has been calculated; and a display unit that displays the first split target data and the selected split related data in association.

2. The processing system according to claim 1, wherein the model provides the target data to the question data and the related data.

3. A processing device comprising: a splitting unit that splits target data to obtain first split target data and second split target data, and splits related data associated with the target data to obtain first split related data and second split related data; a calculation unit that calculates the contribution of the first split related data to the first split target data and the contribution of the second split related data to the first split target data; a selection unit that selects from the first split related data and the second split related data the split related data for which a high contribution to the first split target data has been calculated; and a display unit that displays the first split target data and the selected split related data in association.

4. The processing device according to claim 3, wherein the processing device is connected to a model that provides response data to question data, and the calculation unit inputs the first division-related data to the model and calculates the contribution from the probability that the first division target data is generated.

5. The processing apparatus according to claim 4, wherein the calculation unit calculates the SHAP value calculated from the probability as the degree of contribution.

6. The processing apparatus according to claim 3, wherein the target data is data that the model provides to the question data and the related data, and the related data is data retrieved in relation to the question data.

7. A processing method comprising: a computer dividing target data to obtain first data to be divided and second data to be divided; dividing related data associated with the target data to obtain first related data to be divided and second related data to be divided; calculating the contribution of the first related data to the first data to be divided and the contribution of the second related data to the first data to be divided; selecting the related data from the first and second related data that has been calculated to have a high contribution to the first data to be divided; and displaying the first data to be divided and the selected related data in association.

8. A program for causing a computer to function as the processing device described in any one of claims 3 to 6.

Images