Generated information evaluation device, generated information evaluation method, generated information evaluation program, and recording medium

Abstract

Provided is a generated information evaluation device capable of evaluating contribution of training data in a large-scale trained model subjected to fine tuning.　A generated information evaluation device according to the present disclosure includes: a data acquisition unit that acquires generated data that has been generated by a management target model; a feature extraction unit that extracts a feature portion, from the generated data, by using the management target model; and a contribution evaluation unit that evaluates, on the basis of the feature portion, contribution of a training data source used for training the management target model in the generated data.

Description

Generation Information Evaluation Device, Generation Information Evaluation Method, Generation Information Evaluation Program, and Recording Medium

[0001] The present disclosure relates to a generation information evaluation device, a generation information evaluation method, a generation information evaluation program, and a recording medium.

[0002] In recent years, research has been conducted on constructing dialogue systems such as chatbots using pre-trained models. For example, in Patent Document 1, an acquisition unit that acquires an inquiry sentence, a learning response sentence for a positive example for a learning inquiry sentence, and a learning response sentence for a negative example sampled according to the frequency distribution of the learning response sentence are used. A learned model generated by executing a learning process so that the similarity between the learning inquiry sentence and the learning response sentence for the positive example is large and the similarity between the learning inquiry sentence and the learning response sentence for the negative example is small is used to calculate the similarity between the inquiry sentence and a plurality of candidate response sentences. And an extraction unit that extracts one or more candidate response sentences from the plurality of candidate response sentences based on the similarity are described.

[0003] Japanese Patent Application Laid-Open No. 2021-124824

[0004] In the invention as described in Patent Document 1, learning (fine-tuning) may be performed by giving information such as a specialized book so that an answer to a question can be generated in advance. In such a case, there is a need to evaluate the contribution of the learning source in the generated information. However, in the products of current large-scale learning models, there is a problem that the contribution of each learning data source in the model output cannot be quantitatively grasped, and there is no mechanism for evaluating the contribution by the learning source.

[0005] Therefore, an object of the present disclosure is to provide a generation information evaluation device, a generation information evaluation method, a generation information evaluation program, and a recording medium that can evaluate the contribution of learning data in a large-scale learning model that has been fine-tuned.

[0006] To achieve the above objective, the generated information evaluation apparatus of the present disclosure includes a data acquisition unit, a feature extraction unit, and a contribution evaluation unit, wherein the data acquisition unit acquires generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the feature extraction unit extracts feature portions from the generated data by the managed model, and the contribution evaluation unit evaluates the contribution of the training data source learned by the managed model to the generated data based on the feature portions.

[0007] The generated information evaluation method of this disclosure includes a data acquisition step, a feature extraction step, and a contribution evaluation step, wherein each step is performed by a computer, the data acquisition step is to acquire generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction step is to extract feature portions from the generated data by the managed model, and the contribution evaluation step is to evaluate the contribution of the training data source learned by the managed model in the generated data based on the feature portions.

[0008] The generated information evaluation program of this disclosure includes a data acquisition procedure, a feature extraction procedure, and a contribution evaluation procedure, wherein the data acquisition procedure acquires generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction procedure extracts feature portions from the generated data by the managed model, and the contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portions, and is a program for causing a computer to execute each of these procedures.

[0009] The recording medium of this disclosure is a computer-readable recording medium that records a generated information evaluation program for causing a computer to execute each of the following procedures: a data acquisition procedure, a feature extraction procedure, and a contribution evaluation procedure; the data acquisition procedure acquires generated data generated by a managed model; the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model; the generated data is data generated by the managed model based on given query data; the feature extraction procedure extracts feature portions from the generated data by the managed model; and the contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portions.

[0010] According to this disclosure, the surrogate existence model can be further trained.

[0011] Figure 1 is a block diagram showing the configuration of an example of the generated information evaluation device of the present disclosure. Figure 2 is a block diagram showing an example of the hardware configuration of the generated information evaluation device of the present disclosure. Figure 3 is a flowchart showing an example of processing in the generated information evaluation device of the present disclosure. Figure 4 is a schematic diagram illustrating the processing of the contribution evaluation unit in the generated information evaluation device of the present disclosure. Figure 5 is a block diagram showing the configuration of an example of the generated information evaluation device of the present disclosure. Figure 6 is a flowchart showing an example of processing in the generated information evaluation device of the present disclosure. Figure 7 is a block diagram showing the configuration of an example of the surrogate existence model manufacturing device of the present disclosure. Figure 8 is a block diagram showing an example of the hardware configuration of the surrogate existence model manufacturing device of the present disclosure. Figure 9 is a flowchart showing an example of processing in the surrogate existence model manufacturing device of the present disclosure.

[0012] Next, embodiments of the present disclosure will be described with reference to the drawings. The present disclosure is not limited to the following embodiments. In the following drawings, the same parts are denoted by the same reference numerals. Furthermore, unless otherwise specified, the descriptions of each embodiment can be used interchangeably with those of the others, and unless otherwise specified, the configurations of each embodiment can be combined.

[0013] [Embodiment 1] The generated information evaluation device of this embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of an example of the generated information evaluation device 10 of this embodiment. As shown in Figure 1, the generated information evaluation device 10 (hereinafter also referred to as "this device 10") includes a data acquisition unit 11, a feature extraction unit 12, and a contribution evaluation unit 13. In addition, although not shown, this device 10 may also include, for example, an input unit, an output unit, a display unit and / or a storage unit.

[0014] The device 10 may be, for example, a single device including the aforementioned parts, or it may be a device in which the aforementioned parts can be connected via a communication network. Furthermore, the device 10 can be connected to an external device described later via a communication network. The communication network is not particularly limited and a known network can be used, for example, it may be wired or wireless. Examples of communication networks include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), etc. Examples of wireless communication include Wi-Fi®, Bluetooth®, Local 5G, LPWA, etc. The aforementioned wireless communication may be in the form of direct communication between devices (Ad Hoc communication), infrastructure communication, or indirect communication via an access point. The device 10 may, for example, be incorporated into a server as a system. The device 10 may also be, for example, a personal computer (PC, e.g., desktop or notebook), smartphone, tablet terminal, etc., on which the program disclosed herein is installed. Furthermore, the device 10 may be in the form of cloud computing or edge computing, for example, in which at least one of the aforementioned parts is on a server and the other parts are on a terminal.

[0015] Figure 2 illustrates a block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a central processing unit 101, memory 102, bus 103, storage device 104, input device 105, output device 106, communication device (communication unit) 107, etc. Each part of the device 10 is interconnected via the bus 103 through its respective interface (I / F).

[0016] The central processing unit 101 operates in coordination with other components via controllers (system controller, I / O controller, etc.) and is responsible for the overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program of this disclosure or other programs, and also reads and writes various types of information. Specifically, for example, the central processing unit 101 functions as a data acquisition unit 11, a feature extraction unit 12, and a contribution evaluation unit 13. The device 10 may also include other computing devices such as a CPU, GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), or a combination thereof as computing devices.

[0017] Bus 103 can also be connected to external devices, for example. Examples of such external devices include external storage devices (external databases, etc.), electrocardiographs, printers, external input devices, external display devices, audio output devices such as speakers, external imaging devices such as cameras, and various sensors such as acceleration sensors, geomagnetic sensors, and direction sensors. The device 10 can be connected to an external network (the aforementioned communication network) by a communication device 107 connected to bus 103, for example, and can also be connected to other devices via the external network.

[0018] Memory 102 may be, for example, main memory. When the central processing unit 101 performs processing, memory 102 reads various operational programs, such as the program of this disclosure, stored in the storage device 104 (described later), and the central processing unit 101 receives data from memory 102 and executes the program. The main memory may be, for example, RAM (random access memory). Alternatively, memory 102 may be, for example, ROM (read-only memory).

[0019] The storage device 104 is also called an auxiliary storage device, for example, in relation to the main memory (primary memory). As described above, the storage device 104 stores an operating program including the program of this disclosure. The storage device 104 may be, for example, a combination of a recording medium and a drive for reading and writing to the recording medium. The recording medium is not particularly limited and may be internal or external, for example, an HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc. The storage device 104 may be, for example, a hard disk drive (HDD) in which the recording medium and the drive are integrated, or a solid state drive (SSD). If the device 10 includes, for example, the storage device 104 functions as the storage unit.

[0020] In this device 10, the memory 102 and storage device 104 can also store various types of information, such as log information, information obtained from an external database (not shown) or external devices, information generated by this device 10, and information used by this device 10 when executing processing. At least some of the information may be stored on an external server other than the memory 102 and storage device 104, or it may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

[0021] The device 10 further includes, for example, an input device 105 and an output device 106. The input device 105 may include, for example, a pointing device such as a touch panel, trackpad, or mouse; a keyboard; imaging means such as a camera or scanner; a card reader such as an IC card reader or magnetic card reader; an audio input means such as a microphone; and so on. The output device 106 may include, for example, a display device such as an LED display or liquid crystal display; an audio output device such as a speaker; a printer; and so on. In this embodiment 1, the input device 105 and the output device 106 are configured separately, but the input device 105 and the output device 106 may be configured as an integrated unit, such as a touch panel display.

[0022] Next, an example of the generation information evaluation method of this embodiment will be described based on the flowchart in Figure 3. The generation information evaluation method of this embodiment can be implemented as follows, for example, using the generation information evaluation device 10 shown in Figures 1 and 2. Note that the generation information evaluation method of this embodiment is not limited to the use of the generation information evaluation device 10 shown in Figures 1 and 2.

[0023] First, prior to processing by the device 10, an interaction is performed between the managed model and its responder. The content of the interaction is not particularly limited as long as it is an exchange between the managed model and the responder, and may be, for example, a text exchange such as a text chat, or a voice exchange such as a phone call. The managed model is a model that has been fine-tuned using predetermined training data for a large-scale learning model. The managed model may be, for example, a surrogate existence model described later. The large-scale learning model is, for example, a machine learning model that has been trained using predetermined big data. The large-scale learning model may be, for example, a model that has been trained on big data of natural language (large-scale language model), a model that has been trained on big data of speech (large-scale speech model), or a model that has been trained on big data of images (large-scale image model). In this disclosure, a "surrogate existence model" is, for example, a machine learning model that has been trained to behave like a specific subject. The method for manufacturing the surrogate existence model is not particularly limited and can be manufactured by any method, but for example, it can be manufactured by the method for manufacturing a surrogate existence model described later in this disclosure.

[0024] The data acquisition unit 11 acquires generated data generated by the managed model (S1, evaluation information acquisition step). The generated data is, for example, data generated by the managed model based on given query data. The generated data may be, for example, data generated by one managed model, or data jointly generated by multiple managed models. The query data is not particularly limited and may be, for example, input data when a user of the managed model interacts with the managed model. The data acquisition unit 11 may further acquire the query data, for example. The data acquisition unit 11 may acquire various data (for example, the generated data and the query data) from the managed model, or it may acquire the various data from an external recording medium on which the various data is recorded. The data acquisition unit 11 may store the acquired various data in the storage unit of the device 10, for example.

[0025] The feature extraction unit 12 extracts the feature portion from the generated data according to the managed model (S2, feature extraction step). The feature extraction unit 12 can extract the feature portion by, for example, comparing the provisional response of a large-scale learning model based on the query data with the generated data of the managed model. For example, the feature extraction unit 12 first inputs the query data into a predetermined large-scale learning model to create a provisional response. The feature extraction unit 12 can extract the difference as the feature portion by, for example, comparing the provisional response with the generated data. If the generated data is text data, the feature extraction unit 12 may, for example, compare the provisional response with the generated data as strings to extract the feature portion, or it may compare the provisional response with the generated data in semantic space to extract the feature portion. In the former case, the feature extraction unit 12 can extract the feature portion by, for example, comparing the text itself sentence by sentence between the provisional response and the generated data and checking whether words and / or sentences have been added, changed, or deleted. In the latter case, the feature extraction unit 12 can extract the feature portion by, for example, converting the provisional answer and the generated data into numerical data and comparing the numerical data. More specifically, the feature extraction unit 12 can extract the feature portion by, for example, converting the provisional answer and the generated data into semantic vectors sentence by sentence using a known method such as Word2vec and comparing the semantic vectors. The feature extraction unit 12 may also extract the feature portion by, for example, inputting the provisional answer and the generated data into a large-scale language model and having the large-scale language model compare the provisional answer and the generated data.

[0026] The contribution evaluation unit 13 evaluates the contribution of the training data source learned by the managed model to the generated data based on the feature portion (S3, contribution evaluation step). The contribution evaluation unit 13 can, for example, evaluate the contribution of the training data source learned by the managed model to the generated data with respect to the feature portion based on the training data information. The training data information includes, for example, information that identifies the training data source learned by the managed model during fine-tuning of the managed model. The evaluation of the contribution may be, for example, a qualitative evaluation or a quantitative evaluation. The contribution evaluation unit 13 can, for example, evaluate that the training data source has a contribution if the feature portion exists. The contribution evaluation unit 13 may, for example, evaluate the contribution of the training data source learned by each managed model if the generated data is data jointly generated by multiple managed models. The contribution evaluation unit 13 may evaluate the contribution of each training data source if the generated data is based on multiple training data sources. In this case, the contribution evaluation unit 13 may, for example, evaluate the degree of contribution for each of the multiple training data sources. The degree of contribution may be an absolute or relative evaluation, for example. In this case, the contribution evaluation unit 13 can evaluate the degree of contribution of each learning data source based on the similarity of the learning data source having text similar to the feature portion and the amount of similar text in the generated data.

[0027] The contribution evaluation unit 13 may, for example, have the managed model evaluate the contribution of the training data source in the generated data. In this case, the contribution evaluation unit 13 can evaluate the contribution of the training data source by, for example, inputting a prompt to the managed model to self-evaluate its contribution. Specific examples of the prompt to self-evaluate the contribution include, but are not limited to, "Please tell us what information from the training data source was cited in the previous answer. If there is more than one source cited, please indicate the percentage for each source cited."

[0028] A specific example of the processing of the contribution evaluation unit 13 will be explained using Figure 4, but the processing of the contribution evaluation unit 13 is not limited to the following explanation. First, the sentences contained in the learning data sources (for example, (learning data source A, learning data source B, learning data source C)) are divided into chunks (learning data source A chunk, learning data source B chunk, learning data source C chunk) consisting of multiple sentences. Then, the sentence chunks contained in each learning data source are converted into vectors (learning data source A vector, learning data source B vector, learning data source C vector), and the vectors of each learning data source and each chunk are linked and recorded in the database.

[0029] Next, the contribution evaluation unit 13 divides the feature portion of the generated data 131 into chunks (chunk A, chunk B, chunk C) 132 (132A, 132B, 132C) consisting of multiple sentences. Then, it converts the text of each chunk 132 into vectors (chunk A vector, chunk B vector, chunk C vector) 133 (133A, 133B, 133C). Then, it searches the database and extracts the learning data source vectors (learning data source A vector, learning data source B vector, learning data source C vector) 134 (134A, 134B, 134C) that have a similarity to the vector 132B for each chunk 132A that is equal to or greater than a threshold (for example, 0.7). The contribution evaluation unit 13 can then calculate the contribution of each training data source based, for example, on the number of characters in each chunk 132, the similarity between each chunk vector 133 and the training data source, and the total amount of text in the generated data 131 (300 characters in Figure 4). Specifically, in the example shown in Figure 4, the contribution of training data source A can be calculated as 150 × 0.8 / 300 = 0.4, the contribution of training data source B as 50 × 0.9 / 300 = 0.15, and the contribution of training data source C as 100 × 0.7 / 300 = 0.23. Note that such conversion to vectors, calculation of similarity, and calculation of contribution based on the number of characters and similarity of each chunk require repeatedly performing high-dimensional vector and matrix operations for each training data source and all chunks of generated data. For this reason, it is practically impossible for a human to perform this in their head or with paper and pencil, and it is a computational process that relies on automatic processing by an electronic computer equipped with a CPU or GPU.

[0030] The device 10 may, for example, output the contribution of the evaluated training data source. In this case, the device 10 can, for example, output the training data source that contributed to the generated data and its degree of contribution in association with the generated data. The device 10 may, for example, operate in cooperation with the managed model. In this case, during interaction between the managed model and the user, the device 10 can acquire generated data from the managed model and output the contribution of the training data source to the generated data. This allows the user to confirm which training data source the generated data relies on when interacting with the managed model.

[0031] Furthermore, the device 10 may, for example, detect training data sources that contribute more than a predetermined threshold based on the calculated contribution of each training data source, and output control information to exclude the data corresponding to that training data source from the fine-tuning dataset for subsequent cycles. This suppresses excessive dependence of the model's output on specific training data sources, enabling automated training data selection that reduces overfitting and domain bias. Additionally, the device 10 may, for example, output a control signal to the managed model to change generation parameters such as temperature and top k during decoding when the contribution of a specific training data source in the generated data exceeds a threshold, thereby suppressing excessive generation of representations originating from that source.

[0032] According to this disclosure, it is possible to extract the feature portions generated by the managed model from the generated data produced by the managed model and evaluate the contribution of the training data source to the generated data. For example, the contribution of the training data source can be displayed to the user of the managed model. In general, with large-scale training models, it is difficult to quantitatively grasp the contribution of each training data source, and it is difficult to perform technical management such as analyzing the model's behavior, reviewing or deleting training data, and suppressing harmful information originating from specific sources. According to this disclosure, since it is possible to evaluate the contribution of the training data source to the generated data produced by the managed model, it becomes possible to perform technical management such as analyzing the model's behavior, reviewing or deleting training data, and suppressing harmful information originating from specific sources.

[0033] [Embodiment 2] Embodiment 2 is another example of a generated information evaluation device.

[0034] The generated information evaluation device of this embodiment is the same as the generated information evaluation device 10 of Embodiment 1, except that it includes a revenue information acquisition unit 14 and a revenue distribution unit 15 in addition to the configuration of the generated information evaluation device 10 of Embodiment 1, and the description thereof can be applied accordingly.

[0035] Figure 5 is a block diagram showing an example configuration of the generated information evaluation device 10A of this embodiment. As shown in Figure 5, the generated information evaluation device 10A includes a revenue information acquisition unit 14 and a revenue distribution unit 15 in addition to the configuration of the generated information evaluation device 10 of Embodiment 1. The hardware configuration of the generated information evaluation device 10A is the same as that of the generated information evaluation device 10 of Figure 2, except that the central processing unit 101 has the configuration of the generated information evaluation device 10A of Figure 5 instead of the configuration of the generated information evaluation device 10 of Figure 1.

[0036] The processing of the revenue information acquisition unit 14 and the revenue distribution unit 15 will be explained below with reference to Figure 6. Figure 6 is a flowchart showing an example of processing by this device 10A. The processing of the revenue information acquisition unit 14 and the revenue distribution unit 15 may be inserted at any appropriate position in the flowchart of Figure 3 described in the embodiment 1 above.

[0037] First, S1 to S3 are carried out in the same manner as S1 to S3 of Embodiment 1.

[0038] The revenue information acquisition unit 14 acquires revenue information of the managed model (S4, revenue information acquisition step). The revenue information is, for example, revenue information based on the managed model. The revenue is not particularly limited and may include, but is not limited to, usage fees for the managed model, profits from advertising, etc. The revenue information acquisition unit 14 can acquire the revenue information by, for example, acquiring user payment information in cooperation with the payment function used by the managed model.

[0039] The revenue distribution unit 15 distributes the revenue to the parties involved with the learning data source based on the evaluation of their contributions (S5, revenue distribution process). These parties include, but are not limited to, the creators, editors, and rights holders of the information contained in the learning data source. The rights relating to the learning data source are not particularly limited and include, but are not limited to, intellectual property rights such as patent rights, utility model rights, design rights, trademark rights, and copyrights. These rights may include, for example, licenses such as exclusive licenses and non-exclusive licenses, as well as rights arising from licensing agreements and sales rights. The revenue distribution unit 15 identifies the parties to whom revenue should be distributed based on the evaluation of their contributions and allocates the revenue to them. The revenue distribution unit 15 can distribute the revenue by, for example, coordinating with the payment function used by the managed model and transferring the acquired user payment information to the financial institution accounts of the parties involved; however, the revenue distribution process is not limited to this, and any method can be adopted.

[0040] According to this disclosure, for example, revenue can be distributed based on the contribution of the training data source of the managed model. Therefore, according to this disclosure, for example, even if copyrighted works are used in the fine-tuning of the managed model, revenue from the managed model can be distributed to the authors of those works.

[0041] [Embodiment 3] The generated information evaluation program of this embodiment is a program that causes a computer to execute each step of the generated information evaluation method described above. Specifically, the generated information evaluation program of this embodiment is a program that causes a computer to execute the data acquisition procedure, the feature extraction procedure, and the contribution evaluation procedure.

[0042] The data acquisition procedure acquires generated data generated by the managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction procedure extracts the feature portion from the generated data by the managed model, and the contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portion.

[0043] Furthermore, the generated information evaluation program of this embodiment can also be described as a program that causes a computer to function as a data acquisition procedure, a feature extraction procedure, and a contribution evaluation procedure.

[0044] The generation information evaluation program of this embodiment can incorporate the descriptions in the generation information evaluation device and generation information evaluation method of the present disclosure. Each of the above procedures can be read as "processing" instead of "procedure", for example. Also, the program of this embodiment may be recorded on a computer-readable recording medium, for example. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), flash memory (e.g., SSD (Solid State Drive), USB flash memory, SD / SDHC card, etc.), optical disk (e.g., CD-R / CD-RW, DVD-R / DVD-RW, BD-R / BD-RE, etc.), magneto-optical disk (MO), floppy (registered trademark) disk (FD), etc. Also, the generation information evaluation program of this embodiment (also referred to as a programming product or generation information evaluation program product, for example) may be in a form distributed from an external computer, for example. The above "distribution" may be, for example, distribution via a communication line network or distribution via a device connected by wire. The generation information evaluation program of this embodiment may be installed and executed on the distributed device, or may be executed without being installed.

[0045] [Embodiment 4] The proxy presence model manufacturing apparatus of this embodiment will be described using FIG. 7. FIG. 7 is a block diagram showing the configuration of an example of the proxy presence model manufacturing apparatus 20 of this embodiment. As shown in FIG. 7, the proxy presence model manufacturing apparatus 20 (hereinafter also referred to as "this apparatus 20") includes a knowledge information acquisition unit 21, a construction information extraction unit 22, and a model construction unit 23. Also, although not shown, this apparatus 20 may include, for example, an input unit, an output unit, a display unit, and / or a storage unit.

[0046] The device 20 may be, for example, a single device including the aforementioned parts, or it may be a device in which the aforementioned parts can be connected via a communication network. Furthermore, the device 20 can be connected to an external device described later via a communication network. The communication network is not particularly limited and a known network can be used, for example, it may be wired or wireless. Examples of communication networks include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), etc. Examples of wireless communication include Wi-Fi®, Bluetooth®, Local 5G, LPWA, etc. The aforementioned wireless communication may be in the form of direct communication between devices (Ad Hoc communication), infrastructure communication, or indirect communication via an access point. The device 20 may, for example, be incorporated into a server as a system. Alternatively, the device 20 may be, for example, a personal computer (PC, e.g., desktop or notebook), smartphone, or tablet terminal on which the program disclosed herein is installed. Furthermore, the device 20 may be in the form of cloud computing or edge computing, for example, in which at least one of the aforementioned parts is on a server and the other parts are on a terminal.

[0047] Figure 8 illustrates a block diagram of the hardware configuration of the device 20. The device 20 includes, for example, a central processing unit 201, memory 202, bus 203, storage device 204, input device 205, output device 206, communication device (communication unit) 207, etc. Each part of the device 20 is interconnected via the bus 203 through its respective interface (I / F).

[0048] The central processing unit 201 operates in cooperation with other components, such as a controller (system controller, I / O controller, etc.), to control the entire apparatus 10. In this apparatus 20, for example, the programs of the present disclosure and other programs are executed by the central processing unit 201, and various information is read and written. Specifically, for example, the central processing unit 201 functions as a knowledge information acquisition unit 21, a construction information extraction unit 22, and a model construction unit 23. The apparatus 20 may include other arithmetic units such as a CPU, a GPU (Graphics Processing Unit), and an APU (Accelerated Processing Unit) as an arithmetic unit, or may include a combination thereof.

[0049] The bus 203 can be connected to, for example, an external device. Examples of the external device include an external storage device (external database, etc.), an electrocardiograph, a printer, an external input device, an external display device, an audio output device such as a speaker, an external imaging device such as a camera, and various sensors such as an acceleration sensor, a geomagnetic sensor, and a direction sensor. The apparatus 20 can be connected to an external network (the communication line network) by, for example, a communication device 207 connected to the bus 203, and can also be connected to other devices via the external network.

[0050] The memory 202 includes, for example, a main memory (main storage device). When the central processing unit 201 performs processing, for example, the memory 202 reads various operation programs such as the program of the present disclosure stored in the storage device 204 described later, and the central processing unit 201 receives data from the memory 202 and executes the program. The main memory is, for example, a RAM (Random Access Memory). Further, the memory 202 may be, for example, a ROM (Read Only Memory).

[0051] The storage device 204 is also called an auxiliary storage device, for example, in relation to the main memory (primary memory). As described above, the storage device 204 stores an operating program including the program of this disclosure. The storage device 204 may be, for example, a combination of a recording medium and a drive for reading and writing to the recording medium. The recording medium is not particularly limited and may be internal or external, for example, an HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc. The storage device 104 may be, for example, a hard disk drive (HDD) in which the recording medium and the drive are integrated, or a solid state drive (SSD). If the device 20 includes, for example, the storage device 204 functions as the storage unit.

[0052] In this device 20, the memory 202 and storage device 204 can also store various types of information, such as log information, information obtained from an external database (not shown) or external devices, information generated by this device 20, and information used by this device 20 when executing processing. At least some of the information may be stored, for example, on an external server other than the memory 202 and storage device 204, or distributed and stored across multiple terminals using blockchain technology or the like.

[0053] The device 20 further includes, for example, an input device 205 and an output device 206. The input device 205 may include, for example, a pointing device such as a touch panel, trackpad, or mouse; a keyboard; imaging means such as a camera or scanner; a card reader such as an IC card reader or magnetic card reader; an audio input means such as a microphone; and so on. The output device 206 may include, for example, a display device such as an LED display or liquid crystal display; an audio output device such as a speaker; a printer; and so on. In this embodiment 3, the input device 205 and the output device 206 are configured separately, but the input device 205 and the output device 206 may be configured as an integrated unit, such as a touch panel display.

[0054] Next, an example of the method for manufacturing a surrogate existence model according to this embodiment will be described based on the flowchart in Figure 9. The method for manufacturing a surrogate existence model according to this embodiment can be carried out as follows, for example, using the surrogate existence model manufacturing apparatus 20 shown in Figures 7 and 8. Note that the method for manufacturing a surrogate existence model according to this embodiment is not limited to the use of the surrogate existence model manufacturing apparatus 20 shown in Figures 7 and 8.

[0055] First, the knowledge information acquisition unit 21 acquires the subject's knowledge information (S21, knowledge information acquisition step). The format of the knowledge information is not particularly limited; for example, it may be text information, image information, audio information, or a combination thereof. The knowledge information is, for example, information linked to predetermined information and subject identification information that identifies the creator of the information (the subject). The predetermined information is, for example, information that includes the subject's personality information and explicit knowledge information. The personality information is, for example, information that represents the subject's thoughts from the knowledge information. The personality information is also called, for example, a partial stance. The explicit knowledge information is, for example, the part of the knowledge information excluding the personality information, and includes objective knowledge. If the knowledge information is a book, paper, etc., the explicit knowledge information may, for example, be information such as technical terms and experimental results, but is not limited to these. The subject identification information may, for example, be a name, address, telephone number, email address, identification number (for example, My Number (individual number), etc.). Specific examples of the aforementioned knowledge information include, but are not limited to, books and papers written by the subject, video data of lectures given by the subject, audio data of lectures given by the subject, and image data created by the subject. The knowledge information acquisition unit 21 may, for example, acquire knowledge information recorded in the storage unit of the device 20, or it may acquire the aforementioned knowledge information from outside the device 20 via the input device 205. The knowledge information acquisition unit 21 may, for example, acquire one type of knowledge information of the subject, or it may acquire two or more types. The knowledge information acquisition unit 21 may, for example, record the acquired knowledge information in the storage unit of the device 20.

[0056] The information extraction unit 22 for construction extracts personality information of the subject from the knowledge information (S22, information extraction step for construction). The information extraction unit 22 for construction may also, for example, further extract explicit knowledge information from the knowledge information of the subject. The information extraction unit 22 for construction may, for example, use known natural language processing techniques to extract at least one of the personality information and explicit knowledge information of the subject, or use a large-scale language model to extract at least one of the personality information and explicit knowledge information of the subject. If the knowledge information is textual information such as a book or a paper, the information extraction unit 22 for construction can extract the personality information or explicit knowledge information based, for example, on the end of sentences in a document or the chapter structure of a book. The information extraction unit 22 for construction may, for example, record at least one of the extracted personality information and explicit knowledge information in the storage unit of the device 20. In this case, the information extraction unit 22 for construction can, for example, record the personality information and explicit knowledge information linked to the knowledge information from which they were extracted and the creator identification information of the knowledge information.

[0057] The information extraction unit 22 for construction can, for example, analyze the knowledge information and extract sentences whose sentence ends with a word that expresses the author's thoughts as the personality information. Examples of words that express the author's thoughts include, but are not limited to, words such as "I want to," "I think," "I believe," and "I want." The information extraction unit 22 for construction can also analyze the knowledge information and extract sentences whose sentence ends with a word that indicates explicit knowledge as the explicit knowledge information. Examples of words that indicate explicit knowledge include, but are not limited to, words such as "It is," "It was," and "As a result."

[0058] The information extraction unit 22 for construction may, for example, analyze the knowledge information and extract sentences contained in chapters that describe the author's ideas as personality information. Examples of chapters that describe the author's ideas include the "preface," "introduction," and "afterword."

[0059] The information acquisition unit 22 for construction may, for example, use AI to extract personality information and explicit knowledge information from knowledge information. In this disclosure, AI may refer to, for example, a large-scale learning model called a "foundation model." The foundation model is a machine learning model pre-trained on predetermined big data and is not limited to a large-scale language model (LLM) that has learned natural language, but may also include a large-scale model for speech, a large-scale model for images, and a multimodal model (such as a visual language model) that handles language, images, speech, and video across the board. Furthermore, a configuration may be adopted in which a small-scale language model (SLM) is placed on the terminal side and cooperates with the large-scale model on the cloud side, a configuration that includes search extension generation (RAG) using an external knowledge source, tool execution / function call, agent-oriented control logic, etc. The providers of large-scale language models are not particularly limited. Examples include, but are not limited to, various LLM / multimodal models provided by companies such as OpenAI, Anthropique, Alphabet (Google), META, Microsoft, Cohere, Mistral, xAI, NEC Corporation, and NTT.

[0060] When the information acquisition unit 22 for construction uses AI to extract personality information and explicit knowledge information from knowledge information, the information acquisition unit 22 can cause the AI ​​to extract the user's personality information and explicit knowledge information by inputting extraction instruction information, which instructs the AI ​​to extract personality information and explicit knowledge information, along with the knowledge information. The extraction instruction information may be recorded in the storage unit of the device 10, stored externally, or input by the user each time. Specific examples of the extraction instruction information include, for example, "classifying information into the following two categories: sentences and paragraphs that show the author's thoughts and opinions, and sentences and paragraphs that show knowledge, such as scientific verification results and historical facts." This disclosure is not limited to the above examples.

[0061] Furthermore, the AI ​​may be, for example, a model finely tuned to extract personality information and explicit knowledge information from knowledge information. In this case, for example, fine tuning for extracting personality information and explicit knowledge information from knowledge information can be performed by training the AI ​​with knowledge information and a set of personality information and explicit knowledge information previously extracted from the knowledge information. In this case, the AI ​​may be, for example, a multilayer neural network model having a large number of parameters (e.g., 100,000 or more, and in some cases, several million or more). Such fine tuning processing requires the rapid and repeated execution of a huge amount of numerical calculations, making it practically impossible for a human to perform it in their head or with paper and pencil, and thus relies on automated processing by electronic computers such as processors and GPUs. During the fine tuning, in order to suppress overfitting of the training data, known regularization methods such as L2 regularization, dropout, and early stopping may be combined and applied. This makes it possible to obtain a model that can extract personality information and explicit knowledge information with high generalization performance even from unknown knowledge information, and improves the accuracy and robustness of information extraction compared to simple threshold judgment or rule-based processing.

[0062] Furthermore, the information extraction unit 22 for construction may extract the personality information of the subject by, for example, providing a large-scale language model with the subject's knowledge information and instruction information that instructs the model to extract the subject's personality information from the knowledge information based on the subject's knowledge information, thereby causing the model to extract the personality information from the knowledge information. The large-scale language model is not particularly limited and includes, but is not limited to, OpenAI®'s GPT-3®, GPT-4®, Alphabet Inc. (Google®)'s BERT, LaMDA, PaLM2, META®'s LlaMA, NEC Corporation's LLM, NTT®'s LLM, etc. The instruction information that instructs the model to extract the subject's personality information from the knowledge information based on the subject's knowledge information is not particularly limited as long as it is a document that instructs the model to divide the knowledge information into parts that contain thoughts and parts that contain knowledge. Specific examples of instruction information that instructs the extraction of personality information of the subject from the knowledge information of the subject include, but are not limited to, documents such as, "Classify the information into the following two categories: - Sentences and paragraphs that show the author's thoughts and opinions, such as the author's ideas; - Sentences and paragraphs that show knowledge, such as scientific verification results and historical facts."

[0063] The information extraction unit 22 for construction may, for example, preprocess the knowledge information and extract personality information and explicit knowledge information. Specifically, for example, the information extraction unit 22 for construction may convert the knowledge information into a vectorized embedding vector sequence for each unit text, and then classify or group the knowledge information into personality information and explicit knowledge information based on the embedding vector sequence, thereby extracting personality information and explicit knowledge information based on the knowledge information. The embedding vectors may, for example, contain hundreds to thousands of real-valued elements for each unit text. For this reason, for example, a large number of embedding vectors are generated for the entire knowledge information, and the classification or grouping process consists of a large number of numerical operations, mainly matrix operations, which is a process that is practically impossible for a human to perform in their head or with paper and pencil.

[0064] The model building unit 23 constructs a surrogate existence model that mimics the target person based on the personality information (S23, model building step). The model building unit 23 can construct the surrogate existence model by, for example, providing a large-scale learning model with the personality information and instruction information that instructs the large-scale learning model to construct a surrogate existence model that mimics the target person based on the personality information. The large-scale learning model is, for example, a machine learning model that has been trained using predetermined big data. The large-scale learning model may be, for example, a model that has been trained on big data of natural language (large-scale language model), a model that has been trained on big data of speech (large-scale speech model), or a model that has been trained on big data of images (large-scale image model). The model building unit 23 can construct the surrogate existence model by, for example, providing a large-scale language model, as the large-scale learning model, with the personality information and instruction information that instructs the large-scale learning model to construct a surrogate existence model that mimics the target person based on the personality information. The instruction information is, for example, instruction information (prompt) for generating the behavior of a person handling knowledge. Examples of the aforementioned instructional information include, but are not limited to, documents such as, "When generating text, follow the rules below: - Describe examples that reflect personal information. - Do not reflect personal information in terms of knowledge. - Structure the text such that, for example, you state a fact that is correct as knowledge, and then express an opinion on that fact as personal information."

[0065] Furthermore, the model building unit 23 may generate multiple sets of training data showing the correspondence between the subject's input information and responses based on the extracted personality information. The model building unit 13 may input the training data in mini-batch units into a large-scale learning model, calculate a loss function based on the error between the model's output for the training data and the response, and train a personality model capable of outputting behavior that mimics the subject by iteratively updating a large number of model parameters using gradient descent or a modified algorithm thereof.

[0066] Furthermore, the model building unit 23 may, for example, construct an explicit knowledge model by further training a large-scale language model with the explicit knowledge information. The model building unit 23 can construct the explicit knowledge model by, for example, providing the explicit knowledge information to the large-scale language model and fine-tuning it. The explicit knowledge model is also called, for example, a large-scale language model with domain knowledge.

[0067] The aforementioned surrogate existence model is, for example, a model that has learned the personality information of the subject from the information that constitutes the knowledge information. Therefore, the surrogate existence model has learned, for example, the subject's way of thinking and understanding when handling knowledge, values ​​such as thoughts and beliefs, and how they interact with others (personality). For this reason, the surrogate existence model manufacturing method of this disclosure makes it possible to easily manufacture a model that reflects the thoughts of the information creator. Furthermore, the surrogate existence model manufactured by the surrogate existence model manufacturing method of this disclosure is capable of outputting products that are characteristic of the subject. In other words, according to this disclosure, for example, it becomes possible to construct a model that suppresses personal hallucination. Personal hallucination refers to, for example, a hallucination (illusion) of how to handle knowledge and output policies that the creator of the knowledge information would not say. Therefore, according to the surrogate existence model of this disclosure, for example, it becomes possible to more accurately extract and output the knowledge that knowledge information (e.g., a book) explicitly or implicitly contains. The output that is characteristic of the target person is not particularly limited and may be, for example, text output, voice output, or instructions to a designated machine.

[0068] The model building unit 23 may, for example, use extracted personality information to train a personality model capable of outputting behavior that mimics the subject by iteratively updating the parameters of a large-scale learning model having many parameters based on gradient descent to reproduce the correspondence between the subject's past input information and responses. Similarly, the model building unit 23 may, for example, use extracted explicit knowledge information to train an explicit knowledge model that outputs explicit knowledge information contained in the knowledge information. In this case, the model building unit 23 can, for example, generate proxy behavior information consistent with the subject's personality and output the proxy behavior information by coordinating the trained personality model and the explicit knowledge model. By configuring the personality model and the explicit knowledge model separately in this way, and coordinating the output of explicit knowledge based on the knowledge information with the output based on the personality information, it is possible to reduce the inclusion of personal expressions not included in the knowledge information and improve the consistency and reliability of the generated responses, compared to, for example, simply providing prompts to a general-purpose large-scale language model.

[0069] According to this disclosure, it is possible to create a model that reflects the thoughts of the information creator. Therefore, according to this disclosure, for example, it is possible to construct a surrogate model for individuals with limited human resources (e.g., busy researchers, supervisors, managers, teachers, etc.), thereby reducing the burden on those individuals.

[0070] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, which can be understood by those skilled in the art within the scope of the present disclosure.

[0071] This application claims priority based on Japanese Patent Application No. 2024-229568, filed on 26 December 2024, and incorporates all of its disclosures herein.

[0072] <Note> Some or all of the above embodiments may be described as follows, but are not limited to the following. (Note 1) A generated information evaluation device comprising a data acquisition unit, a feature extraction unit, and a contribution evaluation unit, wherein the data acquisition unit acquires generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction unit extracts feature portions from the generated data by the managed model, and the contribution evaluation unit evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portions. (Note 2) The generated information evaluation device according to Note 1, wherein the data acquisition unit acquires the query data, and the feature extraction unit compares the provisional response of the large-scale learning model based on the query data with the generated data of the managed model to extract the feature portions. (Note 3) The generated information evaluation device according to Note 1 or 2, wherein the contribution evaluation unit evaluates the contribution of the training data source learned by the managed model in the generated data with respect to the feature portion, based on the training data information, and the training data information includes information on the training data source learned by the managed model. (Note 4) The generated information evaluation device according to Note 3, wherein the contribution evaluation unit evaluates the contribution of each training data source when the generated data is based on a plurality of training data sources. (Note 5) The generated information evaluation device according to any one of Notes 1 to 4, comprising a revenue information acquisition unit and a revenue distribution unit, wherein the revenue information acquisition unit acquires revenue information of the managed model, the revenue information is revenue information based on the managed model, and the revenue distribution unit distributes the revenue to the parties involved in the training data source based on the evaluation of the contribution.(Note 6) A method for evaluating generated information, comprising a data acquisition step, a feature extraction step, and a contribution evaluation step, wherein the data acquisition step acquires generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction step extracts feature portions from the generated data by the managed model, and the contribution evaluation step evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portions, each step being performed by a computer. (Note 7) The method for evaluating generated information according to Note 6, wherein the data acquisition step acquires the query data, and the feature extraction step compares the provisional response of the large-scale learning model based on the query data with the generated data of the managed model to extract the feature portions. (Note 8) The generation information evaluation method according to Note 6 or 7, wherein the contribution evaluation step evaluates the contribution of the training data source learned by the managed model in the generated data with respect to the feature portion, based on the training data information, and the training data information includes information on the training data source learned by the managed model. (Note 9) The generation information evaluation method according to Note 8, wherein the contribution evaluation step evaluates the contribution of each training data source if the generated data is based on a plurality of training data sources. (Note 10) The generation information evaluation method according to any one of Notes 7 to 9, comprising a revenue information acquisition step and a revenue distribution step, wherein the revenue information acquisition step acquires revenue information of the managed model, the revenue information is revenue information based on the managed model, and the revenue distribution step distributes the revenue to the parties involved in the training data source based on the evaluation of the contribution.(Note 11) A generated information evaluation program that causes a computer to execute each of the following steps: a data acquisition step, a feature extraction step, and a contribution evaluation step, wherein the data acquisition step acquires generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction step extracts feature portions from the generated data by the managed model, and the contribution evaluation step evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portions. (Note 12) The generated information evaluation program according to Note 11, wherein the data acquisition step acquires the query data, and the feature extraction step compares the provisional response of the large-scale learning model based on the query data with the generated data of the managed model to extract the feature portions. (Note 13) The contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data based on the training data information for the feature portion, and the training data information includes information on the training data source learned by the managed model, as described in Note 11 or 12. (Note 14) The contribution evaluation procedure evaluates the contribution of each training data source if the generated data is based on a plurality of training data sources, as described in Note 13. (Note 15) The generation information evaluation program according to any one of Notes 11 to 14, comprising a revenue information acquisition procedure and a revenue distribution procedure, wherein the revenue information acquisition procedure acquires revenue information of the managed model, the revenue information is revenue information based on the managed model, and the revenue distribution procedure distributes the revenue to the parties involved in the training data source based on the evaluation of the contributions.(Note 16) A computer-readable recording medium that records a generated information evaluation program for causing a computer to execute each of the following procedures: a data acquisition procedure, a feature extraction procedure, and a contribution evaluation procedure, wherein the data acquisition procedure acquires generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction procedure extracts feature portions from the generated data by the managed model, and the contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portions. (Note 17) The recording medium according to Note 16, wherein the data acquisition procedure acquires the query data, and the feature extraction procedure compares the provisional response of the large-scale learning model based on the query data with the generated data of the managed model to extract the feature portions. (Note 18) The recording medium according to Note 16 or 17, wherein the contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data with respect to the feature portion, based on the training data information, and the training data information includes information on the training data source learned by the managed model. (Note 19) The recording medium according to Note 18, wherein the contribution evaluation procedure evaluates the contribution of each training data source if the generated data is based on a plurality of training data sources. (Note 20) The recording medium according to any one of Notes 16 to 19, comprising a revenue information acquisition procedure and a revenue distribution procedure, wherein the revenue information acquisition procedure acquires revenue information of the managed model, the revenue information is revenue information based on the managed model, and the revenue distribution procedure distributes the revenue to the parties involved in the training data source based on the evaluation of the contribution.

[0073] According to this disclosure, it is possible to extract feature portions from the generated data produced by the managed model and evaluate the contribution of the training data source to the generated data. For example, the contribution of the training data source can be displayed to the user of the managed model. For this reason, this disclosure is useful in a wide range of industries that utilize finely tuned large-scale training models.

[0074] 10 Generated Information Evaluation Device 11 Data Acquisition Unit 12 Feature Extraction Unit 13 Contribution Evaluation Unit 14 Revenue Information Acquisition Unit 15 Revenue Distribution Unit 101 Central Processing Unit 102 Memory 103 Bus 104 Storage Device 105 Input Device 106 Output Device 107 Communication Device 20 Proxy Existence Model Manufacturing Device 21 Knowledge Information Acquisition Unit 22 Construction Information Extraction Unit 23 Model Construction Unit 201 Central Processing Unit 202 Memory 203 Bus 204 Storage Device 205 Input Device 206 Output Device 207 Communication Device

Claims

1. A generated information evaluation device comprising a data acquisition unit, a feature extraction unit, and a contribution evaluation unit, wherein the data acquisition unit acquires generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction unit extracts feature portions from the generated data by the managed model, and the contribution evaluation unit evaluates the contribution of the training data source learned by the managed model to the generated data based on the feature portions.

2. The generated information evaluation device according to claim 1, wherein the data acquisition unit acquires the query data, and the feature extraction unit compares the provisional response of a large-scale learning model based on the query data with the generated data of the managed model to extract the feature portion.

3. The contribution evaluation unit evaluates the contribution of the training data source learned by the managed model to the generated data with respect to the feature portion, based on the training data information, and the training data information includes information on the training data source learned by the managed model, according to claim 1 or 2, the generated information evaluation device.

4. The generated information evaluation device according to claim 3, wherein the contribution evaluation unit evaluates the contribution of each learning data source when the generated data is based on a plurality of learning data sources.

5. A generated information evaluation device according to any one of claims 1 to 4, comprising a revenue information acquisition unit and a revenue distribution unit, wherein the revenue information acquisition unit acquires revenue information of the managed model, the revenue information is revenue information based on the managed model, and the revenue distribution unit distributes the revenue to the parties involved in the learning data source based on the evaluation of the contribution.

6. A method for evaluating generated information, comprising a data acquisition step, a feature extraction step, and a contribution evaluation step, wherein each step is performed by a computer, the data acquisition step involves acquiring generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data against a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction step involves extracting feature portions from the generated data by the managed model, and the contribution evaluation step evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portions.

7. The generated information evaluation method according to claim 6, wherein the data acquisition step acquires the query data, and the feature extraction step compares the provisional response of a large-scale learning model based on the query data with the generated data of the managed model to extract the feature portion.

8. The method for evaluating generated information according to claim 6 or 7, wherein the contribution evaluation step evaluates the contribution of the training data source learned by the managed model to the generated data with respect to the feature portion, based on the training data information, and the training data information includes information on the training data source learned by the managed model.

9. The method for evaluating generated information according to claim 8, wherein the contribution evaluation step evaluates the contribution of each learning data source if the generated data is based on a plurality of learning data sources.

10. A method for evaluating generated information according to any one of claims 7 to 9, comprising a revenue information acquisition step and a revenue distribution step, wherein the revenue information acquisition step acquires revenue information of the managed model, the revenue information is revenue information based on the managed model, and the revenue distribution step distributes the revenue to the parties involved in the learning data source based on the evaluation of the contribution.

11. A generated information evaluation program that causes a computer to execute each of the following steps: a data acquisition procedure, a feature extraction procedure, and a contribution evaluation procedure, wherein the data acquisition procedure acquires generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction procedure extracts feature portions from the generated data generated by the managed model, and the contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portions.

12. The generated information evaluation program according to claim 11, wherein the data acquisition procedure acquires the query data, and the feature extraction procedure compares the provisional response of a large-scale learning model based on the query data with the generated data of the managed model to extract the feature portion.

13. The contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data to the feature portion, based on the training data information, wherein the training data information includes information on the training data source learned by the managed model, according to claim 11 or 12.

14. The contribution evaluation procedure, if the generated data is based on a plurality of training data sources, evaluates the contribution of each training data source, according to claim 13.

15. A generated information evaluation program according to any one of claims 11 to 14, comprising a revenue information acquisition procedure and a revenue distribution procedure, wherein the revenue information acquisition procedure acquires revenue information of the managed model, the revenue information is revenue information based on the managed model, and the revenue distribution procedure distributes the revenue to the parties of the learning data source based on the evaluation of the contribution.

16. A computer-readable recording medium that records a generated information evaluation program for causing a computer to execute each of the following procedures: a data acquisition procedure, a feature extraction procedure, and a contribution evaluation procedure, wherein the data acquisition procedure acquires generated data generated by a managed model, the managed model is a model that has been fine-tuned with predetermined training data for a large-scale learning model, the generated data is data generated by the managed model based on given query data, the feature extraction procedure extracts feature portions from the generated data by the managed model, and the contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data based on the feature portions.

17. The recording medium according to claim 16, wherein the data acquisition procedure acquires the query data, and the feature extraction procedure compares the provisional response of a large-scale learning model based on the query data with the generated data of the managed model to extract the feature portion.

18. The recording medium according to claim 16 or 17, wherein the contribution evaluation procedure evaluates the contribution of the training data source learned by the managed model in the generated data with respect to the feature portion, based on the training data information, and the training data information includes information on the training data source learned by the managed model.

19. The recording medium according to claim 18, wherein the contribution evaluation procedure evaluates the contribution of each training data source when the generated data is based on a plurality of training data sources.

20. A recording medium according to any one of claims 16 to 19, comprising a revenue information acquisition procedure and a revenue distribution procedure, wherein the revenue information acquisition procedure acquires revenue information of the managed model, the revenue information is revenue information based on the managed model, and the revenue distribution procedure distributes the revenue to the parties involved in the learning data source based on the evaluation of the contributions.

Images