Learning device, learning method, and learning program

Abstract

A learning device (10) that is to train a conversation model that is specialized to a character or an individual has a character knowledge DB (123) that stores knowledge data that is unique to the character or individual and has been extracted from a conversation log and / or a question / response set that includes speech by the character or individual and a policy optimization learning unit (1316) that uses at least the knowledge data at the character knowledge DB (123) and the conversation log and / or question / response set to train the conversation model that is specialized to the character or individual to generate speech that seems like the character or individual.

Description

Learning Device, Learning Method, and Learning Program

[0001] The present invention relates to a learning device, a learning method, and a learning program.

[0002] In recent years, due to the emergence of large language models (LLMs: Large language Models), dialogue systems that reproduce a specific person or character have attracted attention. Regarding such a dialogue system, methods for realizing a dialogue model specialized for an individual or character, and a method for reproducing a specific person or character by a dialogue prompt (Non-Patent Document 1) have been proposed.

[0003] Yunfan Shao, Linyang Li, Junqi Dai, Xipeng Qiu, “Character-LLM: A Trainable Agent for Role-Playing”, 2023, [Retrieved December 10, 2024], Internet <URL: https: / / aclanthology.org / 2023.emnlp-main.814 / >.

[0004] As problems for realizing a dialogue that reproduces a character by an LLM, a first problem and a second problem can be cited.

[0005] The first problem is that the consistency of character learning and growth is not ensured. The second problem is that hallucinations occur in the character's utterances.

[0006] The first problem will be described. For example, the case where the dialogue data of the target character increases will be described.

[0007] In this case, according to the method described in Non-Patent Document 1, when the amount of character dialogue data is large, there are also limitations in reproduction by the prompt.

[0008] Furthermore, in the case of conventional dialogue models tailored to specific individuals, the model needs to be retrained. In other words, whether the parameters are initialized and trained, or additional training is performed, the model itself is updated to a new one. Therefore, there is no guarantee that the new model will remember what the previous model remembered. That is, in the case of dialogue models tailored to specific individuals, the new model is different from the previous model and becomes a completely new model. As a result, even when the same question is asked, different utterances may be generated than before, and the consistency and continuity of the character cannot be guaranteed, leading to the problem that users perceive the character as different before and after training.

[0009] Let me explain the second challenge. Hallucination refers to the current situation where LLM generates information that differs from common facts, and it is a problem inherent in LLM itself. In particular, hallucination concerning characters is an even deeper problem, as characters must possess correct knowledge about fictional facts that do not generally exist. Therefore, in order to generate utterances from characters that are based on correct knowledge derived from the character, unique improvements other than improving the performance of LLM itself are necessary.

[0010] The present invention has been made in view of the above, and aims to provide a learning device, a learning method, and a learning program that can realize a character- or individual-specific dialogue model that generates utterances consistent with the character's or individual's knowledge while suppressing hallucination.

[0011] To solve the above-mentioned problems and achieve the objectives, the learning device according to the present invention is a learning device for learning a dialogue model specific to a character or individual, and is characterized by comprising: a storage unit that stores knowledge data specific to the character or individual, extracted from a dialogue log and / or question-and-answer collection that includes the utterances of the character or individual; and a learning unit that learns a dialogue model specific to the character or individual so as to generate utterances characteristic of the character or individual, using at least the knowledge data in the storage unit and the dialogue log and / or question-and-answer collection.

[0012] Furthermore, the learning method according to the present invention is a learning method performed by a learning device, which learns a dialogue model specific to a character or individual, wherein the learning device has a storage unit that stores knowledge data specific to the character or individual, extracted from a dialogue log and / or question-and-answer collection that includes the utterances of the character or individual, and the learning method is characterized by including at least a step of learning a dialogue model specific to the character or individual so as to generate utterances that are characteristic of the character or individual, using the knowledge data in the storage unit and the dialogue log and / or question-and-answer collection.

[0013] Furthermore, the learning program according to the present invention causes a computer to perform the step of learning a dialogue model specific to the character or individual so as to generate utterances that are characteristic of the character or individual, using at least knowledge data specific to the character or individual extracted from a dialogue log and / or question-and-answer collection that includes the utterances of the character or individual, and the dialogue log and / or question-and-answer collection.

[0014] According to the present invention, it is possible to realize a character- or individual-specific dialogue model that generates utterances consistent with the character's or individual's knowledge while suppressing hallucination.

[0015] Figure 1 is a diagram showing an example of the configuration of a learning device according to an embodiment. Figure 2 is a diagram illustrating the flow of the learning process according to an embodiment. Figure 3 is a flowchart showing the processing steps of the learning process according to an embodiment. Figure 4 is a diagram showing an example of a prompt. Figure 5-1 is a diagram showing an example of training data. Figure 5-2 is a diagram showing an example of a prompt. Figure 6 is a diagram illustrating the flow of the inference process in an embodiment. Figure 7 is a flowchart showing the processing steps of the inference process in an embodiment. Figure 8 is a diagram showing the results of an evaluation experiment that evaluated reproducibility. Figure 9 is a diagram showing the results of an evaluation experiment that evaluated diversity. Figure 10 is a diagram showing the results of an evaluation experiment that evaluated hallucination. Figure 11 is a diagram showing the results of an evaluation experiment that evaluated the credibility of the character. Figure 12 is a diagram summarizing the relative evaluation scores for reproducibility, diversity, hallucination, and credibility of the character in single-turn responses. Figure 13 is a diagram showing examples of response utterances for each LLM. Figure 14 is a diagram showing the evaluation results for multi-turn responses. Figure 15 is a diagram showing an example of a computer in which a learning device is realized when a program is executed.

[0016] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, in the drawings, identical parts are denoted by the same reference numerals.

[0017] [Embodiment] [Overview of the Embodiment] A character's or individual's knowledge may increase or change from past dialogues and experiences, but their speech policies, such as personality and catchphrases, do not change in the short term. In this embodiment, this point is taken into consideration, and the character's or individual's knowledge is separated from their speech policies. The character's or individual's speech policies are trained in an LLM (Dialogue Model), and the character's or individual's knowledge is incorporated into the LLM's prompts after training.

[0018] As a result, according to this embodiment, LLM can generate character-like utterances while utilizing the knowledge it has input. In other words, according to this embodiment, LLM can autonomously grow based on new knowledge and past dialogues of the character or individual, while maintaining consistency in the character's or individual's utterance policy. Thus, this embodiment solves the first and second problems and realizes character-like dialogue by LLM.

[0019] As an example of this embodiment, we will explain the case where a dialogue model specific to a character is trained, but it is not limited to characters; a dialogue model specific to an individual may also be trained. In other words, when training a dialogue model specific to an individual, one can simply replace "character" with "individual" and perform the same process.

[0020] [Learning Device] The learning device according to the embodiment will now be described. Figure 1 is a diagram showing an example of the configuration of the learning device according to the embodiment. The learning device learns a dialogue model specific to the character.

[0021] As shown in Figure 1, the learning device 10 includes a communication unit 11, a storage unit 12, and a control unit 13. Although not shown in Figure 1, the learning device 10 may also include an input unit such as a keyboard or mouse to receive input such as user operations. Furthermore, the learning device 10 may also include a display unit such as a display to show the generated dialogue model results and dialogue content to the user.

[0022] The communication unit 11 acquires dialogue information from external storage devices on the Internet and performs data communication for outputting generated dialogue models and fine-tuning models to external devices. The communication unit 11 is implemented using a NIC (Network Interface Card) or the like and controls communication via telecommunication lines such as a LAN (Local Area Network) or the Internet. The communication unit 11 can be connected to the network by wired or wireless connection as needed and can send and receive information bidirectionally.

[0023] The memory unit 12 stores data and programs used for various processes by the control unit 13, as well as various data acquired by the operation of the control unit 13. The memory unit 12 is implemented using semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or storage devices such as hard disks and optical discs. As shown in Figure 1, the memory unit 12 includes a response collection memory unit 121, an extraction LLM memory unit 122, a character knowledge database (DB) 123, a base LLM memory unit 124, a fine-tuning LLM memory unit 125, and a character-specific model memory unit 126.

[0024] The response collection storage unit 121 stores dialogue logs and / or question-and-answer sets (FAQs) that contain character utterances. The dialogue logs and / or question-and-answer sets are collected by the collection unit 1311. The response collection storage unit 121 stores response sets generated by the base LLM (described later) (first language model) and response sets generated by the fine-tuning LLM (described later).

[0025] The extraction LLM storage unit 122 stores arbitrary LLMs (extraction LLMs) for extracting character-related knowledge from dialogue logs and / or question-and-answer sets. Note that extraction LLMs can also include web services and APIs (Application Programming Interfaces) from which LLM responses can be obtained.

[0026] The character knowledge database 123 stores character-specific knowledge data extracted from dialogue logs and / or question-and-answer sets that contain the character's utterances. Character-specific knowledge data is, for example, data extracted by the knowledge extraction unit 1312.

[0027] The base LLM memory unit 124 stores the base LLM. The LLM is the learning base used when generating the fine-tuning LLM (described later) and the character-specific LLM (described later). For example, the base LLM memory unit 124 stores a casual conversation model learned using various knowledge information as the base LLM. Note that the casual conversation model referred to here is not limited to a specific model, and may include a model learned as a general-purpose language model or a model learned using a predetermined text.

[0028] The fine-tuning LLM storage unit 125 stores the fine-tuning LLM that has been fine-tuned by the fine-tuning unit 1313 (described later). The fine-tuning LLM is an LLM that has been fine-tuned from the base LLM using the character's dialogue log and / or question-and-answer collection.

[0029] The character-specific model memory unit 126 stores a character-specific model that can reproduce dialogue specific to a particular character. The character-specific model memory unit 126 is an LLM that has been trained by the policy optimization learning unit 1316 (described later).

[0030] The control unit 13 has an internal memory for temporarily storing programs and processing data that define various processing procedures of the learning device 10, and is realized by electronic circuits such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), and integrated circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). As shown in Figure 1, the control unit 13 has a learning unit 131 and an inference unit 132.

[0031] The learning unit 131 trains a character-specific dialogue model (character-specific LLM) that faithfully reproduces the character's responses. The learning unit 131 trains the character-specific LLM by combining the extraction of character-specific knowledge data from dialogue logs and / or question-answer sets with policy optimization using DPO (Direct Preference Optimization).

[0032] The inference unit 132 acquires the user's past dialogue history, and upon receiving the dialogue history, it searches the character knowledge DB 123 for character-specific knowledge data related to the input utterance and dialogue history. Based on the retrieved knowledge data and the character-specific LLM, it generates a character-like utterance in response to the input utterance. The inference unit 132 outputs the generated utterance as a response to the input utterance.

[0033] [Learning Unit] The learning unit 131 includes a collection unit 1311, a knowledge extraction unit 1312, a fine-tuning unit 1313, a base speech generation unit 1314, a fine-tuning speech generation unit 1315, and a policy optimization learning unit 1316.

[0034] The collection unit 1311 collects dialogue logs and / or question-and-answer sets that include character utterances. The collection unit 1311 collects dialogue logs and / or question-and-answer sets from the internet, information processing devices, storage media, etc. The dialogue logs and / or question-and-answer sets are, for example, text data. If the collection unit 1311 collects audio data that includes character utterances, it may perform speech recognition and convert the audio data into text data.

[0035] The knowledge extraction unit 1312 extracts character-specific knowledge data from the dialogue log and / or question-and-answer collection. The knowledge extraction unit 1312 stores the extracted knowledge data in the character knowledge DB 123. The knowledge extraction unit 1312 extracts character-specific knowledge sentences using the extraction LLM of the extraction LLM storage unit 122. By using the existing extraction LLM for knowledge extraction, the knowledge extraction unit 1312 constructs a database of knowledge data specialized only for the target character from the dialogue log and / or question-and-answer collection which contain various information.

[0036] The fine-tuning unit 1313 fine-tunes the base LLM using the character's dialogue log and / or question-and-answer collection, and obtains the fine-tuned LLM.

[0037] The base utterance generation unit 1314 inputs the dialogue history from the dialogue log and / or questions from the question-answer collection into the base LLM, and obtains responses to the input questions from the base LLM. The response sentences (second response candidates) obtained by the base utterance generation unit 1314 are used as training data for a character or individual-specific dialogue model by the policy optimization learning unit 1316. The base utterance generation unit 1314 collects the response sentences generated by the base LLM and obtains them as a base LLM response collection.

[0038] The fine-tuning speech generation unit 1315 inputs the dialogue history from the dialogue log and / or questions from the question-answer collection to the fine-tuning LLM, and obtains responses to the input questions from the fine-tuning LLM. The response sentences (third response candidates) obtained by the fine-tuning speech generation unit 1315 are used as training data for a character-specific dialogue model by the policy optimization learning unit 1316. The fine-tuning speech generation unit 1315 collects the response sentences generated by the fine-tuning LLM and obtains them as a base LLM response collection.

[0039] The policy optimization learning unit 1316 learns a character-specific LLM so as to generate character-like utterances, using at least the knowledge data from the character knowledge DB 123 and the dialogue log and / or question-and-answer collection.

[0040] The policy optimization learning unit 1316 uses, as learning data, one input prepared based on at least the knowledge data in the character knowledge DB 123, and a plurality of response candidates for one ranked input. The policy optimization learning unit 1316 obtains a character-specific LLM by adjusting the parameters of the base LLM so as to easily generate utterances with higher ranks in the learning data. The policy optimization learning unit 1316 adjusts the parameters of the base LLM based on an input and a plurality of response candidates corresponding thereto until a predetermined end condition is satisfied. The policy optimization learning unit 1316 may store the character-specific LLM with adjusted parameters in the storage unit 12, and may also output it to an external device that communicates with the communication unit 11.

[0041] [Inference unit] The inference unit 132 includes a knowledge search unit 1321, a prompt creation unit 1322, and a response generation unit 1323.

[0042] The knowledge search unit 1321 obtains the conversation history with the user so far, and when receiving an utterance input from the user, searches the character knowledge DB 123 for knowledge data related to the input utterance and the conversation history.

[0043] The prompt creation unit 1322 creates an input prompt for the character-specific LLM. The prompt creation unit 1322 instructs the generation of a character-like utterance for the input utterance, and creates an input prompt for the character-specific LLM including the knowledge data and conversation history retrieved by the knowledge search unit 1321.

[0044] The response generation unit 1323 inputs the input prompt to the character-specific LLM to cause the character-specific LLM to generate a character-like utterance for the input utterance. The response generation unit 1323 obtains the utterance output from the character-specific LLM. The response generation unit 1323 outputs the content of the obtained utterance to the user via the display of the learning device 10, an output device such as a speaker or display connected to the learning device 10, or an external device that communicates with the communication unit 11.

[0045] [Flow of Learning Process] Next, the flow of the learning process of the character-specific LLM will be described. FIG. 2 is a diagram for explaining the flow of the learning process according to the embodiment. FIG. 3 is a flowchart showing the processing procedure of the learning process according to the embodiment.

[0046] In the learning by the learning device 10, a dialogue log and / or Q&A set 121A including the utterances of the character, an extraction LLM 122B, and a base LLM 125C serving as a basis are prepared. The extraction LLM 122B targets not only any LLM but also web services and APIs from which responses of the LLM can be obtained.

[0047] In the learning device 10, a collection process (step S11 in FIG. 3) for collecting a dialogue log and / or Q&A set including the utterances of the character by the collection unit 1311, a knowledge extraction process ((1) in FIG. 2, step S12 in FIG. 3), fine-tuning ((2) in FIG. 2, step S13 in FIG. 3), base utterance generation process ((3) in FIG. 2, step S14 in FIG. 3), fine-tuning utterance generation process ((4) in FIG. 2, step S15 in FIG. 3), and policy optimization learning process ((5) in FIG. 2, step S16 in FIG. 3) are performed.

[0048] [Knowledge Extraction Process] In the learning device 10, the knowledge extraction unit 1312 performs a knowledge extraction process of obtaining character-specific knowledge data from the dialogue log and / or Q&A set 121A ((1) in FIG. 2). Specifically, a prompt instructing the extraction of knowledge specific to the character is given to the extraction LLM 122B, and knowledge specific to the character is extracted from the dialogue log and / or Q&A set 121A.

[0049] Figure 4 shows an example of a prompt. The prompt for the extraction LLM 122B includes, as shown in prompt P1 of Figure 4, "an instruction to extract knowledge about the character from the dialogue log or question and answer" and "actual character dialogue or response data". The knowledge extraction unit 1312 inputs this prompt to the extraction LLM 122B. As a result, the knowledge extraction unit 1312 obtains text extracted and generated from the dialogue log and / or question and answer collection 121A by the extraction LLM 122B, and stores the obtained text in the character knowledge DB 123 as knowledge data specific to the character. The knowledge extraction unit 1312 may perform these processes multiple times depending on the amount of dialogue log, etc., to be input.

[0050] Furthermore, after acquiring new knowledge data, the knowledge extraction unit 1312 also checks the accumulated character knowledge DB 123 for inconsistent knowledge. Here, the input prompt instructs the LLM to "extract only inconsistent text from the following list of texts." The knowledge extraction unit 1312 resolves inconsistencies by deleting older knowledge from the extracted text.

[0051] In this way, the knowledge extraction process takes the dialogue log and / or question-and-answer collection 121A and the extraction LLM 122B as input and outputs the character knowledge DB 123.

[0052] [Fine-tuning process] The fine-tuning unit 1313 fine-tunes the base LLM 125C using the character dialogue log and / or question-and-answer collection 121A and obtains the fine-tuned LLM 124D.

[0053] The learning method can employ any commonly used fine-tuning technique for the LLM. In addition to a method that fine-tunes the entire LLM, a method that learns only the additional model as an adapter can also be employed as a fine-tuning technique. The fine-tuned LLM124D obtained here is used in the speech generation process in the fine-tuning speech generation process (Figure 2 (4)).

[0054] [Base Utterance Generation Process] The base utterance generation unit 1314 obtains response sentences to questions in the dialogue log and / or question-answer collection 121A from the base LLM 125C. Here, the response sentences obtained from the base LLM 125C are used as training data for the policy optimization learning process (Figure 2 (5)).

[0055] The prompt to the base LLM125C includes instructions to generate the next utterance or answer to a question, including the dialogue history from the dialogue log and questions from the question-answer set. Here, if there is obvious information such as the character's name or speaking style (for example, always ending sentences with "pyon!"), it can be included in the input prompt. The response obtained in the base utterance generation process is merely the result generated by the base LLM125C. Therefore, the response sentence obtained in the base utterance generation process does not necessarily have to be in line with the character's personality. Also, in the base utterance generation process, the LLM hyperparameters for the response can be set arbitrarily.

[0056] In this way, the base speech generation process takes the dialogue log and / or question-answer collection 121A and base LLM 125C as input and outputs the base LLM response collection 126E obtained from base LLM 125C.

[0057] [Fine-tuning speech generation process] The fine-tuning speech generation unit 1315 obtains response sentences to questions in the dialogue log and / or question-answer collection 121A from the fine-tuning LLM 124D. The prompts for the fine-tuning LLM 124D are the same as the prompts used in the base speech generation process.

[0058] In this way, the fine-tuning speech generation process takes the dialogue log and / or question-answer collection 121A and the fine-tuning LLM 124D as input and outputs the fine-tuning LLM response collection 127F obtained from the fine-tuning LLM 124D.

[0059] [Policy Optimization Learning Process] In the policy optimization learning process, the policy optimization learning unit 1316 takes the dialogue log and / or question and answer collection 121A, the knowledge data from the character knowledge DB 123, the base LLM response collection 126E, the fine-tuning LLM response collection 127F, and the base LLM 125C as input and outputs the character-specific LLM 128G.

[0060] The policy optimization learning unit 1316 learns the base LLM 125C using the dialogue log and / or question and answer set 121A, the knowledge data from the character knowledge DB 123, the base LLM response set 126E, and the fine-tuning LLM response set 127F, and obtains the character-specific LLM 128G.

[0061] Here, a policy-optimized learning method (for example, DPO (Reference 1)) is adopted as the learning method for the character-specific LLM128G. Reference 1: Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, 2023, [Retrieved December 10, 2024], Internet <URL: https: / / arxiv.org / abs / 2305.18290>.

[0062] In DPO, two types of outputs, A and B, are prepared for a given input. The system then determines which output, A or B, is more desirable. For example, if output B is preferred over A, then A and B are classified as preferred and rejected data, respectively, as in "chosen: B, rejected: A".

[0063] The policy optimization learning unit 1316 uses the input-preference-dislike triple data as a set of training data to train the LLM so that it is easier to generate an output that is closer to the preference data.

[0064] When DPO is applied, the most preferred utterances are those actually used by the character in the dialogue log and / or question-and-answer set. The least preferred responses are those from the base LLM125C. The fine-tuned LLM124D responses fall somewhere in between.

[0065] The training data will now be explained. The training unit 131 generates a set of training data in the following three patterns from a single dialogue history by performing the above processing: chosen: actual character response, rejected: base LLM response chosen: actual character response, rejected: fine-tuning LLM response chosen: fine-tuning LLM response, rejected: base LLM response

[0066] Figure 5-1 shows an example of training data. From the knowledge data of the character knowledge DB 123, the actual response sentence R1 of a character to a certain input (for example, a question) is designated as the first response candidate. From the response sentences generated by inputting the same single input to the base LLM 125C, i.e., the base LLM response collection 126E, the response sentence R2 for the same single input is designated as the second response candidate. From the response sentences generated by inputting the same single input to the fine-tuning LLM 124D, i.e., the fine-tuning LLM response collection 127F, the response sentence R3 for the same single input is designated as the third response candidate. These response candidates are used as a set of training data.

[0067] As shown in Figure 5-1, the policy optimization learning unit 1316 obtains a character-specific LLM 128G by adjusting the parameters of the base LLM 125C using training data in which the first response candidate, response sentence R1, is ranked 1st, the third response candidate, response sentence R3, is ranked 2nd, and the second response candidate, response sentence R2, is ranked 3rd.

[0068] The reason for including the fine-tuned LLM124D response here is to avoid the creation of a model that cannot generate desired character-like utterances due to the learning of unintended policies when there is a large discrepancy between the base LLM125C response and the actual character utterance response. When the discrepancy between the base LLM125C response and the actual character utterance response is small, it is not necessarily required to use the fine-tuned LLM124D response, and the implementer may decide whether or not to adopt it at their discretion.

[0069] The policy optimization learning unit 1316 then uses a response generation prompt as the input prompt used for learning. Here, the policy optimization learning unit 1316 also includes knowledge data from the character knowledge DB 123 in the input prompt used for learning.

[0070] Figure 5-2 shows an example of a prompt. As shown in prompt P2 in Figure 5-2, the policy optimization learning unit 1316 includes knowledge data from the character knowledge DB 123 (for example, "I like thunder andagi") as reference knowledge, in addition to the prompt used in the base utterance generation process, as input prompts used for learning.

[0071] The knowledge added here is actually obtained from the dialogue log or question-and-answer set 121A. For example, if four pieces of knowledge, a, b, c, and d, can be extracted from question-and-answer α, the policy optimization learning unit 1316 will include these four pieces of knowledge, a, b, c, and d, in the input prompt where α is the dialogue history in the learning data. This allows the policy optimization learning unit 1316 to learn the character-specific LLM 128G so that it can generate response utterances by referring to the knowledge in the prompt during policy optimization learning.

[0072] The LLM to be trained as a character-specific LLM128G may either train the entire LLM model (base LLM125C) or employ a method that trains only the additional model as an adapter. Similarly, any training method can be applied to the DPO as long as it uses the same input format.

[0073] The policy optimization learning unit 1316 adjusts the parameters of the base LLM 125C for each set of training data to make it easier to generate higher-ranking utterances until a predetermined termination condition is met. Then, by adjusting the parameters of the base LLM 125C for all sets of training data to be learned, the character-specific LLM 128G is obtained. In this way, the policy optimization learning unit 1316 obtains a character-specific LLM 128G that can generate more character-like utterances by performing policy optimization learning to generate actual character utterances rather than conventional LLM utterances.

[0074] [Inference Processing] Next, the flow of inference processing using the character-specific LLM128G will be explained. Figure 6 is a diagram illustrating the flow of inference processing in the embodiment. Figure 7 is a flowchart showing the processing steps of inference processing in the embodiment. In the embodiment, in actual speech generation, the extracted character-specific knowledge data is incorporated as text into the input prompt of the character-specific LLM128G. As a result, in the embodiment, the character-specific LLM128G generates character-like speech while utilizing the input knowledge data.

[0075] The knowledge retrieval unit 1321 acquires the user's dialogue history and, upon receiving utterance input from the user (step S21 in Figure 7), performs a knowledge retrieval process to search for knowledge data related to the input utterance and dialogue history from the character knowledge DB 123 ((1) in Figure 6, step S22 in Figure 7). The knowledge retrieval unit 1321 acquires knowledge data related to the input utterance and dialogue history from the character knowledge DB 123.

[0076] Here, the knowledge retrieval unit 1321 may employ any method as a method for retrieving knowledge about the character. For example, the knowledge retrieval unit 1321 may use a method that uses vector embedding, such as RAG (Reference 2). The knowledge retrieval unit 1321 may also use a method that calculates the similarity between the user's input utterance and the question portion of the question and answer set in an FAQ search format, and uses the knowledge extracted from the question and answer with the highest similarity. Reference 2: Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, Douwe Kiela, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, 2020, [Retrieved December 10, 2020], Internet <URL: https: / / arxiv.org / abs / 2005.11401>.

[0077] The prompt creation unit 1322 performs a prompt creation process to create an input prompt for the character-specific LLM 128G (Figure 6 (2), step S23 in Figure 7). The prompt creation unit 1322 instructs the generation of character-like utterances in response to the input utterance and creates an input prompt for the character-specific LLM 128G that includes knowledge data retrieved by the knowledge retrieval unit 1321 and the dialogue history. The input prompt for the character-specific LLM 128G may be in the same format as prompts P1 and P2 used in the learning process, and may also include basic information such as the character's name in the prompt.

[0078] The response generation unit 1323 inputs the input prompt to the character-specific LLM 128G and performs a response generation process that causes the character-specific LLM 128G to generate a character-like utterance in response to the input utterance (Figure 6 (3), step S24 in Figure 7). Here, the hyperparameters for the response may be set arbitrarily.

[0079] The response generation unit 1323 outputs the response utterance output from the character-specific LLM 128G to the user (step S25 in Figure 7).

[0080] [Evaluation Experiment] An evaluation experiment was conducted to assess the performance of the character-specific LLM128G learned by the learning device 10.

[0081] For the dataset, if the subject is an anime character, we use QA (question and answer) created through role-playing by fans of the character. If the subject is a real person (famous person), we use QA collected through interviews with the person themselves. In the evaluation experiment, "CK" refers to character knowledge created from QA, and "CP" refers to character profiles similarly created from QA.

[0082] The base LLM used in the experiment was Elyza / Llama-3-ELYZA-JP-8B (Reference 3). During model training, 90% of the QA was used as training data, 5% as validation data, and the remaining 5% as test data. Reference 3: Masato Hirakawa, Shintaro Horie, Tomoaki Nakamura, Daisuke Oba, Sam Passaglia, and Akira Sasaki, “elyza / Llama-3-ELYZA-JP-8b”, 2024.

[0083] In the inference process, data retrieval was performed using vector search and keyword search. First, the similarity between the input question and the question from the QA was calculated based on the similarity of embeddings between questions, created by the Multilingual E5 model (Reference 4). If the similarity exceeded a threshold (h=0.9), knowledge data extracted from matching answers in the QA was input to the prompt. If the similarity did not exceed the threshold, a keyword search using BM25 (Reference 5) was performed on the input question and the QA answer, and the top 10 pieces of knowledge were input to the prompt. Reference 4: Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei, “Multilingual E5 Text Embeddings: A Technical Report”, arXiv preprint arXiv:2402.05672, 2024a. Reference 5: Stephen Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford, “Okapi at TREC-3”, 1994.

[0084] For comparison, the following LLMs have been prepared. The base of all LLMs used in these methods is the same LLM.

[0085] [CP] is an LLM created using only the QA character profile for the input prompt.

[0086] [SFT] is a fine-tuned LLM that is fine-tuned using only QA character profiles and QA. However, [SFT] does not learn QA character knowledge (CK).

[0087] [DPO] is an LLM optimized using DPO, but does not learn character knowledge created from QA. [DPO] focuses solely on character profiling and QA for policy optimization.

[0088] [CP+CK] corresponds to extending the character profile (CP) prompts by adding character knowledge (CK) acquired via RAG.

[0089] [SFT+CK] represents a model trained using a fine-tuning LLM, corresponding to prompts that include character profiles (CP) and character knowledge (CK).

[0090] [DPO+CK] corresponds to the character-specific LLM128G learned by the learning device 10. The input prompts for the character-specific LLM128G include both character profiles (CP) and character knowledge (CK).

[0091] First, we evaluated the single-turn response in each LLM in terms of reproducibility, diversity, hallucination, and character authenticity.

[0092] Reproducibility was assessed by calculating the similarity between the generated response and a reference response from QA test data. The assessment was performed using BLEU (Reference 6) and BERTScore (Reference 7) metrics. Reference 6: Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu, “BLEU: a Method for Automatic Evaluation of Machine Translation”, In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics, 2002. Reference 7: Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q.Weinberger, and Yoav Artzi, “BERTSCORE: EVALUATING TEXT GENERATION WITH BERT”, In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.

[0093] Figure 8 shows the results of an evaluation experiment to assess reproducibility. As shown in Figure 8, [DPO] had low reproducibility, while [SFT] performed best. The [DPO+CK] (frame W1) of the embodiment showed high reproducibility, coming very close to [SFT].

[0094] Next, we will discuss diversity. The evaluation metric is distinct-N (Reference 8). Distinct-N is a metric that measures the diversity of text by calculating the ratio of any given n-gram to the total number of n-grams in the generated text.

[0095] Figure 9 shows the results of an evaluation experiment that assessed diversity. As shown in Figure 9, the [DPO+CK] (frame W2) of the embodiment showed the best results in terms of diversity. Reference 8: Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, “A Diversity-Promoting Objective Function for Neural Conversation Models”, In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 110–119, 2016a.

[0096] Next, we will explain hallucination. Hallucination refers to the phenomenon where LLM (Limited Literacy Model) generates inaccurate responses or data despite being inconsistent with the facts or given context. We define it as an instance where the character generates a response that differs from the knowledge or experience they were originally intended to possess.

[0097] Figure 10 shows the results of an evaluation experiment that assessed hallucination. As shown in Figure 10, the character-specific LLM [DPO+CK] (frame W3) of this embodiment showed the lowest hallucination.

[0098] This section explains character authenticity (credibility). Character authenticity is defined as the user's ability to feel as if they are interacting with a specific character through dialogue.

[0099] In the evaluation experiment, to ensure that the LLM could fairly assess the credibility of the characters, the evaluation involved having the LLM provide detailed descriptions of the characters and correcting any inaccuracies. Next, as external knowledge, information from Wikipedia® and QA was obtained to define the credibility of the characters. Based on this information, an evaluation was conducted with a maximum score of 10 points.

[0100] Figure 11 shows the results of an evaluation experiment that assessed the credibility of the characters. As shown in Figure 11, [DPO+CK] (frame W4) in this embodiment received the second highest evaluation after [DPO].

[0101] Figure 12 summarizes the relative evaluation scores for reproducibility, variability, hallucination, and character credibility in single-turn responses.

[0102] In this embodiment, [DPO+CK] combines DPO to improve character authenticity with character knowledge to reduce hallucination while increasing reproducibility. As a result, as shown in Figure 12, [DPO+CK] can generate high-quality responses in terms of reproducibility, diversity, hallucination, and character authenticity, providing a more balanced response compared to using either method alone.

[0103] Figure 13 shows examples of response utterances for each LLM. As shown in Figure 13, the [DPO+CK] (frame W5) of the embodiment can generate a response that effectively captures the character's distinct characteristics and qualities, despite the short response time, thanks to the character's knowledge.

[0104] Next, we evaluated multi-turn responses. For the multi-turn response evaluation, we used ChatGPT®. Each method engaged in a 5-turn dialogue with ChatGPT-4omini, and the dialogue was evaluated using ChatGPT-4o. The evaluation metrics were Naturalness (Natu), Consistency (Cons), Friendliness (Appr), and Character Authenticity (Char).

[0105] Figure 14 shows the evaluation results for multi-turn responses. As shown in Figure 14, [DPO+CK] (frame W6) in this embodiment achieved the highest value in three out of four evaluation indicators. Therefore, [DPO+CK] can generate high-quality dialogue while enhancing character credibility in multi-turn responses, just as it does in single-turn responses.

[0106] [Effects of the Embodiment] In this embodiment, by separating the character's speech model and the character's knowledge data, the LLM can generate character-like speech while utilizing the input knowledge. According to this embodiment, consistency between character learning and the character's speech policy is achieved.

[0107] Here, "character-like speech" refers to elements such as the character's tone of voice and basic personality, which are unlikely to change in the short term. Therefore, even if new data is added, there is no need to retrain the character-specific LLM each time. On the other hand, regarding character knowledge, new knowledge data is acquired and stored as soon as the dialogue log is accumulated, and by including this in the input prompts of the character-specific LLM, it becomes possible to generate character-like speech that immediately reflects the added knowledge data.

[0108] Furthermore, in this embodiment, by using character-specific knowledge data for hallucination, it becomes possible to generate utterances that are consistent with the character's specific knowledge. Therefore, according to the character-specific LLM of this embodiment, it becomes possible to generate utterances that are consistent with the character's knowledge and suppress hallucination.

[0109] As shown in the results of the evaluation experiment described above, the character-specific LLM learned by the learning device 10 according to the embodiment generates utterances consistent with the character's knowledge while suppressing hallucination, thereby realizing dialogue that reproduces the character. Furthermore, the character-specific LLM according to the embodiment can generate high-quality dialogue that suppresses hallucination while enhancing the credibility of the character, in both single-turn and multi-turn scenarios.

[0110] The character-specific LLM utterances described in this embodiment can be used in a variety of applications, including commercial dialogue systems and generative AI (Artificial Intelligence) that automatically generates non-player characters in the metaverse space using only text.

[0111] Furthermore, if there are multiple characters, the learning device 10 can learn a character-specific LLM for each character, and then select the character-specific LLM of the character designated by the user as the conversation partner from among the learned character-specific LLMs and perform inference processing.

[0112] [Regarding the System Configuration of the Embodiment] Each component of the learning device 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific forms of distribution and integration of the functions of the learning device 10 are not limited to those shown in the figure, and all or part of it can be configured by functionally or physically distributing or integrating in any unit according to various loads and usage conditions.

[0113] Furthermore, each process performed in the learning device 10 may be implemented in whole or in part by a CPU, a GPU (Graphics Processing Unit), and a program that is analyzed and executed by the CPU and GPU. Alternatively, each process performed in the learning device 10 may be implemented as hardware using wired logic.

[0114] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated may be changed as appropriate unless otherwise specified.

[0115] [Program] Figure 15 shows an example of a computer in which the learning device 10 is realized when a program is executed. The computer 1000 has, for example, memory 1010 and CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These parts are connected by a bus 1080.

[0116] Memory 1010 includes ROM 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0117] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the learning device 10 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing the same processes as the functional configuration of the learning device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0118] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0119] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0120] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in these embodiments. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on these embodiments are included in the scope of the present invention.

[0121] 10 Learning device 11 Communication unit 12 Memory unit 13 Control unit 121 Response collection memory unit 122 Extraction LLM memory unit 123 Character knowledge DB 124 Base LLM memory unit 125 Fine-tuning LLM memory unit 126 Character-specific model memory unit 131 Learning unit 132 Inference unit 1311 Collection unit 1312 Knowledge extraction unit 1313 Fine-tuning unit 1314 Base speech generation unit 1315 Fine-tuning speech generation unit 1316 Policy optimization learning unit 1321 Knowledge retrieval unit 1322 Prompt creation unit 1323 Response generation unit

Claims

1. A learning device for learning a dialogue model specific to a character or individual, comprising: a storage unit for storing knowledge data specific to the character or individual, extracted from a dialogue log and / or question-and-answer collection that includes the utterances of the character or individual; and a learning unit for learning a dialogue model specific to the character or individual, using at least the knowledge data in the storage unit and the dialogue log and / or question-and-answer collection to generate utterances characteristic of the character or individual.

2. The learning device according to claim 1, characterized in that the learning unit adjusts the parameters of the dialogue model to facilitate the generation of higher-ranked utterances, using at least the knowledge data of the memory unit, one input prepared based on the dialogue log and / or question-answer collection, and a plurality of ranked response candidates for the one input.

3. The learning device according to claim 2, wherein the learning unit sets together one input of knowledge data from the memory unit, a first response candidate from the knowledge data of the memory unit which is the actual response of the character or individual to the one input, a second response candidate generated by inputting the one input into a first language model, and a third response candidate generated by inputting the one input into a second model obtained by fine-tuning the first language model based on the dialogue log and / or question-and-answer collection, and adjusts the parameters of the dialogue model using training data in which the first response candidate is ranked 1st, the third response candidate is ranked 2nd, and the third response candidate is ranked 2nd.

4. The learning device according to claim 2, characterized in that the learning unit includes the knowledge data of the memory unit as reference knowledge in the prompts during the learning of the dialogue model.

5. The learning device according to claim 1, comprising: a search unit that searches for knowledge data related to input utterances and dialogue history from the memory unit; a prompt creation unit that instructs the dialogue model to generate utterances that are characteristic of the character or individual in response to the input utterances, and creates an input prompt for the dialogue model that includes the knowledge data retrieved by the search unit and the dialogue history; and a response generation unit that inputs the input prompt to the dialogue model, obtains the utterances output from the dialogue model, and outputs them.

6. A learning method performed by a learning device for learning a dialogue model specific to a character or individual, wherein the learning device has a storage unit that stores knowledge data specific to the character or individual, extracted from a dialogue log and / or question-and-answer collection that includes the utterances of the character or individual, and the learning method is characterized by including at least a step of learning a dialogue model specific to the character or individual so as to generate utterances characteristic of the character or individual using the knowledge data in the storage unit and the dialogue log and / or question-and-answer collection.

7. A learning program that causes a computer to perform the step of learning a dialogue model specific to the character or individual so as to generate utterances characteristic of the character or individual, using at least the dialogue logs and / or question-and-answer sets, and knowledge data specific to the character or individual extracted from dialogue logs and / or question-and-answer sets containing the utterances of the character or individual.

Images