Generation device, generation method, and generation program

Abstract

This generation device (10) comprises: a generative large language model (LLM) (11) that generates a sentence; and a checking unit (12) that, at a prescribed checking timing, checks whether the sentence generated by the generative LLM (11) includes harmful content.

Description

Generation Device, Generation Method, and Generation Program

[0001] The present invention relates to a generation device, a generation method, and a generation program.

[0002] With the development of computer technology and AI (Artificial Intelligence) research and the use of large corpora, the performance of large language models (LLMs) has been greatly improved, and their application to various tasks such as customer service, chat systems, and sentiment analysis has been rapidly advancing.

[0003] Prompt Guard, [Searched on December 13, 2024], Internet <https: / / github.com / meta-llama / PurpleLlama / blob / main / Prompt-Guard / MODEL_CARD.md> Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Qi Zhang, Xuanjing Huang, “Making Harmful Behaviors Unlearnable for Large Language Models”, arXiv:2311.02105v1 [cs.LG] 2 Nov 2023. Charles O'Neill, Jack Miller, Ioana Ciuca, Yuan-Sen Ting, Thang Bui, “Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Content”, arXiv:2308.13768v1 [cs.CL] 26 Aug 2023.

[0004] As LLMs improve in performance and their social application progresses, an issue called jailbreak, which causes LLMs to produce outputs that are ethically and safely undesirable, has come to be noticed.

[0005] Measures to defend against jailbreaks include prompt guards that detect and block jailbreak prompts in advance, and approaches that create robust LLMs through fine-tuning, but each of these approaches still has challenges in terms of computational cost and accuracy.

[0006] The present invention has been made in view of the above, and aims to provide a generating device, a generating method, and a generating program that can prevent harmful outputs from LLM.

[0007] To solve the above-mentioned problems and achieve the objective, the generation device according to the present invention is characterized by comprising: a first natural language model for generating text; and a checking unit for checking whether the text generated by the first natural language model contains harmful content at a predetermined checking timing.

[0008] Furthermore, the generation method according to the present invention is a generation method performed by a generation device, and is characterized by including a step of checking whether or not a text generated by a first natural language model that generates text contains harmful content at a predetermined check timing.

[0009] Furthermore, the generation program according to the present invention causes a computer to perform the step of checking whether or not the text generated by a first natural language model that generates text contains harmful content at a predetermined check timing.

[0010] According to the present invention, harmful outputs from the LLM can be prevented.

[0011] Figure 1 shows an example of the configuration of a generation device according to an embodiment. Figure 2 is a diagram illustrating the generation process of the generation device shown in Figure 1. Figure 3 is a diagram illustrating the processing procedure of the generation method according to an embodiment. Figure 4 is a diagram showing the input and output of the generated LLM. Figure 5 is a diagram illustrating the pauses in the generated LLM. Figure 6 is a diagram illustrating the pauses in the generated LLM. Figure 7 shows the results of a preliminary experiment. Figure 8 shows an example of a generated sentence. Figure 9 shows the experimental results. Figure 10 shows the experimental results. Figure 11 shows the experimental results. Figure 12 shows the results of an evaluation experiment. Figure 13 shows an example of a computer in which the generation device is realized when a program is executed.

[0012] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0013] [Embodiment] [Prerequisites] Before describing this embodiment, the prerequisites will be explained.

[0014] [LLM Problem Setting] First, we formulate the pre-forgetting state of LLM. We define it as follows: Vocabulary set: V Input token sequence: x∈V N Output token sequence: y∈V M Furthermore, let N be the input token sequence length and M be the output token sequence length.

[0015] The problems that LLM solves can be formulated as shown in equation (1). In equation (1), θ is the model parameter of the LLM.

[0016]

[0017] In LLM, it is difficult to solve the problem in equation (1) exactly. Therefore, an approximate solution to equation (1) is generally generated using sampling techniques such as greedy sampling. The output solution obtained by greedy sampling satisfies equation (2).

[0018]

[0019] Equation (2) shows that when an LLM is given an input prompt and the (i-1)th tokens of the output token sequence, it outputs the token with the highest confidence level when outputting the i-th token.

[0020] [Problem Setting for Avoiding Harmful Content] Let's consider the problem setting of generating a token sequence while avoiding harmful content using LLM. Adding the constraint of avoiding harmful content to the LLM problem setting mentioned earlier, we can formulate it as shown in equation (3).

[0021]

[0022] However, F is a set of undesirable token sequences, which the user can define freely. This problem setting presents a problem with the constraint that the output token sequence of LLM must be generated in such a way that it does not fall into set F.

[0023] [Generation Apparatus] Next, a generation apparatus according to the embodiment will be described. Figure 1 is a diagram showing an example of the configuration of the generation apparatus according to the embodiment.

[0024] The generation device 10 according to this embodiment is a generation device that is realized by loading a predetermined program into a computer or the like, which includes ROM (Read Only Memory), RAM (Random Access Memory), CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The generation device 10 also has a communication interface for sending and receiving various information with other devices connected via a network or the like.

[0025] The generation device 10 includes a generation LLM 11 (first natural language model), a check unit 12 having a check LLM 121 (second natural language model), and a timing monitoring unit 13 (control unit).

[0026] The generation device 10 combines the generation LLM 11 and the check LLM 121, and when it detects harmful content, it attempts to avoid generating harmful content by reverting the generated text and regenerating it each time. Furthermore, the generation device 10 aims to improve the quality of the final generated text by timing the check LLM 121's check to coincide with the timing of the generation LLM 11's pause, and by performing the revert in units of pauses.

[0027] The generated LLM11 is the LLM to be checked. The generated LLM11 generates text according to the prompt.

[0028] The checking unit 12 checks whether the text generated by the generating LLM 11 contains harmful content. The checking unit 12 checks the text generated by the generating LLM 11 at a predetermined checking timing using the checking LLM 121. The checking LLM 121 is a trained LLM that determines whether the input text contains harmful content and outputs the determination result.

[0029] If the generated text contains harmful content, the checking unit 12 deletes the generated text up to the previous check, returns the text to the generating LLM 11, and causes the generating LLM 11 to generate the text again. If the generated text does not contain harmful content, the checking unit 12 returns the text to the generating LLM 11 as is, and causes the generating LLM 11 to generate the rest of the text.

[0030] The timing monitoring unit 13 controls the timing of the check performed by the check unit 12. The timing monitoring unit 13 causes the check unit 12 to check and correct the text generated by the generated LLM 11 at the pause timing, which is the timing when the output probability of the generated LLM 11 is lower than a predetermined threshold.

[0031] [Generation Method] Next, the generation method according to the embodiment will be described. Figure 2 is a diagram illustrating the generation process of the generation apparatus 10 shown in Figure 1. Figure 3 is a diagram showing the processing procedure of the generation method according to the embodiment.

[0032] As shown in Figure 2, the generating LLM 11 generates text according to the input prompt (arrow Y11 in Figure 2) (step S11 in Figure 3).

[0033] The timing monitoring unit 13 determines whether or not it is time to check the text generated by the generation LLM 11 (hereinafter referred to as the generated text) (step S12 in Figure 3).

[0034] If it is not the check timing (step S12: No. in Figure 3), the generation device 10 returns to step S11 and continues generating the document.

[0035] If it is a check timing (step S12 in Figure 3: Yes), the timing monitoring unit 13 inputs the generated text in step S11 to the check unit 12 (arrow Y12 in Figure 2, step S13 in Figure 3).

[0036] The checking unit 12 checks whether the generated text contains harmful content (step S14). The checking unit 12 provides feedback on whether it contains harmful content (NG) or not (OK) (arrow Y13 in Figure 2).

[0037] Specifically, if the document contains harmful content (Step S14 in Figure 3: Yes), the checking unit 12 deletes the generated text up to the previous check (Step S15 in Figure 3), returns the document to the generating LLM 11, and causes the generating LLM 11 to generate the text again (Step S11 in Figure 3).

[0038] If the generated text does not contain harmful content (Step S14: No in Figure 3) and the text generation is not yet complete (Step S16: No in Figure 3), the check unit 12 returns the generated text to the generation LLM 11, causing the generation LLM 11 to generate the next text (Step S11 in Figure 3).

[0039] If text generation is complete (step S16 in Figure 3: Yes), the check unit 12 outputs the generated text as the final output (arrow Y14 in Figure 2, step S17 in Figure 3).

[0040] The algorithm of the processing method shown in FIG. 2 is checked by the checking unit 12 as appropriate for the generated text by the generation LLM 11, and corrections are made as necessary to avoid including harmful content in the final generated text.

[0041] [Algorithm] The algorithm of the processing method shown in FIG. 2 can be formulated as in Equation (4).

[0042] First, the output token sequence of the generation LLM 11 is divided as in Equation (4).

[0043]

[0044] However, it is assumed that a check is inserted at the end of each token sequence S ｉ At this time, the algorithm can be interpreted as solving Equation (5).

[0045]

[0046] [Check Timing] Next, consider the check timing of the checking unit 12.

[0047] FIG. 4 is a diagram showing the input and output of the generation LLM 11. The case where "Tell me about Galileo Galilei." is input to the generation LLM 11 and an output of "He was an Italian physicist." is obtained will be described as an example. When "He was an Italian physicist." is divided into tokens, it becomes "He was an Italian phys ic ist.".

[0048] For example, if checking is done sentence by sentence, the entire sentence has to be regenerated, so the cost of rollback is large. Also, if checking is done token by token, there may be cases where correction is impossible even after rollback. For example, if rollback occurs at "He was an Italian physicist", regeneration will start from "He was an Italian phys ic". Therefore, it is necessary to consider at what timing to perform the check.

[0049] In this embodiment, we propose checking and correcting the timing when the generated LLM11 "takes a breath."

[0050] The timing of a breath in LLM refers to the point at which the LLM output probability for an output token becomes low. The LLM output probability is defined as the LLM output logit normalized by passing it through a softmax function.

[0051] Figures 5 and 6 illustrate the breathing process of the generating LLM. In Figures 5 and 6, we consider the case when generating LLM 11 outputs "He was an Italian physicist." The token sequence "physicist" can be divided into "phys / ic / ist". In this case, the output probability of "phys" from generating LLM 11 is low, while the output probabilities of "ic" and "ist" are high.

[0052] When outputting the token "phys," there are many options other than "phys," and the generating LLM11 is in a state of uncertainty about which token to output from several candidate tokens, so the probability of outputting the token "phys" is low (Figure 5). On the other hand, when it comes to tokens following "phys," there are fewer options, so the probability of the generating LLM11 outputting tokens such as "ic" or "ist" is high.

[0053] The situation in Figure 5 can be interpreted as the LLM being in a state of hesitation (taking a breath) because, when the input to LLM is "He was an Italian," there are many candidate tokens for the following word. In this example, looking at the output logits other than "phys" in LLM, it can be confirmed that "mathe / mat / ic / ian" and "astro / nomer" have a high probability of being output.

[0054] On the other hand, in Figure 6, for the next token generation after "phys" is confirmed ("ic" or "ist"), the output probability is very high, which can be interpreted as LLM not hesitating about the output (not taking a breath). This means that when "He was an Italian phys" is input, the output is almost certain.

[0055] Therefore, in this embodiment, at the breathing timing among the above timings, the checking unit 12 is instructed to check and correct the text generated by the generating LLM 11.

[0056] The pause timing is when the generating LLM 11 is undecided about which token to output from several candidate tokens, and it is considered that it is easier to make corrections when a constraint is imposed to prevent the output of a certain token. Therefore, in this embodiment, by having the check unit 12 check and correct the text generated by the generating LLM 11 at the pause timing, it is expected that the quality of the final generated text will be high while reducing costs. For this reason, in this embodiment, the process is executed assuming that the check and correction at the pause timing is appropriate.

[0057] [Experiment] An evaluation experiment of the generation device 10 was actually conducted.

[0058] [Preliminary Experiment] First, let's explain the preliminary experiment. We visualized when LLM pauses to breathe. The LLM used in the preliminary experiment was Mistral-7B-Instruct-v0.2, the prompt was "Tell me about Galileo Galilei," the output token length was set to 200, and the maximum value of the logit was plotted. Figure 7 shows the results of the preliminary experiment. Figure 8 shows an example of a generated sentence.

[0059] As shown in Figure 7, when the generated text was segmented at points where the output probability was below 0.8, as shown in Figure 8, there were no instances of the text being segmented in the middle of a word. Although this depends on the threshold setting, segmenting at an output probability of "0.8" results in a natural-sounding segmentation. Therefore, in evaluation experiments 1 and 2, the threshold for output probability was set to "0.8," and the timing at which the output probability was lower than "0.8" was used as the pause timing.

[0060] [Evaluation Experiment 1] In this experiment, the quality of the text obtained by the generation device 10 will be evaluated.

[0061] The metric used for quality evaluation is called Perplexity (PPL), which is defined as shown in equation (6).

[0062]

[0063] PPL is an index measured by inputting a token sequence x into LLMθ; the smaller the PPL, the more natural the text is considered to be for that LLM.

[0064] As a baseline to compete with the generator 10, the checks and corrections are performed on a per-token basis and on a per-sentence basis.

[0065] Furthermore, the input prompt is set to "Tell me about Galileo Galilei," and the content to be avoided is set to "Galileo Galilei." The checking unit 12 uses LLM's gpt-4o (registered trademark) to appropriately check whether the generated text is related to "Galileo Galilei."

[0066] The experimental results show the PPL and generated text for those generated by the generator 10 and those generated by the baseline. Figures 9 to 11 show the experimental results. Figure 9 shows the generated text after checking and correcting at the token level. Figure 10 shows the generated text after checking and correcting at the sentence level. Figure 11 shows the generated text by the generator 10.

[0067] As shown in Figure 9, it can be confirmed that checking and correcting at the token level significantly degrades the quality of the generated text. This is evident from the "GalileILEE" at the beginning of the sentence, which shows that it is difficult to correct at the token level, and this is the result of trying to avoid "Galileo Galilei" unnecessarily.

[0068] As shown in Figure 10, checking and correcting each sentence shows that PPL has significantly improved quality compared to checking each token individually, with a score of 5.12, and no particular problems are visible to the human eye.

[0069] Furthermore, as shown in Figure 11, the PPL achieved by the generation device 10 in breath-by-breath checks and corrections is 4.55, confirming that high-quality text can be generated while avoiding "Galileo Galilei".

[0070] [Evaluation Experiment 2] We evaluated whether the generation device 10 could generate high-quality text while avoiding harmful content.

[0071] The Helpful and Harmless (HH) dataset, a set of questions designed to elicit harmful content, was used as the dataset. Llama2-7B was used as the generation LLM11, and GPT-4o was used as the checking mechanism. The baseline was unconstrained LLM, with checking and correction for each sentence. The quality of the generated text was evaluated using loss. Texts generated by unconstrained LLM, texts generated with checking and correction for each sentence, and texts generated by the generation device 10 were evaluated using the GPT-4o harm check (LLM-as-a-Judge).

[0072] Figure 12 shows the results of the evaluation experiment. According to the generation device 10, the percentage of harmful content was very close to that of the generated text that was checked and corrected sentence by sentence, which had the lowest percentage of harmful content, and it was evaluated that harmful content could be avoided. In addition, it was evaluated that the generation device 10 could maintain a high level of text quality.

[0073] [Example] This embodiment can be applied, for example, to a Q&A system using LLM. The procedure of the embodiment will be described below.

[0074] First, the check unit 12 uses the check LLM 121, and the input prompt for the check LLM 121 is set as follows. Also, the breath threshold is set to τ. "Please determine whether the following text contains harmful content. Answer yes if it does, and no if it does not.\nText: {Generated text}"

[0075] In the text generation phase, text generation continues until the output probability of the generated LLM11 falls below τ (until a pause occurs) in response to the user query, and then the program moves to the harmfulness determination phase.

[0076] In the harmfulness determination phase, the check unit 12 is used to determine the harmfulness of the text generated by the generation LLM 11. If it is harmful, the text is deleted up to the last pause and the process moves to the text generation phase. If it is not harmful, the process moves directly to the text generation phase.

[0077] The two upper phases are continued until a text of a specified length is obtained, which is the final output. By applying this embodiment, the Q&A system can be effectively prevented from outputting harmful content.

[0078] [Effects of the Embodiment] As demonstrated in the evaluation experiment, the generation device 10 according to the embodiment combines the generation LLM 11 and the check LLM 121 to check for harmful content, and if harmful content is detected, it retrieves the generated text and regenerates it each time, thereby being able to avoid the generation of harmful content with high accuracy. Furthermore, the generation device 10 was able to improve the quality of the final generated text by setting the timing of the check LLM 121's check to coincide with the timing of the generation LLM 11's pause, and performing the retrieve in units of pauses.

[0079] The breathing timing was explained as the timing when the output probability of the generated LLM11 is lower than a predetermined threshold. This threshold can be fixed or varied.

[0080] For example, LLM may continue to generate meaningless words without being able to verify them. In this case, if the timing monitoring unit 13 determines that no pause timing occurs in the generated LLM 11 for a certain period of time, it raises the threshold by a predetermined value. Also, if the interval between pause timings in the generated LLM 11 is shorter than a predetermined period, the timing monitoring unit 13 determines that the generated LLM 11 is pausing too much and lowers the threshold by a predetermined value.

[0081] Thus, the timing monitoring unit 13 may vary the threshold value of the output probability for determining the breathing timing according to the output status of the LLM.

[0082] Furthermore, the timing monitoring unit 13 may switch the threshold value of the output probability for determining the timing of breaths depending on the field to which the generated text belongs.

[0083] The timing monitoring unit 13 sets the output probability threshold lower than the standard threshold in cases where the field is not included in the training data of the generating LLM 11 (for example, medicine, law, folk art, or languages ​​with few speakers). This is because in such fields, the generating LLM 11 may not answer with confidence, resulting in a situation where it is constantly pausing for breath. Conversely, the timing monitoring unit 13 sets the output probability threshold higher than the standard threshold in cases where the field is frequently included in the training data of the generating LLM 11 (for example, program code or English). In such fields, the generating LLM 11 answers with confidence, so the timing monitoring unit 13 raises the output probability threshold to reliably check and correct generated sentences with low output frequency.

[0084] The timing monitoring unit 13 may also estimate how well the generating LLM 11 is trained on the topic of conversation by observing the output of the generating LLM 11 and performing processing such as taking the average, to determine whether the topic of conversation is a topic that is frequently included in the training data of the generating LLM 11.

[0085] For example, the timing monitoring unit 13 monitors the output probability, and if the average output probability is 0.9, it determines that the topic being discussed is a topic that is frequently included in the training data of the generated LLM 11, and sets the output probability threshold to a value higher than the standard threshold (e.g., 0.8) or the average output probability (e.g., 0.9) (e.g., 0.95). Similarly, for example, the timing monitoring unit 13 monitors the output probability, and if the average output probability is 0.75, it determines that the topic being discussed is not a topic that is frequently included in the training data of the generated LLM 11, and sets the output probability threshold to a value lower than the standard threshold (e.g., 0.8) or the average output probability (e.g., 0.75) (e.g., 0.7).

[0086] [Regarding the System Configuration of the Embodiment] Each component of the generation device 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific forms of distribution and integration of the functions of the generation device 10 are not limited to those shown in the figure, and all or part of it can be configured by functionally or physically distributing or integrating in any unit according to various loads and usage conditions.

[0087] Furthermore, each process performed in the generation device 10 may be implemented in whole or in part by a CPU, a GPU (Graphics Processing Unit), and a program that is analyzed and executed by the CPU and GPU. Alternatively, each process performed in the generation device 10 may be implemented as hardware using wired logic.

[0088] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated may be changed as appropriate unless otherwise specified.

[0089] [Program] Figure 13 shows an example of a computer in which the generation device 10 is realized when a program is executed. The computer 1000 has, for example, memory 1010 and CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These parts are connected by a bus 1080.

[0090] Memory 1010 includes ROM 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0091] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the generation device 10 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing the same processes as the functional configuration in the generation device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0092] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0093] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0094] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in these embodiments. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on these embodiments are included in the scope of the present invention.

[0095] 10 Generating device 11 Generating LLM 12 Checking unit 13 Timing monitoring unit 121 Checking LLM

Claims

1. A generation device comprising: a first natural language model for generating text; and a checking unit for checking whether the text generated by the first natural language model contains harmful content at predetermined check timings.

2. The generation apparatus according to claim 1, characterized in that the checking unit performs a check on the text generated by the first natural language model using a second natural language model that determines whether or not the input text contains harmful content.

3. The generation apparatus according to claim 1, characterized in that, if the text generated by the first natural language model contains harmful content, the checking unit returns the text with the generated text up to the previous checking unit removed to the first natural language model, causing the first natural language model to generate text again, and if the text generated by the first natural language model does not contain harmful content, the checking unit returns the text generated by the first natural language model as is to the first natural language model, causing the first natural language model to generate subsequent text.

4. The generation apparatus according to claim 1, characterized in that the predetermined check timing is the timing at which the output probability of the first natural language model is lower than a predetermined threshold.

5. The generation apparatus according to claim 4, further comprising a control unit for controlling the check timing in the check unit, wherein the control unit varies the predetermined threshold according to the output status of the first natural language model and / or the field to which the generated text of the first natural language model belongs.

6. A generation method performed by a generation device, comprising the step of checking whether a text generated by a first natural language model for generating text contains harmful content at a predetermined check timing.

7. A generation program that causes a computer to perform the following step at a predetermined check timing to check whether the text generated by a first natural language model for text generation contains harmful content.

Images