Token sequence generation device, token sequence generation method, and program

Abstract

This token sequence generation device is for generating an output token sequence of a predetermined length by using a training data set Dpriv. The token sequence generation device includes a token sequence generation unit that executes a probability distribution correction process that includes: first processing for generating a prompt Spub containing an instruction inst and a token sequence -O, and a prompt Spriv (i) (i=1, 2, …, M) containing the instruction inst, the token sequence -O, and a set Dpriv (i) by using subsets Dpriv (1), Dpriv (2), …, Dpriv (M) of the training dataset Dpriv having training data with a label of -y as an element; and second processing for obtaining a token probability distribution pv (i) (i=1, 2, …, M) by correcting a token probability distribution pLLM(v|-O) output by a large language model.

Description

Token sequence generator, token sequence generation method, program

[0001] This invention relates to in-context learning techniques for large-scale language models, and more particularly to token generation techniques that can be used for generating input prompts.

[0002] Neural language models are known to remember training data. Therefore, in large language models (LLMs), there is a risk of training data leaking from the output during in-context learning (ICL), and membership estimation attacks are a real threat to this leakage risk. Here, ICL is a learning method that customizes the output for an input prompt containing instructions and one or more training data points. Training data refers to a pair of content and labels.

[0003] One technique to reduce this leakage risk is to add noise to the probability distribution of tokens output by the LLM. It should be noted that the tokens correspond to the content of the training data. However, simply adding noise to the probability distribution of tokens leads to a significant degradation in accuracy, where the LLM produces incorrect output when the number of candidate tokens is enormous. Therefore, the technique in Non-Patent Literature 1 restricts the vocabulary set whose elements are candidate tokens using instructions. This makes it possible for the technique in Non-Patent Literature 1 to reduce the impact of accuracy degradation due to noise addition.

[0004] Tang, X., Shin, R., Inan, HA, Manoel, A., Mireshghallah, F., Lin, Z., Gopi, S., Kulkarni, J., and Sim, R., “Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation,” The Twelfth International Conference on Learning Representations, 2024.

[0005] However, the technique of Non-Patent Document 1 does not consider the optimization of the probability distribution of tokens, and thus cannot sufficiently suppress accuracy degradation.

[0006] Accordingly, an object of the present invention is to provide a token generation technique that can be used for input prompt generation for in-context learning of large language models, which suppresses accuracy degradation due to noise addition to the token probability distribution by taking into account the optimization of the token probability distribution.

[0007] One aspect of the present invention is a token sequence generation device that uses a training data set D priv to generate an output token sequence of a predetermined length, the token sequence generation device comprising: an initialization unit that sets a token sequence ~O as a token sequence of length 0; and a training data set D that includes, as elements, training data whose label is ~y priv a subset D priv (1) , D priv (2) , ..., D priv (M) a first process of generating a prompt S including an instruction inst and the token sequence ~O pub and a prompt S including the instruction inst, the token sequence ~O, and the set D priv (i) (i=1, 2, ..., M), a probability distribution correction process that executes a second process of obtaining the token probability distribution p priv (i) (i=1, 2, ..., M) by correcting the token probability distribution p LLM (v|~O) output by the large language model v (i) (i=1, 2, ..., M), a noise addition process of obtaining the token probability distribution ^p v (i) by adding noise to the token probability distribution p v (i=1, 2, ..., M), and a set V that maximizes the sum of the probabilities p LLM (v|S pub ) among subsets of the vocabulary set V whose number of elements is k|V| or less (where k satisfies 0<k<1) pubA lexical set restriction process that generates the set V pub Among the tokens included, probability ^p v The token v that maximizes syn The token generation process to find the token v, and the token sequence ~O syn The process includes a token sequence update process that adds tokens to a token sequence, and a token sequence generation unit that repeats this process until the length of the token sequence ~O reaches the predetermined length, and uses the resulting token sequence ~O as the output token sequence.

[0008] According to the present invention, it is possible to generate tokens that suppress the degradation of accuracy due to the addition of noise to the probability distribution of the tokens.

[0009] This is a block diagram showing the configuration of the token sequence generator 100. This is a flowchart showing the operation of the token sequence generator 100. This is a diagram showing an example of the functional configuration of a computer that realizes each device in the embodiment of the present invention.

[0010] The embodiments of the present invention will be described in detail below. Components having the same function will be numbered identically, and redundant explanations will be omitted.

[0011] Before describing each embodiment, the notation methods used in this specification will be explained.

[0012] ^ (caret) represents a superscript. For example, x y^z is y z is a superscript to x, and x y^z is y z This indicates that it is a subscript for x. Also, _ (underscore) represents a subscript. For example, x y_z is y z is a superscript to x, and x y_z is y z This indicates that it is a subscript for x.

[0013] The superscripts "^" and "~" such as ^x and ~x for a given character x should ideally be placed directly above "x", but due to the constraints of notation in the specification, they are written as ^x and ~x.

[0014] <Technical Background> As mentioned above, in ICL, training data is presented as an example in the prompts, so there is a risk that the training data presented as an example may be leaked from the output of the LLM. To reduce this risk, Non-Patent Literature 1 proposes DP-ICL, which implements differential privacy (DP) in ICL. In DP-ICL, noise is added to the probability distribution of tokens, which are the output of the LLM, so there is a trade-off between accuracy degradation and security. However, Non-Patent Literature 1 only experimentally verifies the effect of DP-ICL on suppressing accuracy degradation, and did not clarify the theoretical background of the effect of noise on ICL.

[0015] Therefore, this paper will theoretically clarify the effect of noise on ICL by performing a Bayes analysis on DP-ICL, and then explain the algorithm PTA (Plausible Token Amplification) derived from the insights of the analysis results. Using PTA, it is possible to improve the output accuracy of LLM in DP-ICL.

[0016] First, we will explain the analysis of ICL based on Bayesian analysis, following Reference Non-Patent Document 1. (Reference Non-Patent Document 1: Xie, SM, Raghunathan, A., Liang, P., and Ma, T., “An Explanation of In-Context Learning as Implicit Bayesian Inference,” The Tenth International Conference on Learning Representations, 2022.) Reference Non-Patent Document 1 is interpreted as “ICL customizes the output of LLM to conform to the exemplified training data by performing Bayesian inference of concepts.” Here, a concept is a pattern that is considered to be common to multiple training data exemplified in a prompt. For example, for a prompt containing training data such as “Albert Einstein was German,” “Mahatma Gandhi was India,” etc., a pair with “Name” as content and “Place of Origin” as label is inferred as the concept “Name to Nationality.” As a result, when “Curie was” is input to LLM, LLM will output “Polish.”

[0017] Furthermore, in reference non-patent document 1, regarding the concept, "the larger the log-likelihood ratio of the training data O in the following equation, the more correct the ICL concept θ is." * This demonstrates the proposition that "it is possible to infer that..." Here, p prompt θ represents the probability distribution for generating prompts. Also, θ is the concept θ. * It represents a different concept.

[0018] This proposition is the concept θ in ICL. * This demonstrates that training data specific to this model is useful in terms of improving accuracy.

[0019] Following the example of Non-Patent Document 1, an analysis of DP-ICL reveals that similar propositions concerning the concept also hold for DP-ICL. Furthermore, the following inequality can be obtained, which guarantees that the concept can be correctly inferred in DP-ICL. Here, the right-hand side F includes a term relating to the noise error per token and a term relating to the estimation error per token.

[0020] As can be seen from equation (2), if we can make a modification to increase equation (1) on the left side of the probability distribution of tokens, which is the distribution for sampling tokens, then the inequality in equation (2) becomes more likely to hold.

[0021] Below, we consider a method for generating token sequences based on the above suggestions. To do this, we will examine the correct concept θ common to the tokens included in the token sequence. * The optimization problem to maximize the log-likelihood ratio of can be formulated as follows: Here, p v V represents the probability distribution of the token. pub ~O represents a subset of the lexical set V obtained by a predetermined method. 1:j-1 This represents a token sequence consisting of j-1 tokens. Note that D KL This represents the Kullback-Leibler divergence.

[0022] It is difficult to solve equation (3) directly. Therefore, let p be the probability distribution of the tokens output by the LLM. LLM Assuming the following, the probability distribution of tokens p is given by the following equation. v This can be derived as a solution to equation (3). Here, S pub This consists of the instruction inst and the token sequence ~O 1:j-1 It represents a prompt that includes S. priv This consists of the instruction inst and the token sequence ~O 1:j-1 Training data D, in which the elements are training data with the same label value. priv This represents a prompt that includes a subset of p. LLM (v|~O 1:j-1) is a token sequence ~O consisting of j-1 tokens that have been generated so far. 1:j-1 Note that this represents the probability that the next token is v.

[0023] Equation (4) shows the probability distribution p of the token. LLM weight (p LLM (v|S priv ) / p LLM (v|S pub )) α The probability distribution of tokens p is proportional to the equation obtained by multiplying by the given formula. v This shows that it is good to generate a token sequence using this probability distribution p. v The algorithm obtained using this method is the token sequence generation algorithm PTA. The token sequence generation method, algorithm PTA, in an embodiment of the present invention is shown below.

[0024] <<Token Sequence Generation Algorithm PTA>> ------------------------------------------------------------------ 1: Initialize the token sequence ~O. That is, make the token sequence ~O a token sequence [] of length 0. 2: for j=1, …, T 3: Training data set D priv A set ~D consists of MN training data elements, each with a randomly selected label ~y. priv Generate the set ~D priv By partitioning it into sets of N disjoint elements, we obtain M sets D priv (1) , D priv (2) ,…, D priv (M) Generates: 4: Prompt S containing instruction inst and token sequence ~O pub and instruction inst and token sequence ~O and set D priv (i) Prompt S priv (i) Generate (i=1, 2, …, M). 5: for i=1, …, M 6: Probability distribution p of tokens output by the large-scale language model LLM. LLMusing to obtain the token probability distribution p by the following equation v (i) is obtained. (where V represents a vocabulary set with tokens as elements) 7: end for 8: The token probability distribution ^p is calculated by the following equation v is obtained. (where N(0, 2σ 2 ) represents a normal distribution with a mean of 0 and a variance of 2σ 2 ) 9: A subset V of the vocabulary set V is generated by the following equation pub is generated. (where k satisfies 0<k<1) (where p satisfies 0<p<1) 10: The token v is calculated by the following equation syn is obtained. 11: The token sequence ~O is updated by the following equation. 12: end for 13: Output the token sequence ~O. ------------------------------------------------------------------ Here, T, M, and N are integers of 1 or more, and α is a real number greater than 0. Further, + in Equation (12) represents an append operation.

[0025] The processing of algorithm PTA is briefly described. In the processing from the 3rd line to the 4th line, prompts unrelated to the learning data set D to be protected from data leakage risk priv using the learning data set D priv prompts S unrelated to pub and the learning data set D priv prompts S related to priv (i) (i=1, 2, …, M) are generated. In the processing from the 5th line to the 7th line, the token probability distribution p output by the large language model LLM when the token sequence ~O is input based on Equation (4) LLM the token probability distribution p modified from (v|~O) v (i) is obtained. In the processing of the 8th line, the noise-added token probability distribution ^pv is obtained. In the process at the 9th line, among the subsets of the vocabulary set V with the number of elements being k|V|, the probability p LLM (v|S pub ) the intersection of the set with the maximum sum and the set with the minimum number of elements among the subsets of the vocabulary set V in which the sum of the probability p LLM (v|S pub ) is equal to or greater than a predetermined value (p) is generated as V pub . Thereby, the search range of the process at the 10th line is limited to the subset V pub of the vocabulary set V. In the process at the 10th line, among the tokens included in the set V pub , the token v with the maximum probability ^p v is obtained syn . In the process at the 11th line, the token sequence ~O is updated by adding the token v syn to the token sequence ~O. Then, the token sequence ~O output in the process at the 13th line is a token sequence for which differential privacy is guaranteed.

[0026] Several experiments on algorithm PTA confirmed the following points. (1) In the text classification task, accuracy degradation is suppressed compared with the technique of Non-Patent Document 1. (2) The correct concept can be inferred in ICL. (3) By modifying the probability distribution of tokens before adding noise, tokens with high probabilities are more likely to be output.

[0027] <First Embodiment> A token sequence generation device 100 uses a learning data set D priv to generate an output token sequence used for generating an input prompt in in-context learning of a large-scale language model.

[0028] Hereinafter, the token sequence generation device 100 will be described with reference to FIGS. 1 to 2. FIG. 1 is a block diagram showing the configuration of the token sequence generation device 100. FIG. 2 is a flowchart showing the operation of the token sequence generation device 100. As shown in FIG. 1, the token sequence generation device 100 includes an initialization unit 110, a token sequence generation unit 120, and a recording unit 190. The recording unit 190 is a component that appropriately records information necessary for the processing of the token sequence generation device 100. The learning data set D privInstead of using the token sequence generator 100 as input, the recording unit 190 is pre-programmed with the learning data set D priv You may want to keep a record of it.

[0029] The operation of the token sequence generator 100 will be explained with reference to Figure 2.

[0030] In S110, the initialization unit 110 initializes the token sequence ~O by setting it to a token sequence of length 0.

[0031] In S120, the token sequence generation unit 120 generates the training data set D priv The token sequence generation unit 120 takes the input as input and generates and outputs an output token sequence of a predetermined length. The token sequence generation unit 120 repeats the following processes until the length of the token sequence ~O reaches a predetermined length: (1) probability distribution correction process, (2) noise addition process, (3) vocabulary set restriction process, (4) token generation process, and (5) token sequence update process, and uses the obtained token sequence ~O as the output token sequence. Each of the processes (1) to (5) is as follows.

[0032] (1) A training data set D whose elements are training data with the probability distribution correction processing label ~y. priv Subset D priv (1) , D priv (2) ,…, D priv (M) Using this, the instruction inst and the prompt S containing the token sequence ~O are used. pub and instruction inst and token sequence ~O and set D priv (i) Prompt S priv (i) The first process generates (i=1, 2, …, M), and the probability distribution p of the tokens output by the large-scale language model. LLM By modifying (v|~O), the probability distribution of the token p v (i) This process executes the second step of finding (i=1, 2, …, M).

[0033] (2) Probability distribution p of the noise-added token v (i)A process of adding noise to (i=1, 2, …, M) to obtain the token probability distribution ^p v .

[0034] (3) Vocabulary set restriction processing: A process of generating a set V that maximizes the sum of probabilities p among subsets of the vocabulary set V whose number of elements is k|V| or less (where k satisfies 0<k<1). LLM (v|S pub ) pub .

[0035] (4) Token generation processing: A process of obtaining the token v with the maximum probability ^p among the tokens included in the set V. pub v syn .

[0036] (5) Token sequence update processing: A process of adding the token v to the token sequence ~O. syn .

[0037] Each of the processes (1) to (4) may be, for example, the following processes.

[0038] (1) Probability distribution correction processing: In the first process, a set ~D whose elements are MN pieces of learning data with label ~y randomly selected from the learning data set D priv priv is generated, and the set ~D priv is divided into N disjoint sets each consisting of elements, so as to generate the sets D priv (1) , D priv (2) , …, D priv (M) . Further, in the second process, the token probability distribution p is obtained using the ratio p LLM (v|S priv (i) ) / p LLM (v|S pub ). More specifically, the token probability distribution p (i=1, 2, …, M) is obtained by the following formula. v (i) v (i) (i=1, 2, …, M). (2) Noise addition processing: The token probability distribution ^p is obtained by the following formula.​​​​v is determined. (where N(0, 2σ 2 ) represents a normal distribution with a mean of 0 and a variance of 2σ 2 ) (3) Vocabulary set restriction processing The intersection of sets V1 and V2 obtained by the following equation is generated as set V pub . (where p satisfies 0<p<1) (4) Token generation processing Token v is determined by the following equation syn . According to the embodiments of the present invention, it is possible to generate tokens that suppress accuracy degradation caused by adding noise to the probability distribution of tokens.

[0039] <Supplementary Note> Functions implemented by the components described in this specification may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, that are programmed to implement the described functions. A processor includes transistors and other circuits and is regarded as circuitry or processing circuitry. The processor may be a programmed processor that executes a program stored in a memory.

[0040] In this specification, circuitry, units, and means refer to hardware that is programmed to execute or execute the described functions. The hardware may be any hardware disclosed in this specification, or any hardware known to be programmed to execute or execute the described functions.

[0041] If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0042] The various processes described above can be carried out by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 3, and then causing the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc. to operate.

[0043] The program describing this process can be recorded on a computer-readable recording medium. Any computer-readable recording medium can be used, such as a magnetic recording device, optical disc, magneto-optical recording medium, or semiconductor memory.

[0044] Furthermore, this program may be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs or CD-ROMs on which the program is recorded. Alternatively, the program may be stored in the storage device of a server computer and distributed by transferring the program from the server computer to other computers via a network.

[0045] A computer executing such a program may, for example, first store the program, either recorded on a portable storage medium or transferred from a server computer, in its own memory. Then, when processing is to be executed, the computer reads the program stored in its memory and executes the processing according to the read program. Alternatively, the computer may directly read the program from the portable storage medium and execute the processing according to that program, or it may sequentially execute the processing according to the received program each time a program is transferred to it from a server computer. Furthermore, the processing may be executed using a so-called ASP (Application Service Provider) type service, where the processing function is realized only by issuing execution instructions and obtaining results, without transferring the program from the server computer to this computer.In addition, the processing may be executed using a so-called SaaS (Software as a Service) type service, where a part of the server computer is made available to the user along with the program. Furthermore, the term "program" in this form includes information used for processing by an electronic computer that is equivalent to a program (data, etc., that is not a direct instruction to the computer but has the property of defining the processing of the computer).

[0046] Furthermore, in this configuration, the device is configured by executing a predetermined program on a computer, but at least a part of these processes may be implemented in hardware.

[0047] The present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. Furthermore, the processes described in the above embodiments may not only be executed in chronological order according to the order described, but may also be executed in parallel or individually as needed, depending on the processing capacity of the device performing the process.

Claims

1. Learning data set D syn , v , syn , pub A token sequence generation device that generates an output token sequence of a predetermined length using, comprising: an initialization unit that sets a token sequence ~O as a token sequence of length 0; and a learning data set D whose elements are learning data with label ~y priv Subset D of priv (1) , D priv (2) , ..., D priv (M) A first process that generates a prompt S including the instruction inst and the token sequence ~O pub and a prompt S including the instruction inst, the token sequence ~O, and the set D priv (i) priv (i) (i=1, 2, …, M); a probability distribution correction process that executes a second process for obtaining the token probability distribution p by correcting the token probability distribution p output by the large language model LLM (v|~O) v (i) (i=1, 2, …, M); a noise addition process for obtaining the token probability distribution ^p by adding noise to the token probability distribution p v (i) (i=1, 2, …, M); a vocabulary set restriction process for generating a set V with the maximum sum of probabilities p among subsets of the vocabulary set V whose number of elements is equal to or less than k|V| (where k satisfies 0<k<1) v LLM (v|S pub ) pub ; a token generation process for obtaining the token v with the maximum probability ^p among the tokens included in the set V pub v syn ; and a token sequence update process for adding the token v to the token sequence ~O syn , wherein the process consisting of these is repeated until the length of the token sequence ~O reaches the predetermined length, and the resulting token sequence ~O is used as the output token sequence. A token sequence generation device comprising a token sequence generation unit.​​​​ 2. A token sequence generation device according to claim 1, wherein in the probability distribution correction process, in the first process, the learning data set D priv A set ~D consists of MN training data elements, each with a randomly selected label ~y. priv Generate the set ~D priv By dividing it into sets of N disjoint elements, we can create the set D priv (1) , D priv (2) ,…, D priv (M) Generates the following, and in the second process, ratio p LLM (v|S priv (i) ) / p LLM (v|S pub The probability distribution of tokens p) v (i) A token sequence generator characterized by finding (i=1, 2, …, M).

3. The token sequence generation apparatus according to claim 2, wherein in the second process, the probability distribution of the tokens p is given by the following equation. v (i) Find (i=1, 2, …, M) A token sequence generation device characterized by the following features.

4. A token sequence generation device according to claim 1, wherein the noise addition process is performed by the following equation for the probability distribution of tokens^p v To seek (However, N(0, 2σ) 2 ) has a mean of 0 and a variance of 2σ 2 A token sequence generator characterized by representing a normal distribution.

5. The token sequence generation device according to claim 1, wherein in the vocabulary set restriction processing, an intersection of a set V1 and a set V2 obtained by the following expression is defined as a set V pub to be generated (where p satisfies 0<p<1) The token sequence generation device characterized by the following.

6. A token sequence generation device according to claim 1, wherein the token generation process generates token v by the following formula. syn To seek A token sequence generation device characterized by the following features.

7. Learning data set D priv A token sequence generation method for generating an output token sequence of a predetermined length using the above, comprising: an initialization step of setting token sequence ~O as a token sequence of length 0; and a learning data set D whose elements are learning data with label ~y priv subset D priv (1) , D priv (2) ,..., D priv (M) a first process of generating prompt S including instruction inst and token sequence ~O using pub and prompt S including instruction inst, token sequence ~O, and set D priv (i) , and a probability distribution correction process of executing a second process of obtaining (i=1, 2,..., M) priv (i) by correcting the token probability distribution p output by the large language model LLM (v|~O) v (i) (i=1, 2,..., M), a noise addition process of obtaining token probability distribution ^p by adding noise to the token probability distribution p v (i) (i=1, 2,..., M), a vocabulary set restriction process of generating set V which maximizes the sum of probabilities p v among subsets of vocabulary set V whose number of elements is equal to or less than k|V| (where k satisfies 0<k<1) LLM (v|S pub ), a token generation process of finding the token v that maximizes the probability ^p among the tokens included in set V pub , and a token sequence update process of adding the token v to the token sequence ~O pub , wherein the process consisting of these processes is repeated until the length of the token sequence ~O reaches the predetermined length, and the obtained token sequence ~O is used as the output token sequence v syn syn A token sequence generation method including a token sequence generation step.​​ 8. A program for causing a computer to execute the token sequence generation method described in claim 7.

Images