Decoding device and decoding method

Abstract

This decoding device comprises: a generation unit that generates a set of hypothesis sequences from an input sequence; a first similarity calculation unit that calculates a first similarity, which is the similarity between the input sequence and a case input sequence in a database storing a plurality of items of case data each including a case input sequence, a case hypothesis sequence, and a case quality evaluation score; a second similarity calculation unit that calculates a second similarity, which is the similarity between a case hypothesis sequence and a hypothesis sequence in the set; and a selection unit that selects, from the set, a hypothesis sequence to be used as an output sequence on the basis of the first similarity, the second similarity, and the case quality evaluation score.

Description

Decoding device and decoding method

[0001] This invention relates to a decoding technique that generates appropriate output text from a given input text.

[0002] Conventional decoding methods include maximum a posteriori (MAP) decoding and minimum Bayes risk (MBR) decoding. MAP decoding is currently the most widely used decoding method, but it is known to have the problem of output collapse.

[0003] On the other hand, MBR decoding allows for the selection of the hypothesis that maximizes expected utility from among multiple output hypotheses, making it possible to output a robust text with higher quality and fewer output collapses and errors compared to MAP decoding.

[0004] Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation (Eikema and Aziz, 2020)

[0005] However, since hypothesis selection in MBR decoding is based on expected utility calculated depending on the samples generated by the model, robustness decreases in domains where the model is not sufficiently trained, resulting in a decrease in the quality of the text generated by decoding.

[0006] This invention has been made in view of the above points, and aims to provide a technology for improving the quality of text generated by decoding.

[0007] According to the disclosed technology, there is provided: a generation unit that generates a set of hypothesis sequences from an input sequence; a first similarity calculation unit that calculates a first similarity, which is a similarity between said case input sequence and said input sequence in a database storing a plurality of case data each including a case input sequence, a case hypothesis sequence, and a case quality evaluation score; a second similarity calculation unit that calculates a second similarity, which is a similarity between said case hypothesis sequence and said hypothesis sequence in said set; and a selection unit that selects a hypothesis sequence to be an output sequence from said set based on said first similarity, said second similarity, and said case quality evaluation score.

[0008] According to the disclosed technology, there is provided a technology for improving the quality of text generated by decoding.

[0009] It is a diagram showing an outline of processing when constructing a memory M. It is a diagram showing an outline of processing at the time of decoding. It is a diagram showing an example functional configuration of a decoding device 100. It is a flow chart for explaining the operation of the decoding device 100. It is a diagram showing an example functional configuration of a construction device 200. It is a flow chart for explaining the operation of the construction device 200. It is a diagram showing an example functional configuration of the decoding device 100. It is a diagram showing an example hardware configuration of the device.

[0010] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiments to which the present invention is applied are not limited to the following embodiment.

[0011] In the following embodiment, the input and output targets are texts (= sequences of characters), but the technology according to the present invention is applicable to all general sequences that are not limited to texts.

[0012] (Outline of the embodiment) In the present embodiment, the decoding device 100 described later performs decoding based on case-based decision-making theory. Accordingly, the quality of text generation in a domain can be improved by using information of cases.

[0013] Specifically, information on hypotheses and utilities for each case is stored in a case database from pre-prepared domain data, and during decoding, the case database is referenced and hypothesis selection is performed using utility information of similar cases. Unlike MBR decoding, the expected utility is estimated using correct answer information of actual data.

[0014] Hereinafter, first, a theoretical description will be given of the contents of minimum Bayesian risk decoding, case-based decision theory, and the proposed method (decoding based on case-based decision theory), and then, a specific device configuration and device operation will be described as an embodiment.

[0015] In the text of the present specification, for convenience of description, a normal font is used as the font of characters representing a set. It is clear from the context that the character represents a set. Also, in the text of the present specification, symbols intended to be written at the head of a character are written at the upper left of the character (e.g., ^Y, ￣ s Ｘ ).

[0016] (Minimum Bayesian Risk Decoding) Let an input text in text generation be x∈X and an output text be y∈Y. Here, X, Y⊆V ＊ represent an input space and an output space, respectively, and V ＊ indicates the Kleene closure of a vocabulary V. In MAP decoding, which is a conventional general text generation method, a text generation model θ is used, and an output text y that maximizes the generation probability conditioned on the input text ＭＡＰ = argmax ｈ∈Ｈ p(h|x;θ) is selected. However, since the entire output space Y cannot be searched, an output text is selected from the obtained hypothesis set H⊂Y by beam search or the like.

[0017] On the other hand, in MBR decoding, a hypothesis y that maximizes the expected value (expected utility) of a utility function u:Y×Y→R ＭＢＲ is selected:

[0018] In the above formula (1), y∈Y is reference text obtained in accordance with the true output probability Pr(·|x). As the utility function u, an evaluation index that measures the quality of output text is generally used. Here, since Pr(·|x) is unknown, the expected utility U ＭＢＲ is typically estimated by the Monte Carlo (MC) method as shown in the following formula (2).

[0019] Note that since it is difficult to obtain reference text, in practice, as reference text, a multiset ^Y:={y of pseudo reference text (pseudo references) pseudo-sampled from a text generation model ｉ} ｜＾Ｙ｜ ｉ＝１ ~p(y|x;θ) is used.

[0020] The maximization of expected utility, which is central to the calculation of MBR decoding, is known as Expected Utility Theory (EUT) in the field of decision theory. A decision maker following EUT enumerates the possibilities of multiple conceivable options for the problem currently faced, and selects the action that is expected to provide the maximum utility. Note that the output text of MBR decoding varies greatly depending on the distribution of generated pseudo references. Therefore, in domains where learning of a text generation model is insufficient, expected utility cannot necessarily be estimated robustly.

[0021] (Case-Based Decision Theory) To address such problems, in the field of decision theory, Case-Based Decision Theory (CBDT), which derives the best action based on actually experienced cases in the past, has been proposed. In CBDT, for a problem currently faced, the best action selection is predicted from actions selected under similar problems in the past and rewards obtained at that time. If the set of problems is Q, the set of actions is A, and the reward space is R, the case set is defined as C:=Q×A×R.

[0022] Here, if the problem currently faced is q∈Q, in CBDT, based on the memory M⊆C, an action a according to the following formula ★ ∈A is selected.

[0023] Note that s: Q × Q → [0, 1] is a similarity function that represents the closeness of the problems. Also, 1 ａ＝ａ´ This is an instruction function that outputs 1 when a = a' and 0 when a = a' is not. In other words, in CBDT-based decision-making, we focus on cases where action a was chosen and select the action that maximizes the weighted sum obtained by weighting the reward obtained in those cases with the similarity to the problem.

[0024] (Proposed method: Decoding based on case-based decision theory) In this embodiment, we propose decoding based on case-based decision theory (CBDT decoding) and improve the quality of text generation by utilizing reference data according to the task and domain. In the proposed method, data is stored in advance in memory, which is a collection of cases (a case database described later), and the decoding device 100 selects an output hypothesis while referring to the memory during decoding.

[0025] In this embodiment, since we are dealing with text generation, from here on, we will replace the decision-making problem Q with the input text X, the action A with the output text Y, and the reward R with the output text quality evaluation score R.

[0026] <Memory> Prior to this, the construction device 200, described later, has set up data D consisting of pairs of input text and correct output text (reference text) := {(x ｉ , y ｉ )} ｜Ｄ｜ ｉ＝１ From this, memory M is constructed. Figure 1 shows an overview of the processing during the construction of memory M. First, for each input text x in data D, H hypotheses H are generated. ｘ ⊂Y is generated. That is, by inputting the input text x into a text generation model with parameter θ, H is produced as the output from the text generation model. ｘ To obtain.

[0027] Next, the construction device 200 evaluates the quality of the generated hypothesis text using the reference text y (correct output text) with u, and stores the input text, hypothesis text, and quality evaluation results in memory M ⊆ X × Y × R. M can be expressed as shown in the following equation (5).

[0028] <Decryption> During decoding, the decoding device 100 selects an output hypothesis based on CBDT while referring to a pre-constructed memory. Decryption C is based on a naive approach to CBDT, directly following equation (3). ＢＤＴ N ＡＩＶＥ Therefore, the hypothesis that maximizes the score shown in equation (6) below is selected.

[0029] Here, C ＢＤＴ N ＡＩＶＥ There are two challenges to this. First, various measures that represent the similarity between texts, such as cosine similarity and Euclidean distance, do not necessarily take values ​​in the range [0, 1], thus limiting the measures that can be used for s. Second, the indicator function 1 ｈ＝ｈ´ Therefore, the score cannot be calculated unless the exact same text as the hypothesis h that is currently being output is present in memory M.

[0030] To address these issues, this embodiment uses normalized similarity, which normalizes the similarity between texts, as shown in equations (7) to (9) below, and further uses a score function U that utilizes the similarity between hypotheses h and h' instead of an indicator function. ＣＢＤＴ In this embodiment, C ＢＤＴ N ＡＩＶＥ The use of is not excluded, and the decoding device 100 is C ＢＤＴ N ＡＩＶＥ Decryption may be performed using [a specific method / tool].

[0031]

[0032]

[0033] s in equation (8) Ｘ : X × X → R is a function that represents the similarity between input texts, s Ｙ : Y × Y → R is a function that represents the similarity between the output texts. ￣ s Ｘ : X×X×2 Ｍ →R, ￣ s Ｙ : Y x Y x 2 Ｙ →R is their normalized similarity, τＸ , τ Ｙ This is the temperature parameter for similarity.

[0034] In this embodiment, in CBDT decoding, the decoding device 100 measures the similarity s of the input text. Ｘ After narrowing down the results to the top k input texts x' with high (x, x') values, and selecting only cases ^M⊆M, the aforementioned score function U is used as shown in equation (10) below. ＣＢＤＴ Hypothesis y to maximize ＣＢＤＴ Select this option. Note that memory will contain H hypotheses for each input text x'. ｘ´ Since it has |^M| = Hk,

[0035] Figure 2 shows an overview of the decoding process. The decoding device 100 calculates the similarity between the input text x and the input text in memory. ￣ s Ｘ ) × (Inter-hypothesis similarity between the hypothetical text h obtained by the text generation model from the input text x and the hypothetical text in memory) ￣ s Ｘ The score U of the hypothesis text h is calculated by multiplying the value of () × the quality in memory) for each of the narrowed-down cases (Hk), and then summing them up. ＣＢＤＴ The score is calculated as (h; x, M). The decoding device 100 selects a score U from the set of hypothetical texts obtained by the text generation model from the input text x. ＣＢＤＴ Select the hypothesis text that maximizes (h; x, M) as the output text.

[0036] (MBR Decoding and CBDT Decoding) MBR decoding and CBDT decoding share a common objective: to select a hypothesis that maximizes the utility obtained from a set of hypotheses. On the other hand, these methods are orthogonal, and further quality improvements can be expected by combining them. Therefore, the decoding device 100 in this embodiment may select a hypothesis text as the output text that maximizes the score obtained by linearly combining the scores of both methods, as shown in equation (11) below. This method is called MBR-CBDT decoding.

[0037] However, in equation (11)￣ U ＭＣＭＢＲ This is the U shown in equation (2) ＭＣＭＢＲ After calculating the scores, the data was min-max normalized using the minimum and maximum scores within the hypothesis set H. ￣ U ＣＢＤＴ This is the U shown in equation (7) ＣＢＤＴ After calculating the score, it was min-max normalized using the minimum and maximum scores within the hypothesis set H.

[0038] The configuration and operation of the decoding device 100 and the construction device 200 will be described in detail below as an example. In this embodiment, the decoding device 100 and the construction device 200 are separate devices, but this is just one example, and the decoding device 100 and the construction device 200 may be a single device. For example, the functions of the construction device 200 may be included in the decoding device 100.

[0039] (Decoding device 100) Figure 3 shows an example of the functional configuration of the decoding device 100. As shown in Figure 3, the decoding device 100 includes a generation unit 110, a trained text generation model 120, an inter-hypothesis similarity calculation unit 140, a selection unit 150, a case database 160, and an inter-input similarity calculation unit 170. The case database 160 may be provided outside the decoding device 100.

[0040] Figure 3 also shows the set of hypothetical texts 130 generated by the generation unit 110. The case database 160 is assumed to have been constructed by the construction device 200.

[0041] The operation of the decoding device 100 will be explained according to the steps in the flowchart of Figure 4. In the following explanation of the decoding device 100, the input text will be x, the hypothetical text obtained from the input text x will be h, the input text in the case database 160 will be x', the hypothetical text in the case database 160 obtained from the input text x' will be h', and the quality evaluation score for the hypothetical text h' in the case database 160 will be r'.

[0042] Furthermore, x', h', and r' may be referred to as the case input text, case hypothesis text, and case quality evaluation score, respectively.

[0043] <S101 (Step 101)> Input text x is input to the generation unit 110.

[0044] <S102> The generation unit 110 inputs the input text x to the text generation model 120 and obtains the output from the text generation model 120 as multiple hypothetical texts h. The generation unit 110 outputs the multiple hypothetical texts h as a set of hypothetical texts.

[0045] In subsequent processing, the decoding device 100 uses the case data for the top k (or more) input texts x' with the highest similarity to input text x as case data in the case database 160. The top k (or more) input texts x' with the highest similarity to input text x may also be called "input texts similar to input text x".

[0046] <S103> The inter-hypothesis similarity calculation unit 140 calculates the inter-hypothesis similarity between the hypothesis text h in the hypothesis text set and the hypothesis text h' in the case database 160. More specifically, it calculates the inter-hypothesis similarity using the formula (9) described above.

[0047] If there are H hypothetical texts h' for one input text x', then there are Hk hypothetical texts h' for k input texts x'. In this case, Hk inter-hypothesis similarity scores are calculated for one hypothetical text h.

[0048] <S104> The input similarity calculation unit 170 calculates the input similarity from the input text x and the input text x' in the case database 160. More specifically, it calculates the input similarity using the formula (8) described above.

[0049] If there are k different input texts x', then k input similarity scores are calculated for each input text x.

[0050] <S105> The selection unit 150 calculates the output score of the hypothesis text h from the input similarity, the hypothesis similarity, and the hypothesis score r' in the case database 160. Specifically, the output score of the hypothesis text h is calculated by summing "input similarity × hypothesis similarity × hypothesis score" for the case data of the top k (multiple) input texts x' with similarity to input text x.

[0051] <S106> The decoding device 100 performs the processing in S103 to S105 for each hypothesis text h included in the hypothesis text set.

[0052] The selection unit 150 outputs the hypothesis text h with the best output score among all the output scores obtained for each hypothesis text h.

[0053] (Construction device 200) Figure 5 shows an example of the functional configuration of the construction device 200. As shown in Figure 5, the construction device 200 has a generation unit 210, a trained text generation model 220, and an evaluation unit 240.

[0054] Figure 5 also shows the set of hypothetical texts 230 generated by the generation unit 210. Furthermore, Figure 5 shows the input / correct output text pair set 300 used for input, and the generated case database 400.

[0055] The operation of the construction device 200 will be explained according to the steps in the flowchart in Figure 6. In the following explanation of the construction device 200, the input text will be x, the hypothesis text obtained from the input text x will be h, and the hypothesis score (quality evaluation score) for the hypothesis text h will be r.

[0056] <S201> The generation unit 210 receives input text x from the input / correct output text pair set 300.

[0057] <S202> The generation unit 210 inputs the input text x to the text generation model 220 and obtains the output from the text generation model 220 as multiple hypothetical texts h. The generation unit 210 outputs the multiple hypothetical texts h as a set of hypothetical texts.

[0058] <S203> The evaluation unit 240 calculates a hypothesis score r for each hypothesis text in the generated set of hypothesis texts from the hypothesis text h and the correct output text.

[0059] <S204> The evaluation unit 240 stores the triplicate consisting of input text x, hypothesis text h, and hypothesis score r in the case database 400.

[0060] (Other examples of decoding device 100) As described above, in this embodiment, it is possible to perform MBR-CBDT decoding, which is a combination of MBR decoding and CBDT decoding.

[0061] Figure 7 shows an example configuration of a decoding device 100 that performs MBR-CBDT decoding. As shown in Figure 7, this decoding device 100 has a generation unit 500, a CBDT decoding unit 600, an MBR decoding unit 700, and a selection unit 800.

[0062] The generation unit 500 corresponds to the aforementioned "generation unit 110 + trained text generation model 120".

[0063] The CBDT decoding unit 600 includes an inter-hypothesis similarity calculation unit 140, an inter-input similarity calculation unit 170, a case database 160, and a selection unit 150 in the decoding device 100 shown in Figure 3. However, the selection unit 150 may be included in the selection unit 700 described later. The CBDT decoding unit 600 processes the score U obtained in the processing steps S103 to S105 described above. ＣＢＤＴ Normalize (h; x, ^M) and score ￣ U ＣＢＤＴ Calculate (h; x, ^M).

[0064] The MBR decoding unit 700 uses the aforementioned prior art to calculate the expected utility U ＭＣＭＢＲ (h;^Y) is calculated. Furthermore, the MBR decoding unit 700 U ＭＣＭＢＲ (h;^Y) normalized score ￣ U ＭＣＭＢＲ Calculate (h;^Y).

[0065] The selection unit 700, as shown in equation (11) above, selects from the set of hypothetical texts generated by the generation unit 500, ￣ U ＣＢＤＴ (h; x, ^M) +￣ U ＭＣＭＢＲ Select the hypothesis text h that maximizes "(h;^Y)" and output it as the output text.

[0066] (Example Hardware Configuration) Any of the devices described in this embodiment (decoding device 100, construction device 200) can be realized, for example, by having a computer execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0067] In other words, the device can be realized by using hardware resources such as the CPU and memory built into a computer to execute a program corresponding to the processing performed by the device. The program can be recorded on a computer-readable recording medium (such as portable memory), saved, and distributed. It can also be provided via a network, such as the Internet or email.

[0068] Figure 8 shows an example of the hardware configuration of the computer described above. The computer in Figure 8 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., all of which are interconnected by bus B. The computer may also be equipped with a GPU.

[0069] The program that enables processing on the computer is provided on a recording medium 1001, such as a CD-ROM or memory card. When the recording medium 1001 containing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001; it may also be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files and data.

[0070] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when a program startup command is received. The CPU 1004 implements the functions related to the memory device 1003 according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) etc., based on a program. The input device 1007 consists of a keyboard and mouse, buttons, or a touch panel, etc., and is used to input various operation commands. The output device 1008 outputs the calculation results.

[0071] (Summary of Embodiments, Effects, etc.) As described above, the technology described in this embodiment uses actual case information in calculating the score for selecting hypothesis texts, making it possible to improve the robustness of text generation in domains where the model output is unstable.

[0072] Furthermore, since CBDT decoding in this embodiment is an orthogonal method to MBR decoding, it is possible to combine the two. By using CBDT decoding and MBR decoding together, it becomes possible to generate text of even higher quality than with MBR decoding alone.

[0073] We conducted German-English translation experiments in five domains and confirmed that the proposed method, using CBDT decoding, can produce higher quality text than MAP decoding. Furthermore, we confirmed that MBR-CBDT decoding can generate higher quality text than MBR decoding.

[0074] The following additional information is disclosed regarding the embodiments described above.

[0075] <Notes> (Note 1) A decoding device comprising: a memory and at least one processor connected to the memory, wherein the processor generates a set of hypothesis sequences from an input sequence, calculates a first similarity which is the similarity between the case input sequence and the input sequence in a database storing a plurality of case data having a case input sequence, a case hypothesis sequence, and a case quality evaluation score, calculates a second similarity which is the similarity between the case hypothesis sequence and the hypothesis sequence in the set, and selects a hypothesis sequence to be used as the output sequence from the set based on the first similarity, the second similarity, and the case quality evaluation score. (Note 2) The decoding device according to Note 1, wherein the processor calculates the sum of the values ​​obtained by multiplying the first similarity, the second similarity, and the case quality evaluation score for a plurality of case data for a plurality of case input sequences similar to the input sequence, and selects the hypothesis sequence that maximizes the sum as the output sequence. (Note 3) The decoding device according to Note 1, wherein the processor selects as the output sequence a hypothesis sequence that maximizes the sum of a score based on the product of the first similarity, the second similarity, and the case quality evaluation score, and an expected utility score calculated by minimum Bayesian risk decoding. (Note 4) A decoding method executed by the decoding device, comprising: a generation step of generating a set of hypothesis sequences from an input sequence; a first similarity calculation step of calculating a first similarity, which is the similarity between the case input sequence and the input sequence in a database storing a plurality of case data sets having a case input sequence, a case hypothesis sequence, and a case quality evaluation score; a second similarity calculation step of calculating a second similarity, which is the similarity between the case hypothesis sequence and the hypothesis sequence in the set; and a selection step of selecting a hypothesis sequence to be used as the output sequence from the set based on the first similarity, the second similarity, and the case quality evaluation score.

[0076] Although this embodiment has been described above, the present invention is not limited to this specific embodiment, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims.

[0077] 100 Decoding device 110 Generation unit 120 Trained text generation model 140 Hypothesis similarity calculation unit 150 Selection unit 160 Case database 170 Input similarity calculation unit 200 Construction device 210 Generation unit 220 Trained text generation model 240 Evaluation unit 300 Input / ground truth output text pair set 400 Case database 500 Generation unit 600 CBDT decoding unit 700 MBR decoding unit 800 Selection unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A decoding device comprising: a generation unit that generates a set of hypothesis sequences from an input sequence; a first similarity calculation unit that calculates a first similarity, which is the similarity between the case input sequence and the input sequence in a database storing a plurality of case data sets having a case input sequence, a case hypothesis sequence, and a case quality evaluation score; a second similarity calculation unit that calculates a second similarity, which is the similarity between the case hypothesis sequence and the hypothesis sequence in the set; and a selection unit that selects a hypothesis sequence to be used as the output sequence from the set based on the first similarity, the second similarity, and the case quality evaluation score.

2. The decoding device according to claim 1, wherein the selection unit calculates the sum of the values ​​obtained by multiplying the first similarity, the second similarity, and the case quality evaluation score for multiple case data for multiple case input sequences similar to the input sequence, and selects the hypothesis sequence that maximizes the sum as the output sequence.

3. The decoding device according to claim 1, further comprising a decoding unit that performs minimum Bayesian risk decoding, wherein the selection unit selects as the output sequence a hypothetical sequence that maximizes the sum of a score based on the product of the first similarity, the second similarity, and the case quality evaluation score, and the expected utility score calculated by the decoding unit.

4. A decoding method performed by a decoding device, comprising: a generation step of generating a set of hypothesis sequences from an input sequence; a first similarity calculation step of calculating a first similarity, which is the similarity between the case input sequence and the input sequence in a database storing a plurality of case data sets having a case input sequence, a case hypothesis sequence, and a case quality evaluation score; a second similarity calculation step of calculating a second similarity, which is the similarity between the case hypothesis sequence and the hypothesis sequence in the set; and a selection step of selecting a hypothesis sequence to be used as the output sequence from the set based on the first similarity, the second similarity, and the case quality evaluation score.

Images