Text evaluation device and text evaluation device system
The text evaluation device and system address the limitation of existing methods by generating questions from original text information to assess summary coverage, enhancing evaluation efficiency and accuracy.
Patent Information
- Application Number
- PCT/JP2025/017028
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2026-11-12
AI Technical Summary
Existing summary evaluation methods, such as QAGS, fail to assess the extent to which important information from the original text is covered in the summary, as they generate questions directly from the summary and do not account for information missing in the summary.
A text evaluation device and system that acquires original and summary texts, generates questions based on important information from the original text, and evaluates the summary by comparing answers derived from both texts, determining the coverage of important information in the summary.
Effectively evaluates the degree to which important information from the original text is included in the summary, improving summary evaluation methods by avoiding the need for reference summaries and ensuring comprehensive coverage assessment.
Smart Images

Figure JP2025017028_12112026_PF_FP_ABST
Abstract
Description
Text Evaluation Device and Text Evaluation Device System
[0001] The present invention relates to a text evaluation device and a text evaluation device system.
[0002] In recent years, research on techniques for summarizing texts using machine learning has been actively conducted. Methods for automatically evaluating generated summaries are also being studied. In automatic summary evaluation, a technique using a pre-prepared reference summary is known. Patent Document 1 describes an example of such a technique.
[0003] However, the method of Patent Document 1 requires preparing a reference summary to be used as a correct answer. Preparing an appropriate reference summary incurs costs.
[0004] As an example of a technique for evaluating text summaries without using reference summaries, a technique called QAGS is described in Non-Patent Document 1. In QAGS, questions are generated from the summary, and then the summary is evaluated based on the answers to the questions
[0005] Japanese Unexamined Patent Application Publication No. 2019-148934
[0006] Alex Wang, Kyunghyun Cho, Mike Lewis, "Asking and Answering Questions to Evaluate the Factual Consistency of Summaries", [online], April 8, 2020, [retrieved on March 25, 2025], Internet <URL: https: / / arxiv.org / abs / 2004.04228>
[0007] However, the conventional techniques have a problem that there is room for improvement in summary evaluation methods.
[0008] For example, in the method (QAGS) of Non-Patent Document 1, since questions are generated from the summary, it is possible to evaluate whether the content included in the summary is appropriate, but it is not possible to evaluate to what extent the important information included in the original text is covered in the summary.
[0009] This invention was made to solve these problems and aims to provide a technique for evaluating the degree to which important information contained in the original text is covered in the summary.
[0010] An example of a text evaluation device according to the present invention comprises: an acquisition unit that acquires original text and a summary text obtained by summarizing the original text; and an evaluation unit that evaluates the summary text based on the original text, wherein the evaluation unit acquires at least one question based on at least one important piece of information contained in the original text; the evaluation unit acquires a first answer to each of the at least one question based on the question and the original text; and acquires a second answer to the question based on the question and the summary text; and the evaluation unit evaluates the summary text based on the first and second answers to each of the at least one question.
[0011] An example of a text evaluation system according to the present invention is a text evaluation system comprising the above-described text evaluation device and a terminal, wherein the evaluation unit transmits evaluation result information relating to the results of evaluating the summarized text to the terminal, and the terminal comprises a receiving unit that receives the evaluation result information from the text evaluation device and a display unit that displays the evaluation result information.
[0012] According to the text evaluation device and text evaluation device system of the present invention, the method for evaluating summaries is improved.
[0013] In particular, it allows us to evaluate the degree to which important information contained in the original text is comprehensively covered in the summary.
[0014] Other effects will be described in connection with the embodiments and modifications described herein and in the drawings.
[0015] A schematic configuration example of the text evaluation system 1 according to Embodiment 1. A configuration example of the text evaluation device 10 in Figure 1. A configuration example of the terminal 20 in Figure 1. A flowchart illustrating an example of processing by the text evaluation device 10 in Figure 1. A specific example of the first prompt D31 among prompt D3 in Figure 2. A specific example of the second prompt D32 among prompt D3 in Figure 2. A specific example of the third prompt D33 among prompt D3 in Figure 2. A specific example of the fourth prompt D34 among prompt D3 in Figure 2. A diagram illustrating an overview of processing by a text evaluation device according to the prior art.
[0016] Hereinafter, embodiments of the present invention will be described based on the attached drawings.
[0017] [1. Embodiment 1] [1-1. Overall Outline] Figure 1 shows an example of the schematic configuration of a text evaluation system 1 according to Embodiment 1. The text evaluation system 1 includes a text evaluation device 10. In relation to the text evaluation device 10, a terminal 20 and an LLM 30 (Large Language Model) are provided. The text evaluation device 10 is connected to the terminal 20 and the LLM 30 so as to be able to communicate via a network.
[0018] The text evaluation system 1 may include a terminal 20. The text evaluation system 1 may also include an LLM 30.
[0019] The text evaluation device 10 is a device that evaluates the summarized text. The terminal 20 is a terminal that manages the text evaluation device 10. For example, the terminal 20 receives the results of the evaluation of the summarized text from the text evaluation device 10. The LLM 30 processes the questions used to evaluate the summary.
[0020] [1-2. Configuration of the Text Evaluation Device] Figure 2 shows an example of the configuration of the text evaluation device 10. The text evaluation device 10 comprises a processing device 11, a storage device 12, an input device 13, a display device 14, a communication device 15, and a bus 19 that connects these devices to each other.
[0021] The processing unit 11 includes at least one CPU (Central Processing Unit). At least one CPU is an example of at least one processor. At least one processor is an example of at least one computer. The text evaluation device 10 may be a single computer.
[0022] The storage device 12 is a recording medium that can be read by the processing device 11. The storage device 12 includes at least one memory. The storage device 12 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory) and VRAM (Video Random Access Memory).
[0023] The memory device 12 stores the program PG1. The program PG1 includes at least one instruction. The memory device 12 also stores the original text D1, the summary text D2, the prompt D3, the important information D4, the candidate question D5, the question D6, and the answer D7.
[0024] Program PG1 is a program for operating the text evaluation device 10. The processing unit 11 reads program PG1 from the storage device 12. By executing program PG1, the processing unit 11 functions as an acquisition unit 111 and an evaluation unit 112. At least one of the acquisition unit 111 and the evaluation unit 112 may be composed of circuits such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array).
[0025] The input device 13 is an input device that accepts input from an external source (e.g., a keyboard, mouse, microphone, switch, button, or sensor). The display device 14 is a display device that displays information to the outside (e.g., various display panels such as a liquid crystal display panel or an organic EL display panel). The input device 13 and the display device 14 may be configured as an integrated unit (e.g., a touch panel).
[0026] The communication device 15 is equipped with a communication interface that can connect to a network and communicates with other devices connected to the network using wireless or wired communication. The communication device 15 may also perform short-range wireless communication such as Bluetooth® or Wi-Fi®.
[0027] Bus 19 is a wiring configuration for communicating information. Bus 19 connects the processing unit 11, the storage device 12, the input device 13, the display device 14, and the communication device 15 to each other. Bus 19 may consist of a single bus, or it may consist of various buses provided between each device.
[0028] [1-3. Terminal Configuration] Figure 5 shows an example of the configuration of terminal 20. Terminal 20 comprises a processing unit 21, a storage device 22, an input device 23, a display device 24, a communication device 25, and a bus 29 that connects these devices to each other.
[0029] The processing unit 21 includes at least one CPU (Central Processing Unit). At least one CPU is an example of at least one processor. At least one processor is an example of at least one computer. The terminal 20 may be a single computer.
[0030] The storage device 22 is a recording medium that can be read by the processing device 21. The storage device 22 includes at least one memory. The storage device 22 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory) and VRAM (Video Random Access Memory). The storage device 22 stores program PG2. Program PG2 includes at least one instruction.
[0031] Program PG2 is a program for operating terminal 20. The processing unit 21 reads program PG2 from storage device 22. By executing program PG2, the processing unit 21 functions as a transmission unit 211, a reception unit 212, and a display unit 213. At least one of the transmission unit 211, reception unit 212, and display unit 213 may be composed of circuits such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array).
[0032] The input device 23 is an input device that accepts input from the outside (for example, a keyboard, mouse, microphone, switch, button, or sensor). The display device 24 is a display device that displays information to the outside (for example, various display panels such as a liquid crystal display panel or an organic EL display panel). The input device 23 and the display device 24 may be configured as an integrated unit (for example, a touch panel).
[0033] The communication device 25 is equipped with a communication interface that can connect to a network and communicates with other devices connected to the network using wireless or wired communication. The communication device 25 may also perform short-range wireless communication such as Bluetooth® or Wi-Fi®.
[0034] Bus 29 is a wiring configuration for communicating information. Bus 29 connects the processing unit 21, the storage device 22, the input device 23, the display device 24, and the communication device 25. Bus 29 may consist of a single bus, or it may consist of various buses provided between each device.
[0035] [1-4. LLM Configuration] An example of the LLM30 configuration is not specifically illustrated, but will be explained below. The LLM30 is configured using at least one computer. The LLM30 receives natural language data, including prompts, as input. The LLM30 outputs a natural language response to the received input.
[0036] LLM30 is a pre-trained machine learning model that uses, for example, a large amount of text data to output appropriate natural language responses to natural language inputs. Various publicly known models can be used as LLM30. For example, GPT, BERT, PaLM, and Llama models are publicly known, but other models may also be used.
[0037] [1-5. Processing Overview] The processing overview of the text evaluation device 10 according to this embodiment will be explained using Figure 4. A summary text D2 is prepared as a summary of the original text D1. The acquisition unit 111 of the text evaluation device 10 acquires the original text D1 and the summary text D2. The evaluation unit 112 of the text evaluation device 10 evaluates the summary text D2 based on the original text D1. For example, it evaluates how good a quality summary the summary text D2 is as a summary of the original text D1.
[0038] In Figure 4, the dashed line in the original text D1 contains the text "It will be sunny in the afternoon (tomorrow)." The summarized text D2 in Figure 4 does not include the corresponding text. In this respect, the summarized text D2 in Figure 4 fails to cover important information and can be said to be of low quality as a summary.
[0039] Question D6 is text representing a question obtained in relation to the source text D1. In particular, Question D6 is generated based on important information contained in the source text D1. Important questions will be discussed later. In Figure 4, Question D6 includes the question "Q3. What will the weather be like tomorrow afternoon?" which corresponds to the dashed line portion of the source text D1.
[0040] The first answer D71 is the answer D7 to question D6, generated by referring to the original text D1. The dashed portion of the first answer D71 in Figure 4 includes the answer "A3. It's sunny." to the above question "Q3," based on the original text D1.
[0041] The second answer D72 is the answer generated from the answer D7 to question D6, referring to the summary text D2. Since the summary text D2 in Figure 4 does not contain information about the weather tomorrow afternoon, the dashed line portion of the second answer D72 in Figure 4 contains the answer "A3. Unknown." as the answer to the above question "Q3" based on the summary text D2.
[0042] The evaluation unit 112 evaluates the summary text D2 based on the similarity between the first answer D71 and the second answer D72. In the example in Figure 4, the content of the answer "A3" differs between the first answer D71 and the second answer D72, so in this respect, the first answer D71 and the second answer D72 are not similar. In other words, the summary text D2 does not adequately cover the important information contained in the original text D1. For this reason, the summary text D2 is evaluated as having low quality as a summary.
[0043] In the following, for comparison, an overview of processing by a text evaluation device according to the conventional art will be described using FIG. 10. In the text evaluation device according to the conventional art, questions are generated based on the summary text D2. For this reason, no question is generated for information not included in the summary text D2. Therefore, unlike the example in FIG. 4, no question relating to the dashed line portion of the original text D1, that is, the weather tomorrow afternoon, is generated.
[0044] As a result, the summary text D2 is evaluated based only on the information contained in the summary text D2. For this reason, the information in the dashed line portion that is not covered in the summary text D2 does not affect the evaluation. Therefore, the answer to the question is the same when referring to the original text D1 and when referring to the summary text D2. Accordingly, the summary text D2 is evaluated as having high quality as a summary.
[0045] As described above, the text evaluation device 10 (FIG. 4) according to the present embodiment can evaluate the degree to which important information contained in the original text D1 is covered in the summary text D2, unlike the text evaluation device (FIG. 10) according to the conventional art. In this way, the evaluation method for the summary text D2 is improved.
[0046] [1-6. Processing Details] Hereinafter, detailed examples of processing by the text evaluation device 10 according to the present embodiment will be described with reference to FIGS. 5 to 9. The text handled by the text evaluation device 10 is expressed, for example, in Japanese. In the present embodiment, the text is in Japanese. In the present embodiment, the texts transmitted and received between the text evaluation device 10 and the LLM 30 are also in Japanese.
[0047] Note that, as a modification, the text handled by the text evaluation device 10 may be expressed in a language other than Japanese. For example, the text handled by the text evaluation device 10 may be expressed in English or Chinese. Furthermore, the text handled by the text evaluation device 10 may include a mixture of multiple languages. For example, in at least one of the original text D1, summary text D2, prompt D3, important information D4, candidate question D5, question D6, answer D7, etc., the main part may be expressed in Japanese, and the other parts may be expressed in English.
[0048] FIG. 5 is a flowchart illustrating an example of processing performed by the text evaluation device 10 according to the present embodiment. FIG. 6 shows a specific example of the first prompt D31 among the prompts D3 used by the text evaluation device 10. FIG. 7 shows a specific example of the second prompt D32 among the prompts D3 used by the text evaluation device 10. FIG. 8 shows a specific example of the third prompt D33 among the prompts D3 used by the text evaluation device 10. FIG. 9 shows a specific example of the fourth prompt D34 among the prompts D3 used by the text evaluation device 10.
[0049] In each of the prompts shown in FIG. 6 to FIG. 8, the portions indicated by the {{}} symbols represent variables. The evaluation unit 112 can appropriately assign suitable values to each variable, either alone or in cooperation with the LLM 30. Alternatively, the evaluation unit 112 can appropriately replace each variable with a suitable character string, either alone or in cooperation with the LLM 30.
[0050] The processing in FIG. 5 is started, for example, in response to the text evaluation device 10 receiving an instruction for evaluating a summary from the terminal 20.
[0051] In the processing of FIG. 5, first, the acquisition unit 111 acquires the original text D1 (step S1). The original text D1 is the text that is the target of summary processing. The original text D1 includes, for example, at least one or more sentences. The original text D1 is received, for example, from the terminal 20. Alternatively, the original text D1 may be stored in the storage device 12 in advance.
[0052] Next, the evaluation unit 112 inputs the first prompt D31 to the LLM 30 (step S2). As shown in Figure 6, the first prompt D31 includes an instruction text (first instruction text) for commanding the acquisition of important information D4 from the source text D1. In the example of the first prompt D31 shown in Figure 6, {{input}} is a variable that refers to text data as the content of the source text D1. That is, the first prompt D31 includes the source text D1.
[0053] In the example in Figure 6, the command is to obtain important information D4 by using the expression, "Extract items that are considered important from a business perspective from the minutes given in the input text." In this expression, "minutes given in the input text" refers to the original text D1. Also, "items that are considered important from a business perspective" refers to important information D4. Thus, the first prompt D31 includes the original text D1 and the first command text that commands the acquisition of important information D4 from the original text D1.
[0054] Furthermore, the expression indicating important information D4 in the first instruction text is not limited to "items considered important from a business perspective." For example, the perspective may not be limited to business. Alternatively, other perspectives may be specified. The expression indicating important information D4 may be any of the following: "items considered important," "important information," "important matters," "important elements," "important agenda items," "important topics," etc.
[0055] The first instruction text may include information representing the number or quantity of important information D4. For example, {{question_number}} in Figure 6 is a variable representing the number of important information D4 to be obtained. The value of {{question_number}} may be received from terminal 20. Alternatively, the value of {{question_number}} may be predetermined as a fixed value.
[0056] The first instruction text may include information representing the limiting conditions imposed when obtaining the important information D4. In the example in Figure 6, the limiting conditions imposed when obtaining the important information D4 are described immediately after the expression "#condition".
[0057] As an example of a limiting condition, the first instruction text may include text that instructs each of the important pieces of information D4 to be expressed with a length greater than or equal to a predetermined threshold length. The predetermined threshold length may be expressed, for example, by the number of characters. In the example in Figure 6, this limiting condition is described by the expression "50 characters or more". The predetermined threshold length may also be expressed by the number of words. Such a limiting condition ensures that the important pieces of information D4 are text of sufficient length.
[0058] As an example of a limiting condition, the first instruction text may include text that instructs the important information D4 to be presented in bulleted list format. In the example in Figure 6, this limiting condition is described by the expression "bulleted list".
[0059] As an example of a limiting condition, the first instruction text may include text that instructs not to include people's names in important information D4. In the example in Figure 6, this limiting condition is described by the expression "do not include people's names." Generally, the speaker's name is often not important in text. Such a limiting condition allows us to exclude unimportant information from important information D4.
[0060] As a variation, the first instruction text may include text that instructs not to include the speaker's name in important information D4. With this variation, names of people other than the speaker are not excluded from important information D4. For example, if the original text D1 is a text that describes a specific person, the name of the person being described can be included in important information D4 while excluding the speaker's name. In this way, relatively important names can be included in important information D4.
[0061] As an example of a limiting condition, the first instruction text may include text that instructs that important information D4 should not contain multiple items that overlap with each other. In the example in Figure 6, this limiting condition is described by the expression, "Never extract any duplicate points." Such a limiting condition prevents a single item from being repeatedly included in important information D4.
[0062] Although not shown in Figure 5, after step S2, in response to receiving the first prompt D31, the LLM 30 generates at least one piece of important information D4. The LLM 30 outputs the generated important information D4 to the text evaluation device 10.
[0063] The evaluation unit 112 of the text evaluation device 10 receives important information D4 from the LLM 30 and acquires at least one important piece of information D4 (step S3). In this way, the evaluation unit 112 acquires important information D4 from the LLM 30 as a response to the first prompt D31. By using the LLM 30, important information D4 with appropriate content can be easily acquired.
[0064] Next, the evaluation unit 112 inputs a second prompt D32 to the LLM 30 (step S4). As shown in Figure 7, the second prompt D32 includes an instruction text (second instruction text) for instructing the generation of at least one candidate question D5 based on at least one important piece of information D4. The candidate question D5 is, for example, a candidate question whose answer can be determined based on the important piece of information D4. In the example of the second prompt D32 shown in Figure 7, {{input}} is a variable that refers to text data as the content of the important piece of information D4. That is, the second prompt D32 includes the important piece of information D4.
[0065] Here, the important information D4 and the candidate question D5 may have a one-to-one correspondence, or they may not have a one-to-one correspondence.
[0066] In the example in Figure 7, the expression "You are given the minutes of a business meeting as input. From each item in the input, generate {{question_number}} questions that meet the following conditions." instructs the system to generate at least one question as candidate question D5. Thus, the second prompt D32 includes important information D4. The second prompt D32 also includes a second instruction text to instruct the system to generate at least one candidate question D5 based on the important information D4.
[0067] The second instruction text may include information representing the number of candidate question D5s. For example, {{question_number}} in Figure 7 is a variable representing the number of candidate question D5s to be generated. The value of {{question_number}} may be received from terminal 20. Alternatively, the value of {{question_number}} may be predetermined as a fixed value. Note that the value of {{question_number}} in the second instruction text (Figure 7) and the value of {{question_number}} in the first instruction text (Figure 6) described above may be the same or different.
[0068] The second instruction text may include information representing the limiting conditions imposed when generating the candidate question D5. In the example in Figure 7, the limiting conditions imposed when generating the candidate question D5 are described immediately after the expression "#condition".
[0069] As an example of a limiting condition, the second instruction text may include text specifying that for each of at least one candidate question D5, only one answer can be determined based on the information contained in the important information D4. In the example in Figure 7, this limiting condition is described by the expression "a question that exists in the input text and whose answer is definitely narrowed down to one".
[0070] As an example of a limiting condition, the second instruction text may include text that instructs the system to designate questions concerning particularly important matters among the important information D4 as candidate questions D5. In the example in Figure 7, this limiting condition is described by the phrase "questions concerning matters considered important in the input text." According to such a limiting condition, in subsequent processing, candidate questions D5 will be generated as questions concerning particularly important matters among the important information D4.
[0071] Although not shown in Figure 5, after step S4, the LLM 30 generates a candidate question D5 in response to receiving the second prompt D32. The LLM 30 outputs the generated candidate question D5 to the text evaluation device 10.
[0072] The evaluation unit 112 of the text evaluation device 10 acquires a candidate question D5 by receiving it from the LLM 30 (step S5). In this way, the evaluation unit 112 acquires at least one candidate question D5 from the LLM 30 as a response to the second prompt D32. By using the LLM 30, an appropriate candidate question D5 can be easily acquired.
[0073] Here, the candidate question D5 is generated based on important information D4. Furthermore, important information D4 is information contained in the original text D1. Therefore, it can be said that the evaluation unit 112 obtains at least one candidate question D5 based on the important information D4 contained in the original text D1.
[0074] Next, the evaluation unit 112 inputs a third prompt D33 to the LLM 30 (step S6). As shown in Figure 8, the third prompt D33 includes an instruction text (third instruction text) that commands the deletion of one of the question candidates whose content overlaps with that of the other. That is, the third instruction text includes text that commands the deletion of either the first question candidate or the second question candidate if the content of the first question candidate among at least one or more question candidates D5 overlaps with the content of the second question candidate among at least one or more question candidates D5.
[0075] In the example of the third prompt D33 shown in Figure 8, {{input}} is a variable that refers to the text data representing the content of the candidate question D5. Thus, the third prompt D33 includes at least one candidate question D5 and the third instruction text.
[0076] Although not shown in Figure 5, after step S6, the LLM 30 generates question D6 in response to receiving the third prompt D33. Here, the LLM 30, for example, deletes zero or more question candidates that overlap with other question candidates from among the question candidates D5, and generates question D6 which includes all the question candidates that were not deleted. Note that "deleting zero or more question candidates" means that if there are no question candidates that overlap with each other, no question candidates are deleted. The LLM 30 outputs the generated question D6 to the text evaluation device 10.
[0077] In the example shown in Figure 4, three questions D6 are presented. Note that in the example in Figure 4, the original text D1 and questions D6 are highly simplified for the purpose of explaining the operation of the text evaluation device 10.
[0078] The evaluation unit 112 of the text evaluation device 10 acquires question D6 by receiving it from the LLM 30 (step S7). In this way, the evaluation unit 112 acquires at least one question D6 from the LLM 30 as a response to the third prompt D33. By using the LLM 30, it is possible to generate question D6 while appropriately considering the semantic overlap of the candidate question D5.
[0079] Here, question D6 is generated based on candidate question D5. Therefore, it can be said that the evaluation unit 112 obtains at least one question D6 based on at least one candidate question D5. Furthermore, candidate question D5 is generated based on important information D4, and important information D4 is information contained in the original text D1. Therefore, it can be said that the evaluation unit 112 obtains at least one question D6 based on important information D4 contained in the original text D1.
[0080] Next, the evaluation unit 112 obtains an answer D7 to question D6 based on the question D6 and the source text D1 (step S8). That is, for each of at least one question D6, the evaluation unit 112 obtains a first answer D71 as the answer D7 to question D6 based on the question D6 and the source text D1. As mentioned above, each of the at least one question D6 was included in at least one candidate question D5. In the example in Figure 4, a total of three first answers D71 are shown, corresponding to each of the three questions D6.
[0081] In step S8, a specific method for the evaluation unit 112 to obtain the first answer D71 can be appropriately designed by a person skilled in the art. For example, an LLM 30 may be used. That is, the evaluation unit 112 may first send a prompt to the LLM 30 that includes a question D6, a source text D1, and an instruction text that commands the LLM 30 to generate an answer to question D6 based on the source text D1. The evaluation unit 112 may then obtain the first answer D71 from the LLM 30 as a response to this prompt.
[0082] After step S8, the acquisition unit 111 acquires the summary text D2 (step S9). The summary text D2 is a summary of the original text D1. The summary text D2 includes, for example, at least one sentence. The summary text D2 is received, for example, from the terminal 20. Alternatively, the summary text D2 may be stored in the storage device 12 beforehand. Note that this step S9 may be executed at any point before the subsequent step S10. For example, step S9 may be executed simultaneously with step S1, immediately after step S1, or in parallel with step S1.
[0083] Next, the evaluation unit 112 obtains an answer D7 to question D6 based on the question D6 and the summary text D2 (step S10). That is, for each of at least one question D6, the evaluation unit 112 obtains a second answer D72 as the answer D7 to question D6 based on the question D6 and the summary text D2.
[0084] In the example in Figure 4, a total of three second answers D72 are shown, corresponding to each of the three questions D6.
[0085] In step S10, a specific method for the evaluation unit 112 to obtain the second answer D72 can be appropriately designed by a person skilled in the art. For example, LLM30 may be used, similar to step S8.
[0086] After step S10, the evaluation unit 112 inputs the fourth prompt D34 to the LLM 30 (step S11). As shown in Figure 9, the fourth prompt D34 includes an instruction text (fourth instruction text) that commands the evaluation of the similarity between the two input sentences. In the example in Figure 9, the similarity is expressed as an integer between 1 and 5. Also, in the example in Figure 9, a larger value is output as the similarity when the two input sentences are very similar to each other.
[0087] Note that the example in Figure 9 shows an example where the answer D7 is expressed in the form of a "sentence." However, if the answer D7 includes information in a format other than a sentence, a person skilled in the art may modify the fourth instruction text according to the format of the answer D7.
[0088] In the example in Figure 9, the contents of the two input statements are not explicitly referenced, but the evaluation unit 112 may input the first response D71 and the second response D72 to the LLM 30 as two input statements associated with the fourth prompt D34. The fourth prompt D34 may also include variables that reference text data representing the contents of the first response D71 and the second response D72, respectively.
[0089] In the example in Figure 9, the entirety of the first answer D71 and the entirety of the second answer D72 are each input into LLM30 as a single sentence. In this case, one numerical value is output as the similarity score, which is the evaluation result. As a variation, each of the answers included in the first answer D71 and each of the answers included in the second answer D72 may be input into LLM30 as a pair. That is, for example, the pair of "A1. It's sunny." from the first answer D71 and "A1. It's sunny." from the second answer D72 in Figure 4, the pair of "A2. It's raining." from the first answer D71 and "A2. It's raining." from the second answer D72, and the pair of "A3. It's sunny." from the first answer D71 and "A3. It's unknown." from the second answer D72 may each be input into LLM30 as subjects for similarity evaluation. In this case, the similarity score, which is the evaluation result, will be output the same number of times as the number of questions D6.
[0090] Although not shown in Figure 5, after step S11, the LLM 30 determines the similarity score in response to receiving the fourth prompt D34. The LLM 30 outputs the determined similarity score to the text evaluation device 10.
[0091] The evaluation unit 112 of the text evaluation device 10 acquires the similarity score by receiving it from the LLM 30 (step S12). In this way, the evaluation unit 112 acquires the similarity score from the LLM 30 as a response to the fourth prompt D34.
[0092] If the similarity is high, that is, if the first answer D71 and the second answer D72 are very similar to each other, then it can be considered that the original text D1 and the summary text D2 contain substantially the same information, and therefore the summary text D2 can be said to be a good summary of the original text D1. On the other hand, if the similarity is low, that is, if the first answer D71 and the second answer D72 are very different to each other, then it can be considered that the original text D1 and the summary text D2 contain different information, and therefore the summary text D2 cannot be said to be a good summary of the original text D1.
[0093] Here, by using LLM30 to determine similarity, it becomes possible to make a similarity judgment that takes into account the meaning of each answer. As a variation, similarity may be determined without using LLM30. For example, similarity may be determined based on the matching rate of characters that appear in common to the two answers. However, in such a variation, even if the conclusions of the two answers are different, the similarity may be high if the long expressions included in the answers are the same. For this reason, it is preferable to use LLM30, especially from the viewpoint of determining similarity while taking into account the meaning of the answers.
[0094] Next, the evaluation unit 112 evaluates the summary text D2 based on the similarity (step S13). For example, the similarity obtained in step S12 may be used directly as the evaluation result of the summary text D2. Alternatively, an expression representing the evaluation result may be determined based on the similarity obtained in step S12. The expression representing the evaluation result may include at least one of the following: a number, a string, an image, etc. In this way, the evaluation unit 112 obtains information (evaluation result information) that represents the result of evaluating the summary text D2. The evaluation result information is information relating to the result of evaluating the summary text D2.
[0095] The evaluation results information may include, for example, the similarity between the entire first response D71 and the entire second response D72. Furthermore, the evaluation results information may also include, for example, the similarity between each of the questions D6.
[0096] In this way, the evaluation unit 112 evaluates the summary text D2 based on the first answer D71 and the second answer D72 for each of at least one question D6.
[0097] Next, the evaluation unit 112 transmits the evaluation result information to the terminal 20 (step S14). In step S14, the terminal 20 receives the evaluation result information from the text evaluation device 10. Next, the terminal 20 displays the evaluation result information on the display device 24 (step S15). This allows the user of the text evaluation device 10 to recognize the evaluation result of the summary text D2 on the terminal 20.
[0098] In step S14 described above, the evaluation unit 112 may, in addition to or instead of transmitting the evaluation result information to the terminal 20, output the evaluation result information in another format. For example, the evaluation unit 112 may display the evaluation result information on the display device 14. The evaluation unit 112 may store the evaluation result information in the storage device 12.
[0099] As described above, the text evaluation device 10 and text evaluation system 1 according to this embodiment improve the method of evaluating summaries. In particular, it is possible to evaluate the degree to which important information D4 contained in the original text D1 is covered in the summarized text D2.
[0100] Furthermore, since reference summaries are not required as correct answers, it is possible to efficiently evaluate a large number of summary texts D2.
[0101] Furthermore, since candidate question D5 is not generated directly from the original text D1 but is generated via important information D4, it is possible to suppress the generation of candidate question D5 from unimportant parts of the original text D1. For this reason, even if the summary text D2 does not contain unimportant information, the evaluation of the summary text D2 will not decrease.
[0102] The first instruction text of the first prompt D31 includes text that instructs the system to represent each of the important pieces of information D4 with a length equal to or greater than a predetermined threshold length. Therefore, each of the important pieces of information D4 contains sufficient information useful for generating the candidate question D5. As a variation, the first instruction text may not include such text.
[0103] The first instruction text of the first prompt D31 includes text that instructs the system to express important information D4 in bullet points. Therefore, when generating the candidate question D5, the contents of important information D4 can be considered without omission. As a variation, the first instruction text may not include such text.
[0104] The first command text of the first prompt D31 includes text that commands the exclusion of personal names from important information D4. Therefore, non-important information can be excluded from important information D4. As a variation, the first command text may not include such text.
[0105] The first command text of the first prompt D31 includes text instructing that the important information D4 should not contain multiple pieces of content that overlap with each other. This prevents a situation where certain items are given undue importance compared to other items due to repeated descriptions in the important information D4. This improves the accuracy of the evaluation. As a variation, the first command text may not include such text.
[0106] The second instruction text of the second prompt D32 includes text specifying that for each of the at least one candidate question D5, only one answer can be determined based on the information contained in the important information D4. As a result, variations in the answers are suppressed, and when the original text D1 and the summary text D2 contain substantially the same information, the similarity between the first answer D71 and the second answer D72 will be high. Therefore, the summary text D2 can be evaluated appropriately.
[0107] [2. Modifications] In the above-described embodiment 1, the LLM 30 is provided separately from the text evaluation device 10. In the modification, the LLM 30 may be provided as part of the text evaluation device 10.
[0108] Multiple LLMs may be provided. Furthermore, different LLMs may be used at each stage of processing. In Embodiment 1, the same LLM 30 is used in steps S2, S4, S6, S11, etc., but other LLMs may be used in at least one of these steps.
[0109] Furthermore, LLM30 may be omitted. For example, the text evaluation device 10 may use a language processing technique different from known LLM to obtain at least one of the important information D4, question candidates D5, questions D6, and answers D7.
[0110] Steps S6 and S7 may be omitted. In that case, none of the questions included in candidate question D5 will be deleted, so candidate question D5 and question D6 will have the same content. For example, candidate question D5 will be treated as question D6.
[0111] [3. Others] (1) Each function illustrated in Figures 2 and 3 can be implemented by any combination of hardware and software. The method of implementing each function is not particularly limited. Each function may be implemented using one physically or logically coupled device, or it may be implemented using a device configured by directly or indirectly connecting two or more physically or logically separated devices (for example, using wired, wireless, etc.). Each function may be implemented by combining the above one device or the above multiple devices with software.
[0112] (2) In this disclosure, the term “apparatus” may be replaced with other terms such as circuit, device or unit.
[0113] (3) In each embodiment and each modification, the storage device may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., compact disc, digital multipurpose disc, Blu-ray® disc), a smart card, flash memory (e.g., card, stick, key drive), a floppy® disk, a magnetic strip, etc. The program may also be transmitted from a network via a telecommunications line.
[0114] (4) Each embodiment and each modification is based on LTE (Long Term Evolution), LTE-A (LTA-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), 6th generation mobile communication system (6G), xth generation mobile communication system (xG) (where x is, for example, an integer or decimal), FRA (Future Radio Access), NR (new Radio), New radio access (NX), Future generation radio access (FX), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20 may apply to at least one system utilizing UWB (Ultra-WideBand), Bluetooth®, or other appropriate systems, and to next-generation systems extended, modified, created, or defined based thereon. Alternatively, multiple systems may be applied in combination (e.g., a combination of at least one of LTE and LTE-A with 5G).
[0115] (5) The processing procedures, sequences, or flowcharts illustrated in each embodiment and each modification may be in any order, as long as they do not contradict each other. For example, the methods described in this disclosure present various step elements in an exemplary order and are not limited to the specific order presented.
[0116] (6) In each embodiment and each modification, the input and output information may be stored in a specific location (e.g., memory) or managed using a management table. The input and output information may be overwritten, updated, or appended to. The output information may be deleted. The input information may be transmitted to other devices.
[0117] (7) In each embodiment and each modification, the determination may be based on a value represented by one bit (0 or 1), on a Boolean value (true or false), or on a numerical comparison (for example, a comparison with a predetermined value).
[0118] (8) The programs illustrated in each embodiment and each modification should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, or functions, whether they are called software, firmware, middleware, microcode, or hardware description languages or by other names. Furthermore, software, or instructions, etc., may be transmitted or received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technology (such as coaxial cable, fiber optic cable, twisted pair, and digital subscriber line (DSL)) and wireless technology (such as infrared, microwave, etc.), at least one of these wired and wireless technologies is included in the definition of a transmission medium.
[0119] (9) The information described in each embodiment and each modification may be represented using any of the various different technologies. For example, the data, information, etc. that may be referred to throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields, magnetic particles, optical fields, photons, or any combination thereof. Notwithstanding the terms described herein and the terms necessary for understanding this disclosure, terms may be replaced with terms having the same or similar meanings.
[0120] (10) In each embodiment and each variation, the terms “system” and “network” are used interchangeably.
[0121] (11) In each embodiment and each variation, the terminal is, for example, a mobile station. A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or several other appropriate terms.
[0122] (12) A mobile station may also be called a transmitting device, receiving device, or communication device. A mobile station may also be a device mounted on a mobile body, or the mobile body itself. A mobile body means a movable object. The speed of movement of a mobile body is arbitrary. A mobile body can be stopped. A mobile body includes, but is not limited to, vehicles, transport vehicles, automobiles, motorcycles, bicycles, connected cars, excavators, bulldozers, wheel loaders, dump trucks, forklifts, trains, buses, handcarts, rickshaws, ships and other watercraft, airplanes, rockets, satellites, drones (registered trademark), multicopters, quadcopters, balloons, and things mounted on them. A mobile body may be a mobile body that moves autonomously based on operational commands. A mobile body may be a vehicle (e.g., a car, an airplane, etc.), an unmanned mobile body (e.g., a drone, an autonomous vehicle, etc.), or a robot (manned or unmanned). A mobile station also includes devices that do not necessarily move during communication operations. For example, the mobile station could be an IoT (Internet of Things) device such as a sensor.
[0123] (13) In each embodiment and each variation, the term “decision” may encompass a wide variety of actions. “Decision” may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiry (e.g., searching in a table, database or another data structure), and ascertaining. “Decision” may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and accessing (e.g., accessing data in memory). “Decision” may also include resolving, selecting, choosing, establishing, and comparing. In other words, "decision" can include considering that some action has been "decided." Furthermore, "decision" can be reinterpreted as "assuming," "expecting," or "considering."
[0124] (14) In each embodiment and each modification, the term “connected,” or any variation thereof, means any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access.” As used in the present disclosure, two elements may be considered to be “connected” or “coupled” to each other using at least one of one or more wires, cables, and printed electrical connections, and, in some non-limiting and non-exclusive examples, electromagnetic energy having wavelengths in the radio frequency domain, microwave domain, and optical (both visible and invisible) domain.
[0125] (15) In each embodiment and each modification, the phrase “based on” does not mean “based solely on” unless otherwise specified. In other words, the phrase “based on” means both “based solely on” and “based at least on.”
[0126] (16) Any reference to elements using the designations “first” and “second” as used in this disclosure does not limit the quantity or order of those elements in general. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the first and second elements do not imply that only two elements may be adopted or that the first element must precede the second element in any way.
[0127] (17) Where “include,” “including,” and variations thereof are used in this disclosure or claims in each embodiment and each variation, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, where the term “or” is used in this disclosure or claims, it is intended not to be an exclusive OR.
[0128] (18) Where articles are added in the translation, for example, a, an, and the in English, the disclosure may include the fact that the noun following these articles is plural.
[0129] (19) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a given value, or other corresponding information.
[0130] (20) In this disclosure, the term “A and B are different” may mean “A and B are different from each other.” The term may also mean “A and B are each different from C.” Terms such as “separate” and “combined” may be interpreted in the same way as “different.”
[0131] (21) Each aspect / embodiment described herein may be used individually, in combination, or switched between as needed during implementation. Furthermore, notification of certain information (e.g., notification that "it is X") is not limited to explicit notification, but may also be implicit (e.g., by not providing such notification).
[0132] (22) It will be obvious to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented in modified and altered forms without departing from the spirit and scope of the invention as defined by the claims. Accordingly, the descriptions herein are for illustrative purposes only and are not intended to be restrictive in any way to the present invention. Furthermore, multiple embodiments selected from those illustrated herein may be combined.
[0133] [4. Aspects understood from the above embodiments or modifications] The following aspects can be understood from at least one of the above embodiments or modifications.
[0134] [4-1. First Embodiment] The text evaluation device according to the first embodiment comprises: an acquisition unit that acquires original text and summarized text obtained by summarizing the original text; and an evaluation unit that evaluates the summarized text based on the original text, wherein the evaluation unit acquires at least one question based on at least one important piece of information contained in the original text; the evaluation unit acquires a first answer to each of the at least one question based on the question and the original text; and acquires a second answer to the question based on the question and the summarized text; and the evaluation unit evaluates the summarized text based on the first and second answers to each of the at least one question.
[0135] This embodiment improves the method for evaluating summaries. In particular, it allows for the evaluation of the degree to which important information contained in the original text is covered in the summary.
[0136] [4-2. Second Embodiment] According to the text evaluation device of the second embodiment, the evaluation unit inputs a first prompt to a large-scale language model, the first prompt includes the original text and a first command text for commanding the acquisition of at least one important piece of information from the original text, and the evaluation unit acquires at least one important piece of information from the large-scale language model as a response to the first prompt.
[0137] According to this embodiment, by using LLM, important information of appropriate content can be easily obtained.
[0138] [4-3. Third Embodiment] According to the text evaluation device of the third embodiment, the first instruction text includes text that instructs that each of the at least one important piece of information be expressed with a length equal to or greater than a threshold length.
[0139] According to this embodiment, each of the important pieces of information D4 contains sufficient information useful when generating the candidate questions D5.
[0140] [4-4. Fourth Embodiment] According to the text evaluation device of the fourth embodiment, the first instruction text includes text that instructs the expression of at least one important piece of information in bulleted list format.
[0141] According to this embodiment, when generating candidate questions D5, the contents of important information D4 can be taken into full consideration.
[0142] [4-5. Fifth Embodiment] According to the text evaluation device of the fifth embodiment, the first command text includes text that commands not to include a person's name in at least one important piece of information.
[0143] According to this configuration, unimportant information can be excluded from important information.
[0144] [4-6. Sixth Embodiment] According to the text evaluation device of the sixth embodiment, the first instruction text includes text that instructs that the at least one important piece of information not include multiple pieces of content that overlap with each other.
[0145] According to this embodiment, the situation in which certain matters are given undue importance compared to other matters due to repeated descriptions in important information D4 is avoided.
[0146] [4-7. Seventh Embodiment] According to the text evaluation device of the seventh embodiment, the evaluation unit inputs a second prompt to a large language model, the second prompt includes at least one important piece of information and a second command text for commanding the generation of at least one question candidate based on the at least one important piece of information, the evaluation unit obtains at least one question candidate from the large language model as a response to the second prompt, and the evaluation unit obtains at least one question based on the at least one question candidate.
[0147] According to this embodiment, appropriate question candidates can be easily obtained by using LLM.
[0148] [4-8. Eighth Embodiment] According to the text evaluation device of the eighth embodiment, the second instruction text includes text that specifies that for each of the at least one candidate question, only one answer can be determined based on the information contained in the at least one important piece of information.
[0149] This configuration suppresses inconsistencies in responses, allowing for a more accurate evaluation of the summarized text.
[0150] [4-9. Ninth Embodiment] According to the text evaluation device of the ninth embodiment, the evaluation unit inputs a third prompt to a large-scale language model, the third prompt includes at least one question candidate and a third command text, the third command text includes text that commands the deletion of either the first question candidate or the second question candidate if the content of the first question candidate among the at least one question candidate overlaps with the content of the second question candidate among the at least one question candidate, and the evaluation unit obtains at least one question from the large-scale language model as a response to the third prompt.
[0151] According to this embodiment, questions can be generated while appropriately considering semantic overlaps among candidate questions.
[0152] [4-10. Tenth Embodiment] The text evaluation system according to the tenth embodiment is a text evaluation system comprising a text evaluation device according to the first embodiment and a terminal, wherein the evaluation unit transmits evaluation result information relating to the result of evaluating the summarized text to the terminal, and the terminal comprises a receiving unit that receives the evaluation result information from the text evaluation device and a display unit that displays the evaluation result information.
[0153] According to this embodiment, the user of the text evaluation device can recognize the evaluation result of the summarized text on the terminal.
[0154] 1...Text evaluation system, 10...Text evaluation device, 20...Terminal, 30...LLM, 111...Acquisition unit, 112...Evaluation unit, 211...Transmission unit, 212...Receiver unit, 213...Display unit, D1...Original text, D2...Summary text, D3...Prompt (D31...First prompt, D32...Second prompt, D33...Third prompt, D34...Fourth prompt), D4...Important information, D5...Question candidate, D6...Question, D7...Answer (D71...First answer, D72...Second answer), PG1...Program, PG2...Program.
Claims
1. A text evaluation device comprising: an acquisition unit that acquires original text and a summary text which is a summary of the original text; and an evaluation unit that evaluates the summary text based on the original text, wherein the evaluation unit acquires at least one question based on at least one important piece of information contained in the original text; the evaluation unit acquires a first answer to each of the at least one question based on the question and the original text; the evaluation unit acquires a second answer to the question based on the question and the summary text; and the evaluation unit evaluates the summary text based on the first and second answers to each of the at least one question.
2. The text evaluation device according to claim 1, wherein the evaluation unit inputs a first prompt to a large-scale language model, the first prompt includes the source text and a first command text for commanding the acquisition of at least one important piece of information from the source text, and the evaluation unit acquires at least one important piece of information from the large-scale language model as a response to the first prompt.
3. The text evaluation device according to claim 2, wherein the first instruction text includes text that instructs the representation of each of the at least one important piece of information with a length equal to or greater than a threshold length.
4. The text evaluation device according to claim 2, wherein the first instruction text includes text that instructs the representation of at least one important piece of information in bullet points.
5. The text evaluation device according to claim 2, wherein the first instruction text includes text that instructs not to include a person's name in at least one important piece of information.
6. The text evaluation device according to claim 2, wherein the first instruction text includes text that instructs that the at least one important piece of information not include multiple pieces of content that overlap with each other.
7. The text evaluation device according to claim 1, wherein the evaluation unit inputs a second prompt to a large-scale language model, the second prompt includes at least one important piece of information and a second instruction text for instructing the model to generate at least one candidate question based on the at least one important piece of information, the evaluation unit obtains at least one candidate question from the large-scale language model as a response to the second prompt, and the evaluation unit obtains at least one question based on the at least one candidate question.
8. The text evaluation device according to claim 7, wherein the second instruction text includes text specifying that for each of the at least one candidate question, only one answer can be determined based on the information contained in the at least one important piece of information.
9. The text evaluation device according to claim 7, wherein the evaluation unit inputs a third prompt to a large-scale language model, the third prompt includes at least one of the question candidates and a third command text, the third command text includes text that commands the deletion of either the first question candidate or the second question candidate if the content of the first question candidate among the at least one of the question candidates overlaps with the content of the second question candidate among the at least one of the question candidates, and the evaluation unit obtains at least one of the questions from the large-scale language model as a response to the third prompt.
10. A text evaluation system comprising a text evaluation device according to claim 1 and a terminal, wherein the evaluation unit transmits evaluation result information relating to the result of evaluating the summarized text to the terminal, and the terminal comprises a receiving unit for receiving the evaluation result information from the text evaluation device, and a display unit for displaying the evaluation result information.
