Information provision device, information provision method, and information provision program
The information providing apparatus enhances language model performance in specialized tasks by using multiple LLM instances to generate reflections based on case studies and user evaluations, resulting in improved document generation accuracy.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- NT T INC
- Filing Date
- 2024-12-06
- Publication Date
- 2026-06-11
AI Technical Summary
Conventional language models struggle to improve inference performance in tasks requiring high specialization due to insufficient knowledge compared to expert perspectives.
An information providing apparatus and method that utilizes multiple instances of language models (LLM) to generate reflections based on acquired information, including case studies and user evaluations, to enhance the generation process.
Improves the inference performance of language models in specialized tasks by providing higher quality reflections, leading to more accurate document generation.
Smart Images
Figure JP2024043316_11062026_PF_FP_ABST
Abstract
Description
Information providing apparatus, information providing method, and information providing program
[0001] The present invention relates to an information providing apparatus, an information providing method, and an information providing program.
[0002] As a technique of Prompting for improving the inference performance of an LLM, there is Reflection (see, for example, Non-Patent Document 1). The effectiveness of Reflection has been confirmed in various tasks such as report generation and document summarization.
[0003] Reflection is a technique for obtaining better generated text by causing the LLM to reflect on (including criticism, for example) the document generated by the LLM and then causing it to perform generation considering the reflection again. Reflection is used in various cases due to its high versatility.
[0004] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao, "Reflexion: Language Agents with Verbal Reinforcement Learning", [online], [searched on November 25, 2024], Internet <URL: https: / / arxiv.org / abs / 2303.11366>
[0005] However, in the conventional technology, it may be difficult to improve the inference performance of the model in tasks with high specialization.
[0006] In Reflection, the LLM generates reflection based on its own knowledge and the like. Therefore, in the case of a task that requires high specialization such as a security report, it is difficult to generate reflection from the perspective of an expert alone. This is because the knowledge possessed by the LLM is insufficient compared to the knowledge of an expert.
[0007] Therefore, an object of the present invention is to improve the inference performance of the model in tasks with high specialization.
[0008] To solve the aforementioned problems, the information providing device of the present invention is characterized by comprising: an acquisition unit that acquires information related to a query requesting the generation of a document by a language model; a document generated by the language model in response to the query; a reflection generation unit that causes the language model to generate reflections based on the information acquired by the acquisition unit; and a response proposal generation unit that causes the language model to generate a document based on the reflections and the query.
[0009] According to the present invention, the inference performance of a model can be improved in highly specialized tasks.
[0010] Figure 1 is a diagram illustrating the overview of the information provision device. Figure 2 is a diagram showing an example configuration of the information provision device according to the first embodiment. Figure 3 is a flowchart showing the processing flow of the first embodiment. Figure 4 is a diagram showing an example of a prompt for generation. Figure 5 is a diagram showing an example of a prompt for evaluation. Figure 6 is a diagram showing an example configuration of a computer that executes the information provision program.
[0011] The embodiments for carrying out the present invention will be described below with reference to the drawings. The present invention is not limited to these embodiments.
[0012] [First Embodiment] Reflection, as described in Non-Patent Document 1, will be explained. Reflection is a method that outputs a response generated by LLM in response to an input query. In Reflection, in response to a query from the user, LLM first generates a proposed response. Then, another instance of LLM generates a reflection on the generated proposed response. The LLM of the instance that generated the proposed response updates the proposed response based on the reflection. In Reflection, a cycle including the generation of a proposed response and the generation of a reflection is repeated.
[0013] The information-providing device of the first embodiment repeats a cycle that includes generating response proposals and generating reflections, similar to Reflection. However, the information-providing device aims to improve the accuracy of the reflections by providing LLM with information to assist in generating the reflections.
[0014] The overview of the information provision device will be explained using Figure 1. Figure 1 is a diagram illustrating the overview of the information provision device.
[0015] The information providing device processes data using LLM21, LLM22, and LLM23 shown in Figure 1. LLM21, LLM22, and LLM23 may be different instances based on the same LLM. Instances are obtained by assigning roles to the base LLM via prompts. Note that LLM21, LLM22, and LLM23 may be different LLMs. An LLM is an example of a language model. For example, LLM is ChatGPT (registered trademark).
[0016] LLM21 is assigned the role of generating a response proposal from a query, or from a query and reflection. LLM22 is assigned the role of generating reflection from a response proposal and case study. LLM23 is assigned the role of searching for case studies. LLM21, LLM22, and LLM23 may also be called Generator LLM, Reflector LLM, and Search LLM, respectively.
[0017] For example, the proposed response, reflection, and case study may all be information expressed in natural language text. The proposed response, reflection, and case study may also be referred to as documents, etc. The proposed response, reflection, and case study may be in any format as long as they are in a format that can be interpreted using LLM.
[0018] The case database is a database that stores information useful to users (hereinafter referred to as "cases"). For example, a case is data that links past user queries to LLM, LLM's responses to those queries, and user evaluations of those responses. In addition, data that received high user evaluations may be stored in the case database. Another example is data that links documents, reports, and other texts created by experts with the conditions under which such texts were created.
[0019] As shown in Figure 1, LLM21 receives a response request query input from user U1 (step S1). Based on the response request query, LLM21 generates a response (step S5).
[0020] Furthermore, LLM23 accepts a search query input from user U1 (step S2). LLM23 converts the search query into a query for searching an external information source (case database). For example, LLM23 understands the task from the search query input in natural language and converts the search query into a logical expression using keywords based on its understanding. Alternatively, for example, LLM23 understands the task from the search query input in natural language and converts the search query into a vector for vector search based on text similarity based on its understanding.
[0021] LLM23 uses the converted search query to search the case database (step S3). Alternatively, the conversion by LLM23 may be omitted, and the search query entered by user U1 may be used directly for the search.
[0022] The search query is expected to be related to the response request query entered in step S1. For example, if the response request query is "Please propose effective measures to combat global warming," then a search query such as "Please search for reports on global warming created by experts in the past" might be entered.
[0023] The cases obtained by searching the case database, i.e., similar cases 33, are provided to the LLM 22 (step S4).
[0024] LLM21 generates a proposed response 31 based on the proposed response request query (step S5). LLM22 generates a reflection 32 based on the proposed response 31 and similar cases 33 (step S6).
[0025] LLM22 is given a pre-defined role by prompts to generate reflections. For example, the prompts input to LLM22 may indicate that it should generate a document criticizing response proposal 31 if similar case 33 is considered correct, a document pointing out errors in response proposal 31 based on similar case 33, a document for improving response proposal 31 by referring to similar case 33, etc., as reflections.
[0026] For example, reflection 32 may point out the differences between similar case 33 and response 31. If response 31 is "Measure A is effective against global warming," then reflection 32 may be generated as "Measure B is more highly regarded than A as a measure against global warming."
[0027] The generated reflection 32 is added to and stored in the reflection log. The information providing device stores the reflection log. The information providing device inputs the response proposal request query, along with the reflection log, into the LLM 21. The LLM 21 then considers the reflection log and generates a response proposal 31 based on the response proposal request query.
[0028] For example, if the response request query is "Please propose effective measures to combat global warming," and reflection 32 is "Measure B is more highly regarded than measure A as a measure to combat global warming," then a response 31 such as "Measures A or B are effective against global warming" may be generated. This can be said to be a result of the statement "Measure A is effective against global warming" being altered by the influence of reflection 32.
[0029] The information provider may provide the reflection log to the LLM21 as RAG (Retrieval-Augmented Generation). RAG means providing highly reliable external information to the LLM in order to improve the accuracy of the response.
[0030] The information provider repeatedly performs the following processes: generating a response proposal 31 based on the response proposal request query and the reflection log (step S5), and generating reflections 32 based on the response proposal 31 and similar cases 33 (step S6). After the repetition is complete, the last generated response proposal 31, i.e., the latest response proposal 31, is output as the final response proposal 34.
[0031] The termination condition for the loop may be defined in advance by a prompt entered into the prompt of LLM22. If the termination condition is met, LLM22 does not generate reflection 32. If reflection 32 is not generated, the information providing device 10 outputs a final response proposal.
[0032] The termination conditions may include the following: the process has been repeated a certain number of times; a certain amount of time has elapsed since the response request query was entered; or there are no longer any items to generate as reflection 32.
[0033] The configuration of the information providing device is shown in Figure 2. Figure 2 is a diagram showing an example of the configuration of the information providing device according to the first embodiment. As shown in Figure 2, the information providing device 10 has a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15.
[0034] The communication unit 11 is a module for data communication with other devices. The communication unit 11 is, for example, a NIC (Network Interface Card). The input unit 12 is an interface connected to input devices such as a mouse and a keyboard. The output unit 13 is an interface connected to output devices such as a speaker and a display.
[0035] The storage unit 14 stores data, programs, etc., that are referenced when the control unit 15 performs various processes. The storage unit 14 is implemented by semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs.
[0036] The memory unit 14 stores model information 141 and case database 142. Model information 141 consists of model parameters. For example, if the model is a neural network, then model information 141 consists of parameters such as weights and biases. In this embodiment, the model is an LLM (Long Life Model).
[0037] As explained in Figure 1, Case Study DB142 is a database that stores information (case studies) that is useful to the user.
[0038] The control unit 15 is responsible for controlling the entire information providing device 10. The functions of the control unit 15 are realized, for example, by the CPU (Central Processing Unit) executing a program stored in the memory unit 14. The control unit 15 includes a response proposal generation unit 151, an acquisition unit 152, and a reflection generation unit 153.
[0039] The response answer generation unit 151 generates a response answer using an LLM based on the response answer request query and reflection. For example, the response answer generation unit 151 executes steps S1 and S5 in FIG. 1.
[0040] The acquisition unit 152 acquires cases related to the response answer request query from the case DB 142. For example, the acquisition unit 152 executes steps S3 and S4 in FIG. 1.
[0041] For example, the case DB 142 stores a combination of a query that requests generation of a response answer by an LLM, the response answer generated by the LLM in response to the query, and the user's evaluation of the response answer. The acquisition unit 152 acquires, from the case DB 142, a combination whose similarity to the input query is greater than or equal to a threshold value. The similarity may be obtained by comparing vectors obtained by converting texts using a method such as Word2vec.
[0042] Further, the acquisition unit 152 may filter the cases in the case DB 142 in advance to those with high user evaluations. For example, the acquisition unit 152 acquires, from the case DB 142, a combination whose similarity to the input query is greater than or equal to a threshold value and whose evaluation is greater than or equal to a threshold value. Note that the evaluation may be a score set by the user for the case, or may be the ratio of users who determined that the case is good among a plurality of users.
[0043] The reflection generation unit 153 generates a reflection based on the response answer and the case. For example, the reflection generation unit 153 executes step S6 in FIG. 1.
[0044] The processing flow of the information providing apparatus 10 will be described using FIG. 3. FIG. 3 is a flowchart showing the processing flow of the first embodiment.
[0045] As shown in FIG. 3, first, the response answer generation unit 151 receives a document generation request (for example, a response answer request query) (step S101). Also, the acquisition unit 152 acquires information on similar documents from the DB (for example, the case DB 142) (step S102).
[0046] Next, the response answer generation unit 151 generates a document based on the generation request (step S103). Subsequently, the reflection generation unit 153 generates a reflection based on the generated document and the acquired information (step S104).
[0047] Furthermore, the response answer generation unit 151 generates a document based on the reflection and the generation request (step S105).
[0048] If the reflection has not ended (step S106; No), the information providing apparatus 10 returns to step S104 and repeats the process. If the reflection has ended (step S106; Yes), the information providing apparatus 10 outputs the finally generated response answer (step S107) and ends the process.
[0049] [Effect of the First Embodiment] As described so far, the acquisition unit 152 acquires information related to a query that requests generation of a document by a language model. The reflection generation unit 153 causes the language model to generate a reflection based on the document generated by the language model in response to the query and the information acquired by the acquisition unit 152. The response answer generation unit 151 causes the language model to generate a document based on the reflection and the query.
[0050] Thus, since the information acquired by the acquisition unit 152 is used for generation of the reflection, the information providing apparatus 10 can improve the inference performance of the model in tasks that require high expertise, which were difficult to handle in Reflection.
[0051] Furthermore, the acquisition unit 152 acquires combinations from a database that stores queries requesting document generation by the language model, documents generated by the language model in response to the queries, and user evaluations of the documents, where the similarity to the input query is above a threshold. The reflection generation unit 153 causes the language model to generate reflections based on the documents generated by the language model in response to the input query and the combinations acquired by the acquisition unit 152. The response proposal generation unit 151 causes the language model to generate a document based on the reflections and the input query. In addition, the acquisition unit 152 may acquire combinations from the database where the similarity to the input query is above a threshold and the evaluation is above a threshold. This allows the information providing device 10 to generate reflections using higher quality information.
[0052] [Other Embodiments] The case database 142 may implement an LLM Agent. In this case, the case database 142 stores the order in which the Tools are called when generating the response. Tool is an example of a tool used to generate the response. Chain is an example of a source of information referenced for generating the response (for example, LangChain described in Reference 1). The acquisition unit 152 acquires cases from the case database 142 that include the tools used to generate the document and the sources of information referenced for generating the document.
[0053] Reference 1: Applications that can reason. Powered by LangChain. (URL: https: / / www.langchain.com / )
[0054] In this way, the information providing device 10 can generate more accurate reflections by utilizing the tools and information sources.
[0055] [Experiment] This section describes the experiment conducted to confirm the effectiveness of the embodiment. Papers often include an abstract and a main text. In the experiment, the main text of a paper was provided to LLM (Likely a Generator Module) to generate an abstract.
[0056] First, 200 combinations of full text and abstract were collected from PubMed (see, for example, Reference 2). Of these 200, 100 were stored in Case DB142. The remaining 100 were used for evaluation (for which abstracts were generated).
[0057] Reference 2: PubMed (URL: https: / / pubmed.ncbi.nlm.nih.gov / )
[0058] The prompts entered into the LLM for response generation are shown in Figure 4. Figure 4 shows an example of the prompts used for generation. The text of the paper being evaluated goes after "## Article".
[0059] Then, the evaluation LLM performed a comparison of the Abstract generated by a standard LLM, the Abstract generated by Reflection (Non-Patent Literature 1), and the Abstract generated by the embodiment.
[0060] The LLM used for evaluation received prompts like those shown in Figure 5, which included the author's abstract, i.e., the abstract that was originally published in the paper. Figure 5 shows an example of the evaluation prompts.
[0061] The experimental results showed that, for 42 out of 100 abstracts, the LLM generated by the embodiment was deemed the best. Furthermore, 37 abstracts generated by reflection and 21 abstracts generated by the standard LLM were deemed the best. Both the generation and evaluation LLMs used in the experiment were GPT-4o-mini.
[0062] [System Configuration, etc.] Furthermore, the components of each part shown in the diagram are functional concepts and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown in the diagram, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. In addition, all or any part of the processing functions performed by each device can be realized by a CPU and the program executed on that CPU, or by hardware using wired logic.
[0063] Furthermore, among the processes described in the embodiments described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.
[0064] [Program] The information providing device 10 described above can be implemented by installing a program (information providing program) as packaged software or online software on a desired computer. For example, by having the computer run the above program, the computer can be made to function as the information providing device 10. The term "computer" here includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), as well as terminals such as PDA (Personal Digital Assistant).
[0065] Figure 6 shows an example configuration of a computer that runs an information provision program. Computer 1000 has, for example, memory 1010 and CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0066] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.
[0067] The hard disk drive 1090 stores, for example, the OS 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define each process executed by the information providing device 10 are implemented as program modules 1093 in which executable code for a computer is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to the functional configuration of the information providing device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).
[0068] Furthermore, the data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.
[0069] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.
[0070] 10 Information provision device 11 Communication unit 12 Input unit 13 Output unit 14 Storage unit 15 Control unit 21, 22, 23 LLM 31 Draft response 32 Reflection 33 Similar cases 34 Final draft response 141 Model information 142 Case database 151 Draft response generation unit 152 Acquisition unit 153 Reflection generation unit
Claims
1. An information providing device comprising: an acquisition unit that acquires information related to a query requesting the generation of a document by a language model; a reflection generation unit that causes the language model to generate reflections based on the document generated by the language model in response to the query and the information acquired by the acquisition unit; and a response proposal generation unit that causes the language model to generate a document based on the reflections and the query.
2. The information providing device according to claim 1, characterized in that the acquisition unit acquires combinations from a DB storing combinations of queries requesting document generation by a language model, documents generated by the language model in response to the queries, and user evaluations of the documents, the combinations having a similarity to the input query that is greater than or equal to a threshold; the reflection generation unit causes the language model to generate reflections based on the documents generated by the language model in response to the input query and the combinations acquired by the acquisition unit; and the response proposal generation unit causes the language model to generate a document based on the reflections and the input query.
3. The information providing device according to claim 2, characterized in that the acquisition unit acquires combinations from the DB in which the similarity to the input query is above a threshold and the evaluation is above a threshold.
4. The information providing device according to claim 1, characterized in that the acquisition unit acquires combinations from a DB storing combinations of queries requesting document generation by a language model, documents generated by the language model in response to the queries, tools used to generate the documents, information sources referenced for generating the documents, and user evaluations of the documents, the combinations having a similarity to the input query that is greater than or equal to a threshold; the reflection generation unit causes the language model to generate reflections based on the documents generated by the language model in response to the input query and the combinations acquired by the acquisition unit; and the response proposal generation unit causes the language model to generate a document based on the reflections and the input query.
5. An information provision method performed by an information provision device, comprising: an acquisition step of acquiring information related to a query that requests the generation of a document by a language model; a reflection generation step of causing the language model to generate reflections based on a document generated by the language model in response to the query and the information acquired by the acquisition step; and a response proposal generation step of causing the language model to generate a document based on the reflections and the query.
6. An information provision program characterized by causing a computer to execute: an acquisition step of acquiring information related to a query that requests the generation of a document by a language model; a reflection generation step of causing the language model to generate reflections based on the document generated by the language model in response to the query and the information acquired in the acquisition step; and a response proposal generation step of causing the language model to generate a document based on the reflections and the query.