Retrieval-augmented generation apparatus and method using generative language model, and computer-readable recording medium having program for executing same in computer recorded thereon

Abstract

This retrieval-augmented generation apparatus using a generative language model comprises: a plan generation unit; a query, re-planning, and answer generation unit; a generated answer classification unit; and a database management system. The plan generation unit receives a natural language query and a schema of a database and generates a query generation plan using the generative language model. The query, re-planning, and answer generation unit receives the natural language query, the schema of the database, and the query generation plan, and generates one of a database query, a re-plan, or an answer by using the generative language model. The generated answer classification unit classifies whether the output of the query, re-planning, and answer generation unit is the database query, the re-plan, or the answer. The database management system includes the database for receiving the database query and outputting a query execution result.

Description

Search augmentation generation device and method using a generative language model, and a computer-readable recording medium having a program for executing the same on a computer

[0001] The present invention relates to a search augmentation generation device and method using a generative language model and a computer-readable recording medium having a program for executing the same on a computer. More specifically, the invention relates to a search augmentation generation device and method using a generative language model that can be utilized for business decision-making and a computer-readable recording medium having a program for executing the same on a computer.

[0002] In many business situations, decision-making plays a crucial role in the success of an organization. Here, decision-making involves the process of analyzing data and selecting the most suitable alternative to ultimately achieve specific goals.

[0003] Generally, making such decisions requires performing the following three steps: (1) planning what analysis is needed for the decision, (2) retrieving the necessary data through queries, and (3) making a decision (i.e., providing an answer) based on the data obtained from the previous steps. Of these steps, the first one is the most difficult and has the limitation that it is still handled by humans.

[0004] The objective of the present invention, which is to solve the above-mentioned problems, is to provide a search augmentation generation device and method using a generative language model that can improve answer accuracy by separating the plan generation unit and the query, replanning, and answer generation units.

[0005] Another objective of the present invention is to provide a computer-readable recording medium having a program recorded thereon for executing a search augmentation generation method using the generative language model on a computer.

[0006] A search augmentation generation device using a generative language model according to an embodiment for achieving the above-described objective of the present invention includes a plan generation unit, a query, replanning, and answer generation unit, a generated answer classification unit, and a database management system. The plan generation unit receives a natural language query and a schema of a database and generates a query generation plan using a generative language model. The query, replanning, and answer generation unit receives the natural language query, the schema of the database, and the query generation plan and generates one of a database query, a replan, and an answer using the generative language model. The generated answer classification unit distinguishes whether the output of the query, replanning, and answer generation unit is the database query, the replan, or the answer. The database management system includes the database that receives the database query and outputs the query execution result.

[0007] In one embodiment of the present invention, the plan generation unit may further receive a business rule including a text description of a formula referenced in a decision. The query, replan, and answer generation unit may further receive the business rule.

[0008] In one embodiment of the present invention, the search augmentation generation device may further include a database management system interface that outputs the database query to the database and receives the query execution result from the database.

[0009] In one embodiment of the present invention, when the output of the query, replanning, and answer generation unit is the answer, the answer can be recognized as an answer to the natural language query and execution can be terminated.

[0010] In one embodiment of the present invention, when the output of the query, replanning, and answer generation unit is the database query, the database query is output to the database to obtain the query execution result from the database, and the query, replanning, and answer generation unit can be operated again using the query execution result as input.

[0011] In one embodiment of the present invention, when the output of the query, replanning, and answer generation unit is the replan, the query, replanning, and answer generation unit can be operated again by using the replan as the input to the query, replanning, and answer generation unit.

[0012] In one embodiment of the present invention, the replanning may include an increasing group in which the number of steps increases compared to the original plan after the replanning, an identical group in which the number of steps is the same compared to the original plan after the replanning, and a decreasing group in which the number of steps decreases compared to the original plan after the replanning.

[0013] In one embodiment of the present invention, the replanning may include a reorder that includes a case where the order of steps is rearranged.

[0014] In one embodiment of the present invention, the replanning may include a replace that includes cases where some steps are replaced with new steps.

[0015] In one embodiment of the present invention, the replanning may include a change target, which includes cases where the target of the lookup operation or the target of the calculation operation is changed.

[0016] In one embodiment of the present invention, the replanning may include an add look-up, which includes a case where a new look-up operation is added to the original plan.

[0017] In one embodiment of the present invention, the replanning may include an add calculation, which includes a case where a new calculation task is added to the original plan.

[0018] In one embodiment of the present invention, the replanning may include lookup and calculation additions (Add both actions), which include cases where a new lookup action and a new calculation action are added to the original plan within a single replanning process.

[0019] In one embodiment of the present invention, the replanning may include a divide to sub-steps, which includes a case where one step of the original plan is divided into sub-steps corresponding to detailed tasks.

[0020] In one embodiment of the present invention, the replanning may include deletion, including cases where some steps are deleted from the original plan.

[0021] In one embodiment of the present invention, the replanning may include a merger in which some steps of the original plan are summarized into a single step or merged.

[0022] In one embodiment of the present invention, the search augmentation generation device may perform a decision question answer. The decision question answer may include a locating scenario and a building scenario.

[0023] In one embodiment of the present invention, where c is a country and profit(c) is the profit of said country, the goal of the decision-making in the locating scenario is the amount of change in the profit of said country c. It may be to select a trading node that can maximize it.

[0024] In one embodiment of the present invention, when c is a country, n is a trading node, src is a source node, dest is a destination node, h is a home node, the set of countries is C, the set of TradingFlow tuples is F, TPR is the trading power ratio, TPcountry is the trading power of the country at the trading node, TPtotal is the sum of the trading powers of all countries at the trading node, IV is the amount of value coming from neighboring source nodes, flow is the flow weight from the source node to the destination node, and LV is the amount of value produced within the node, the business rule of the locating scenario is It may include.

[0025] In one embodiment of the present invention, when a good is g and CP is the current price of said good, the goal of the decision of the building scenario may be to minimize the current price CP(g) of said good for a given good g.

[0026] In one embodiment of the present invention, CO is the current quantity of the good produced in the building, b is the building, Sup is the set of supply, Dem is the set of demand, TD is the total demand, TS is the total supply, PD is the minimum quantity of demand for the good, MD is the maximum quantity of the good that can be consumed by the building, and BP is the reference price of the good, the business rule of the building scenario is that when the building (b) is expanded, CO(g, b) for the good (g) increases. It may include.

[0027] A search augmentation generation method using a generative language model according to an embodiment for achieving the above-described objective of the present invention comprises: a planning step of receiving a natural language query and a schema of a database and generating a query generation plan using a generative language model; an answer step of receiving the natural language query, the schema of the database and the query generation plan and generating any one of a database query, a replan, and an answer using the generative language model; a step of distinguishing whether the output of the answer step is the database query, the replan, or the answer; and a search step of receiving the database query and outputting a query execution result.

[0028] In one embodiment of the present invention, if the output of the answer step is the answer, the answer may be recognized as an answer to the natural language query and the execution may be terminated.

[0029] In one embodiment of the present invention, when the output of the answer step is the database query, the database query is output to the database to obtain the query execution result from the database, and the answer step can be operated again using the query execution result as input.

[0030] In one embodiment of the present invention, if the output of the answer step is the replan, the answer step can be operated again using the replan as input.

[0031] In one embodiment of the present invention, a program for executing a search augmentation generation method using the generative language model on a computer may be recorded on a computer-readable recording medium.

[0032] According to the search augmentation generation device and method using a generative language model according to the embodiment of the present invention as described above, and the computer-readable recording medium having a program for executing the same on a computer, a decision-making problem can be effectively solved without additional learning using only the database schema and natural language query.

[0033] By separating the plan generation unit and the query, replanning, and answer generation units, and by following the query generation plan generated by the plan generation unit when generating a query, the omission of queries for resolving the natural language query can be reduced.

[0034] When generating the above query, the accuracy of the answer can be improved by following the query generation plan generated by the above plan generation unit.

[0035] FIG. 1 is a diagram showing an example of a decision question-answer performed by a search augmentation generation device using a generative language model according to the present embodiment.

[0036] Figure 2 is a table representing the schema of a relational database for a locating scenario among the decision question responses of Figure 1.

[0037] Figure 3 is a table representing the schema of a relational database for a building scenario among the decision question responses of Figure 1.

[0038] Figure 4 is a diagram showing the building scenario among the decision question-and-answer of Figure 1.

[0039] Figure 5 is a table showing statistics for the database of decision question answers of Figure 1.

[0040] FIG. 6a is a diagram showing an example of a search augmentation generation device using the generative language model of FIG. 1.

[0041] FIG. 6b is a diagram showing an example of a search augmentation generation device using the generative language model of FIG. 1.

[0042] Figure 7a is a diagram illustrating a search augmentation generation method using a generative language model of a comparative example.

[0043] FIG. 7b is a diagram illustrating a search augmentation generation method using a generative language model according to the present embodiment.

[0044] FIG. 8 is a diagram showing the reasoning of a search augmentation generation method using a generative language model of a comparative example, which performs a locating scenario among the decision question responses of FIG. 1, and the reasoning of a search augmentation generation method using a generative language model according to the present embodiment.

[0045] FIGS. 9a to 9c are tables showing statistics and examples of replanning performed by the search augmentation generation device of FIG. 1.

[0046] FIG. 10 is a table showing the accuracy of a search augmentation generation method using a generative language model of a comparative example and a search augmentation generation method using a generative language model according to the present embodiment.

[0047] FIG. 11 is a table showing the omission of data analysis in a search augmentation generation method using a generative language model of a comparative example and a search augmentation generation method using a generative language model according to the present embodiment.

[0048] With respect to the embodiments of the present invention disclosed in the text, specific structural or functional descriptions are provided merely for the purpose of explaining the embodiments of the present invention, and the embodiments of the present invention may be implemented in various forms and should not be interpreted as being limited to the embodiments described in the text.

[0049] The present invention is susceptible to various modifications and may take various forms, and specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the invention to the specific disclosed forms, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. Similar reference numerals have been used for components in the description of each drawing.

[0050] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0051] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0052] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0053] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0054] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. Identical components in the drawings are given the same reference numerals, and redundant descriptions of identical components are omitted.

[0055] FIG. 1 is a diagram showing an example of a decision question-response performed by a search augmentation generation device using a generative language model according to the present embodiment. FIG. 2 is a table showing the schema of a relational database for a locating scenario among the decision question-responses of FIG. 1. FIG. 3 is a table showing the schema of a relational database for a building scenario among the decision question-responses of FIG. 1. FIG. 4 is a diagram showing the building scenario among the decision question-responses of FIG. 1. FIG. 5 is a table showing statistics for the database of the decision question-response of FIG. 1.

[0056] Referring to FIGS. 1 through 5, a large language model (LLM) may be utilized in this specification as a solution for decision-making that requires complex data analysis. Decision question-answering can be defined as a task of answering the optimal decision (dbest) for a decision query (Q), a business rule (R), and a database (D). Since there is no benchmark to review decision question-answering, a decision question-answering benchmark (DQA) may be defined in this specification. The decision question-answering may include two scenarios: a locating scenario and a building scenario. The locating scenario and the building scenario may be built based on two video games (Europa Universalis IV and Victoria 3) that have nearly the same goal as the decision question-answering. To effectively process the decision question-answering, this specification includes a method called iterative plan-then-retrieval augmented generation. After the above iterative planning, the generative language model based on the search augmentation generation method generates a decision plan in the first step, and the searcher generates a query for data analysis in the second step.

[0057] As described above, the following three steps must be performed for decision-making: (1) planning what analysis is required for the decision, (2) retrieving necessary data through a query, and (3) making a decision (i.e., providing an answer) based on the data obtained in the preceding process. Among these steps, step (1) is the most difficult and has the limitation that it is currently handled by humans. This specification discloses the possibility of replacing the human role with a large-scale language model to perform all steps, including step (1) as well as steps (2) and (3), from beginning to end.

[0058] To achieve the above objective, the present specification includes a decision question-answering task for the generative language model. The decision question-answering task may be defined as a question-answering task that takes a database (D), a business rule (R), and a decision query (Q) as inputs and generates an optimal decision as output.

[0059] Figure 1 is an example of a decision question-answer, where the point shown in "Step 2" of Figure 1 represents a transaction node, and "Profit" in the "Deccan" box represents the potential profit change resulting from each decision. Since the potential profit change is not stored in the database, it must be calculated from the database. In Figure 1, each country has only one main (home) transaction node. In the table within Figure 1, the underlined column names represent the keys of the table.

[0060] Figure 1 illustrates a trade competition situation in the Age of Discovery within the game Europa Universalis IV. This is an example of a decision-making question-answer. Each nation decides which trading city (i.e., node) to place merchants in to maximize profits from the main (home) trading node. In this example, a decision-making large-scale language model analyzed an international trade database and decided to place merchants in Doab to maximize profits from Deccan, the home trading node of the nation BAH.

[0061] Additionally, this specification includes a DQA, which is a benchmark for decision question answers. The DQA includes locating scenarios and building scenarios. The locating scenario consists of decision queries such as "Which trade node should the merchant be placed?" The building scenario consists of queries such as "How much wood should be supplied to the factory?" Since it is difficult to construct a DQA using actual business data, a benchmark was constructed by extracting game data for 301 specific situations from two video games, Europa Universalis IV and Victoria 3, which accurately reproduce real business situations. To eliminate randomness in the game and implement the benchmark, a game simulator was developed to record the decision processes for the 301 situations. This result is used as annotations for the DQA queries.

[0062] The above steps (2) and (3) of the decision-making process can be replaced with a large-scale language model, and in particular, can be based on a search augmentation generative technique. In this method, the searcher finds external data highly relevant to the query and transmits it to the language model, and the language model can generate an answer based on this. To solve more complex problems that require performing additional searches using the searched results, an iterative search augmentation generative technique may be utilized.

[0063] However, conventional search augmentation generative-based methodologies focus primarily on knowledge-based question-answering tasks and do not focus on decision-making question-answering tasks. Consequently, search augmentation generative-based methodologies are not very efficient in step (1), namely, in generating a decision plan. For example, in FIG. 1, the language model for decision-making can infer what analysis should be performed to maximize the benefits of Deccan. However, conventional search augmentation generative-based methodologies can focus only on, for example, identifying what Deccan is.

[0064] To address these limitations, the present specification includes a method called iterative plan-then-retrieval augmented generation, which extends iterative retrieval augmented generation techniques to decision-making question-answering. The iterative plan-then-retrieval augmented generation-based language model can first examine a data schema and a query to generate a plan for the required type of analysis (planning phase). Next, it can generate and present a query to retrieve data (retrieval phase). Finally, after evaluating whether a new plan needs to be generated for further analysis, it can repeat the planning phase and the retrieval phase (replanning phase) or make a decision based on the data (answering phase).

[0065] This specification includes the following:

[0066] Defines decision-making questions and answers, a new challenge that requires both planning and data analysis for decision-making.

[0067] It includes a decision question answer benchmark called DQA, which has two scenarios: a locating scenario and a building scenario.

[0068] It includes the iterative planning-after-search augmentation method, a search augmentation generation technique that can significantly improve the decision-making capabilities of large-scale language models.

[0069] It has been demonstrated that the iterative planning-after-search augmentation generation method of this specification is much more effective than the conventional iterative search augmentation generation method for decision question answers.

[0070] In this specification, the decision query response may be defined as a task that answers the optimal decision (dbest) given a structured database (D) having a decision query (Q), business rules (R), and a schema (S). The decision query (Q) includes a text goal that the user intends to achieve through the decision (dbest), and the business rules (R) include a text description of a formula referenced in the decision (dbest).

[0071] It is assumed that the above database (D) is too large to accommodate all of the language model's inputs at once. Therefore, it is assumed that the language model retrieves data from the database (D) by presenting a query for data analysis (data analysis query).

[0072] For example, the above database (D) may be a labeled property graph (LPG) database or a relational database (RDB). The labeled property graph refers to a graph having attributes on its edges and nodes. The labeled property graph database may simply be named a graph database (GDB).

[0073] When the database (D) of the above locating scenario is the relational database (RDB), the database (D) of the above locating scenario may include the four tables of FIG. 2. The tables of FIG. 2 can also be easily represented as a graph database (GDB) by considering the tuples of "TradingFlow" and "NodeCountry" as edges and the tuples of "TradingNode" and "Country" as vertices.

[0074] Figure 2 is a relational database (RDB) schema for a locating scenario, where bold text represents values ​​that depend on user decisions and other values. Some columns have been omitted.

[0075] There are several business rules for decision-making. The column values ​​in bold can be calculated based on user decisions according to the rules (e.g., the location of a trader) and other values. When a user places a trader on a trading node for a specific country, the flow from said trading node to said country's home node increases. In the rules below, c is the country, n is the trading node, src is the source node, dest is the destination node, and h is the home node. Additionally, the set of countries is C, and the set of TradingFlow tuples is F. TPR stands for Trading Power Ratio. As shown in Figure 2, TPcountry is the trading power of a country at the trading node, TPtotal is the sum of the trading power of all countries at the trading node, IV is the amount of value coming from neighboring source nodes, flow is the flow weight from the source node to the destination node, and LV is the amount of value produced within the node.

[0076] [Formula 1]

[0077]

[0078] [Equation 2]

[0079]

[0080] [Equation 3]

[0081]

[0082] In the above locating scenario, the goal of the decision is the amount of change in profit of a given target country c ( It is to select a trading node that can maximize ).

[0083] When the database (D) of the above building scenario is the relational database (RDB), the database (D) of the above building scenario may include the four tables of FIG. 3. The tables of FIG. 3 can also be easily represented as a graph database (GDB) by considering tuples of "Demand" and "Supply" as edges and tuples of "Goods" and "Building" as vertices.

[0084] Figure 3 is a relational database (RDB) schema for a building scenario, with some columns omitted.

[0085] The basic business rule is that when a decision maker expands factory building b, CO(g, b) for good (g) increases. As shown in Figure 3, CO is the current quantity of the good produced by the building. In the rules below, g is the good and b is the building. Also, the set of supply is Sup, and the set of demand is Dem. TD represents total demand, and TS represents total supply. As shown in Figure 3, PD is the minimum quantity demanded for the good, MD is the maximum quantity of the good that can be consumed by the building, CP is the current price of the good, and BP is the base price of the good. The key business rules are as follows:

[0086] [Equation 4]

[0087]

[0088] [Formula 5]

[0089]

[0090] [Equation 6]

[0091]

[0092] In the above building scenario, the goal of the decision is to minimize CP(g) for a given good g.

[0093] Figure 4 shows an example of a building scenario. In Figures 4 (a), (b), and (c), circles represent furniture, squares represent wood, and triangles represent hardwood. In Figure 4, the goal is to increase productivity and lower the price of furniture by determining where to expand the factory between B1 and B2.

[0094] As shown in FIG. 5, the DQA of this specification may include 301 pairs of databases (D) and decision queries (Q). Of these, 200 pairs are for locating scenarios and 101 pairs are for building scenarios. Since each database (D) may have two versions, a relational database (RDB) and a graph database (GDB), for the same query (Q), a total of 602 databases (D) may be provided for the DQA benchmark. FIG. 5 shows some statistics of the databases of the DQA.

[0095] FIG. 6a is a diagram showing a search augmentation generation device using the generative language model of FIG. 1. FIG. 6b is a diagram showing a search augmentation generation device using the generative language model of FIG. 1. FIG. 7a is a diagram showing a search augmentation generation method using the generative language model of a comparative example. FIG. 7b is a diagram showing a search augmentation generation method using the generative language model according to the present embodiment. FIG. 8 is a diagram showing the reasoning of the search augmentation generation method using the generative language model of a comparative example and the reasoning of the search augmentation generation method using the generative language model according to the present embodiment, which perform a locating scenario among the decision question responses of FIG. 1. FIG. 9a to 9c are tables showing statistics and examples of replanning performed by the search augmentation generation device of FIG. 1.

[0096] Referring to FIGS. 1 to 9c, a search augmentation generation device using the generative language model includes a plan generation unit (100), a query, replan, and answer generation unit (200), a generated answer classification unit (300), and a database management system (400).

[0097] Referring to FIG. 6a, the plan generation unit (100) receives a natural language query (Q) and a schema (S) of a database (D), and generates a query generation plan (PL) using a generative language model.

[0098] The above query, replanning, and answer generation unit (200) receives the above natural language query (Q), the above schema (S) of the above database (D), and the above query generation plan (PL), and generates one of a database query (QU), a replanning (RPL), and an answer (A) using the above generative language model. The above replanning (RPL) may be a modified query generation plan.

[0099] The above-mentioned generation answer distinguishing unit (300) distinguishes whether the output of the above-mentioned query, replanning, and answer generation unit (200) is the above-mentioned database query (QU), the above-mentioned replanning (RPL), or the above-mentioned answer (A).

[0100] The above database management system (400) includes the database (D) that receives the database query (QU) and outputs the query execution result (QUR).

[0101] The search augmentation generation device may further include a database management system interface (500) that outputs the database query (QU) to the database (D) and receives the query execution result (QUR) from the database (D).

[0102] For example, if the output of the above query, replanning, and answer generation unit (200) is the above answer (A), the above answer (A) can be recognized as an answer to the above natural language query (Q) and execution can be terminated.

[0103] For example, if the output of the above query, replanning, and answer generation unit (200) is the above database query (QU), the above database query (QU) is output to the above database (D) to obtain the above query execution result (QUR) from the above database (D), and the above query, replanning, and answer generation unit (200) can be operated again using the above query execution result (QUR) as input.

[0104] For example, if the output of the above query, replan, and answer generation unit (200) is the above replan (RPL), the above replan can be used as the input of the above query, replan, and answer generation unit to re-operate the above query, replan, and answer generation unit (200).

[0105] The embodiment of FIG. 6b is identical to the embodiment of FIG. 6a except that the plan generation unit (100) and the query, replan, and answer generation unit (200) further receive business rules (R).

[0106] Referring to FIG. 6b, the plan generation unit (100) receives a natural language query (Q), a schema (S) of a database (D), and a business rule (R), and generates a query generation plan (PL) using a generative language model.

[0107] The above query, replanning, and answer generation unit (200) receives the above natural language query (Q), the above schema (S) of the above database (D), the above business rule (R), and the above query generation plan (PL), and generates one of a database query (QU), a replanning (RPL), and an answer (A) using the above generative language model.

[0108] The above business rule (R) may include a text description of a formula referenced in decision-making.

[0109] For decision question answering, conventional search augmentation generation methods attempt to find the answer to the optimal decision (dbest) for a given (Q, S, R) through a single type of inference utilizing results retrieved from a database (D) via a data analysis query. Figure 7a illustrates this inference process. When a search is performed only once in the database (D), this process is called single-turn search augmentation generation. Otherwise, when a search is performed multiple times, this process is called iterative search augmentation generation.

[0110] On the other hand, the iterative planning-after-search augmentation generation method of the present embodiment attempts to answer the optimal decision (dbest) through two types of reasoning. The first type of reasoning is to generate a plan, and the second type of reasoning is to respond based on results retrieved from the database (D) according to the data analysis query. To reduce the side effects of using separate language models, the present embodiment may construct a single language model capable of performing both types of reasoning. The language model may be activated by adding plan and re-plan commands. FIG. 7b illustrates the reasoning procedure of the iterative planning-after-search augmentation generation method. The reasoning procedure may include (1) a planning step, (2) a search and answer step, and (3) a re-planning step.

[0111] (1) Planning stage: In this stage, the language model takes (Q, S, R) as input and generates an initial plan for data analysis. The initial plan describes a series of data analyses necessary for decision-making and to be performed in the search stage. The left part of Fig. 7b shows an example.

[0112] (2) Search and Answer Phases: Unlike conventional search augmentation generation techniques, the language model receives not only (Q, S, R) but also an initial plan as input. Therefore, it can generate data analysis queries for decision-making much more effectively than conventional search augmentation generation techniques. Figure 8 shows how a search augmentation generation-based language model generates queries differently from conventional search augmentation generation techniques after iterative planning. The query can be transmitted to a database via SQL or Cypher through a search augmentation generation interface. The query results can be used to infer whether replanning is required or whether additional searching is needed for better decision-making. Through a reverse linkage with the planning phase, the planning and search phases can be performed iteratively until the language model determines that further analysis for decision-making is no longer necessary.

[0113] (3) Replanning stage: Replanning may be performed when the initial plan is not sufficient to solve the decision problem. Instructions may be provided to the language model to evaluate the current plan by referring to the results of each search stage so that the language model can decide whether to replan. Consequently, the language model can generate a new plan to perform additional analysis or modify the direction of the previous analysis by taking the current plan and query results, as well as (Q, S, R), as input.

[0114] As shown in FIGS. 9a to 9c, all replanning cases can be categorized into three groups: an Increase group, a Same group, and a Decrease group. The Increase group refers to cases where the number of steps increases compared to the original plan after replanning. The Same group refers to cases where the number of steps remains the same compared to the original plan after replanning. The Decrease group refers to cases where the number of steps decreases compared to the original plan after replanning.

[0115] Each of the above categories can be further divided into the following subcategories:

[0116] * Reordering includes cases where the order of steps is rearranged.

[0117] * Replace includes cases where some steps are replaced with new steps.

[0118] * Change target includes cases where the target of an operation, such as a lookup or calculation, is changed.

[0119] * Add look-up includes cases where a new look-up task is added to the original plan.

[0120] * Add calculation includes cases where a new calculation task is added to the original plan.

[0121] * Add both actions includes cases where new lookup actions and new calculation actions are added to the original plan within a single replanning process.

[0122] Divide to sub-steps includes cases where a single step of the original plan is divided into sub-steps corresponding to detailed tasks.

[0123] * Deletion includes cases where some steps are removed from the original plan.

[0124] * Merging includes cases where some steps of the original plan are summarized or merged into a single step.

[0125] For some queries, replanning is performed two or more times, so the replanning may fall into several categories. In the locating scenario above, the search augmentation generation method after iterative planning of this embodiment may primarily perform "Add look-up" replanning. In the building scenario above, in most cases, "Divide to sub-steps" may be the main strategy of replanning.

[0126] FIG. 10 is a table showing the accuracy of a search augmentation generation method using a generative language model of a comparative example and a search augmentation generation method using a generative language model according to the present embodiment.

[0127] Referring to FIGS. 1 through 10, FIG. 10 shows the experimental results regarding the accuracy of each method for DQA. Each accuracy is the average of the accuracy of the relational database (RDB) and the graphical database (GDB). Compared to conventional iterative search augmentation generation (Iterative RAG), the decision performance of the iterative plan-after-search augmentation generation (PlanRAG) of the present embodiment improved by 15.8% in the locating scenario and by 7.4% in the building scenario. This clearly demonstrates how effective the iterative plan-after-search augmentation generation of the present embodiment is for decision-making tasks. The reason the iterative plan-after-search augmentation generation is relatively more effective in the locating scenario than in the building scenario is that the building scenario requires longer traversal than the locating scenario, and consequently, the planning phase of the building scenario becomes more difficult than the planning phase of the locating scenario. The accuracy of Single-turn Search Augmentation (Single-turn RAG) in building scenarios is very low, because building scenarios require generating very complex queries that are difficult to reason about in a single step. The aforementioned Single-turn Search Augmentation failed to retrieve results from the database in over 60% of queries in locating scenarios and over 95% of queries in building scenarios. Figure 10 shows that accuracy decreases when replanning is not performed in Search Augmentation after iterative planning (PlanRAG-LM w / o RP), specifically decreasing by 10.8% in queries in locating scenarios and 0.9% in queries in building scenarios. This result demonstrates the importance of the replanning step in assisting the decision-making tasks of decisioners (language models) using the Search Augmentation after iterative planning technique.

[0128] FIG. 11 is a table showing the omission of data analysis in a search augmentation generation method using a generative language model of a comparative example and a search augmentation generation method using a generative language model according to the present embodiment.

[0129] Referring to FIGS. 1 through 11, each scenario has several important values ​​that must be queried or calculated to answer a query. For example, the important values ​​in the locating scenario may include IV and TPtotal. For example, the important values ​​in the building scenario may include CO and PD. To analyze why the language model for iterative search augmentation generation after planning of this embodiment (PlanRAG-LM) is more effective than the conventional language model for iterative search augmentation generation (IterRAG-LM), the ratio of missing data analysis when querying or calculating important values ​​was measured. In the locating scenario, IV and TPtotal were used as the basis, and in the building scenario, CO and PD were used as the basis. In FIG. 11, the omission rates of the language model for search augmentation generation after iterative planning (PlanRAG-LM) of the present embodiment were low at 1.3% and 21.8%, respectively, whereas the omission rates of the conventional language model for search augmentation generation after iterative planning (IterRAG-LM) were higher at 3.3% and 33.2%, respectively. This means that even if the conventional language model for search augmentation generation after iterative planning (IterRAG-LM) can perform inference perfectly, its accuracy is lower than that of the language model for search augmentation generation after iterative planning (PlanRAG-LM) of the present embodiment. According to FIG. 11, the language model for search augmentation generation after iterative planning (PlanRAG-LM) of the present embodiment can achieve 98.7% accuracy in the locating scenario, but the actual accuracy in FIG. 10 is lower than that. This is because, in addition to the analysis of missing data, the inference (including planning) itself is very difficult.

[0130] For example, the search augmentation generation method using the generative language model according to the present embodiment can be performed by a computing device.

[0131] According to one embodiment of the present invention, a computer-readable recording medium may be provided on which a program for executing a search augmentation generation method using the generative language model according to the embodiments is recorded. The above-described method can be written as a program executable on a computer and can be implemented on a general-purpose digital computer that operates the program using a computer-readable medium. In addition, the structure of data used in the above-described method may be recorded on a computer-readable medium through various means. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or may be those known and available to a person skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The above-mentioned hardware device may be configured to operate as one or more software modules to perform the operation of the present invention.

[0132] In addition, the search augmentation generation method using the aforementioned generative language model can also be implemented in the form of a computer program or application executed by a computer and stored on a recording medium.

[0133] According to the present embodiment, decision-making problems can be effectively solved using only the database schema and natural language queries without additional learning.

[0134] By separating the plan generation unit and the query, replanning, and answer generation units, and by following the query generation plan generated by the plan generation unit when generating a query, the omission of queries for resolving the natural language query can be reduced.

[0135] When generating the above query, the accuracy of the answer can be improved by following the query generation plan generated by the above plan generation unit.

[0136] According to the present invention, the accuracy of the answer can be improved by separating the plan generation unit and the query, replanning, and answer generation units.

[0137] As described above, although the present invention has been explained with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the invention as set forth in the following claims.

Claims

1. A plan generation unit that receives a natural language query and a database schema, and generates a query generation plan using a generative language model; A query, replanning, and answer generation unit that receives the above natural language query, the above schema of the above database, and the above query generation plan, and generates any one of a database query, replanning, and answer using the above generative language model; A generation answer distinguishing unit that distinguishes whether the output of the above query, replanning, and answer generation unit is the above database query, the above replanning, or the above answer; and A search augmentation generation device utilizing a generative language model, comprising a database management system including the database that receives the above database query and outputs the query execution result.

2. In paragraph 1, the plan generation unit further receives a business rule including a text description of a formula referenced in decision-making, and A search augmentation generation device utilizing a generative language model characterized by the above query, replanning, and answer generation unit further receiving the above business rules.

3. A search augmentation generation device using a generative language model, characterized in that, in claim 1, it further includes a database management system interface that outputs the database query to the database and receives the query execution result from the database.

4. A search augmentation generation device using a generative language model, characterized in that, in the case where the output of the query, replanning, and answer generation unit is the answer, the answer is recognized as an answer to the natural language query and execution is terminated.

5. A search augmentation generation device using a generative language model, characterized in that, in the case where the output of the query, replanning, and answer generation unit is the database query, the database query is output to the database to obtain the query execution result from the database, and the query, replanning, and answer generation unit is operated again using the query execution result as input.

6. A search augmentation generation device using a generative language model, characterized in that, in the case where the output of the query, replan, and answer generation unit is the replan, the replan is used as the input to the query, replan, and answer generation unit to re-operate the query, replan, and answer generation unit according to claim 1.

7. In Paragraph 6, the above replan is An increase group in which the number of steps increases compared to the original plan after the above replanning; The same group having the same number of steps compared to the original plan after the above replanning; and A search augmentation generation device using a generative language model characterized by including a reduction group in which the number of steps is reduced compared to the original plan after the above replanning.

8. In Paragraph 6, the above replan is A search augmentation generation device using a generative language model characterized by including a re-order that includes cases where the order of steps is rearranged.

9. In Paragraph 6, the above replan is A search augmentation generation device using a generative language model characterized by including a replace that includes cases where some steps are replaced with new steps.

10. In Paragraph 6, the above replan is A search augmentation generation device using a generative language model characterized by including a change target, which includes cases where the target of a lookup operation or the target of a calculation operation is changed.

11. In Paragraph 6, the above replan is A search augmentation generation device using a generative language model characterized by including an add look-up, which includes a case where a new look-up operation is added to the original plan.

12. In Paragraph 6, the above replan is A search augmentation generation device using a generative language model characterized by including an "add calculation" that includes cases where a new calculation task is added to the original plan.

13. In Paragraph 6, the above replan is A search augmentation generation device using a generative language model characterized by including lookup and calculation addition (Add both actions), which includes cases where a new lookup action and a new calculation action are added to the original plan within a single replanning process.

14. In Paragraph 6, the above replan is A search augmentation generation device using a generative language model characterized by including a divide to sub-steps, which includes cases where one step of the original plan is divided into sub-steps corresponding to detailed tasks.

15. In Paragraph 6, the above replan is A search augmentation generation device using a generative language model characterized by including a deletion (Delete) that includes cases where some steps are deleted from the original plan.

16. In Paragraph 6, the above replan is A search augmentation generation device using a generative language model characterized by including a merge that includes cases where some steps of the original plan are summarized into a single step or merged.

17. In paragraph 1, the search augmentation generation device performs decision question answering, and A search augmentation generation device utilizing a generative language model characterized by the above decision question answer including locating scenarios and building scenarios.

18. In Paragraph 17, where c is a country and profit(c) is the interest of said country, the goal of the decision-making in said locating scenario is the amount of change in said interest of said country c. A search augmentation generation device using a generative language model characterized by selecting a transaction node that can maximize .

19. In Paragraph 18, where c is a country, n is a trading node, src is a source node, dest is a destination node, h is a home node, the set of countries is C, the set of TradingFlow tuples is F, TPR is the trading power ratio, TPcountry is the trading power of the said country at the said trading node, TPtotal is the sum of the trading power of all countries at the said trading node, IV is the amount of value coming from the neighboring said source nodes, flow is the flow weight from the said source node to the said destination node, and LV is the amount of value produced within the node, the business rule of the said locating scenario is A search augmentation generation device utilizing a generative language model characterized by including 20. A search augmentation generation device using a generative language model, characterized in that, in claim 17, when a good is g and CP is the current price of said good, the goal of the decision of said building scenario is to minimize the current price CP(g) of said good for a given said good g.

21. In paragraph 20, CO is the current quantity of the said good produced in the building, b is the building, Sup is the set of supplies, Dem is the set of demands, TD is total demand, TS is total supply, PD is the minimum quantity of demand for said good, MD is the maximum quantity of said good that can be consumed by said building, and BP is the reference price of said good, The business rules of the above building scenario are When expanding the above building (b), CO(g, b) for the above good (g) increases; A search augmentation generation device utilizing a generative language model characterized by including 22. A planning step that receives a natural language query and a database schema, and generates a query generation plan using a generative language model; A response step of receiving the above natural language query, the above schema of the database, and the above query generation plan, and generating any one of a database query, replanning, and answer using the above generative language model; A step of distinguishing whether the output of the above answer step is the above database query, the above replanning, or the above answer; and A search augmentation generation method using a generative language model comprising a search step that receives the above database query and outputs the query execution result.

23. A search augmentation generation method using a generative language model, characterized in that, in the case where the output of the answer step in paragraph 22 is the answer, the answer is recognized as an answer to the natural language query and the execution is terminated.

24. A search augmentation generation method using a generative language model, characterized in that, in the case where the output of the answer step is the database query in claim 22, the database query is output to the database to obtain the query execution result from the database, and the answer step is operated again using the query execution result as input.

25. A search augmentation generation method using a generative language model, characterized in that, in the case where the output of the answer step is the replan, the answer step is re-operated using the replan as input.

26. A computer-readable recording medium having a program recorded thereon for executing the method of any one of paragraphs 22 through 25 on a computer.

Images