Method, system, and computer-readable storage medium for generating multi-document-based question / answer pair to evaluate system using llm

Abstract

The present invention relates to a method and system for generating a multi-document-based question / answer pair for evaluating a system using an LLM and, more specifically, to a method and system for generating a multi-document-based question / answer pair for evaluating a system using an LLM, the method and system generating the question / answer pair by inputting, into a deep learning-based language model, each node in a dataset in which a plurality of nodes including document content for a plurality of documents are arranged hierarchically.

Description

A method, system, and computer-readable storage medium for generating multidocument-based question / answer pairs for evaluating a system using LLM

[0001] The present invention relates to a method and system for generating question / answer pairs based on multiple documents for evaluating a system using LLM, and more specifically, to a method and system for generating question / answer pairs based on multiple documents for evaluating a system using LLM, wherein each node in a dataset in which multiple nodes containing document content for multiple documents are layered is input into a deep learning-based language model to generate question / answer pairs.

[0002]

[0003] Retrieval-Augmented Generation (RAG) is a natural language processing technology developed to overcome the limitations of Large Language Models (LM). Since LLMs generate answers based on previously trained data when a question is input, if an untrained question is input, the LLM generates an answer by combining the most probable information from its existing data. This process can lead to "hallucinations," where false or outdated information is presented as fact. RAG is one method designed to mitigate these hallucinations by referencing external data to generate answers, thereby increasing the accuracy of the responses.

[0004] Meanwhile, companies utilize RAG systems to create AI systems based on LLM, such as chatbots, for internal use or commercialization. The core of these AI systems is to derive answers appropriate to the intent of a question based on vast amounts of data. Therefore, to evaluate such AI systems accurately, evaluation data of varying difficulty levels must exist according to the difficulty of answering the questions. Under these circumstances, there is a demand for technology capable of more accurately evaluating systems based on LLM.

[0005]

[0006] The present invention aims to provide a method and system for generating question / answer pairs based on multiple documents for evaluating a system using LLM, and more specifically, to generate question / answer pairs based on multiple documents by inputting each node into a deep learning-based language model in a dataset in which multiple nodes containing document content for multiple documents are layered.

[0007]

[0008] In order to solve the above problems, one embodiment of the present invention provides a method for generating question / answer pairs based on multiple documents for evaluating a system using LLM, comprising: a question generation step of determining a question node that serves as the basis for question generation in a dataset including a tree structure in which the document contents of a plurality of documents are structured in a plurality of nodes, and inputting text information of the determined question node into a question generator including a deep learning-based language model to derive question information; and an answer generation step of inputting text information of a node related to the question node according to a preset criterion among the question node, the question information, and the child nodes of the question node into an answer generator including a deep learning-based language model to derive answer information; wherein for a plurality of nodes of the dataset, the upper node includes a summary document output by inputting text information of one or more clustered child nodes into a deep learning-based language model.

[0009] In one embodiment of the present invention, the dataset comprises a tree structure composed of multiple nodes that are hierarchically structured into levels having upper and lower concepts, wherein the document content included in a plurality of documents is included in a predetermined level among the lower layers of the tree structure, the nodes included in a first tree structure structured with the document content of each document, and the remaining nodes are included in a second tree structure structured based on the document content of the root node corresponding to the top-level node of the first tree structure for each document, and the upper node in the tree structure may include a form in which the document content of one or more lower nodes is summarized.

[0010] In one embodiment of the present invention, the method for generating the question / answer pair further comprises a document structuring step for generating the dataset, wherein the document structuring step may include: a leaf node generation step for separating each of a plurality of documents into chunks and generating a plurality of leaf nodes including text information and embedding values ​​for each chunk; a document internal structuring step for generating a tree structure for a single document by clustering the plurality of leaf nodes based on the embedding values ​​of each leaf node and generating a parent node including a single summary document derived based on the clustered leaf nodes; and a document external structuring step for generating a structured dataset for the plurality of documents by clustering root nodes corresponding to the top-level nodes of each document based on the embedding values ​​and generating a parent node based on the clustered root nodes.

[0011] In one embodiment of the present invention, the answer generation step involves, for a question generation node for determining a question node, selecting the node with the highest degree of relevance to the question node among the sub-nodes of the question generation node after the question node and the question node have been determined, and inputting the text information of the selected node and the question node into the answer generator to derive answer information, wherein the degree of relevance may be derived based on the embedding value of each node.

[0012] In one embodiment of the present invention, the answer generation step selects a leaf node corresponding to the lowest node of the question node among the child nodes of the question node, and inputs the selected leaf node and the question node into the answer generator to derive answer information, wherein the answer information may include source information of a document regarding the leaf node.

[0013] In one embodiment of the present invention, the answer generation step may, when selecting a node related to the question node among the sub-nodes of the question node, derive a degree of correlation with the question node based on the embedding value of the question node and the embedding value of each node included in the sub-nodes of the question node, and select a predetermined number of nodes in order of high correlation with the question node.

[0014] In one embodiment of the present invention, the question generation step may determine a node corresponding to a parameter received from a user terminal among a plurality of nodes existing within a dataset as a question node.

[0015] In one embodiment of the present invention, the question generation step may derive question information based on the difficulty information for the plurality of nodes that are hierarchically divided into levels, the parameter received from the user terminal to derive question information includes difficulty information, and determine the level of the question node based on the difficulty information for each of the nodes, determine one or more question nodes among the nodes included in the determined level, and then derive question information based on the question node.

[0016] In one embodiment of the present invention, the question generation step may include keyword information for parameters received from a user terminal to derive question information, determine one or more nodes related to the keyword information among the plurality of nodes as question nodes, and derive question information based on the question nodes.

[0017] In one embodiment of the present invention, the question generation step may derive question information by including document count information as a parameter received from a user terminal to derive question information, and for the plurality of nodes referencing one or more documents, selecting one or more nodes among the plurality of nodes that refer to a number of documents corresponding to the document count information, determining one or more question nodes related to a preset keyword among the selected nodes, and then deriving question information based on the question nodes.

[0018] In one embodiment of the present invention, the question generation step can derive question information by determining one or more nodes included in the level corresponding to the level information among a plurality of nodes that reference a specific document corresponding to the document information as question nodes, and the parameter received from a user terminal to derive question information includes document information including a title for one or more specific documents and level information of a node for each of the plurality of nodes hierarchically arranged into levels, and can derive question information based on the question nodes.

[0019] In order to solve the above problems, in one embodiment of the present invention, a server system for performing a method of generating question / answer pairs based on multiple documents for evaluating a system using LLM is provided, comprising: a question generation unit that determines a question node that serves as the basis for question generation in a dataset including a tree structure in which the document contents of a plurality of documents are structured in a plurality of nodes, and inputs text information of the determined question node into a question generator including a deep learning-based language model to derive question information; and an answer generation unit that inputs text information of a node among the question node, the question information, and a node with high correlation to the question node among the child nodes of the question node into an answer generator including a deep learning-based language model to derive answer information; wherein for a plurality of nodes of the dataset, the upper node includes an output value that is output by inputting one or more clustered child nodes into a deep learning-based language model.

[0020] In order to solve the above-mentioned problem, in one embodiment of the present invention, a computer-readable storage medium for implementing a method for generating question / answer pairs based on multiple documents for evaluating a system using LLM executed in a server system, wherein the computer-readable storage medium comprises computer-executable instructions that cause the server system to perform the following steps, and the following steps include: a question generation step of determining a question node that serves as the basis for question generation in a dataset comprising a tree structure in which the document contents of a plurality of documents are structured in a plurality of nodes, and inputting text information of the determined question node into a question generator comprising a deep learning-based language model to derive question information; The present invention provides a computer-readable storage medium comprising: an answer generation step of deriving answer information by inputting text information of a node related to the question node into an answer generator including a deep learning-based language model according to a preset standard among the question node, the question information, and the child nodes of the question node; wherein, for a plurality of nodes of the dataset, the upper node includes an output value that is output by inputting one or more clustered child nodes into a deep learning-based language model.

[0021]

[0022] According to one embodiment of the present invention, the effect of deriving question information and answer information using a deep learning-based language model can be achieved.

[0023] According to one embodiment of the present invention, the degree of correlation between nodes is derived based on the embedding value of each node, thereby enabling the effect of clustering nodes with high correlation.

[0024] According to one embodiment of the present invention, the effect of clustering nodes based on the matching rate of metadata included in each node can be achieved.

[0025] According to one embodiment of the present invention, the effect of deriving a summary document summarizing one or more documents using a deep learning-based language model can be achieved.

[0026] According to one embodiment of the present invention, by creating a tree structure in which the root node for each document is a leaf node, the effect of expanding a dataset for multiple documents can be achieved.

[0027] According to one embodiment of the present invention, the effect of deriving question information corresponding to parameters input by a user through a question generation step can be achieved.

[0028] According to one embodiment of the present invention, each of a plurality of documents can be chunked to separate the chunks, and the effect of creating leaf nodes for each chunk can be achieved.

[0029] According to one embodiment of the present invention, it is possible to achieve the effect of evaluating the quality of a large language model-based system based on question / answer pairs generated in a server system.

[0030]

[0031] FIG. 1 schematically illustrates the connection configuration of a server system and the steps of a method for generating question / answer pairs according to an embodiment of the present invention.

[0032] FIG. 2 schematically illustrates the process of performing a leaf node generation step according to one embodiment of the present invention.

[0033] FIG. 3 schematically illustrates the process of performing a document internal structuring step according to one embodiment of the present invention.

[0034] FIG. 4 schematically illustrates the process of performing the document external structuring step according to one embodiment of the present invention.

[0035] FIG. 5 schematically illustrates information included in a node and metadata according to an embodiment of the present invention.

[0036] FIG. 6 schematically illustrates the process of performing a question generation step according to one embodiment of the present invention.

[0037] FIG. 7 schematically illustrates the process of performing the step of generating a question based on parameters according to one embodiment of the present invention.

[0038] FIG. 8 schematically illustrates information included in parameters according to one embodiment of the present invention.

[0039] FIG. 9 schematically illustrates the process of performing the answer generation step according to one embodiment of the present invention.

[0040] FIG. 10 schematically illustrates the process of performing the step of generating answer information based on a question generation node according to one embodiment of the invention.

[0041] FIG. 11 schematically illustrates the process of performing the step of generating answer information based on leaf nodes according to one embodiment of the present invention.

[0042] FIG. 12 schematically illustrates the process of performing the step of generating answer information based on the correlation with a question node according to one embodiment of the present invention.

[0043] FIG. 13 illustrates the internal configuration of a computing device according to one embodiment of the present invention.

[0044]

[0045] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0046] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0047] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0048] The "user terminal" mentioned below may be implemented as a computer or portable terminal capable of connecting to a server or other terminals via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal may include, for example, all types of handheld-based wireless communication devices that ensure portability and mobility, such as smartphones, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet), and BLE Beacon (Bluetooth Low Energy Beacon) terminals. In addition, the “network” can be implemented as a wired network such as a Local Area Network (LAN), Wide Area Network (WAN), or Value Added Network (VAN), or as any type of wireless network such as a mobile radio communication network or a satellite communication network.

[0049] FIG. 1 schematically illustrates the connection configuration of a server system (1000) and the steps of performing a method for generating question / answer pairs according to one embodiment of the present invention.

[0050] Schematically, FIG. 1 (a) illustrates the connection configuration and internal configuration of a server system (1000), and FIG. 1 (b) illustrates the steps of a method for generating question / answer pairs.

[0051] Specifically, FIG. 1(a) illustrates the connection configuration of a server system (1000) that performs the method of generating the question / answer pair, a user terminal (2000), a question generator (3000), and an answer generator (4000).

[0052] The method for generating the above question / answer pairs can be performed in a server system (1000) comprising one or more processors and one or more memories, and as illustrated in FIG. 1 (a), the server system (1000) can communicate with a user terminal (2000), a question generator (3000), and an answer generator (4000) to perform the method for generating the above question / answer pairs.

[0053] At this time, the server system (1000) can receive parameters for generating question information from a user terminal (2000), the server system (1000) can input text information of a question node into a question generator (3000) to derive question information from the question generator (3000), and the server system (1000) can input text information of a question node, question information, and a node with the highest degree of correlation with the question node among the sub-nodes of the question node into an answer generator (4000) to derive answer information from the answer generator (4000).

[0054] In one embodiment of the present invention, the connection configuration of the user terminal (2000) and the server system (1000) may include a form in which the server system (1000) is accessed through the user terminal (2000), which includes a computer or portable terminal used by the user.

[0055] Additionally, as illustrated in FIG. 1 (a), the server system (1000) comprises: a question generation unit (100) that performs a question generation step in which a question node is determined as the basis for question generation in a dataset (1100) that includes a tree structure in which the document contents of a plurality of documents are structured in a plurality of nodes, and inputs text information of the determined question node into a question generator (3000) that includes a deep learning-based language model to derive question information; an answer generation unit (200) that performs an answer generation step in which the question node, the question information, and text information of a node among the sub-nodes of the question node that has a high degree of correlation with the question node are input into an answer generator (4000) that includes a deep learning-based language model to derive answer information; a document structuring unit (300) that performs a document structuring step to generate the dataset (1100); and the dataset (1100).

[0056] More specifically, each component included in the server system (1000) illustrated in FIG. 1 performs the role of controlling the operation of the server system (1000) that performs the method of generating question / answer pairs of the present invention.

[0057] The question generation unit (100) of the above server system (1000) can determine a question node that serves as the basis for question generation in a dataset (1100) that includes a tree structure in which the document contents of a plurality of documents are structured in a plurality of nodes, and input the text information of the determined question node into a question generator (3000) that includes a deep learning-based language model to derive question information.

[0058] In one embodiment of the present invention, the question information may include a plurality of questions for each of the plurality of nodes. The question information generated through the question generation step may be included in the questions of the question / answer pair of the present invention, and subsequently, the question / answer pair may be generated by generating answers to the questions included in the question information through the answer generation step.

[0059] The answer generation unit (200) of the above server system (1000) can derive answer information by inputting the question node, the question information, and text information of a node among the sub-nodes of the question node that is highly related to the question node into an answer generator (4000) that includes a deep learning-based language model.

[0060] In one embodiment of the present invention, the answer information may include a plurality of answers for each of the question information. The answer information generated through the answer generation step may be included in the answer of the question / answer pair of the present invention.

[0061] The document structuring unit (300) of the above server system (1000) can generate the above dataset (1100), and the document structuring step comprises: a leaf node generation step of separating each of a plurality of documents into chunks and generating a plurality of leaf nodes including text information and embedding values ​​for each chunk; a document internal structuring step of generating a tree structure for a single document by clustering the plurality of leaf nodes based on the embedding values ​​of each leaf node and generating a parent node including a single summary document derived based on the clustered leaf nodes; and a document external structuring step of generating a structured dataset (1100) for the plurality of documents by clustering root nodes corresponding to the top-level nodes of each document based on the embedding values ​​and generating a parent node based on the clustered root nodes.

[0062] In one embodiment of the present invention, the document structuring step is performed prior to the question generation step to generate a dataset (1100) including a plurality of nodes capable of generating question information. Accordingly, after generating a dataset (1100) for a plurality of documents through the document structuring step, the question generation step can derive question information, and based on the derived question information, the answer generation step can derive answer information.

[0063] The dataset (1100) of the above server system (1000) can be generated through the document structuring step and includes a tree structure composed of multiple nodes that are hierarchically structured into levels having upper and lower concepts, where the document content included in multiple documents is hierarchically structured.

[0064] Nodes included in one or more levels of the lower hierarchy of the above tree structure are included in a first tree structure structured with the document content of each document, and the remaining nodes are included in a second tree structure structured based on the document content of the root node corresponding to the top-level node of the first tree structure for each document.

[0065] In the tree structure above, the upper node includes a form in which the document content of one or more child nodes is summarized, and for a plurality of nodes of the dataset (1100), the upper node may include a summary document that is output by inputting text information of one or more clustered child nodes into a deep learning-based language model.

[0066] Preferably, in the present invention, inputting one of the nodes into the question generator (3000) or the answer generator (4000) may correspond to inputting all information included in the node and may correspond to inputting text information included in the node.

[0067] For example, inputting a question node into the question generator (3000) corresponds to inputting multiple pieces of information including text information among text information, embedding values, and metadata included in the question node.

[0068] As illustrated in Fig. 1(b), the method for generating the question / answer pair can be performed by determining a question node among a plurality of nodes included in the dataset (1100) for a dataset (1100) which includes a tree structure in which the document contents of a plurality of documents are structured in a plurality of nodes included in a server system (1000), and inputting the text information of the determined question node into a question generator (3000) which includes a deep learning-based language model to derive question information (S100).

[0069] In addition, the text information of the question node, the question information, and the node with the highest correlation to the question node among the sub-nodes of the question node can be input into an answer generator (4000) that includes a deep learning-based language model to derive answer information (S200).

[0070] Therefore, question / answer pairs can be generated based on the question information and answer information derived from multiple documents, and a system using LLM can be evaluated based on the generated question / answer pairs.

[0071] Preferably, a system using an LLM outside the server system (1000) can be verified or evaluated by utilizing a question / answer pair including question information and answer information generated through the method of generating a question / answer pair of the present invention.

[0072] For example, the system using an LLM outside the server system (1000) can be evaluated by calculating the correlation or similarity between the answer generated when the question included in the question / answer pair is input into a system using an LLM outside the server system (1000) and the answer included in the question / answer pair.

[0073] In addition, when the questions included in the above question / answer pair are input into a system using an LLM outside the server system (1000) to separate the chunks, the system using an LLM outside the server system (1000) can be evaluated by comparing the separated chunks with text information included in the node of the present invention, and the system using an LLM outside the server system (1000) can be evaluated by comparing the metadata generated in the server system (1000) of the present invention with the metadata generated in the system using an LLM outside the server system (1000).

[0074] FIG. 2 schematically illustrates the process of performing a leaf node generation step according to one embodiment of the present invention.

[0075] Generally, the leaf node generation step can separate each of the multiple documents into chunks and generate multiple leaf nodes containing text information and embedding values ​​for each chunk.

[0076] Specifically, when a plurality of documents are input into the server system (1000), a first tree structure is generated for each of the plurality of documents, and a second tree structure for the plurality of documents is generated based on the first tree structure for each document. At this time, a tree structure combining the first tree structure and the second tree structure may exist in the dataset (1100).

[0077] In one embodiment of the present invention, in order to generate the first tree structure, a plurality of nodes included in the first tree structure may be generated, and among the plurality of nodes, the lowest node may correspond to a leaf node. Accordingly, the first tree structure may be generated by generating the leaf node and then generating the parent node of the leaf node.

[0078] As illustrated in FIG. 2, each of the plurality of documents can be separated into chunks, and Document #1 of FIG. 2 may correspond to one document included in the plurality of documents. Document #1 can be chunked to separate the document content included in Document #1 into one or more chunks, and (1) to (5) of FIG. 2 may correspond to each chunk.

[0079] Preferably, the chunking may include a process of meaningfully grouping information into chunks, and may perform chunking based on paragraphs, sentences, words, etc., within the corresponding document content. Accordingly, the chunk may correspond to a text form, the leaf node may correspond to a unit formed by cutting the document content, and may include text information corresponding to the chunk.

[0080] Subsequently, five leaf nodes can be created based on each of the five chunks separated from Document #1, and one chunk may correspond to the document content included in one leaf node. Therefore, one node referencing Document #1 may contain a part of the document content included in Document #1, and the leaf node may contain text information corresponding to that document content.

[0081] In one embodiment of the present invention, when a document is separated into one or more chunks, a leaf node may contain text because it contains one chunk, and the content of the document may be reflected as is without being summarized. Therefore, by combining all one or more leaf nodes for a document, the entire content of the document may be included. At this time, the number of leaf nodes may be generated differently for each document.

[0082] Preferably, in a tree structure generated for a single document, the leaf node is included in the lowest hierarchy and may correspond to the lowest node within the tree structure.

[0083] FIG. 3 schematically illustrates the process of performing a document internal structuring step according to one embodiment of the present invention.

[0084] Schematically, FIG. 3(a) illustrates a tree structure generated for each of a plurality of documents, and FIG. 3(b) illustrates the process of performing the step of generating the tree structure.

[0085] Specifically, the document internal structuring step can generate a tree structure for a single document by clustering the plurality of leaf nodes based on the embedding values ​​of each leaf node and generating a parent node containing a single summary document derived based on the clustered leaf nodes.

[0086] As illustrated in FIG. 3(a), a tree structure can be generated by structuring the document content of each of the plurality of documents including Document #1 to Document #N through a document structuring step, and the generated tree structure can be included in the dataset (1100).

[0087] Preferably, one tree structure can be generated for one document, and since there are N documents from Document #1 to Document #N in FIG. 3 (a), N tree structures corresponding to each of the multiple documents can be generated, and the N tree structures can be included in the dataset (1100). The tree structure includes multiple nodes hierarchically arranged into levels having a superordinate concept and a subordinate concept, each hierarchy can be represented as a level, and one or more nodes can be included in one hierarchy.

[0088] Accordingly, in the tree structure illustrated in FIG. 3(a), Level 1 may contain 5 nodes labeled 1, 2, 3, 4, and 5, Level 2 may contain 3 nodes labeled 6, 7, and 8, and Level 3 may contain 2 nodes labeled 9 and 10. In this case, since the lowest node in the tree structure corresponds to a leaf node and the highest node corresponds to a root node, the 5 nodes included in Level 1 in the tree structure may correspond to leaf nodes, and the 2 nodes included in Level 3 may correspond to root nodes.

[0089] In Figure 3(a), when the 10 nodes included in the tree structure are referred to as Node #1 to Node #10, three nodes included in Level 2 can be generated by clustering the leaf nodes based on the embedding values ​​of the five leaf nodes.

[0090] For example, Node #6 can be created by clustering Node #3 and Node #5, and the created Node #6 may include a single summary document derived by summarizing the document content contained in Node #3 and Node #5, respectively. In this case, a single summary document can be derived by inputting the text information contained in Node #3 and Node #5, respectively, into a deep learning-based language model specialized in summarization.

[0091] Therefore, Node #7 can be created by clustering Node #1, Node #4, and Node #5, and Node #8 can be created by clustering Node #2 and Node #3. Additionally, Node #9 can be created by clustering Node #6 and Node #7, and Node #10 can be created by clustering Node #6 and Node #8.

[0092] Preferably, a parent node containing a summary document can be generated by clustering multiple child nodes to derive a single summary document. By repeatedly performing this process, a tree structure for a single document is generated.

[0093] As illustrated in FIG. 3(b), to generate a tree structure for a single document, multiple chunks can first be separated from a single document by referring to FIG. 2. The text information of each chunk can be input into an embedding model, and chunks with high relevance can be clustered based on the output embedding values. Thus, among the multiple text information included in the content of a single document, text information with high relevance can be clustered. Subsequently, a summary document can be generated by inputting the clustered chunks into a deep learning-based language model.

[0094] At this time, each chunk may correspond to a leaf node, and a generated summary document may correspond to a parent node of a leaf node. A parent node of a leaf node may contain text information summarized from text information of one or more leaf nodes.

[0095] FIG. 4 schematically illustrates the process of performing the document external structuring step according to one embodiment of the present invention.

[0096] In summary, the document external structuring step can generate a structured dataset (1100) for the plurality of documents by clustering root nodes corresponding to the top-level nodes of each document based on embedding values ​​and generating top nodes based on the clustered root nodes.

[0097] As illustrated in FIG. 4, a tree structure can be generated by structuring the document content of each of the plurality of documents including Document #1 to Document #N through the document structuring step, and a larger tree structure generated based on each generated tree structure can be included in the dataset (1100).

[0098] Preferably, one tree structure can be generated for one document, and since there are N documents from Document #1 to Document #N in FIG. 4, N tree structures corresponding to each of the multiple documents can be generated, and the N tree structures can be included in the dataset (1100). At this time, each of the N tree structures referencing one document may correspond to a small tree, and a big tree referencing all N documents can be generated by combining multiple small trees based on each of the N tree structures.

[0099] Accordingly, a tree structure referencing a single document corresponds to a first tree structure and a small tree, and when the small tree referencing Document #1 in FIG. 4 is called Small Tree #1, it corresponds to a form in which the top layer, the root layer, is generated from the lowest layer, the leaf layer, of Small Tree #1, and one or more nodes included in the root layer may be included in the root node. Each of Small Tree #1 to Small Tree #N includes a root node, and when a hierarchical tree structure is created by making the root nodes included in each of Small Tree #1 to Small Tree #N into leaf nodes, a second tree structure can be generated.

[0100] At this time, the first tree structure and the second tree structure combined can be referred to as a big tree. The root layer, which is the highest level of the first tree structure, can be the leaf layer, which is the lowest level of the second tree structure, and in the big tree formed by combining the first tree structure and the second tree structure, the lowest level can be the leaf layer and the highest level can be the upper layer. Therefore, the big tree corresponds to a tree structure that references all of the plurality of documents. In one embodiment of the present invention, the number of root nodes in the small tree and the big tree may be one or more.

[0101] Preferably, the tree structure included in the dataset (1100) includes a plurality of nodes that are hierarchically structured into levels having upper and lower concepts for the document content included in a plurality of documents, and in the tree structure, the upper node includes a form in which the document content of one or more lower nodes is summarized.

[0102] In addition, nodes included in one or more levels of the lower hierarchy of the above tree structure are included in a first tree structure structured with the document content of each document, and the remaining nodes may be included in a second tree structure structured based on the document content of the root node corresponding to the top node of the first tree structure for each document.

[0103] Accordingly, the tree structure in the present invention corresponds to a single large tree structure including the first tree structure and the second tree structure, the first tree structure corresponds to a small tree generated for a single document, the second tree structure corresponds to a tree structure generated for a plurality of documents based on a root node included in the first tree structure, and the tree structure of the present invention may correspond to a big tree including the first tree structure and the second tree structure.

[0104] In this case, since the second tree structure includes multiple nodes that reference multiple documents, it can generate question information or answer information based on grounds derived from more documents than the first tree structure which includes multiple nodes that reference a single document. Therefore, more accurate and specific question information or answer information can be generated.

[0105] In the present invention, since the tree structure is generated by linking multiple documents into one, when deriving question information through the tree structure, questions likely to be asked by actual human users can be derived, rather than simple enumeration questions or questions for simple information verification. Furthermore, when deriving answer information through the tree structure, the exact source can be verified as to which of the multiple documents the answer refers to and which chunk of that document the answer refers to, so that answers with clear sources can be generated even if the number of referenced documents increases.

[0106] If the system is not structured in a tree structure such as the aforementioned Big Tree, it may take a long time or be inefficient because all of the aforementioned multiple documents must be referenced to generate question information or answer information. However, if question information or answer information is generated by referencing only the documents related to the nodes input into the question generator (3000) or answer generator (4000) through a tree structure such as the present invention, without the need to refer to all of the aforementioned multiple documents, then less time may be required and question information or answer information can be generated efficiently based on detailed and accurate information.

[0107] Additionally, the question and answer information generated using a tree structure such as the above-mentioned big tree can include more accurate and specific information, and since question / answer pairs can be generated by difficulty level or attribute through the above-mentioned tree structure, it can be utilized when evaluating a system using an LLM outside the server system (1000), such as verifying the question or answer ability of a system using an LLM outside the server system (1000).

[0108] FIG. 5 schematically illustrates information included in a node and metadata according to an embodiment of the present invention.

[0109] Schematically, FIG. 5(a) illustrates the information contained in each node, and FIG. 5(b) illustrates the information contained in the metadata.

[0110] As illustrated in FIG. 5(a), each node included in the dataset (1100) may include text information, embedding values, and metadata. The dataset (1100) of the present invention includes a tree structure in which the document contents of a plurality of documents are structured in a plurality of nodes, and each of the plurality of nodes includes text information corresponding to the document contents of one or more related documents among the plurality of documents.

[0111] Additionally, each of the above multiple nodes includes an embedding value, and the embedding value corresponds to an output value derived by inputting text information of the corresponding node into an embedding model. Based on the embedding value of each node, one or more nodes included in the same level can be clustered. At this time, the degree of correlation between one or more nodes can be derived based on the embedding value, and nodes with high correlation can be clustered together.

[0112] In one embodiment of the present invention, the plurality of nodes may include metadata including source information and table of contents information of a document for each leaf node. The metadata of a parent node is generated based on the metadata of one or more child nodes, and when clustering one or more nodes included at the same level, the nodes may be clustered based on the matching rate of the metadata of each node.

[0113] As illustrated in FIG. 5(b), the metadata may include source information and table of contents information. Each of the plurality of documents may include source information corresponding to the source of the content of the document or table of contents information corresponding to the table of contents of the document. Accordingly, if the document includes source information or table of contents information, a node referencing the document may have the source information or table of contents information as the metadata of the node.

[0114] In this case, the source information may include the source when the document content has a source outside of the plurality of documents, and may include information that indicates which chunk of the document content of the document referenced by the node has the source.

[0115] In one embodiment of the present invention, metadata may not be included in all nodes, may not exist depending on the document referenced by the node, and may include either the source information or the table of contents information. Additionally, the metadata may further include author information corresponding to the author of the document, or subject information including the subject of the document.

[0116] Therefore, since a higher matching rate of the above metadata corresponds to a more similar document, when clustering one or more nodes included in the same level, nodes can be clustered based on the matching rate of the metadata of each node. Additionally, multiple documents can be filtered or classified based on metadata, and when selecting a specific node, the node can be selected by filtering the documents that the node references.

[0117] For example, when receiving keyword information from a user terminal (2000) and selecting a node corresponding to the keyword information, a node related to the keyword information can be found based on the metadata of each node.

[0118] FIG. 6 schematically illustrates the process of performing a question generation step according to one embodiment of the present invention.

[0119] In summary, the question generation step can determine a question node that serves as the basis for question generation in a dataset (1100) that includes a tree structure in which the document contents of multiple documents are structured in multiple nodes, and input the text information of the determined question node into a question generator (3000) that includes a deep learning-based language model to derive question information.

[0120] Specifically, when a question node is determined among a plurality of nodes included in a tree structure included in a dataset (1100), text information of the determined question node can be input into a question generator (3000) to derive question information. The question generator (3000) includes a deep learning-based language model and is a model that derives questions based on input data, and can be located inside or outside the server system (1000).

[0121] If the determined question node corresponds to a leaf node, the text information of the question node includes text that does not summarize the document content, and if the question node does not correspond to a leaf node, the text information of the question node may include text that summarizes the document content. Therefore, when generating question information, the question information may be generated based on the unsummarized document content or a summarized document containing the document content.

[0122] At this time, the tree structure in which the question node is determined may include the small tree and the big tree. When the question node is determined in the big tree, question information can be derived by referring to more documents than when the question node is determined in the small tree, so there is an effect of generating more specific questions.

[0123] If a question node is determined in a big tree, selecting a question node at a higher level of the tree structure can generate question information that references a larger number of documents; consequently, when generating answer information, a larger number of documents may also be required to provide a response. Therefore, as the level of the question node increases, the attributes of the question may shift from a more comprehensive question to a more specific one.

[0124] In one embodiment of the present invention, the question node is a node that serves as the basis for generating a question, and by inputting the text information of the question node into the question generator (3000), a question can be generated based on the document content corresponding to the text information, and the question node may include one or more nodes.

[0125] The question node can be determined in various ways, such as being determined from a user terminal (2000), being determined arbitrarily by a server system (1000), being determined according to a rule set in the server system (1000), being determined based on a question generation node selected by the user terminal (2000), being determined based on a question generation node arbitrarily determined by the server system (1000), or having all nodes included in the dataset (1100) determined as question nodes once.

[0126] As illustrated in FIG. 6, a tree structure structured into three levels of hierarchy including nodes #1 to #10 exists in the dataset (1100), and (1) may correspond to a question node. When node #6 corresponding to (1) is determined to be a question node, the text information of node #6 can be input into a question generator (3000) to derive question information.

[0127] Preferably, the question information may correspond to a question included in a question / answer pair generated based on multiple documents to evaluate a system using LLM. Accordingly, the question information derived from the question generator (3000) may be included in a question for evaluating a system using LLM located outside the server system (1000).

[0128] FIG. 7 schematically illustrates the process of performing the step of generating a question based on parameters according to one embodiment of the present invention.

[0129] In summary, FIG. 7 (a) illustrates the process of performing the step of receiving parameters from a user terminal (2000), and FIG. 7 (b) illustrates the process of performing the step of determining a question node corresponding to the received parameters.

[0130] Specifically, the question generation step may determine a node corresponding to a parameter received from a user terminal (2000) among a plurality of nodes existing within the dataset (1100) as a question node. Accordingly, the parameter may be received from the user terminal (2000) to determine the question node, and the question node may be determined according to the user's input.

[0131] As illustrated in FIG. 7(a), the server system (1000) may receive parameters for determining a question node from a user terminal (2000). The parameters may include one or more of difficulty information, keyword information, document count information, document information, and level information, and may include information for deriving question information or information for determining a question node, so that a question node may be determined based on the parameters.

[0132] As illustrated in FIG. 7(b), when a parameter for deriving question information is input to the server system (1000), a question node corresponding to the parameter can be determined in the tree structure of the dataset (1100). At this time, the question node may include one or more nodes, and the question information derived according to the question node may include one or more questions.

[0133] The tree structure illustrated in FIG. 7(b) includes 10 nodes including nodes #1 to #10, and the node corresponding to the parameter received by the server system (1000) from the user terminal (2000) can be determined as the question node. Accordingly, question information can be derived by reflecting the user's selection, and since the user can indirectly determine the question node, the user can generate the type of question they want.

[0134] FIG. 8 schematically illustrates information included in parameters according to one embodiment of the present invention.

[0135] In summary, the question generation step may determine a node corresponding to a parameter received from a user terminal (2000) among a plurality of nodes existing within the dataset (1100) as a question node. Accordingly, the parameter may be received from the user terminal (2000) to determine the question node, and the question node may be determined according to the user's input.

[0136] As illustrated in FIG. 8, the parameter may include one or more of difficulty information, keyword information, document count information, document information, and level information, and may include information for deriving question information or information for determining a question node.

[0137] In one embodiment of the present invention, the question generation step may derive question information by determining the level of a question node based on the difficulty information for a plurality of nodes that are layered into levels, and determining one or more question nodes among the nodes included in the determined level, and then deriving question information based on the question nodes.

[0138] Accordingly, when difficulty information is received from a user terminal (2000), a question node can be determined among one or more nodes included in a level corresponding to the received difficulty information. The difficulty information may correspond to the difficulty of the generated question information, and since the difficulty of the question information may increase as the level of the node increases, a node included in a higher hierarchy in the tree structure can be determined as the question node as the difficulty of the question information received from the user terminal (2000) increases.

[0139] At this time, if there are multiple nodes corresponding to the input difficulty information, the server system (1000) may arbitrarily determine a question node among the multiple nodes, generate multiple questions by making all of the multiple nodes question nodes, or receive another parameter from the user terminal (2000) and determine a question node based on one or more parameters.

[0140] When inputting the difficulty information at the user terminal (2000), the level or hierarchy of the question node, the difficulty of the question, information on how comprehensive or specific the question is, or the attributes of the question may be input. Accordingly, the difficulty information includes various information that can determine the level of the question node.

[0141] The question generation step described above can derive question information by inputting parameters from a user terminal (2000) to derive question information, including keyword information, determining one or more nodes among the plurality of nodes related to the keyword information as question nodes, and deriving question information based on the question nodes.

[0142] Accordingly, when keyword information is received from a user terminal (2000), a question node can be determined among the nodes related to the received keyword information. The keyword information may correspond to a keyword included in the text information of the question node or may correspond to a keyword related to the generated question information, and among a plurality of nodes existing in the dataset (1100), one or more nodes that include the keyword received from the user terminal (2000) or include content related to the keyword may be determined as the question node.

[0143] At this time, if there are multiple nodes related to the input keyword information, the server system (1000) may arbitrarily determine a question node among the multiple nodes, generate multiple questions by making all of the multiple nodes question nodes, or receive another parameter from the user terminal (2000) and determine a question node based on one or more parameters.

[0144] When inputting the keyword information at the user terminal (2000), the user may input the keyword in the form of text containing words or sentences, and when the server system (1000) provides multiple keywords related to the multiple documents, the user may select one or more keywords. Accordingly, the keyword information may include various forms of text related to question nodes or question information, and may include one or more keywords.

[0145] In addition, when determining a question node based on input keyword information, a node can be determined as a question node if the node's text information contains the corresponding keyword or if the node's text information is related to the corresponding keyword.

[0146] The question generation step described above can derive question information by including document count information as a parameter received from a user terminal (2000) to derive question information, and for the plurality of nodes referencing one or more documents, selecting one or more nodes among the plurality of nodes that refer to a number of documents corresponding to the document count information, determining one or more question nodes related to a preset keyword among the selected nodes, and then deriving question information based on the question nodes.

[0147] Accordingly, when document count information is received from a user terminal (2000), a question node can be determined among the nodes that reference a number of documents corresponding to the received document count information. The document count information corresponds to information that allows each node to know how many documents it references among the plurality of documents, and among the plurality of nodes existing in the dataset (1100), one or more nodes that reference the number of documents received from the user terminal (2000) can be determined as question nodes.

[0148] At this time, if there are multiple nodes related to the input document count information, the server system (1000) may arbitrarily determine a question node among the multiple nodes, generate multiple questions by making all of the multiple nodes question nodes, or receive another parameter from the user terminal (2000) and determine a question node based on one or more parameters.

[0149] When inputting the document count information at the user terminal (2000), the user may input a specific number. Accordingly, the document count information may include information that can determine how many documents the question node will refer to to generate question information. Preferably, each of the plurality of nodes may include information that can determine how many documents the node was generated by referring to.

[0150] For example, if the document count information is entered as 3, the question node can be determined among the nodes that reference 3 documents in the tree structure of the dataset (1100). At this time, since the first tree structure references one document and the second tree structure references multiple documents, if 1 is entered as the document count information, the question node can be determined among the nodes included in the first tree structure, and if a number of 2 or more is entered as the document count information, the question node can be determined among the nodes included in the second tree structure.

[0151] The question generation step described above can derive question information by inputting parameters from a user terminal (2000) to derive question information, the parameters include document information including the title of one or more specific documents, and level information of a node, and for the plurality of nodes hierarchized into each level, one or more nodes included in the level corresponding to the level information among the plurality of nodes referencing specific documents corresponding to the document information are determined as question nodes, and question information can be derived based on the question nodes.

[0152] Accordingly, when document information and level information are received from a user terminal (2000), a question node can be determined by finding a node that is included in the level corresponding to the received level information among the nodes that reference a specific document corresponding to the received document information. The document information corresponds to information containing the titles of one or more documents that each node among the plurality of documents references, and the level information corresponds to information containing the level of each node in a tree structure hierarchically organized into each level.

[0153] If only the document information is received as the above parameter, the node referencing the specific document corresponding to the document information may correspond to all nodes included in the first tree generated based on the specific document, so the level information may also be received from the user terminal (2000) in order to select which level of node among the nodes included in the first tree will be determined as the question node.

[0154] At this time, if there are multiple nodes corresponding to the input document information and level information, the server system (1000) may arbitrarily determine a question node among the multiple nodes, generate multiple questions by making all of the multiple nodes question nodes, or receive another parameter from the user terminal (2000) and determine a question node based on one or more parameters.

[0155] When inputting the document information at the user terminal (2000), the user may input the title of a specific document in the form of text containing words or sentences, and when the server system (1000) provides the titles of the plurality of documents, the user may select one or more document information. Accordingly, the document information may include text related to a question node or question information, and may include one or more document titles.

[0156] Additionally, when inputting the level information at the user terminal (2000), the user may input a specific number, and if the server system (1000) provides a level of a tree structure, the user may select one of the multiple levels. Accordingly, the level information may include information that indicates which level the question node is included in.

[0157] Preferably, the parameters input from the user terminal (2000) may include one or more of difficulty information, keyword information, document count information, document information, and level information, and as the number of input parameters increases, question information closer to the question desired by the user can be generated or more specific question information can be generated.

[0158] FIG. 9 schematically illustrates the process of performing the answer generation step according to one embodiment of the present invention.

[0159] In summary, the answer generation step can derive answer information by inputting the text information of the question node, the question information, and the node with high correlation to the question node among the sub-nodes of the question node into an answer generator (4000) that includes a deep learning-based language model.

[0160] Specifically, after question information is generated based on a question node, the text information of the question node, the question information, and the node with the highest correlation to the question node among the sub-nodes of the question node can be input into an answer generator (4000) to derive answer information. The answer generator (4000) includes a deep learning-based language model and is a model that derives an answer to the question based on the input question and data, and can be located inside or outside the server system (1000).

[0161] In one embodiment of the present invention, among the sub-nodes of the question node, the node with the highest degree of relevance to the question node may be referred to as the answer node. If the question node and the answer node correspond to leaf nodes, the text information of the question node and the answer node includes text that does not summarize the document content, and if the question node and the answer node do not correspond to leaf nodes, the text information of the question node and the answer node may include text that summarizes the document content.

[0162] Therefore, when generating answer information, since the answer information must be generated based on the unsummarized document content to increase the accuracy of the answer, if the above question node and the above answer node do not correspond to leaf nodes, the leaf nodes of each of the above question node and the above answer node can be additionally input into the answer generator (4000) to generate answer information.

[0163] At this time, the tree structure in which the answer node is determined may include the small tree and the big tree. When the answer node is determined in the big tree, answer information can be derived by referring to more documents than when the answer node is determined in the small tree, so there is an effect of generating a more accurate answer.

[0164] If an answer node is determined in a big tree, answer information referencing a larger number of documents can be generated as the answer node is selected from the upper hierarchy of the tree structure; and since the answer node is determined from among the child nodes of the question node, as the level of the question node increases, more specific and accurate answers can be derived from more comprehensive answers.

[0165] In one embodiment of the present invention, the answer node is a node that serves as the basis for generating an answer, and by inputting the text information of the answer node into the answer generator (4000), an answer to the question information can be generated based on the document content corresponding to the text information, and the answer node may include one or more nodes.

[0166] The above-mentioned answer node corresponds to the node among the child nodes of the above-mentioned question node that has the highest relevance to the above-mentioned question node, and said relevance can be derived based on the embedding value of each node. The text information of each child node of the above-mentioned question node can be input into an embedding model, and based on the output embedding value, a single node with the highest relevance to the above-mentioned question node can be determined as the above-mentioned answer node.

[0167] As illustrated in FIG. 9, a tree structure structured into three levels of hierarchy including nodes #1 to #10 exists in the dataset (1100), where (1) corresponds to a question node and (2) corresponds to an answer node. When node #6 corresponding to (1) is determined to be a question node, node #3, which is the node with the highest degree of correlation to node #6 among nodes #1 to #5 included as sub-nodes of node #6, may be determined to be an answer node. Accordingly, text information of node #6 and node #3 can be input into an answer generator (4000) to derive answer information, and text information of node #6 and text information of node #3 can be input into an answer generator (4000) to derive answer information.

[0168] Preferably, the above answer information may correspond to an answer included in a question / answer pair generated based on multiple documents to evaluate a system using LLM. Accordingly, the above answer information derived from the above answer generator (4000) may be included in an answer for evaluating a system using LLM located outside the server system (1000).

[0169] FIG. 10 schematically illustrates the process of performing the step of generating answer information based on a question generation node according to one embodiment of the invention.

[0170] In summary, the answer generation step involves, for a question generation node for determining a question node, selecting a node with high relevance to the question node among the sub-nodes of the question generation node after the question node and the question node have been determined, and inputting the text information of the selected node and the question node into the answer generator (4000) to derive answer information, wherein the relevance may be derived based on the embedding value of each node.

[0171] Specifically, in the case where a question generation node for determining a question node is first determined in the server system (1000), and a question node is determined based on the question generation node and question information based on the question node is generated, the node with the highest degree of correlation to the question node among the sub-nodes of the question generation node is selected, and the text information of the selected node and the question node are input into the answer generator (4000) to derive answer information.

[0172] In one embodiment of the present invention, among the sub-nodes of the question generation node, the node with the highest degree of correlation with the question node may be referred to as the answer generation node. When the question generation node, the question node, and the answer generation node correspond to leaf nodes, the text information of the question generation node, the question node, and the answer generation node includes text that does not summarize the document content, and when the question generation node, the question node, and the answer generation node do not correspond to leaf nodes, the text information of the question generation node, the question node, and the answer generation node may include text that summarizes the document content.

[0173] Therefore, when generating answer information, since the answer information must be generated based on the unsummarized document content to increase the accuracy of the answer, if the above question generation node, the above question node, and the above answer generation node do not correspond to leaf nodes, the leaf nodes of the question generation node, the above question node, and the above answer generation node can be additionally input into the answer generator (4000) to generate answer information.

[0174] At this time, the tree structure in which the answer generation node is determined may include the small tree and the big tree. When the answer generation node is determined in the big tree, answer information can be derived by referring to more documents than when the answer generation node is determined in the small tree, so there is an effect of generating a more accurate answer.

[0175] If an answer generation node is determined in BigTree, the higher the hierarchy of the tree structure from which the answer generation node is selected, the more documents referenced in the answer information can be generated. Furthermore, since the answer generation node is determined from among the child nodes of the question generation node, as the level of the question generation node increases, more specific and accurate answers can be derived from more comprehensive answers.

[0176] In one embodiment of the present invention, the answer generation node is a node that serves as the basis for answer generation, and by inputting the text information of the answer generation node into the answer generator (4000), an answer to the question information can be generated based on the document content corresponding to the text information, and the answer generation node may include one or more nodes.

[0177] The above-mentioned answer generation node corresponds to the node among the child nodes of the above-mentioned question generation node that has the highest correlation with the above-mentioned question node, and the correlation can be derived based on the embedding value of each node. The single node with the highest correlation with the above-mentioned question node can be determined as the above-mentioned answer generation node based on the embedding value output by inputting the text information of each child node of the above-mentioned question generation node into an embedding model.

[0178] As illustrated in FIG. 10, a tree structure structured into three levels of hierarchy including nodes #1 to #10 exists in the dataset (1100), where (1) corresponds to a question generation node, (2) corresponds to a question node, and (3) corresponds to an answer generation node. When node #10 corresponding to (1) is a question generation node, node #6, which is a question node, can be determined based on node #10. In one embodiment of the present invention, when a question node is determined based on a question generation node, it can be determined according to relevance, keywords, hierarchy, etc.

[0179] The answer generation node can be determined as the node with the highest correlation to Node #6 among Nodes #2, #3, #5, #6, and #7, which are child nodes of Node #10 (corresponding to the question generation node), excluding Node #6 (corresponding to the question node). If Node #3 has the highest correlation to Node #6 (the question node), Node #3 can be determined as the answer generation node.

[0180] Accordingly, the text information of Node #6 and Node #3 can be input into the answer generator (4000) to derive answer information, and the text information of Node #6 and Node #3 can be input into the answer generator (4000) to derive answer information.

[0181] In one embodiment of the present invention, the question generation node is a node that serves as the basis for determining the question node, and the question node can be determined based on the question generation node. The question generation node can be determined in various ways, such as being selected by a user terminal (2000) or arbitrarily determined by a server system (1000). The question generation node can be included in the upper layer of the question node or in the lower layer of the question node.

[0182] At this time, for a question node determined by a question generation node, the node with the highest correlation to the question node among the sub-nodes of the question generation node is selected, and when the text information of the selected node and the question node are input into the answer generator (4000) to derive answer information, if the document referenced by the question generation node and the document referenced by the question node correspond to different documents, the answer information can be generated based on evidence derived from more documents than when the answer information is generated based only on the question node. Therefore, a question generation node to determine the question node can be selected first.

[0183] FIG. 11 schematically illustrates the process of performing the step of generating answer information based on leaf nodes according to one embodiment of the present invention.

[0184] In summary, the answer generation step involves selecting a leaf node corresponding to the lowest level node of the question node among the child nodes of the question node, and inputting the selected leaf node and the question node into the answer generator (4000) to derive answer information, wherein the answer information may include source information of a document regarding the leaf node.

[0185] Specifically, after question information is generated based on a question node, a leaf node corresponding to the lowest node of the question node is selected, and the selected leaf node and the question node are input into the answer generator (4000) to derive answer information. At this time, the answer information may include source information of the document referenced by the selected leaf node. The source information may include the source when the content of the document has a source outside of the plurality of documents, and may include information that identifies which chunk of the document content referenced by the node has the source.

[0186] In one embodiment of the present invention, the lowest node among the sub-nodes of the question node corresponds to a leaf node, and since the text information of the leaf node includes text that does not summarize the document content, more accurate answer information can be generated based on the leaf node. In this case, if the source information is included in the metadata of the leaf node, answer information can be generated based on the document content of the document referenced by the leaf node and the document content of another document corresponding to the source information.

[0187] If the leaf node includes multiple nodes, one of the multiple leaf nodes can be selected, or all of the multiple leaf nodes can be input together into the answer generator (4000) to generate answer information.

[0188] In this case, to select the leaf node of the above question node, regardless of whether the question node is a node located in the first tree referencing a single document or a node located in the second tree referencing multiple documents, the leaf node of the question node corresponds to a node referencing a single document. Therefore, since the leaf node can contain the content of the document as is rather than a summary document, when generating answer information based on the leaf node, accurate answer information can be generated by referencing the content of the document as is.

[0189] As illustrated in FIG. 10, a tree structure structured into three levels of hierarchy including nodes #1 to #10 exists in the dataset (1100), (1) corresponds to a question node, and (2) may correspond to a leaf node of the question node. When node #9 corresponding to (1) is a question node, node #5 may be selected among the leaf nodes corresponding to the lowest node of node #9. In one embodiment of the present invention, the leaf nodes corresponding to the lowest node of node #9 may include node #1, node #3, node #4, and node #5.

[0190] Accordingly, when inputting a question node and a leaf node into the answer generator (4000), one or more of Node #1, Node #3, Node #4, and Node #5 may be input, and when determining which leaf node among the multiple leaf nodes to be input into the answer generator (4000), it may be determined based on the degree of relevance to the question node or keywords included in the text information of the corresponding leaf node. If one or more leaf nodes are determined based on the degree of relevance to the question node, the degree of relevance to the question node may be calculated based on the embedding values ​​of the question node and the leaf node, respectively.

[0191] Accordingly, if Node #5 is determined as one leaf node to be input into the answer generator (4000), Node #9 and Node #5 can be input into the answer generator (4000) to derive answer information, and in this case, text information and metadata of each node are included, and text information of Node #9 and text information of Node #5 can be input into the answer generator (4000) to derive answer information.

[0192] FIG. 12 schematically illustrates the process of performing the step of generating answer information based on the correlation with a question node according to one embodiment of the present invention.

[0193] In summary, the answer generation step may, when selecting a node with high relevance to the question node among the child nodes of the question node, derive the degree of relevance to the question node based on the embedding value of the question node and the embedding value of each node included in the child nodes of the question node, and select a predetermined number of nodes in order of high relevance to the question node.

[0194] Specifically, after question information is generated based on a question node, answer information can be generated by selecting a node with high relevance to the question node among the child nodes of the question node. In this case, when there are multiple child nodes of the question node, the top N nodes among the child nodes of the question node can be selected in order of high relevance to the question node. In this case, N is a natural number greater than or equal to 1.

[0195] As illustrated in FIG. 12, a tree structure structured into three levels of hierarchy including nodes #1 to #10 exists in the dataset (1100), and (1) may correspond to a question node. When node #8 corresponding to (1) is a question node, the child nodes of node #8 may include node #2, node #3, and node #5.

[0196] If the question node and two answer nodes with high relevance to the question node are to be determined among the question node's child nodes, the top two nodes among Node #2, Node #3, and Node #5 that have the highest relevance to Node #8 can be determined as answer nodes. In this case, the relevance to Node #8 can be calculated based on the embedding values ​​of Node #2, Node #3, Node #5, and Node #8, and when the three nodes are sorted in order of highest relevance, only the top two nodes can be determined as answer nodes. In this case, if the node with the highest relevance to Node #8 is Node #5 and the node with the next highest relevance is Node #3, then Node #3 and Node #5 can be selected.

[0197] In addition, when determining a single answer node, only the node with the highest correlation to Node #8 can be determined as the answer node.

[0198] Therefore, when selecting a node with high correlation to the question node among the sub-nodes of the question node, one or more nodes corresponding to the top N with high correlation to the question node can be selected as shown in Fig. 12.

[0199] Preferably, only the top N nodes with high relevance to the question node can be selected, and all child nodes of the question node can be selected. If the question node and multiple selected nodes among the question node's child nodes are input into the answer generator (4000) to generate answer information, multiple answer information can be generated based on each of the selected multiple nodes, and a single answer information utilizing all of the question node and the selected multiple nodes can be generated.

[0200] In one embodiment of the present invention, when determining an answer node among the sub-nodes of a question node, when determining an answer generating node among the sub-nodes of a question generating node, and when selecting one or more nodes to be input to an answer generator (4000) among the leaf nodes of a question node, one or more nodes can be determined through steps such as those shown in FIG. 12.

[0201] FIG. 13 illustrates, in an exemplary manner, the internal configuration of a computing device (11000) according to one embodiment of the present invention.

[0202] The server system (1000) mentioned in the description of FIG. 1 may include components of the computing device (11000) illustrated in FIG. 13, which will be described later.

[0203] As illustrated in FIG. 13, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600).

[0204] Specifically, the memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (11000).

[0205] At this time, access to the memory (11200) from other components, such as the processor (11100) or the peripheral device interface (11300), can be controlled by the processor (11100). The processor (11100) may be composed of a single or multiple units and may include processors in the form of GPUs and TPUs to improve computational processing speed.

[0206] The above peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (11000) to the processor (11100) and the memory (11200). The processor (11100) can perform various functions for the computing device (11000) and process data by executing a software module or instruction set stored in the memory (11200).

[0207] The input / output subsystem (11400) may connect various input / output peripheral devices to the peripheral device interface (11300). For example, the input / output subsystem (11400) may include a controller for connecting peripheral devices such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral device interface (11300). According to another aspect, the input / output peripheral devices may be connected to the peripheral device interface (11300) without passing through the input / output subsystem (11400).

[0208] The power circuit (11500) may supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0209] The communication circuit (11600) may enable communication with another computing device using at least one external port. Alternatively, as described above, the communication circuit (11600) may enable communication with another computing device by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal, as needed.

[0210] The embodiment of FIG. 13 is merely an example of the computing device (11000), and the computing device (11000) may have some components shown in FIG. 13 omitted, additional components not shown in FIG. 13 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include a touchscreen or sensors in addition to the components shown in FIG. 13, and the communication circuit (1160) may include a circuit for RF communication of various communication methods (Wi-Fi, 3G, LTE, 5G, 6G, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (11000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0211] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file upon a request from the user terminal.

[0212] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0213] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be standardized and stored or executed in a standardized manner on a networked computing device. Software and data may be stored on one or more computer-readable recording media.

[0214] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0215] According to one embodiment of the present invention, the effect of deriving question information and answer information using a deep learning-based language model can be achieved.

[0216] According to one embodiment of the present invention, the degree of correlation between nodes is derived based on the embedding value of each node, thereby enabling the effect of clustering nodes with high correlation.

[0217] According to one embodiment of the present invention, the effect of clustering nodes based on the matching rate of metadata included in each node can be achieved.

[0218] According to one embodiment of the present invention, the effect of deriving a summary document summarizing one or more documents using a deep learning-based language model can be achieved.

[0219] According to one embodiment of the present invention, by creating a tree structure in which the root node for each document is a leaf node, the effect of expanding a dataset for multiple documents can be achieved.

[0220] According to one embodiment of the present invention, the effect of deriving question information corresponding to parameters input by a user through a question generation step can be achieved.

[0221] According to one embodiment of the present invention, each of a plurality of documents can be chunked to separate the chunks, and the effect of creating leaf nodes for each chunk can be achieved.

[0222] According to one embodiment of the present invention, it is possible to achieve the effect of evaluating the quality of a large language model-based system based on question / answer pairs generated in a server system.

[0223] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below are also within the scope of the claims.

Claims

1. A method for generating multidocument-based question / answer pairs for evaluating a system using LLM, A question generation step of determining a question node that serves as the basis for question generation in a dataset containing a tree structure in which the document contents of multiple documents are structured in multiple nodes, and inputting the text information of the determined question node into a question generator including a deep learning-based language model to derive question information; and The method includes an answer generation step of deriving answer information by inputting text information of a node related to the question node into an answer generator including a deep learning-based language model according to a preset criterion among the question node, the question information, and the sub-nodes of the question node. A method for generating question / answer pairs, wherein for a plurality of nodes of the above dataset, the parent node inputs text information of one or more clustered child nodes into a deep learning-based language model and includes a summary document output.

2. In Claim 1, The above dataset is, The document content included in multiple documents comprises a tree structure consisting of multiple nodes hierarchically arranged at respective levels having superordinate and subordinate concepts, and Nodes included in one or more pre-set levels among the lower layers of the above tree structure are included in a first tree structure structured with the document content of each document, and the remaining nodes are included in a second tree structure structured based on the document content of the root node corresponding to the top-level node of the first tree structure for each document, and A method for generating question / answer pairs in the above tree structure, wherein the parent node includes a form in which the document content of one or more child nodes is summarized.

3. In Claim 1, The method for generating the above question / answer pairs is, It further includes a document structuring step for generating the above dataset, and The above document structuring step is, A leaf node generation step of separating each of a plurality of documents into chunks and generating a plurality of leaf nodes containing text information and embedding values ​​for each chunk; A document internal structuring step for generating a tree structure for a single document by clustering the plurality of leaf nodes based on the embedding value of each leaf node and generating a parent node including a single summary document derived based on the clustered leaf nodes; and A method for generating question / answer pairs, comprising: a document external structuring step of clustering root nodes corresponding to the top-level nodes of each document based on embedding values, and generating a structured dataset for the plurality of documents by generating a parent node based on the clustered root nodes.

4. In Claim 1, The above answer generation step is, A method for generating question / answer pairs, wherein, for a question generation node for determining a question node, after the question node and the question generation node are determined, the node with the highest correlation to the question node among the child nodes of the question generation node is selected, and the text information of the selected node and the question node are input into the answer generator to derive answer information, wherein the correlation is derived based on the embedding value of each node.

5. In Claim 1, The above answer generation step is, A method for generating a question / answer pair, wherein among the child nodes of the above question node, a leaf node corresponding to the lowest level node of the above question node is selected, and the selected leaf node and the above question node are input into the above answer generator to derive answer information, wherein the answer information includes source information of a document regarding the leaf node.

6. In Claim 1, The above answer generation step is, A method for generating question / answer pairs, wherein when selecting a node related to the question node among the child nodes of the question node, the degree of correlation with the question node is derived based on the embedding value of the question node and the embedding value of each node included in the child nodes of the question node, and a predetermined number of nodes are selected in order of high correlation with the question node.

7. In Claim 1, The above question generation step is, A method for generating a question / answer pair, wherein a node corresponding to a parameter received from a user terminal among a plurality of nodes existing within the above dataset is determined as the question node.

8. In Claim 1, The above question generation step is, To derive the above question information, the parameters received from the user terminal include difficulty information, and For the plurality of nodes layered at each level, the level of the question node is determined based on the difficulty information, and A method for generating question / answer pairs by determining one or more question nodes among the nodes included in a determined level, and then deriving question information based on the said question nodes.

9. In Claim 1, The above question generation step is, To derive the above question information, the parameters received from the user terminal include keyword information, and Among the plurality of nodes mentioned above, one or more nodes related to the keyword information are determined as question nodes, and A method for generating question / answer pairs that derive question information based on the above question nodes.

10. In Claim 1, The above question generation step is, To derive the above question information, the parameters received from the user terminal include document count information, and For the plurality of nodes referencing one or more documents, among the plurality of nodes, one or more nodes referencing a number of documents corresponding to the document count information are selected, and A method for generating question / answer pairs by determining one or more question nodes related to a preset keyword among selected nodes, and then deriving question information based on the said question nodes.

11. In Claim 1, The above question generation step is, To derive the above question information, the parameters received from the user terminal include document information containing the title of one or more specific documents, and level information of the node, and For the plurality of nodes hierarchized into respective levels, among the plurality of nodes referencing a specific document corresponding to the document information, one or more nodes included in the level corresponding to the level information are determined as question nodes, and A method for generating question / answer pairs that derive question information based on the above question nodes.

12. A server system that performs a method for generating multi-document-based question / answer pairs for evaluating a system using LLM, A question generation unit that determines a question node serving as the basis for question generation in a dataset containing a tree structure in which the document contents of multiple documents are structured in multiple nodes, and inputs text information of the determined question node into a question generator including a deep learning-based language model to derive question information; and An answer generation unit that derives answer information by inputting text information of a node related to the question node into an answer generator including a deep learning-based language model according to a preset standard among the question node, the question information, and the sub-nodes of the question node; A server system comprising, for a plurality of nodes of the above dataset, an output value obtained by inputting one or more clustered child nodes into a deep learning-based language model for the upper node.

13. A computer-readable storage medium for implementing a method for generating multi-document-based question / answer pairs for evaluating a system using LLM executed on a server system, The above computer-readable storage medium includes computer-executable instructions that cause the server system to perform the following steps, and The steps below are: A question generation step of determining a question node that serves as the basis for question generation in a dataset containing a tree structure in which the document contents of multiple documents are structured in multiple nodes, and inputting the text information of the determined question node into a question generator including a deep learning-based language model to derive question information; and The method includes an answer generation step of deriving answer information by inputting text information of a node related to the question node into an answer generator including a deep learning-based language model according to a preset criterion among the question node, the question information, and the sub-nodes of the question node. A computer-readable storage medium comprising, for a plurality of nodes of the above dataset, an output value obtained by inputting one or more clustered child nodes into a deep learning-based language model for a parent node.

Images