Response generation method and system

Abstract

The present invention relates to a response generation method and system, and provides a response generation method and system using generative artificial intelligence (AI) or a large language model (LLM).

Description

Answer generation method and system

[0001] The present invention relates to a method and system for generating answers, and provides a method and system for generating answers using Generative Artificial Intelligence (Generative AI) or a Large Language Model (LLM).

[0002] The dictionary definition of artificial intelligence is a technology that realizes human learning, reasoning, perception, and natural language understanding abilities through computer programs. This artificial intelligence has achieved rapid development through deep learning.

[0003] Along with the advancement of such deep learning technology, Generative AI technology has recently been attracting attention. More specifically, Generative AI models can generate new data in various forms, such as text, images, and voice, from given data, offering a different dimension of application possibilities compared to simply classifying or predicting existing data.

[0004] Accordingly, as it has become possible to automatically generate sentences, images, and voices that were previously generated by humans using generative AI models, services using generative AI (e.g., ChatGPT) have demonstrated responsiveness and accuracy that differentiate them from existing chatbot services and are receiving great attention worldwide.

[0005] Furthermore, driven by the advancement of artificial intelligence, various language models have been developed. These models have reached a level where they not only recognize text and understand its meaning but also extract and classify information from vast amounts of text-based data, such as documents, and even generate text directly.

[0006] These language models are actively utilized in various fields and exist in diverse areas where text-based tasks can be performed, such as search engines, document creation (e.g., resume writing, report writing, posting, etc.), free conversation on various topics, data parsing from given text (e.g., data summarization, classification, etc.), provision of expertise, programming, and converting given sentences into sentences of an appropriate style.

[0007] In this regard, large-scale language models (LLMs) have recently emerged that understand and generate human language through prior training on vast amounts of text data. Unlike chatbots, which are manually built and provide only limited answers, large-scale language models are demonstrating innovation in the artificial intelligence market by communicating naturally, almost like humans, and providing fast and accurate information.

[0008] Despite the continuous advancement of such generative AI and large-scale language models, research on utilizing generative AI or large-scale language models in various industries and service sectors is still ongoing.

[0009] For example, in the manufacturing processes of displays, semiconductors, and electronic components, defects can occur at multiple stages, and these defects can directly affect product yield, reliability, and delivery deadlines. In particular, in processes handling large panels or precision components, identifying the cause of defects is difficult. Since defects occur due to complex factors such as process conditions, material properties, equipment conditions, and the process environment, the process of identifying causes and establishing countermeasures is extremely challenging. Furthermore, internal company documents containing defect data generated during the manufacturing process are a repository of important domain knowledge, but searching and utilizing them is difficult and inefficient. External models cannot be used due to security concerns, and the performance of models may be degraded due to the specificity of internal terminology.

[0010] As such, there is still a need for methods to utilize generative AI or large-scale language models to proactively prevent quality issues during product development or to resolve them early by rapidly analyzing causes and taking corrective actions when issues arise.

[0011] The present invention aims to provide a method and system for generating answers that can efficiently provide customized information corresponding to the user's query intent.

[0012] More specifically, the present invention aims to provide an answer generation method and system capable of providing accurate and high-quality answers to user queries, while simultaneously providing a service environment optimized for the user.

[0013] Furthermore, the present invention is intended to provide a method and system for generating answers that provide a user-customized environment and enable the user to receive high-quality information.

[0014] Furthermore, the present invention aims to provide a method and system for generating answers that enable the prior prevention and rapid response to quality issues during product development, and support rapid cause analysis and corrective measures when quality issues occur.

[0015] To solve the problem described above, a computerized answer generation method according to the present invention, comprising the following steps, may include: receiving a user query from a user terminal; searching for documents related to the user query using a document search module; extracting at least one specific document from the searched documents based on a pre-set criterion; generating an answer to the user query based on the specific document in a large-scale language model (LLM) trained with training data to be optimized for a pre-set domain; and providing information about the specific document along with the answer to the user terminal.

[0016] In an embodiment, the method further includes a step of extracting core content that serves as a basis for generating an answer to the user query from the extracted specific document, and in the search step, to search for documents related to the user query, a core keyword is extracted from the user query, and documents related to the user query can be searched using the extracted core keyword.

[0017] In an embodiment, in the search step, the document search module searches for documents related to the user query based on the similarity between the core keyword extracted from the user query and the main keyword included in the document, and in the step of extracting at least one specific document, the at least one specific document can be extracted among the searched documents according to the pre-set criteria.

[0018] In the embodiment, the core content may include at least one of a core sentence and a core paragraph that serve as the basis for generating an answer to the user query.

[0019] In an embodiment, in the step of extracting the core content, at least one of the core sentence and the core paragraph that serve as the basis for generating an answer to the user query can be extracted from the specific document.

[0020] In an embodiment, the method further includes the step of configuring a prompt to be input into the large-scale language model based on at least one of the user query and the specific document, and in the step of generating an answer to the user query, the large-scale language model may use the prompt to generate the answer corresponding to the user query intent.

[0021] In an embodiment, it may be configured to include at least one of the prompt, the user query, and the key content extracted from the specific document.

[0022] In an embodiment, if the user query or the specific document contains at least one specific technical term among the technical terms of the pre-set domain, the prompt may be configured to include at least one of the user query, key content extracted from the specific document, and information regarding the specific technical term.

[0023] In an embodiment, the information regarding the specific technical term may include at least one of the definition information, abbreviation information, synonym information, and a description related to the use of the specific technical term.

[0024] In an embodiment, the information regarding the specific document includes at least one of a sentence or paragraph used to generate the answer, summary information of the specific document, and the name of the specific document, and in the step of providing the information regarding the specific document together with the answer, at least one of the sentence or paragraph used to generate the answer, summary information of the specific document, and the name of the specific document may be provided to the user terminal together with the answer.

[0025] In an embodiment, the method further includes the step of receiving user feedback regarding the answer generated through the large-scale language model from the user terminal, wherein the user feedback may include first feedback or second feedback related to the user's evaluation of the answer.

[0026] In an embodiment, the method further includes the step of storing the received user feedback in a specified repository and the step of retraining the large-scale language model using the user feedback stored in the specified repository, wherein the retraining step may be a step of inducing the user to learn the style of the answer preferred by the user by reflecting the user feedback in the retraining.

[0027] In an embodiment, the training data includes at least one of a target question-and-answer data set, a domain glossary containing specialized terms of the pre-set domain, a question-and-answer data set augmented based on at least one document, and a preference data set, and in the search step, the user query can be mapped to the domain glossary to search for documents related to the user query.

[0028] In an embodiment, the target question-and-answer data set is configured to include at least one of a query, a document search result related to the query, a target answer to the query, and a reference document used to generate the target answer, and the augmented question-and-answer set may be constructed based on the reference document used to generate the target answer to the query included in the target question-and-answer data set.

[0029] In an embodiment, the domain glossary is configured to include at least one of definition information, abbreviation information, and a description related to the use of each of the technical terms, and the preference dataset may be configured to include multiple answers of different styles generated for the same query.

[0030] In an embodiment, the preference dataset can induce the artificial intelligence model to learn the style of a specific answer preferred by the user among a plurality of answers of different styles.

[0031] In an embodiment, when the extracted specific document includes a plurality of specific documents, the large-scale language model (LLM) can generate an answer to the user query based on the plurality of specific documents and provide information about each of the plurality of documents along with the answer to the user terminal.

[0032] An answer generation system according to the present invention, comprising a memory configured to store executable instructions and one or more processors configured to perform operations by executing one or more instructions, receives a user query from a user terminal, searches for documents related to the user query using a document search module, extracts at least one specific document from the searched documents based on a preset criterion, generates an answer to the user query based on the specific document using a large-scale language model (LLM) trained with training data to be optimized for a preset domain, and can provide information about the specific document along with the answer to the user terminal.

[0033] A program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored on a computer-readable recording medium, and may include instructions for performing the steps of: receiving a user query from a user terminal; searching for documents related to the user query using a document search module; extracting at least one specific document from the searched documents based on a pre-set criterion; generating an answer to the user query based on the specific document in a large-scale language model (LLM) trained with training data to be optimized for a pre-set domain; and providing information about the specific document along with the answer to the user terminal.

[0034] As described above, according to the answer generation method and system of the present invention, by searching for documents with high similarity to a user query and providing an appropriate answer to the user query along with documents with high similarity to the user query, the user can intuitively recognize the necessary information and understand it more quickly. In particular, the present invention visually provides information regarding the documents utilized in the process of generating the answer, thereby allowing the user to intuitively verify the source and reliability of the answer and to quickly and accurately search for and receive the necessary information.

[0035] Furthermore, according to the answer generation method and system of the present invention, optimal answers to user queries related to said domain can be generated by utilizing a large-scale language model (LLM) trained with training data to be optimized for a pre-set domain. Through this, users can easily solve problems without having to ask other users or search through past data when they need specialized knowledge related to a specific domain. In addition, the present invention ensures security by resolving security issues such as the leakage of technology, information, and sensitive data to the outside through a generative artificial intelligence built to search for knowledge information specialized for a specific company or domain, and improves work efficiency by enhancing specialized knowledge and strengthening users' work capabilities.

[0036] Furthermore, the present invention can improve manufacturing quality management by establishing an advanced system that comprehensively utilizes various internal materials of a specific company, such as not only defective documents but also quality control, security documents, and technical data. The present invention overcomes the limitations of existing keyword search methods and can efficiently provide customized information that matches the intent of user queries through Generative AI or Large Language Models (LLM). In other words, by utilizing quality documents and data that are assets of a specific company, the present invention enables the proactive prevention and rapid response to quality issues during new product development, and in particular, can provide rapid root cause analysis and corrective measures when quality issues arise. Through this, it can contribute to maximizing the efficiency of the quality management process, securing customer trust by resolving defect issues early during the development phase, and improving the level of on-time delivery. Thus, the present invention can contribute to the advancement of the quality management process and the establishment of a user-centric system.

[0037] FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented.

[0038] FIG. 2 illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system, as an embodiment of a computing system in which the present invention can be implemented.

[0039] Figure 3 illustrates an example of a block diagram from another perspective of a computing device, which is one of the components of a computing system.

[0040] FIGS. 4a and FIGS. 4b are conceptual diagrams for explaining an answer generation system according to the present invention.

[0041] FIGS. 5A and FIGS. 5B illustrate examples of raw data according to the present invention.

[0042] FIG. 6 is a flowchart illustrating a method for generating an answer according to the present invention.

[0043] FIGS. 7a, FIGS. 7b, FIGS. 7c, FIGS. 8a, FIGS. 8b, FIGS. 8c, FIGS. 9a, and FIGS. 9b are conceptual diagrams for explaining a method for generating an answer according to the present invention.

[0044] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not have distinct meanings or roles in themselves. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0045] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0046] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0047] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0048] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0049] Hereinafter, the present invention will be examined in more detail with reference to the attached drawings. FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. FIG. 2 illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system as an embodiment of a computing system in which the present invention can be implemented. FIG. 3 illustrates an example of a block diagram of a computing device in another aspect that is one of the components of a computing system.

[0050] Furthermore, FIGS. 4a and 4b are conceptual diagrams for explaining an answer generation system according to the present invention. FIGS. 5a and 5b illustrate examples of raw data according to the present invention. FIG. 6 is a flowchart for explaining an answer generation method according to the present invention, and FIGS. 7a, 7b, 7c, 8a, 8b, 8c, 9a and 9b are conceptual diagrams for explaining an answer generation method according to the present invention.

[0051] Meanwhile, FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. In this regard, the answer generation system (100) according to the present invention can be implemented through a computing device described below and can perform data processing related to the answer generation method described in this specification.

[0052] Referring to FIG. 1, a computing system (10000) that performs a method for generating an answer using Generative AI or a Large Language Model (LLM) for a user query according to one embodiment of the present invention may include at least one computing device. At this time, the at least one computing device may be a single processor or a multiprocessor computing device.

[0053] The components of at least one computing device of the present invention may include various hardware components such as one or more processors, memory, other hardware, and a system bus (not shown) that connects various system components so that they can transmit and receive data to and from each other (e.g., telecommutatively connected, physically connected, electrically connected), and the components of at least one computing device are not limited thereto and may be very diverse.

[0054] Meanwhile, at least one computing device included in a computing system (10000) that performs a method of generating an answer using generative artificial intelligence or a large-scale language model for a user query may be connected to communicate via a network (1070). For example, at least one computing device included in the computing system (10000) may be clustered or may be part of a local area network (LAN). Additionally, at least one computing device may be part of a wide area network (WAN) or connected to at least one of a client-server network and a peer-to-peer network within the cloud.

[0055] Meanwhile, when at least one computing device is used in at least one of a network environment and a cloud computing environment, the at least one computing device may be connected to at least one of a public and private network through a network interface or adapter. In one embodiment, other communication connection devices, such as a modem, may be used to establish communication through the network. The modem may be at least one of an internal modem and an external modem, and may be connected to a system bus through a network interface or a specific mechanism, etc. A wireless network component consisting of an interface and an antenna may be coupled to the network through a device such as an access point, a peer computer, etc. In the present invention, the method of connecting at least one computing device to communicate through the network (1070) is not limited, and it may be connected to communicate in a manner different from the described example.

[0056] Furthermore, other computer-type devices and / or systems not shown in FIG. 1 may also interact technically with at least one computing device or other system through one or more connections to the network (1070) via a network interface. Here, the network interface may include network interface equipment such as a physical network interface controller (NIC) or a virtual network interface (VIF).

[0057] The network (1070) of the present invention may include various forms such as the Internet, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™ RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), etc., and in the present invention, data transmission may be performed based on standard communication protocols such as TCP / IP, HTTP, SSL, etc.

[0058] A computing system (10000) that performs a method for generating an answer using a generative artificial intelligence or a large-scale language model for a user query according to the present invention may include at least one of a user computing device (1010), a training computing system (1050), and a server computing system (1030).

[0059] A user computing device (1010) according to the present invention may be understood as a computing device comprising at least one processor (1011) and a memory (1012) that perform a method of generating an answer using a generative artificial intelligence or a large-scale language model for a user query. For example, the user computing device (1010) may include at least one computing device among a smartphone, a smart TV, a laptop computer, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, and a head-mounted display).

[0060] At least one processor (1011) constituting the user computing device (1010) may include one or more general-purpose processors and / or one or more special-purpose processors. For example, at least one processor (1011) constituting the user computing device (1010) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), an application integrated circuit, an application semiconductor (ASIC), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or electrical units for performing other functions, or a plurality of electrically connected processors.

[0061] Furthermore, at least one processor (1011) may be configured to execute computer-readable instructions contained in memory (1012) and / or other instructions described herein.

[0062] The memory (1012) constituting the user computing system (1010) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media and / or other types of physically durable storage media.

[0063] For example, memory (1012) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and may include web storage of a server that performs the storage function of memory on the internet. Such memory (1012) may store data and instructions necessary for the operation of an application in which the at least one processor (1011) generates an answer to a user query using generative artificial intelligence or a large-scale language model.

[0064] A user computing device (1010) may include one or more user input components (1021) that detect user input. For example, the user input component (1021) may also be referred to as a user interface module. The user input component (1021) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of user input component (1021). In this case, the user input component (1021) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user. Meanwhile, the user of the present invention may refer to an automated agent, script, playback software, etc., that operates on behalf of one or more people.

[0065] A user can interact with a computing system (10000) including at least one computing device through input text, touch, voice, movement, computer vision, gestures and / or other forms of input / output using a user input component (1021). For example, the user input component (1021) may include one or more of a command line interface (CLI), a graphical user interface (GUI), a natural user interface (NUI), a voice command interface and / or other user interface (UI) representations.

[0066] Between the user input component (1021) and the user computing device (1010), one or more application programming interface (API) calls may be made based on user input received from the user interface and / or network.

[0067] Here, the expression "based on" may be interpreted to include cases where it is based on the use of a specific configuration, modified from, derived from, influenced by, dependent on, or otherwise derived from a specific configuration. In some embodiments, an API call may be configured for a specific API, which may be interpreted or converted into an API call configured for another API. Here, an API may refer to a defined interface or connection between computers or between computer programs.

[0068] In one embodiment, the user computing device (1010) may store at least one machine learning model (1020). For example, the user computing device (1010) may be various machine learning models, such as a plurality of neural networks (e.g., deep neural networks) that perform a method of generating answers using generative artificial intelligence or large-scale language models for user queries, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0069] According to an embodiment of the present invention, a user computing device (1010) may perform a method for generating an answer using a generative artificial intelligence or a large-scale language model for a user query by using a local or / and external machine learning model (1020). Alternatively, the user computing device (1010) may perform a method for generating an answer using a generative artificial intelligence or a large-scale language model for a user query by using a machine learning model (1040) provided by a server.

[0070] In addition, according to another embodiment of the present invention, a server computing system (1030) communicating with a user computing device (1010) may provide information on an answer to a user query and at least one document with high similarity to the user query to the user computing device (1010) on an application or / and the web, in accordance with a user's request received through the user computing device (1010).

[0071] In addition, according to another embodiment of the present invention, at least a part of a user computing device (1010) and a server computing system (1030) are linked together to perform a method for generating an answer to a user query using generative artificial intelligence or a large-scale language model, thereby providing the user (or user) with an answer to the user query and at least one document that has high similarity to the user query.

[0072] Additionally, according to various embodiments of the present invention, a user computing device (1010) and / or a server computing system (1030) may learn machine learning models (1020, 1040) performed in a method for generating answers using generative artificial intelligence or a large-scale language model for user queries through interaction with a training computing system (1050) that is communicatedly connected via a network (1070). In this case, the training computing system (1050) may be a computing system separate from the server computing system (1030). Alternatively, in some embodiments, the training computing system (1050) may be part of the server computing system (1030) or part of the user computing device (1010).

[0073] Meanwhile, the server computing system (1030) may include at least one processor (1031) and memory (1032). Here, the processor (1031) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an application integrated circuit, an application semiconductor (ASIC), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions. For example, at least one processor (1031) may include a circuit and a transistor configured to execute instructions from memory (1032).

[0074] The memory (1032) constituting the server computing system (1030) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media, and / or other types of physically durable storage media. For example, the memory (1032) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and combinations thereof, and may include web storage of a server that performs the storage function of memory over the internet. Additionally, the server computing system (1030) may further include a data storage (data store). For example, the data storage may be composed of at least one of a relational database, a NoSQL database, a data warehouse, and a local file system.

[0075] In the memory (1032) constituting the server computing system (1030) according to the present invention, data and instructions necessary for the operation of an application in which at least one processor (1031) generates an answer to a user query using generative artificial intelligence or a large-scale language model may be stored.

[0076] In one embodiment, the server computing system (1030) may be composed of a single device or a plurality of computing devices, and these may be configured to operate according to a sequential or parallel computing architecture. Additionally, a distributed processing system may be configured with a plurality of networked devices.

[0077] Meanwhile, the training computing system (1050) may include at least one processor (1051) and memory (1052). The model trainer (1060) is a logical component that executes the training of at least one machine learning model (1020, 1040) and may be implemented in the form of hardware, firmware, or software. For example, the model trainer (1060) may be executed by the processor (1051) after loading training data (1061) stored in a storage device into memory (1052). For example, the model trainer (1060) may be configured to execute one or more operations (e.g., model training, model reconstruction, model validation, model testing) on ​​at least one machine learning model.

[0078] The machine learning model of the present invention may include at least one of a statistical model, an algorithm, a neural network (NN), a convolutional neural network (CNN), a generative neural network (GNN), a Word2Vec model, a Bag of Words model, a TF-IDF (document frequency-inverse document frequency) model, a GPT (Generative Pre-trained Transformer) model (or other autoregressive models), a PPO (Proximal Policy Optimization) model, a nearest neighbor model (e.g., a k-nearest neighbor model), a linear regression model, a K-means clustering model, a Q-learning model, a TD (Temporal Difference) model, a Deep Adversarial Network model, and all other types of models further described herein.

[0079] Specifically, the model trainer (1060) may execute operations to train a machine learning model, and said operations may include at least one of adding, removing, and modifying model parameters. At this time, the training of the machine learning model may be at least one of supervised learning, semi-supervised learning, and unsupervised learning. In one embodiment, the training of the machine learning model may include the step of repeatedly inputting training data (1061) based on epochs and repeatedly performing the machine learning model training process configured in this way. Here, an epoch may refer to a unit in which the entire set of training data (1061) undergoes forward and backpropagation processing once. In some implementations, different levels of training methods (e.g., supervised learning, semi-supervised learning, unsupervised learning) may be used for different epochs.

[0080] The training data (1061) of the present invention may include input data and / or data previously output from at least one machine learning model (e.g., recursive learning feedback).

[0081] At least one parameter of a machine learning model may include at least one of a seed value, a model node, a model layer, an algorithm, a function, connections between different machine learning models, connections between parameters, machine learning model constraints, and other digital components that influence the output of the machine learning model. In this case, model connections between different machine learning models may include or represent relationships between model parameters and / or models, which may be dependent or interdependent, hierarchical, and / or static or dynamic. The combinations and configurations of model parameters described herein may be too complex to be maintained or utilized by human cognitive abilities.

[0082] In the present invention, the machine learning parameters described according to the embodiments are not limited, and a single machine learning model may further include a plurality of model parameters.

[0083] Meanwhile, FIG. 2 illustrates an example of a block diagram of a computing device (1100) that may be included in a user computing device (1010), a server computing system (1030), and a training computing system (1050), as an embodiment of a computing system (10000) in which the present invention can be implemented.

[0084] As illustrated in FIG. 2, the computing device (1100) may include at least one application (e.g., Application 1 to Application N), and each of the at least one application may include a machine learning library and a model execution environment for performing a method of generating answers using generative artificial intelligence or a large-scale language model for machine learning-based user queries. The at least one application included in the computing device (1100) may communicate with sensors, context managers, device state managers, or additional component(s) within the computing device (1100) via an Application Programming Interface (API). In one embodiment, the at least one application may interface with device components, such as receiving sensor data or state data or transmitting prediction results to an output device via a public or private API.

[0085] Meanwhile, FIG. 3 illustrates an example of a block diagram in another aspect of a computing device (1200), which is one of the components of a computing system (10000) that performs a method of generating an answer using a generative artificial intelligence or a large-scale language model for a user query according to an embodiment of the present invention.

[0086] A computing device (1200) according to the present invention may include at least one application (e.g., Application 1 to Application N), and at least one application may communicate with a central intelligence layer (1210). Each application may interact with a shared model within the central intelligence layer (1210) through an API (e.g., a common API).

[0087] The central intelligence layer (1210) includes one or more machine learning models and may share them among multiple applications or provide them independently to each. In one embodiment, the central intelligence layer (1210) may be integrated as part of an operating system or implemented as a separate logical layer.

[0088] Additionally, the central intelligence layer (1210) can communicate with the central device data layer (1220). The central device data layer (1220) can integrate and store documents and data, such as those containing knowledge information specialized for a specific enterprise or domain, stored within the computing device (1200), and provide them as input data necessary to generate answers to user queries. Each device component (e.g., sensor, state manager, etc.) can communicate with the central device data layer (1220) via private APIs, etc.

[0089] The technology described herein may be composed of a single or multiple computing devices, and a machine learning model that performs a method for generating answers using generative artificial intelligence or a large-scale language model for user queries may be executed sequentially or in parallel on one component or multiple distributed components. Data storage, machine learning models, and applications may be distributed and operated locally or over a network, and these configurations can be flexibly applied to various system architectures.

[0090] Meanwhile, the present invention relates to a method and system for generating answers that can efficiently provide customized information corresponding to the intent of a user's query. More specifically, the present invention relates to a method and system for generating answers that can provide accurate and high-quality answers to user queries while simultaneously providing a service environment optimized for the user.

[0091] The answer generation system according to the present invention performs answer generation based on Generative AI or a Large Language Model (LLM), and may also be referred to as an “answer generation platform based on Generative AI or a Large Language Model.” However, the “Large Language Model” may also be referred to as a Generative Model, a Foundation Model, etc.

[0092] Meanwhile, as illustrated in FIG. 4a, the answer generation system (100) according to the present invention may include at least one of an input unit (110), an output unit (120), a communication unit (130), a storage unit (140), a document search module (150), a large-scale language model (160), a feedback collection unit (170), and a control unit (180). However, the components of the answer generation system (100) according to the present invention are not limited thereto and may further include various hardware components that perform the same or similar roles as described in the description of the present specification.

[0093] The answer generation system (100) according to the present invention may include at least one processor and at least one memory containing computer program code. In this case, the memory may perform the role of the storage unit (140) above. In the present invention, the memory and the program code may cooperate with the processor to perform a series of processes described below.

[0094] Although not illustrated, the answer generation system (100) according to the present invention may include one or more processors, and such processors may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processor, tensor processing unit (TPU), graphics processing unit (GPU), neural network processing unit (NPU), application integrated circuit, application semiconductor (ASIC), field programmable gate array (FPGA), quantum processing unit (or quantum processor, QPU), etc.). One or more processors may be configured to execute instructions, computer-readable instructions, and / or other instructions described herein that are stored (or included) in the storage unit (140). The answer generation method and system according to the present invention may perform data processing described below in cooperation with memory and at least one processor. The processor may perform a series of operations and data processing using data and information stored in memory. In this case, memory may be a component of the storage unit (140).

[0095] In addition, the answer generation system (100) according to the present invention can perform data processing and computation processes using quantum gates, quantum entanglement, and quantum superposition states, taking into consideration implementation in a quantum computer environment. For example, the present invention can perform parallel computations based on qubits, and such quantum computations can operate complementarily with existing classical computers.

[0096] Such quantum computers may include parallel computation using qubits and high-speed data processing devices utilizing quantum entanglement, and hardware-based computational optimization using FPGAs and ASICs is possible. In addition, quantum computers may utilize quantum processors capable of qubit-based parallel computation, and data processing efficiency can be improved through a hybrid structure with existing classical computers.

[0097] Meanwhile, the input unit (110) can be configured in various ways as a means of data input. For example, the input unit (110) can be configured to receive user input. The input unit (110) can be configured to receive user input from a user terminal (10). Here, “receiving input” may mean receiving an input signal (or selection signal) corresponding to the user’s input based on input made by the user through the input unit configuration provided in the user terminal (10).

[0098] Here, the user terminal (10) may include at least one of a mobile phone, a smartphone, a notebook computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, and a wearable device (e.g., a smartwatch, a smart glass, a head-mounted display).

[0099] In addition, the input unit (110) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user.

[0100] The input unit (110) may also be referred to as a user interface module. The input unit (110) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of input unit (110).

[0101] Here, user input may include documents, text, images (or videos), voice, etc. In this case, the answer generation system (100) may further include a module that converts voice into text.

[0102] Next, the output unit (120) can output information through an output unit configuration (e.g., a display unit, a touch screen, a speaker, etc.) provided in a user terminal (10) linked to the answer generation system (100) according to the present invention. For example, the output unit (120) can output a page (or service page, 1000) linked to the answer generation system (100) according to the present invention to the display unit of the user terminal (10). In addition, the output unit (120) does not necessarily mean a hardware means, but can be understood as a channel for outputting results to a user.

[0103] Next, the communication unit (130) may be connected via a wireless or wired network to a user terminal (10), a server (e.g., a central server, an external server, etc.), a device, and at least one network, etc., to receive or transmit overall data and information necessary for the operation of the answer generation system (100) according to the present invention.

[0104] The communication unit (130) can support various communication methods depending on the communication standard of the communicating device.

[0105] For example, the communication unit (130) may be configured to communicate with a communication target using at least one of the following technologies: WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™ RFID (Radio Frequency Identification), Infrared Communication (Infrared Data Association; IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus).

[0106] Next, the storage unit (140, or memory) serves to store various data related to the present invention and may include one or more non-transient computer-readable storage media that can be read and / or accessed by at least one of one or more processors.

[0107] One or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage unit (140) may be implemented using a single physical device (e.g., one optical, magnetic, organic, or other memory or disk storage device), whereas in other examples, the storage unit (140) may be implemented using two or more physical devices.

[0108] The storage unit (140) may include computer-readable instructions and additional data. The storage unit (140) may include a storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functions of the device and network.

[0109] Furthermore, at least a portion of the storage unit (140) may be a cloud storage or a cloud server. At least a portion of the data corresponding to user input received from the input unit (110) and the training data (or training data set) may be stored in the storage unit (140).

[0110] For example, the storage unit (140) may store data input from a specific company (or user) using the answer generation system (100, or answer generation service) according to the present invention, or knowledge information (or data) related to a pre-set domain (e.g., manufacturing process and defect analysis). The storage unit (140) may store various materials (e.g., documents (or defective documents), quality control, security documents, technical data, etc.) of the specific company (or within the specific company). That is, the storage unit (140) may store documents and data that have been converted into assets of the specific company (or within the specific company).

[0111] That is, the storage unit (140) is sufficient as a space where information necessary for the operation of the answer generation system (100) according to the present invention is stored, and it can be understood that there are no restrictions on the physical space.

[0112] Furthermore, the storage unit (140) may store a computer program including computer program instructions. Furthermore, the storage unit (140) may store a computer program including computer program instructions that control the operation of the system (100) or control the operation of the control unit (180) when loaded into the processor of the system (100).

[0113] Next, the document search module (150) may be configured to perform the role of searching for at least one document (e.g., TOP-K) related to (or associated with) the user query (or question). The document search module (150) may search for documents that the large-scale language model (160) needs to summarize regarding the user query and deliver the text of the appropriate documents, or search for and extract (or derive) documents related to the user query so that the large-scale language model (160) can use (or utilize) them as material for the summary (or material for generating answers).

[0114] In one embodiment, the document search module (150) can search for documents related to the user query among the documents stored in the specified storage based on the similarity between the search term keyword (or core keyword) extracted from the user query and the main keyword included in each of the documents stored in the specified storage (e.g., storage unit (140, or memory), document DB, etc.).

[0115] The documents related to the user query searched through the document search module (150) may include one document or multiple (or multiple) documents. Furthermore, in this specification, the document search module (150) may also be referred to as a “document search unit,” a “document search engine,” or a “document search model,” etc.

[0116] Next, the large-scale language model (160) may be configured to perform the role of generating an optimal answer (or a high-quality answer) to a user query. The large-scale language model (160) can generate an appropriate answer that matches the user's query intent by using documents (or related documents) that have high similarity (or relevance, association, etc.) to the user's query retrieved through the document search module (150).

[0117] In one embodiment, the large-scale language model (160) is a model trained with training data to be optimized for a pre-set domain, and when a user query related to the domain is input, it can generate an optimal answer to the user query.

[0118] In another embodiment, the large-scale language model (160) can integrate documents retrieved through the document search module (150) and describe and summarize appropriate answers in sentence form. Let us assume that multiple documents (e.g., 5) related to a user query are retrieved from the document search module (150). In this case, the large-scale language model (160) can classify the retrieved multiple documents into content that matches the user's query intent and describe all of the multiple documents in sentence form.

[0119] That is, the large-scale language model (160) can perform the role of integrating multiple (or multiple) documents related to a user query to summarize important content, describing all multiple documents in sentence form, or generating answers that match the user's query intent (e.g., distinguishing between causes / measures, excluding content unrelated to the query and documents with low relevance, etc.).

[0120] Next, the feedback collection unit (170) may be configured to perform the role of collecting (or receiving) user feedback (or user feedback data) for answers generated from the large-scale language model (160).

[0121] For example, the feedback collection unit (170) may receive user feedback regarding the answer based on input made through the user terminal (10) to which the answer is provided. In this case, the user feedback may include at least one of a first feedback (e.g., good) and a second feedback (e.g., bad) related to the user's evaluation of the answer.

[0122] Next, the control unit (180) can perform the role of controlling the overall operation of the answer generation system (100) related to the present invention. The control unit (180) can process signals, data, information, etc. that are input or output through the components of the answer generation system (100) described above, or perform a series of data processing to provide or process appropriate information and functions to the user. The control unit (180) can be physically implemented by the processor described above.

[0123] Meanwhile, FIG. 4b illustrates an example of a major component and a data processing process of an answer generation system (100) according to the present invention. A component is an independent unit that constitutes a system and may refer to an element that performs a specific function and is combined through an interface with other components to form the entire system. That is, a component is an independent and / or modularized unit that performs a specific function and, as a basic element constituting the system, enables the entire operation through interaction with other components.

[0124] First, a user interface (e.g., “UI Web Server”, 100a) may be configured to allow a user to input a query (400) and check an answer (400a) to the query (400). For example, when a query (400) is input in an environment such as a web browser or application, the answer generation system (100) may visualize and output an answer (400a) to the query (400) through the user interface (100a). This user interface (100a) may be directly connected to a backend processing module.

[0125] Next, the control module (100b) may be configured to perform the role of controlling the entire process for generating and providing an answer (400a) to a query (400) input by a user. In the present invention, the control module (100b) may be a component included in the control unit (180), or it may be understood as the control unit (180) itself.

[0126] The control module (100b) can perform data preprocessing to convert the user’s query (400) into a form that a large-scale language model (e.g., “EXAONE”, 160) can understand through prompt engineering, or to enable a document search module (e.g., “KeyLook”, 150) and / or a large-scale language model (160) to find and summarize specific documents and content related to the query (400).

[0127] In one embodiment, the control module (100b) can process documents or data to fit the input format of the large-scale language model (160) (e.g., “LLM Preprocessing Data”). The control module (100b) can convert documents or data into a format suitable for input into the large-scale language model (160), or can textify and structure various original documents (e.g., PPT, Excel, reports, etc.) (e.g., Latex, Plain text, etc.). Additionally, the control module (100b) can standardize raw data (or raw data, original data) from various sources into a format that the large-scale language model (160) can process. Furthermore, the control module (100b) can recognize and / or normalize specific (or special) formats (e.g., Latex, Patent documents, etc.) to preserve the structural meaning of the query content.

[0128] In another embodiment, the control module (100b) can perform data preprocessing in a form that can be utilized by the document search module. The control module (100b) can build a search index based on the document's keywords, metadata, location information, etc., or perform at least one of indexing, tokenization, and normalization for information retrieval (IR) optimization (e.g., “IR Preprocessing Data”).

[0129] That is, the control module (100b) can appropriately convert user queries through prompt engineering to optimize them so that the large-scale language model (160) can understand them (e.g., distinguishing defect causes / actions according to the intent of the query, adding summary instructions, etc.), or transmit (or forward) preprocessed data or documents or converted data or documents to the document search module (150) or the large-scale language model (160).

[0130] In this regard, FIGS. 5a and 5b illustrate examples of various documents subject to preprocessing. For example, FIG. 5a illustrates an example of a table (or table, table data, tabular data, etc., 510) included in a first document (e.g., Excel document) among documents stored in a specified repository.

[0131] Here, a table can be structured to include rows (or records, tuples, samples, etc.) and columns (or features, columns, fields, characteristics, attributes, variables, predictors, etc.). For example, a table refers to a data structure composed of rows and columns and can be a unit for storing and managing data in a relational database. Tables can be used to systematically organize and manage data, such as Excel sheets or database tables. Here, a column refers to a set of data arranged vertically in a table, and each column can represent a specific attribute (or field) of the table. Additionally, a row refers to a single line of data arranged horizontally in a table, and each row can represent a single record or tuple. In other words, each row represents a single record in the table, and each column can represent a specific attribute of that record.

[0132] In this case, the table (510) included in the first document may include a plurality of columns (e.g., column 1 (defect phenomenon), column 2 (defect cause), column 3 (measures), column 4 (main keyword)) and a row containing a value corresponding to each of the plurality of columns.

[0133] Additionally, FIG. 5b illustrates an example of a slide (or page, screen, etc., 520) included in a second document (e.g., a PPT document) among the documents stored in a specified repository.

[0134] In one embodiment, the control module (100b) can convert text using location coordinates and row / column information of the document. The control module (100b) can identify groups belonging to the same column on the horizontal axis based on the location coordinates of objects (e.g., tables / text boxes, etc.) within the slide (520). For table-type objects, the control module (100b) can convert text into a UNIX format using row / column information, and can convert text by arranging text similarly to human visual information using the coordinates (x, y) of the object.

[0135] In another embodiment, to perform effective preprocessing, the control module (100b) first extracts the type and detailed information of an object stored within the slide (520) of the first document, ii) separates the objects into individual objects and then extracts coordinate information and detailed information of the objects, iii) performs a clustering process to sort the text in the same order as a human reads, iv) sorts and extracts text within a group in Top-Down order and text between groups in Left-Right order, and non-text objects such as images may be removed at this stage. Additionally, when determining clustering, the control module (100b) may determine the group objects as individual groups rather than recognizing them as a single number (e.g., 1), or may consider non-text objects such as images. Additionally, the control module (100b) may appropriately reorder when extracting text by considering the directionality of the object placement.

[0136] In another embodiment, when all cells of a table (510) in which multiple cells are merged are output, the control module (100b) generates a preprocessing code that converts the table contents into text using a rule-based method, and can use this to preprocess and store the content in multiple different forms suitable for the characteristics of the document search module (150) and the large-scale language model (160), respectively. In this case, the data converted into “Latex” is input to the document search module (150) with the content preserved, and the data converted into “Plain Text” is input to the large-scale language model (160) with the content preserved.

[0137] Furthermore, the roles performed in the document search module (150) and the large-scale language model (160), respectively, have been explained in more detail above, so they will be omitted to avoid duplication of explanation.

[0138] Meanwhile, the present invention is intended to provide a method and system for generating answers that can efficiently provide customized information corresponding to the user's query intent. More specifically, the present invention is intended to provide a method and system for generating answers that can provide accurate and high-quality answers to user queries, while simultaneously providing a service environment optimized for the user. Below, we will examine in more detail how to construct training data to be used for training a large-scale language model (160) that generates answers to user queries, and how to train the large-scale language model (160) using this data.

[0139] As illustrated in FIG. 4b, a large-scale language model (160) that generates an answer (400a) to a user query (400) in an answer generation system (100) may be a model trained with at least one training data (300). More specifically, the large-scale language model (160) may be a model trained with the training data (300) to be optimized for a pre-set domain. Here, a domain may refer to an area where specialized knowledge, data, terms, documents, etc. belonging to (or used in) a specific industry, technology, or business field are concentrated, and may include, for example, specific fields such as manufacturing processes, defect analysis, medical, financial, or legal fields. That is, a domain may refer to an area (or specialized field) where knowledge, data, terms, documents, etc. belonging to a specific specialized field are concentrated.

[0140] Although there may be various examples included in the pre-established domain in the present invention, for the sake of convenience of explanation, this specification describes the invention on the premise that the pre-established domain is related to “manufacturing process and defect analysis (e.g., display manufacturing process and defect analysis).” As an example, good products (normal product data) or defective data (or defective product data) generated during the manufacturing process, and related internal documents, reports, sets of terms, etc., may be understood to belong to the manufacturing process and defect analysis domain.

[0141] In this regard, as described above, the answer generation system (100, or answer generation service) according to the present invention may store data input (or collected) from a specific company (or user) using the answer generation system (100) or knowledge information (or data) related to a pre-set domain (e.g., manufacturing process). Various materials (e.g., documents (or defective documents), quality control, security documents, technical data, etc.) of a specific company (or within a specific company) may be stored in a pre-specified storage unit (e.g., storage unit (140, or memory), document DB, etc.). That is, documents and data that have been asseted within a specific company (or within a specific company) may be stored in the storage unit (140). For example, documents of various formats (e.g., PPT, Excel, DB, etc.) related to defect analysis within a specific company may be collected and stored in a pre-specified storage unit.

[0142] In this case, the control unit (180) can index the stored documents. Here, indexing may be a process of creating an index by structuring the content of the document to enable searching. Alternatively, indexing may be a process of creating an index table by processing the content of the document to enable searching. For example, the control unit (180) may extract text and metadata (e.g., title, author, date, etc.) from the document and perform at least one preprocessing step among stop word removal (e.g., particles, conjunctions, etc.), morphological analysis, and keyword extraction. Then, the control unit (180) may store the preprocessed (or refined) words or keywords in the form of an index (e.g., mapping how many times and at what location the word “bad” appears in a document). At this time, the indexing tool may be a document search module (150), and as a result of indexing, searching for documents using the document search module (150) may be possible.

[0143] Referring again to FIG. 4b, the training data (300) used for training the large-scale language model (160) may vary. For example, the training data (300) may include at least one of a target question-and-answer data set (310), a domain glossary (320), an augmented question-and-answer data set (330), and a preference data set (340).

[0144] First, a method for generating a target question-and-answer data set (or target Q&A data set, 310) will be described. In the present invention, the target question-and-answer data set (310) may also be named a “model question-and-answer data set (or model Q&A data set).”

[0145] In one embodiment, the control unit (180) may receive a user query from a user terminal (10) to generate a target question-response data set (310). Alternatively, the control unit (180) may generate a query related to a pre-configured domain (e.g., manufacturing process) using a generative model or a large-scale language model. In this case, the control unit (180) may generate (or configure) a query based on actual documents related to the pre-configured domain.

[0146] Next, the control unit (180) can search for documents related to the query using the document search module (150). The document search module (150) searches for documents related to the query and can extract at least one document among the searched documents that has high relevance (or relevance, similarity, etc.) to the query. Alternatively, the document search module (150) can extract multiple documents (e.g., TOP-K) among the searched documents that have high relevance to the query.

[0147] Furthermore, the control unit (180) can generate a target answer (or best answer) for a query using a large-scale language model (160). The large-scale language model (160) can generate a target answer for a query based on extracted documents. That is, the large-scale language model (160) can generate a summary answer for a query through extracted documents.

[0148] This target question-answer dataset (310) may be configured to include at least one of a query, a document search result related to the query, a target answer (or a summary target answer) for the query, and a reference document used to generate the target answer. In this case, the target question-answer dataset (310) may be configured such that at least one of the query, the document search result related to the query, the target answer (or a summary target answer) for the query, and the reference document used to generate the target answer forms a pair. Through this, the present invention can obtain accurate and high-quality training samples to be used for training (e.g., fine-tuning) a large-scale language model (160).

[0149] Next, the control unit (180) may receive a domain terminology set (320) related to a pre-set domain from a specific company or user (or a user belonging to a specific company) using the answer generation system (100, or answer generation service) according to the present invention. Alternatively, the control unit (180) may collect a domain terminology set (320) related to a pre-set domain from a server linked to the answer generation system (100) according to the present invention (e.g., a server of a specific company). However, in addition to the examples mentioned above, there may be various ways in which the domain terminology set (320) is received and / or collected in the present invention.

[0150] The domain glossary (320) may be configured to include at least one technical term used in (or belonging to) a pre-set domain. For example, if the pre-set domain is “manufacturing process and defect analysis,” the domain glossary (320) may include technical terms actually used in the field of manufacturing process and defect analysis.

[0151] This domain glossary (320) may not simply list terms (or words), but may include at least one of definitions (or definition information), abbreviations (or abbreviation information), and descriptions (or usage context, sentence-type description including usage context, etc.) for each of the specialized terms (or multiple specialized terms) used in the pre-established domain.

[0152] For example, the domain glossary (320) may include at least one of definition information for a first technical term (e.g., lifting) used in a pre-established domain (e.g., a phenomenon where two layers separate due to poor adhesion in a lamination process), abbreviation information (e.g., none), synonym information (e.g., none), and a description related to the use of the first technical term (e.g., mainly used to refer to poor adhesion during the OLED panel lamination process).

[0153] As another example, the domain glossary (320) may include at least one of definition information for a second technical term (e.g., PP) used in a pre-established domain (e.g., meaning the entire preprocessing process performed before the main manufacturing stage), abbreviation information (e.g., Pre-process), synonym information (e.g., Preprocessing), and a description related to the use of the second technical term (e.g., used in reports such as “the defect rate increased due to contamination that occurred in the PP stage”).

[0154] Next, we will look at how to generate an augmented question-and-answer data set (330).

[0155] The control unit (180) can improve the generalization performance of the large-scale language model (160) and expand the diversity (or volume) of the training data (300). To this end, the control unit (180) can generate an augmented question-and-answer data set (330) based on the target question-and-answer data set (310).

[0156] The control unit (180) can augment the target question-and-answer data set (310) using a generative model or a large-scale language model. More specifically, the control unit (180) can augment the target question-and-answer data set (310) based on documents related to at least one query included in the target question-and-answer data set (310). That is, the control unit (180) can construct an augmented question-and-answer data set (330) based on reference documents related to the query included in the target question-and-answer data set (310).

[0157] In one embodiment, the control unit (180) inputs a document (or a plurality of documents) related to a specific query included in the target question-answer data set (310) into a generative model or a large-scale language model, and the generative model or the large-scale language model can generate an augmented query (or a new query) similar to the specific query. Then, the control unit (180) inputs the augmented query into the generative model or the large-scale language model, and the generative model or the large-scale language model can generate an answer to the augmented query based on a document related to the augmented query (i.e., a document related to the specific query). Then, the control unit (180) can construct an augmented question-answer data set (330) comprising at least one of the augmented query, a document search result related to the augmented query, an answer (or summary answer) to the augmented query, and a reference document (or a plurality of reference documents) used to generate an answer to the augmented query. In this case, the augmented question-answer data set (330) may be configured such that at least one of the augmented query, the document search result related to the augmented query, the answer to the augmented query, and the reference document used to generate the answer forms a pair. Through this, the present invention can secure a large amount of training data to improve the generalization performance of the large-scale language model (160) that is the subject of training.

[0158] Next, we will look at how to generate a preference data set (340).

[0159] The control unit (180) can generate a preference dataset (340) configured to include multiple answers of different styles for the same query in order to induce a large-scale language model (160) to learn the answer style preferred by actual users.

[0160] In one embodiment, the control unit (180) can generate at least one of a first-style first answer (e.g., Wafer defect analysis can be performed through correlation analysis between process log data and inspection equipment measurement results. For example...) and a second-style second answer (e.g., The process of analyzing defects in a wafer process is similar to an investigation looking for clues at a crime scene. First...) by using a generative model or a large-scale language model for the same query (e.g., How can defect patterns occurring in a wafer process be analyzed?). The control unit (180) can receive a user's selection from a user terminal (10) for a specific preferred answer among multiple answers of different styles generated for the same query (e.g., First answer, Second answer, etc.). In this case, the preference data set (340) may include preference information for each of the multiple answers of different styles (e.g., preferring the first answer and not preferring the second answer). Such preference information may exist mapped to each of the first answer and the second answer, and the large-scale language model (160) may be induced to learn the style of the answer preferred by the user based on the preference information. At this time, if the first answer is selected as the preferred answer preferred by the user, the control unit (180) may induce the large-scale language model (160) to learn the style of the first answer.

[0161] This preference data set (340) may be configured to include at least one of a specific query, multiple answers of different styles to said specific query (e.g., a first answer and a second answer, etc.), and a preference for each of the multiple answers. In this case, the preference data set (340) may be configured such that at least one of the specific query, multiple answers of different styles to said specific query (e.g., a first answer and a second answer, etc.), and a preference for each of the multiple answers forms a pair.

[0162] Meanwhile, the control unit (180) can train a large-scale language model (160) using training data (300).

[0163] In one embodiment, the control unit (180) can fine-tune the large-scale language model (160) using the target question-answer data set (310) to obtain an understanding and answer structure for documents related to a pre-set domain.

[0164] In another embodiment, the control unit (180) may train the large-scale language model (160) using a domain glossary (320) to understand specialized terms related to a pre-set domain and to improve the quality of the answer of the large-scale language model (160) to a query related to the domain. The domain glossary (320), which includes specialized terms related to the domain, may be input in parallel with the target question-and-answer dataset (310) and / or the augmented question-and-answer dataset (330) during the training of the large-scale language model (160) and reflected in the training of the large-scale language model (160).

[0165] In another embodiment, the control unit (180) can train the large-scale language model (160) using an augmented question-and-answer dataset (330) to improve the generalization performance of the large-scale language model (160) and to enable it to respond to various types of queries.

[0166] In another embodiment, the control unit (180) may train a large-scale language model (160) using a preference dataset (340) to improve the naturalness and usefulness of the answers. The control unit (180) may perform Direct Preference Optimization (DPO) tuning on the large-scale language model (160) using the preference dataset (340) so that the large-scale language model (160) learns the answer style preferred by actual users among multiple answers of different styles generated for the same query.

[0167] In this way, the learned large-scale language model (160) can increase domain understanding and improve the accuracy of response sentences. That is, the learned large-scale language model (160) can accurately interpret the context and technical terms of the document, and can generate natural responses that correspond to the user's query intent and match the user's preferences. Finally, the learned large-scale language model (160) can be deployed as a chatbot-type interface (or chatbot-type UI) and utilized effectively.

[0168] Meanwhile, an answer generation system (100) including a large-scale language model (160) trained using the training data described above can provide information about a specific document along with an optimal answer to a user query to a user terminal at the inference stage, as illustrated in FIG. 6, through the process of receiving a user query from a user terminal (S610), the process of searching for documents related to the user query using a document search module (S620), the process of extracting at least one specific document from the searched documents based on a preset standard (S630), and the process of generating an answer to the user query based on the specific document in a large-scale language model (LLM) trained with training data to be optimized for a preset domain (S640) (S650).

[0169] In the present invention, interaction with a user can be achieved through a chatbot (chatbot, or conversational artificial intelligence, conversational agent, etc.) linked with an answer generation system (100). In this case, the chatbot may be implemented based on a learned large-scale language model (160). However, the implementation form of the chatbot in the present invention is merely an example, and it is obvious that it can be implemented in various forms other than a chatbot.

[0170] First, in the present invention, a user query to be the subject of answer generation can be received through a conversation with a user using a user terminal (10) as a medium. A large-scale language model (160) can receive a user query (e.g., guidance message, guidance information, etc.) to the user terminal (10) so that the user query to be the subject of answer generation is specified, and can receive the user query from the user terminal (10). For example, as shown in FIG. 8a, a control unit (180) can provide guidance information (e.g., “Hello. How can I help you, OO?”, 701) to the user terminal (10) through a large-scale language model (160) included in the answer generation system (100) (or linked with the answer generation system (100)) in order to receive the user query. Alternatively, the control unit (180) may provide guidance information (701) for receiving a user query through a service page (1000) linked to an answer generation system (100) provided (or output) to the user terminal (10).

[0171] The control unit (180) can receive a user query (e.g., “What is the cause of the OOO tearing defect?”, 710) corresponding to the user input based on the input received from the user terminal (10) through the service page (1000) (e.g., selection of a graphic object (1000a) associated with the user query receiving function). Then, the control unit (180) can extract key keywords (e.g., OOO tearing, defect, etc.) from the user query (710) to search for documents related to the user query (710), and can search for documents related to the user query using the extracted key keywords.

[0172] Specifically, the control unit (180) can input a core keyword extracted from a user query (710) into the document search module (150). Subsequently, as illustrated in FIGS. 7a and 7b, the document search module (150) can search for documents related to the user query (710) from a specified repository based on (or derived from) the core keyword extracted from the user query (710). In this case, as seen above, the documents stored in the specified repository (or the text (e.g., words, paragraphs, etc.), objects, etc.) contained therein may exist as vectorized documents through a preprocessing process.

[0173] The document search module (150) can search for documents related to a user query based on the similarity between a core keyword extracted from a user query (710) and a major keyword included in the document. For example, as illustrated in FIG. 5a, at least one column (e.g., column 4) among a plurality of columns (or columns, features, etc.) of a table (510) included in the document may correspond to a major keyword, and the table (510) may be configured to include at least one row (or a plurality of rows containing values ​​corresponding to major keywords) that includes a value corresponding to the major keyword.

[0174] More specifically, the document search module (150) can compare the similarity between a core keyword extracted from a user query (710) and a major keyword within a document, and search for documents related to the user query (710) based on the result of the similarity comparison between the two keywords. For example, as illustrated in FIG. 7b, the document search module (150) can compare the similarity between a core keyword (or core keyword vector) extracted from a user query (710) and a major keyword (or major keyword vector) contained in a vectorized document stored in a specified repository, and search for documents (720a to 720n) in which the similarity (or similarity score) satisfies a pre-set threshold value (e.g., 0.85). Here, the core keyword vector or major keyword vector may refer to a vector in which the core keyword extracted from the query or the major keyword contained in the document is converted into coordinates in a multidimensional numerical space. The core keyword vector and the major keyword vector are each configured to reflect semantic characteristics, and the major keyword vector can be used to calculate the relevance of the document by comparing similarity with the core keyword vector.

[0175] And, the document search module (150) can extract at least one specific document from the searched documents (720a to 720n) based on a pre-set criterion. The document search module (150) can extract at least one specific document from the searched documents (720a to 720n) according to a pre-set criterion. For example, the document search module (150) can extract a plurality of specific documents (e.g., the first specific document (731) to the Nth specific document (735)) from the searched first document (e.g., “Document 1”, 720a) to the Nth document (e.g., “Document N”, 720n) according to a pre-set criterion. In this case, the pre-set criterion may be related to the TOP-K documents among the searched documents that have the highest similarity to the user query (710).

[0176] Alternatively, if the user query (710) includes specific technical terms used in a pre-configured domain, the document search module (150) can search for and extract documents related to the user query (710) based on the specific technical terms. For example, the document search module (150) can map the specific technical terms included in the user query (710) to a domain glossary (320) to search for documents related to the user query, and extract at least one specific document among the searched documents according to a pre-configured criterion.

[0177] As such, the output of the document search module (150) may include a plurality of specific documents (731 to 735) related to the user query (710).

[0178] Furthermore, the control unit (180) may acquire (or collect) specific documents (731 to 735) extracted through the document search module (150) and extract key content that serves as the basis for generating an answer to a user query (710) from the extracted specific documents (731 to 735). Here, the key content may include at least one of a key sentence (e.g., OOO tearing is a defect where the PAD lifts off the substrate and occurs during PTN inspection after the panel photo process) and a key paragraph (e.g., to address OOO tearing, surface cleaning power enhancement, metal adhesive layer reinforcement, etc. were performed...). For example, the key sentence or key paragraph may also be understood as a sentence or paragraph containing a key keyword extracted from the user query (710). That is, the control unit (180) can extract at least one of the key sentence and key content that serve as the basis for generating an answer to a user query (710) from a specific document.

[0179] Meanwhile, the control unit (180) can configure (or generate) a prompt (or input prompt) to be input into a large-scale language model (160) based on at least one of a user query and a specific document. As an example, the prompt may be configured as in the embodiment below (see Table 1 and Table 2 below).

[0180] postfix = Summarize this briefly.Exaone input = Document(top-5) + postfix

[0181] prefix = 'This article is about {question}. Based on this, summarize the following postfix = '{summary}' Example input = prefix + document(top-1) + postfix

[0182] Specifically, as illustrated in FIGS. 7a and 7b, the control unit (180) can configure a prompt (740) to be input to a large-scale language model (160) through prompt engineering. Prompt engineering may refer to a technique for designing and optimizing prompts so that the large-scale language model (160) can generate a desired response to the input (i.e., to optimize the output of the large-scale language model (160).

[0183] In this case, the prompt (740) may be configured to include a user query (710) and specific documents (731 to 735) related to the user query (710). More specifically, the prompt (740) may be configured to include at least one of the user query (710) and a plurality of specific documents (731 to 735) related to the user query (710) or a core content extracted from each of the plurality of specific documents (731 to 735).

[0184] At this time, if at least one specific technical term among the technical terms of a domain set in the user query (710) or specific documents (731 to 735) is included, the prompt (740) may be configured to include at least one of the core content extracted from the user query (710) and specific documents (731 to 735) and information (710a) about the specific technical term (e.g., OOO tearing). The information (710a) about the specific technical term (e.g., OOO tearing) may include at least one of the definition information of the specific technical term (e.g., OOO tearing is a defect where the PAD lifts off the substrate...), abbreviation information, synonym information, and a description related to the use of the specific technical term (e.g., in internal defect reports, “OOO tearing” is used to refer to the OOO phenomenon and, unlike lifting, is classified as damage caused by mechanical stress...). In this case, when a specific technical term is included in the prompt (740) and processed as input to the large-scale language model (160), the control unit (180) may input the specific technical term into the large-scale language model (160) by highlighting or emphasizing it.

[0185] That is, when encoding a query and a document, the control unit (180) can identify (or search for) a definition of a specialized term that matches the specialized term appearing in the query or the main keyword included in the document, and include it in the input of the large-scale language model (160). In this way, the present invention can induce domain interpretation of the specialized term by the large-scale language model (160) by detecting the specialized term included in the query or document in advance and configuring information about the specialized term in the prompt to be input into the large-scale language model. The large-scale language model (160), learned through the domain terminology dictionary (320), can generate an answer by processing the specialized term without confusing it with other general knowledge.

[0186] Meanwhile, as illustrated in FIGS. 7a and 7c, the control unit (180) can process the prompt (740) as input to the large-scale language model (160). As seen above, the large-scale language model (160) may be a model trained with training data (300) to be optimized for a pre-set domain.

[0187] A large-scale language model (160) can generate an answer corresponding to the intent of the user query (710) (e.g., “OOO peeling is a defect where the PAD lifts off the substrate and mainly occurs during PTN inspection after the panel photo process...”, 750) by using at least one of the user query (710) included in the prompt (740), key content extracted from a specific document (or each of a plurality of specific documents (731 to 735)), and information (710a) about specific technical terms. For example, based on the intent of the user query (710) including the cause of the defect regarding “OOO tearing,” the answer (750) may include at least one of the causes of the defect regarding “OOO tearing” corresponding to the intent of the user query (e.g., 1. tearing due to reduced metal adhesion, 2. tearing due to reduced cleaning power before metal adhesion, 3. tearing due to a difference in thickness between insulating films...) and a solution (or measure) to resolve (or take action) the defect (e.g., “As a measure for this, surface cleaning power enhancement, metal adhesion layer reinforcement and deposition / heat treatment conditions optimization, insulating film thickness uniformization and stress relief...”).

[0188] And, the control unit (180) may obtain an answer (750) generated from a large-scale language model (160) and provide information about a specific document along with the answer (750) to a user terminal. At this time, the control unit (180) may provide information about each of a plurality of specific documents (e.g., a first specific document (731), a second specific document (732), a Nth specific document (735), etc.) to the user terminal (10) along with an answer (750) generated based on the plurality of specific documents (731 to 735) in the large-scale language model (160).

[0189] In this case, information regarding a specific document is included in the specific document and may include at least one of the sentence or paragraph used to generate the answer, summary information of the specific document, and the name of the specific document. For example, as illustrated in FIG. 8b, information regarding each of a plurality of specific documents (731 to 735) may be provided to the user terminal (10) on which the service page (1000) is output, along with an answer to a user query (710) (e.g., “OOO tearing is a defect where the PAD lifts off the substrate and mainly occurs during PTN inspection after the panel photo process. During PTN inspection, the OOO tearing phenomenon may have various causes depending on various factors...”, 750). At this time, the information for each of the plurality of specific documents (731 to 735) may include at least one of the name (731a, 732a, 735a) of each of the plurality of specific documents (731 to 735), a sentence or paragraph included in each of the plurality of specific documents and used to generate the answer (750), and summary information (731b, 732b) of each of the plurality of specific documents (731 to 735). Here, at least one of the sentences, paragraphs, keywords (or words) used to generate the answer (750) in the summary information (731b, 732b) may be highlighted and displayed on a part of the user terminal (10, or service page (1000)) (e.g., refer to parts such as “OOO peeling is a defect where the PAD lifts off the substrate and occurs during PTN inspection after the panel photo process”, “reduction in metal adhesion, reduction in cleaning power, difference in thickness between insulating films..”, “enhanced surface cleaning power, reinforcement of metal adhesive layer”).

[0190] At this time, the display ranking (or sorting ranking) of each of the plurality of specific documents (731 to 735) provided along with the answer (750) may be displayed (or sorted) differently according to a preset standard (or display standard, sorting standard) and provided on the user terminal (10). For example, the preset standard may be related to a document defining a specific technical term included in the user query (710). The control unit (180) may display (or sort) the first specific document (e.g., “<Document 1> Report on OOO tearing phenomenon_0910...”, 731) among the plurality of specific documents (731 to 735) that satisfies the preset standard on the user terminal (10) in priority.

[0191] Additionally, in one area of ​​the user terminal (10, or service page (1000)), graphic objects (750a, 750b, 750c) may be provided that are associated with a user query (710) and have a function to move to a specific document used to generate an answer (750). By selecting the graphic objects, the user can move to (or view) a specific document associated with the query or used to generate an answer.

[0192] Furthermore, as illustrated in FIG. 7c, the control unit (180) can receive user feedback (760) regarding the answer (750) from the user terminal (10) to which the answer (750) was provided, using the feedback collection unit (170). This can also be understood as receiving the user's evaluation result regarding the answer (750).

[0193] For example, as illustrated in FIG. 8c, a graphic object (760a, 760b) associated with a function to receive user feedback (760) may be provided in a region of the service page (1000). The user feedback may include at least one of first feedback (e.g., good) or second feedback (e.g., bad) related to a user's evaluation of the answer (750). Such user feedback may also be understood as a user evaluation of the quality or usefulness of the answer itself. Or, it may be understood as an evaluation of the user's preference regarding the style of the answer.

[0194] In this case, the graphic objects (760a, 760b) may include multiple graphic objects (760a, 760b) having different types (or types, attributes, characteristics, etc.) associated with different user feedback receiving functions. For example, one area of ​​the service page (1000) may include at least one of a first graphic object (760a) associated with a first feedback receiving function and a second graphic object (760b) associated with a second feedback receiving function. The first graphic object (760a) and the second graphic object (760b) are each formed to have a first visual appearance and a second visual appearance, respectively, so that the user can intuitively recognize and perform feedback on the answer (750).

[0195] The feedback collection unit (170) can obtain user feedback regarding the answer (750) based on input received from the user terminal (10). For example, the feedback collection unit (170) can receive first feedback regarding the answer (750) based on the selection of a first graphic object (760a) associated with the first feedback receiving function among a plurality of graphic objects (760a, 760b) from the user terminal (10). Then, the control unit (180) can store the user feedback (first feedback) collected through the feedback collection unit (170) in a designated storage, and use the user feedback stored in the designated storage to retrain (or update, fine-tune, additional training, etc.) the large-scale language model (160). That is, user feedback can be stored (or accumulated) in a specified repository and used for retraining (e.g., DPO tuning) of a large-scale language model (160), or added to a preference dataset (340) and used for DPO tuning of a large-scale language model (160). That is, the control unit (180) can incorporate user feedback into the retraining of the large-scale language model (160) to induce the user to learn the style of answers preferred by the user.

[0196] In this way, the present invention can monitor user feedback on answers in real time, periodically update a large-scale language model (160) by reflecting user feedback, and provide a user environment tailored to the style of individual users.

[0197] Meanwhile, in the present invention, an optimal answer to a query can be provided by utilizing not only a specific domain but also user information (or specific company information). For example, in the answer generation system (100), user information (or specific company information) utilizing the answer generation service provided by the answer generation system (100) may be stored (or registered).

[0198] Such user information may exist in various forms. For example, user information regarding a user employed by a specific company may include at least one of a username, age, department, position (or rank), and information on ongoing projects.

[0199] For example, as illustrated in FIG. 9b, the control unit (180) may receive a user query (e.g., “What is lifting?”, 910) corresponding to the user input based on the input received from the user terminal (10) on which the service page (1000) is output (e.g., selection of a graphic object (1000a) associated with a user query receiving function). In this case, the term “lifting” may be used in various domains (e.g., floor lifting, tile lifting, paint lifting, wallpaper lifting, etc.) in addition to a pre-set domain (e.g., manufacturing process).

[0200] The control unit (180) can input a user query (910) and user information of the user who entered the user query (910) into the document search module (150). The document search module (150) can search for documents related to the user query (910) based on the user information and extract at least one specific document from the searched documents according to pre-set criteria. For example, the document search module (150) can search for and extract at least one specific document among the documents stored in a pre-specified repository that is related to the user information and contains technical terms of a specific domain included in the user query, based on the fact that the user's ongoing project information is related to a "process optimization project."

[0201] And, the control unit (180) can configure a prompt to be input into a large-scale language model (160) through prompt engineering and input the prompt into the large-scale language model (160) to obtain an answer to a user query (910) generated from the large-scale language model (160). At this time, information about specific documents may be provided to the user terminal (10) along with the answer. For example, as shown in FIG. 9b, to the user terminal (10) where the service page (1000) is displayed, information about each of a plurality of specific documents may be provided along with an answer generated using user information and documents extracted based on user information for the user query (910) (e.g., “Hello. Manager OO of the OO Department! Based on the “Process Optimization Project” currently in progress, the “lifting” you asked about is a phenomenon in which an interlayer gap occurs due to adhesion failure during the thin film (or lamination) process...” 920). At this time, the information for each of the multiple specific documents may include at least one of the name of each of the multiple specific documents (901a, 902a, 905a), a sentence or paragraph included in each of the multiple specific documents and used to generate the answer (920), and summary information of each of the multiple specific documents (901b, 902b).

[0202] Additionally, in one area of ​​the user terminal (10, or service page (1000)), graphic objects (920a, 920b, 920c) may be provided that are associated with a user query (910) and have a function to move to a specific document used to generate an answer (920). By selecting the graphic objects, the user can move to (or view) a specific document associated with the query or used to generate an answer.

[0203] Furthermore, the control unit (180) can receive user feedback regarding the answer (920) from the user terminal (10) to which the answer (920) was provided, by using the feedback collection unit (170). For example, as shown in FIG. 9b, a graphic object (930a, 930b) associated with a function for receiving user feedback (930) may be provided in a part of the service page (1000), thereby allowing user feedback regarding the answer (920) to be received. Since details regarding user feedback have been described in more detail above, they will be omitted to avoid duplication of explanation.

[0204] As described above, according to the answer generation method and system of the present invention, by searching for documents with high similarity to a user query and providing an appropriate answer to the user query along with documents with high similarity to the user query, the user can intuitively recognize the necessary information and understand it more quickly. In particular, the present invention visually provides information regarding the documents utilized in the process of generating the answer, thereby allowing the user to intuitively verify the source and reliability of the answer and to quickly and accurately search for and receive the necessary information.

[0205] Furthermore, according to the answer generation method and system of the present invention, optimal answers to user queries related to said domain can be generated by utilizing a large-scale language model (LLM) trained with training data to be optimized for a pre-set domain. Through this, users can easily solve problems without having to ask other users or search through past data when they need specialized knowledge related to a specific domain. In addition, the present invention ensures security by resolving security issues such as the leakage of technology, information, and sensitive data to the outside through a generative artificial intelligence built to search for knowledge information specialized for a specific company or domain, and improves work efficiency by enhancing specialized knowledge and strengthening users' work capabilities.

[0206] Furthermore, the present invention can improve manufacturing quality management by establishing an advanced system that comprehensively utilizes various internal materials of a specific company, such as not only defective documents but also quality control, security documents, and technical data. The present invention overcomes the limitations of existing keyword search methods and can efficiently provide customized information that matches the intent of user queries through Generative AI or Large Language Models (LLM). In other words, by utilizing quality documents and data that are assets of a specific company, the present invention enables the proactive prevention and rapid response to quality issues during new product development, and in particular, can provide rapid root cause analysis and corrective measures when quality issues arise. Through this, it can contribute to maximizing the efficiency of the quality management process, securing customer trust by resolving defect issues early during the development phase, and improving the level of on-time delivery. Thus, the present invention can contribute to the advancement of the quality management process and the establishment of a user-centric system.

[0207] Meanwhile, the present invention described above can be implemented based on a quantum computer. The present invention implemented based on a quantum computer may include a qubit-based quantum processor and quantum memory, and may include software and hardware interfaces optimized for quantum computation.

[0208] Quantum processors in quantum computers utilize qubits to efficiently process complex operations through parallel computation, quantum entanglement, and quantum superposition, which cannot be performed by the binary bits of classical computers. Quantum processors process data using quantum gates and can provide exponential speed improvements for specific problems.

[0209] Meanwhile, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a computer-readable medium (or recording medium).

[0210] Furthermore, the present invention described above can be implemented as computer-readable code or instructions on a medium on which a program is recorded. That is, the present invention can be provided in the form of a program.

[0211] Meanwhile, computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0212] Furthermore, the computer-readable medium may be a server or cloud storage that includes a storage and is accessible to an electronic device via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage via wired or wireless communication.

[0213] A computer program may reach the system (100) through various suitable transmission mechanisms. The transmission mechanism may be, for example, a computer-readable storage medium, a computer program product, a memory device, a recording medium such as a CD-ROM or DVD, or a product that tangibly embodies the computer program. The transmission mechanism may be a signal configured to reliably transmit the computer program through air or an electrical connection. The system (100) may propagate or transmit the computer program as a computer data signal.

[0214] Furthermore, references to 'computer-readable storage media,' 'computer program products,' 'computer programs embodied in a tangible form,' etc., or to 'controller,' 'computer,' 'processor,' etc., should be understood to include not only computers with various architectures such as single / multi-processor architectures and sequential (Von Neumann) / parallel architectures, but also specialized circuits such as Field-Programmable Gate Arrays (FPGAs), Application Specific Circuits (ASICs), signal processing units, and other devices. References to computer programs, instructions, code, etc., should be understood to include software for programmable processors or firmware, such as programmable content for hardware devices, whether it is instructions for a processor or configuration settings for a fixed-function device, gate array, or programmable logic device.

[0215] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, namely a CPU (Central Processing Unit), and no special limitations are placed on its type.

[0216] Meanwhile, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

1. In a computerized method comprising the following, A step of receiving a user query from a user terminal; A step of searching for documents related to the above user query using a document search module; A step of extracting at least one specific document from the searched documents based on a pre-established standard; A step of generating an answer to the user query based on the specific document in a large-scale language model (LLM) trained with training data to be optimized for a pre-configured domain; and A method for generating an answer characterized by including the step of providing information about the specific document along with the answer to the user terminal.

2. In Paragraph 1, The method further includes the step of extracting key content from the aforementioned extracted specific document that serves as the basis for generating an answer to the aforementioned user query, and In the above-mentioned search step, In order to search for documents related to the above user query, core keywords are extracted from the above user query, and A method for generating an answer characterized by searching for documents related to the user query using the extracted core keywords.

3. In Paragraph 2, In the above-mentioned search step, In the above document search module, documents related to the user query are searched based on the similarity between the core keyword extracted from the user query and the main keyword included in the document, and In the step of extracting at least one specific document mentioned above, A method for generating an answer characterized by extracting at least one specific document among the documents searched above according to the previously set criteria.

4. In Paragraph 2, The key points above are, A method for generating an answer characterized by including at least one of a core sentence and a core paragraph that serve as the basis for generating an answer to the above user query.

5. In Paragraph 4, In the step of extracting the key content mentioned above, A method for generating an answer characterized by extracting at least one of the core sentence and the core paragraph, which serve as the basis for generating an answer to the user query, from the aforementioned specific document.

6. In Paragraph 1, The method further includes the step of configuring a prompt to be input into the large-scale language model based on at least one of the above user query and the above specific document, In the step of generating an answer to the above user query, A method for generating an answer characterized by using the prompt in the above-described large-scale language model to generate the above-described answer corresponding to the user's query intent.

7. In Paragraph 6, The above prompt is, A method for generating an answer characterized by being configured to include at least one of the core content extracted from the above user query and the above specific document.

8. In Paragraph 6, If the above user query or the above specific document includes at least one specific technical term among the technical terms of the above-set domain, The above prompt is, A method for generating an answer characterized by being configured to include at least one of the above user query, the core content extracted from the above specific document, and the above specific technical term information.

9. In Paragraph 8, Information regarding the specific technical terms mentioned above is, A method for generating an answer characterized by including at least one of definition information, abbreviation information, synonym information, and a description related to the use of the specific technical term.

10. In Paragraph 1, Information regarding the specific document mentioned above is, Included in the aforementioned specific document, comprising at least one of a sentence or paragraph used to generate the aforementioned answer, summary information of the aforementioned specific document, and the name of the aforementioned specific document, and In the step of providing information about the specific document mentioned above along with the above answer, A method for generating an answer characterized by providing at least one of a sentence or paragraph used to generate the answer, summary information of the specific document, and the name of the specific document to the user terminal along with the answer.

11. In Paragraph 1, The method further includes the step of receiving user feedback regarding the answer generated through the large-scale language model from the user terminal. A method for generating an answer characterized in that the above user feedback includes a first feedback or a second feedback related to the user's evaluation of the above answer.

12. In Paragraph 10, The step of storing the received user feedback in a specified repository; and The method further includes the step of retraining the large-scale language model using the user feedback stored in the aforementioned specified repository, and The above relearning step is, A method for generating answers characterized by a step of inducing the user to learn the preferred style of answer by reflecting the user feedback in the relearning.

13. In Paragraph 1, The above training data is, It includes at least one of a target question-and-answer dataset, a domain glossary containing specialized terms of the aforementioned pre-set domain, a question-and-answer dataset augmented based on at least one document, and a preference dataset. In the above-mentioned search step, A method for generating an answer characterized by mapping the above user query to the above domain glossary and searching for documents related to the above user query.

14. In Paragraph 13, The above target question-and-answer dataset is, It is configured to include at least one of a query, a document search result related to the query, a target answer to the query, and a reference document used to generate the target answer. The above augmented question and answer set is, A method for generating answers characterized by being constructed based on the reference document used to generate a target answer for the above query included in the above target question-answer dataset.

15. In Paragraph 13, The above domain glossary is, It is configured to include at least one of definition information, abbreviation information, and a description related to the use of each of the above technical terms. The above preference data set is, An answer generation method characterized by being configured to include multiple answers of different styles generated for the same query.

16. In Paragraph 15, The above preference data set is, A method for generating answers characterized by inducing the artificial intelligence model to learn the style of a specific answer preferred by the user among a plurality of answers of different styles.

17. In Paragraph 1, If the above extracted specific document includes multiple specific documents, In the above-mentioned large-scale language model (LLM), an answer to the user query is generated based on the plurality of specific documents, and A method for generating an answer characterized by providing information regarding each of the plurality of documents along with the answer to the user terminal.

18. A system comprising memory configured to store executable instructions and one or more processors configured to perform operations by executing one or more instructions, The above system is, Receive a user query from a user terminal, and Using a document search module, documents related to the above user query are searched, and Based on pre-established criteria, at least one specific document is extracted from the above-mentioned searched documents, and Using a large-scale language model (LLM) trained with training data to be optimized for a pre-configured domain, an answer to the user query is generated based on the specific document, and An answer generation system characterized by providing information about the specific document along with the answer to the user terminal.

19. A program that is executed by one or more processes in an electronic device and stored on a computer-readable recording medium, The above program is, A step of receiving a user query from a user terminal; A step of searching for documents related to the above user query using a document search module; A step of extracting at least one specific document from the searched documents based on a pre-established standard; A step of generating an answer to the user query based on the specific document in a large-scale language model (LLM) trained with training data to be optimized for a pre-configured domain; and A program stored on a computer-readable recording medium characterized by including instructions that perform the step of providing information about the specific document along with the answer to the user terminal.

Images