Method for generating output data of language model on basis of target information and electronic device

Abstract

Disclosed is a method, performed by at least one processor, for generating output data of a language model based on target information, the method including the steps of: acquiring a first query from a user; generating a first target response data group corresponding to the first query in a data structure including a plurality of similar response data groups classified on the basis of similarity; and acquiring output data for the first query from an LLM-based generation model on the basis of the first query and the first target response data group.

Description

Method for generating output data of a language model based on target information and electronic device

[0001] The present disclosure relates to a method for generating output data of a language model based on target information and an electronic device.

[0002] Recently, in the field of natural language processing technology, techniques are being developed to optimize model performance based on large language models (LMs) to meet user requirements. In addition, as LLM models have become capable of processing multimodal data, there is a growing demand for training tasks that extract specific information from specific images.

[0003] However, since the sentence generation process of the LLM model involves repeatedly selecting the next token, there is a problem in that the probability of outputting incorrect information (hallucination) increases when the model receives a query it does not know. Additionally, there is a problem in that the LLM model cannot provide an accurate answer in the case of information it has not learned (e.g., up-to-date or confidential information).

[0004] On the other hand, most companies intend to use LLM models that derive accurate answers based on their unique internal data. However, if individual companies fine-tune LLM models to obtain their own unique models, there is a difficulty in that tuning LLM models with vast parameters requires massive amounts of training data and resources.

[0005] Accordingly, there is a need for the development of language model-based technologies that accurately generate output data based on target information.

[0006] The present disclosure provides a method for generating output data of a language model based on target information and an electronic device for solving the above-mentioned problems.

[0007] The present disclosure may be implemented in various ways, including a computer-readable non-transient recording medium that records instructions for execution in a method, device (system), and / or computer.

[0008] According to one embodiment of the present disclosure, a method for generating output data of a language model based on target information, performed by at least one processor, may include the steps of: obtaining a first query from a user; generating a first target response data group corresponding to the first query within a data structure comprising a plurality of similar response data groups classified based on similarity; and obtaining output data for the first query from an LLM-based generative model based on the first query and the first target response data group.

[0009] According to one embodiment, a method for generating output data of a language model based on target information further comprises the step of generating the data structure, and the step of generating the data structure may include the step of calculating a similarity between a specific response data included in the data structure and each of the remaining plurality of response data, and the step of generating the similar response data group by comparing the calculated similarity with a predetermined threshold value.

[0010] According to one embodiment, the step of calculating the similarity may include the step of calculating an embedding vector for each of the specific response data and the plurality of response data, and the step of calculating the similarity between the embedding vector for the specific response data and the embedding vector for each of the plurality of response data.

[0011] According to one embodiment, the step of calculating the similarity may include the step of calculating the similarity through the output of the artificial neural network model by inputting the response data pairs into the LLM-based artificial neural network model.

[0012] According to one embodiment, a method for generating output data of a language model based on target information further comprises the step of generating the data structure, wherein the step of generating the data structure may include: generating an embedding vector for each of a plurality of response data included in the data structure; generating one or more reference vectors having the same dimension as the embedding vector; calculating a hash value for each of the plurality of response data based on the embedding vector and the one or more reference vectors; and classifying response data having the same hash value into the same similar response data group based on the calculated hash value.

[0013] According to one embodiment, the step of calculating a hash value for each of the plurality of response data may include: a step of performing a dot product operation between an embedding vector for a specific response data and each of the one or more reference vectors; and a step of generating a hash value having a length equal to the number of the one or more reference vectors according to the result of the dot product operation.

[0014] According to one embodiment, a plurality of similar response data groups included in the data structure can be stored by matching each with corresponding metadata.

[0015] According to one embodiment, the metadata may consist of one or more key keywords included in a specific similar response data group.

[0016] According to one embodiment, the one or more major keywords may be composed of words whose term frequency is greater than or equal to a threshold frequency, calculated by calculating the term frequency within each response data included in the specific similar response data group.

[0017] According to one embodiment, the one or more main keywords may consist of keywords derived by inputting response data included in the specific similar response data group into an LLM-based artificial neural network model.

[0018] According to one embodiment, the data structure further includes a basic response data group, and the step of generating the first target response data group may include: determining a first response data group corresponding to the first query within the basic response data group; determining a candidate similar response data group for the first query based on metadata of each of the plurality of similar response data groups; determining a second response data group corresponding to the first query within the candidate similar response data group; and generating a first target response data group including the first response data group and the second response data group.

[0019] According to one embodiment, a method for generating output data of a language model based on target information may include the steps of: obtaining a second query that follows a first query from a user; generating a new query by concatenating the first query and the second query; generating a second target response data group corresponding to the new query; and obtaining output data for the new query from an LLM-based generative model based on the new query, the first target response data group, and the second target response data group.

[0020] A computer-readable, non-transient recording medium may be provided that records instructions for executing on a computer a method for generating output data of a language model based on target information according to one embodiment of the present disclosure.

[0021] According to one embodiment of the present disclosure, an electronic device comprises a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and the at least one program may include instructions for obtaining a first query from a user, generating a first target response data group corresponding to the first query within a data structure including a plurality of similar response data groups classified based on similarity, and obtaining output data for the first query from an LLM-based generative model based on the first query and the first target response data group.

[0022] According to one embodiment, the at least one program may further include instructions for generating the data structure by calculating the similarity between each of the remaining plurality of response data and a specific response data included in the data structure, and generating the similar response data group by comparing the calculated similarity with a predetermined threshold value.

[0023] According to one embodiment, the at least one program may further include instructions for calculating similarity by calculating an embedding vector for each of the specific response data and the plurality of response data, and calculating a similarity between an embedding vector for the specific response data and an embedding vector for each of the plurality of response data.

[0024] According to one embodiment, the at least one program may further include instructions for generating the data structure by generating an embedding vector for each of the plurality of response data included in the data structure, generating one or more reference vectors having the same dimension as the embedding vector, calculating a hash value for each of the plurality of response data based on the embedding vector and the one or more reference vectors, and classifying response data having the same hash value into the same similar response data group based on the calculated hash value.

[0025] According to one embodiment, the at least one program may further include instructions for obtaining a second query that follows the first query from a user, generating a new query by concatenating the first query and the second query, generating a second target response data group corresponding to the new query, and obtaining output data for the new query from an LLM-based generation model based on the new query, the first target response data group, and the second target response data group.

[0026] According to some embodiments of the present disclosure, even if a number of documents of similar form are registered, such a number of documents are stored through an appropriate data structure, so that an answer based on a language model can be efficiently output for a specific query.

[0027] In addition, according to some embodiments of the present disclosure, even when a subsequent query is obtained for a preceding query, the probability of an inaccurate answer output by the LLM model can be reduced and a more accurate answer can be provided by generating target information to generate more accurate output data within a given context.

[0028] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art to which the present disclosure pertains (referred to as "person skilled in the art") from the description in the claims.

[0029] Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein similar reference numerals indicate similar elements, but are not limited thereto.

[0030] FIG. 1 is a diagram illustrating an exemplary electronic device for generating output data of a language model based on target information according to one embodiment of the present disclosure.

[0031] FIG. 2 is a schematic diagram showing a configuration in which an information processing system is connected to communicate with a plurality of user terminals in relation to data processing according to one embodiment of the present disclosure.

[0032] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to one embodiment of the present disclosure.

[0033] FIG. 4 is a diagram illustrating a method for generating output data of a language model based on target information according to one embodiment of the present disclosure.

[0034] FIGS. 5 and 6 are drawings illustrating response data included in a specific similar response data group according to one embodiment of the present disclosure.

[0035] FIG. 7 is a drawing for explaining a method for generating a data structure according to one embodiment of the present disclosure.

[0036] FIG. 8 is a diagram illustrating, in an exemplary manner, the process of generating a target response data group according to one embodiment of the present disclosure.

[0037] FIG. 9 is a diagram illustrating a method for generating output data for a second query that follows a first query according to one embodiment of the present disclosure.

[0038] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding well-known functions or configurations will be omitted if there is a risk that the gist of the present disclosure may be unnecessarily obscured.

[0039] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0040] The advantages and features of the disclosed embodiments and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments provided are merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention.

[0041] The terms used in this specification will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected to be as generally used as possible, taking into account their functions in this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0042] In this specification, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout the specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0043] Additionally, the terms 'module' or 'part' as used in the specification refer to software or hardware components, and the 'module' or 'part' performs certain roles. However, the meaning of 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside in an addressable storage medium or configured to run on one or more processors. Thus, as an example, the 'module' or 'part' may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and the functions provided within the 'module' or 'part' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0044] According to one embodiment of the present disclosure, a ‘module’ or ‘part’ may be implemented as a processor and memory. The term ‘processor’ should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the term ‘processor’ may refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term ‘processor’ may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such configurations. Additionally, the term ‘memory’ should be broadly interpreted to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as Random Access Memory (RAM), Read-Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable-Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), Flash Memory, Magnetic or Marked Data Storage Devices, Registers, etc. If a processor can read information from memory and / or write information to memory, the memory is said to be in an electronic communication state with the processor. Memory integrated into a processor is in an electronic communication state with the processor.

[0045] In addition, terms such as first, second, A, B, (a), (b), etc. used in the following embodiments are used merely to distinguish one component from another, and the essence, order, or sequence of the said component is not limited by such terms.

[0046] Additionally, in the following embodiments, where it is stated that one component is 'connected', 'coupled', or 'joined' to another component, it should be understood that the component may be directly connected or joined to the other component, but that another component may also be 'connected', 'coupled', or 'joined' between each component.

[0047] Additionally, the terms 'comprises' and / or 'comprising' as used in the following embodiments do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0048] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0049] FIG. 1 is a diagram illustrating an exemplary electronic device (100) for generating output data of a language model based on target information according to one embodiment of the present disclosure. Referring to FIG. 1, the electronic device (100) can generate output data using an LLM-based language model (110) based on a database having a predetermined data structure.

[0050] A Large Language Model (LLM) refers to an artificial neural network model that has been trained on a vast amount of text data in advance. An LLM-based language model (110) is composed of a large neural network structure to perform natural language processing (NLP) tasks and can generate responses to basic queries based on previously trained data.

[0051] The electronic device (100) may include memory and at least one processor. However, the configuration of the electronic device (100) is not limited thereto. According to various embodiments, the electronic device (100) may further include at least one other component in addition to the components described above. For example, the electronic device (100) may further include a communication circuit (or communication module) for communication with an external electronic device.

[0052] A processor may be connected to memory and configured to execute at least one computer-readable program contained in memory. For example, the processor may execute software (or a program) to control at least one other component (e.g., a hardware or software component) of an electronic device (100) connected to the processor and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor may load instructions or data received from other components (e.g., a communication circuit) into volatile memory, process the instructions or data stored in volatile memory, and store the resulting data in non-volatile memory.

[0053] The memory may store various data used by at least one component (e.g., a processor) of the electronic device (100). The data may include, for example, input data or output data for software (or programs) and related instructions. The memory may include volatile memory or non-volatile memory.

[0054] At least one program executed by the processor may include instructions associated with generating output data using a language model. In the following description, the processor is described as performing a certain function, but this is for the convenience of explanation, and the function performed by the processor can be understood as the processor executing instructions included in at least one program stored in memory.

[0055] In FIG. 1, the LLM-based language model (110) is shown as existing outside the electronic device (100), but according to the embodiment, the LLM-based language model (110) may be stored in the memory of the electronic device (100).

[0056] FIG. 2 is a schematic diagram showing a configuration in which an information processing system (230) is connected to communicate with a plurality of user terminals (210_1, 210_2, 210_3) in relation to data processing according to one embodiment of the present disclosure. The information processing system (230) may include system(s) capable of providing a data processing service (e.g., a language model-based service). In one embodiment, the information processing system (230) may include one or more server devices and / or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to the data processing service, or one or more distributed computing devices and / or distributed databases based on cloud computing services. For example, the information processing system (230) may include separate systems (e.g., servers) for the data processing service.

[0057] Data processing services, etc. provided by the information processing system (230) can be provided to the user through a data processing application, a web browser application, etc. installed on each of the multiple user terminals (210_1, 210_2, 210_3).

[0058] Multiple user terminals (210_1, 210_2, 210_3) can communicate with an information processing system (230) through a network (220). The network (220) can be configured to enable communication between the multiple user terminals (210_1, 210_2, 210_3) and the information processing system (230). Depending on the installation environment, the network (220) may be configured as a wired network such as Ethernet, Power Line Communication, telephone line communication devices and RS-serial communication, a mobile communication network, a Wireless LAN (WLAN), Wi-Fi, Bluetooth and ZigBee, or a combination thereof. The communication method is not limited and may include not only communication methods utilizing communication networks that the network (220) may include (e.g., mobile communication network, wired internet, wireless internet, broadcasting network, satellite network, etc.) but also short-range wireless communication between user terminals (210_1, 210_2, 210_3).

[0059] For example, multiple user terminals (210_1, 210_2, 210_3) can transmit a data processing request and a command associated with a user request for data processing to an information processing system (230) through a network (220), and the information processing system (230) can receive this.

[0060] In FIG. 2, a mobile phone terminal (210_1), a tablet terminal (210_2), and a PC terminal (210_3) are illustrated as examples of user terminals, but are not limited thereto. The user terminals (210_1, 210_2, 210_3) may be any computing device capable of wired and / or wireless communication and capable of installing and running data processing applications. For example, user terminals may include smartphones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, wearable devices, IoT (Internet of Things) devices, VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc. Additionally, FIG. 2 illustrates three user terminals (210_1, 210_2, 210_3) communicating with an information processing system (230) through a network (220), but is not limited thereto, and may be configured so that a different number of user terminals communicate with an information processing system (230) through a network (220).

[0061] FIG. 3 is a block diagram showing the internal configuration of a user terminal (210) and an information processing system (230) according to an embodiment of the present disclosure. The user terminal (210) may refer to any computing device capable of executing data processing applications, etc., and capable of wired / wireless communication, and may include, for example, the mobile phone terminal (210_1), tablet terminal (210_2), PC terminal (210_3) of FIG. 2. As illustrated, the user terminal (210) may include a memory (312), a processor (314), a communication module (316), and an input / output interface (318). Similarly, the information processing system (230) may include a memory (332), a processor (334), a communication module (336), and an input / output interface (338). As illustrated in FIG. 3, the user terminal (210) and the information processing system (230) may be configured to communicate information and / or data through the network (220) using their respective communication modules (316, 336). Additionally, the input / output device (320) may be configured to input information and / or data to the user terminal (210) or output information and / or data generated from the user terminal (210) through the input / output interface (318).

[0062] The memory (312, 332) may include any non-transient computer-readable recording medium. According to one embodiment, the memory (312, 332) may include a non-perishable permanent mass storage device such as ROM (read-only memory), a disk drive, an SSD (solid-state drive), or a flash memory. As another example, a non-perishable permanent mass storage device such as ROM, an SSD, a flash memory, or a disk drive may be included in the user terminal (210) or the information processing system (230) as a separate permanent storage device distinct from the memory. Additionally, the memory (312, 332) may store an operating system and at least one program code (e.g., code for an application associated with a data processing service).

[0063] These software components may be loaded from a computer-readable recording medium separate from memory (312, 332). This separate computer-readable recording medium may include a recording medium that can be directly connected to the user terminal (210) and the information processing system (230), for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. As another example, the software components may be loaded into memory (312, 332) via a communication module (316, 336) rather than a computer-readable recording medium. For example, at least one program may be loaded into memory (312, 332) based on a computer program (e.g., an application associated with a data processing service, etc.) that is installed by files provided through a network (220) by developers or a file distribution system that distributes installation files for the application.

[0064] The processor (314, 334) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (314, 334) by memory (312, 332) or a communication module (316, 336). For example, the processor (314, 334) may be configured to execute instructions received according to program code stored in a recording device such as memory (312, 332).

[0065] The communication module (316, 336) may provide a configuration or function for the user terminal (210) and the information processing system (230) to communicate with each other via the network (220), and may provide a configuration or function for the user terminal (210) and / or the information processing system (230) to communicate with another user terminal or another system (e.g., a separate cloud system). For example, a request or data (e.g., a data processing request or data, etc.) generated by the processor (314) of the user terminal (210) according to program code stored in a recording device such as memory (312) may be transmitted to the information processing system (230) via the network (220) under the control of the communication module (316). Conversely, a control signal or command provided under the control of the processor (334) of the information processing system (230) can be received by the user terminal (210) through the communication module (336) and the network (220) via the communication module (316) of the user terminal (210).

[0066] The input / output interface (318) may be a means for interfacing with an input / output device (320). As an example, the input device may include a device such as a camera including an audio sensor and / or an image sensor, a keyboard, a microphone, or a mouse, and the output device may include a device such as a display, a speaker, or a haptic feedback device. As another example, the input / output interface (318) may be a means for interfacing with a device in which the configuration or function for performing input and output is integrated into one, such as a touchscreen. Although the input / output device (320) is depicted in FIG. 3 as not being included in the user terminal (210), it is not limited thereto and may be configured as a single device with the user terminal (210). Additionally, the input / output interface (338) of the information processing system (230) may be a means for interfacing with a device (not shown) for input or output that is connected to the information processing system (230) or that the information processing system (230) may include. In FIG. 3, the input / output interface (318, 338) is shown as an element configured separately from the processor (314, 334), but is not limited thereto, and the input / output interface (318, 338) may be configured to be included in the processor (314, 334).

[0067] The user terminal (210) and the information processing system (230) may include more components than those of FIG. 3. However, it is not necessary to clearly illustrate most of the conventional technical components. In one embodiment, the user terminal (210) may be implemented to include at least some of the input / output devices (320) described above. Additionally, the user terminal (210) may further include other components such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, etc. For example, if the user terminal (210) is a smartphone, it may include components that are generally included in a smartphone, and may be implemented to include various components such as an accelerometer, a gyroscope, a microphone module, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration.

[0068] According to one embodiment, the processor (314) of the user terminal (210) may be configured to operate a data processing application or a web browser application that provides data processing services. At this time, program code associated with the application may be loaded into the memory (312) of the user terminal (210). While the application is operating, the processor (314) of the user terminal (210) may receive information and / or data provided from an input / output device (320) through an input / output interface (318) or receive information and / or data from an information processing system (230) through a communication module (316), and may process the received information and / or data and store it in the memory (312). Additionally, such information and / or data may be provided to the information processing system (230) through the communication module (316).

[0069] While the data processing application is in operation, the processor (314) may receive voice data, text, images, videos, etc., that are input or selected through an input device such as a touch screen, keyboard, audio sensor and / or image sensor, camera, microphone, etc., connected to the input / output interface (318), and may store the received voice data, text, images and / or videos, etc. in memory (312) or provide them to an information processing system (230) through a communication module (316) and a network (220). In one embodiment, the processor (314) may receive user input input through an input device and provide data / requests corresponding to the received user input to an information processing system (230) through a network (220) and a communication module (316).

[0070] The processor (314) of the user terminal (210) can transmit information and / or data to an input / output device (320) through an input / output interface (318) and output it. For example, the processor (314) of the user terminal (210) can output the processed information and / or data through an output device (320), such as a display output device (e.g., touch screen, display, etc.) or a voice output device (e.g., speaker).

[0071] The processor (334) of the information processing system (230) may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals (210) and / or a plurality of external systems. The information and / or data processed by the processor (334) may be provided to the user terminals (210) through a communication module (336) and a network (220).

[0072] FIG. 4 is a diagram illustrating a method for generating output data of a language model based on target information according to one embodiment of the present disclosure.

[0073] Referring to FIG. 4, the electronic device (100) can obtain a first query from a user (S410). In the present disclosure, the query may be text data, and may be text regarding a question for which a response is sought based on target information, such as, for example, "What is the loan amount available for an apartment in Cheonan?", "What is the maximum amount of interim loan for an apartment?".

[0074] The electronic device (100) can generate a first target response data group corresponding to a first query within a data structure that includes a plurality of similar response data groups classified based on similarity (S420). In the present disclosure, response data may be text data provided to produce output data for such a query, and may be a set of text such as a loan-related explanatory document or data regarding an apartment in Cheonan. In the present disclosure, the term "response data" may be used interchangeably with terms to refer to a specific set of text, such as a document or a text chunk, depending on the description. Response data may be data provided along with the query so that an LLM-based artificial neural network model outputs an accurate answer to the query. In other words, in the present disclosure, response data may be a data search space provided along with the query so that an artificial neural network model derives a more accurate output.

[0075] The similar response data group of the present disclosure may be a set of response data comprising one or more response data, and may be a set of response data clustered based on the similarity of each response data. The data structure of the present disclosure may include one or more such similar response data groups, and the structure and method of generating thereof will be described in detail below.

[0076] In the present disclosure, a target response data group may be a set of texts containing one or more response data as information provided to a language model along with a query.

[0077] Next, the electronic device (100) can obtain output data for the first query from an LLM-based generative model based on the first query and the first target response data group (S430). The electronic device (100) of the present disclosure can obtain a more accurate output from the LLM-based generative model by inputting not only the query corresponding to the query but also the target response data group as target information.

[0078] In the present disclosure, the electronic device (100) may perform the task of converting text data into an embedding vector for mathematical operations on text data. Specifically, the electronic device (100) may perform a tokenization task of dividing text data into fixed units and an embedding task of converting individual tokens into vector values ​​that can be recognized and processed by a computer. For example, a Byte Pair Encoding (BPE) technique may be used for the tokenization task. Generally, BPE is a technique that creates a vocabulary by dividing words into character or Unicode units and generating tokens by merging characters or Unicodes that appear consecutively according to their frequency of appearance in the vocabulary. The embedding task is the task of converting each token generated through the tokenization task into an embedding vector, and may be generated by various techniques such as Glove, FastText, Word2Vec, etc.

[0079] According to one embodiment of the present disclosure, an electronic device (100) can generate a data structure including a plurality of similar response data groups.

[0080] FIGS. 5 and 6 are exemplary drawings illustrating response data (hereinafter referred to as "documents") included in a specific similar response data group according to one embodiment of the present disclosure.

[0081] Referring to FIGS. 5 and 6, the first document (510) and the second document (520) are both documents related to an apartment interim payment loan, but they are documents with different details such as interest rates, periods, and applicable subjects.

[0082] According to some embodiments below, the electronic device (100) of the present disclosure can generate a similar response data group by generating an embedding vector for each document (510, 520) and calculating a similarity based on the generated vector.

[0083] In a first embodiment regarding the generation of a data structure including a plurality of similar response data groups, the electronic device (100) can calculate the similarity between a specific response data and each of the remaining plurality of response data and generate a similar response data group based thereon.

[0084] Specifically, the electronic device (100) can calculate similarity using at least one of the following measurement methods: Euclidean distance measurement, Manhattan distance measurement, Herbie distance measurement, Minkowski distance measurement, Mahalanobis distance measurement, cosine similarity measurement, and Jaccard similarity measurement. For example, the calculation of similarity according to the cosine similarity measurement method can be expressed as a mathematical formula as shown in Equation 1 below.

[0085]

[0086] In mathematical formula 1 represents an embedding vector for one of the two response data in a pair, and represents the embedding vector for the remaining response data. Assuming that the embedding vector of Document A is expressed as [1, 1, 1, 1, 1, 1, 1, 0, 0] and the embedding vector of Document B is expressed as [0, 0, 1, 1, 1, 1, 1, 1], the dot product of the two vectors is 5 (= (1·0)+(1·0)+(1·1)+(1·1)+(1·1)+(1·1)+(1·1)+(0·1)+(0·1)), and the magnitude of each of the two vectors is In this case, the cosine similarity of the two vectors can be calculated as 5 / 7.

[0087] The electronic device (100) can generate a similar response data group by comparing the cosine similarity calculated between paired documents with a predetermined threshold (i.e., a similarity threshold). For example, let's assume that the similarity threshold is 0.7, the cosine similarity between response data A and response data B is 0.9, and the cosine similarity between response data A and response data C is 0.5. In this case, the electronic device (100) can determine response data A and response data B, which have a similarity greater than or equal to the similarity threshold, as similar response data. Response data A may have multiple similar response data in addition to response data B, and these similar response data may form a single similar response data group.

[0088] In an additional embodiment, the electronic device (100) can calculate the similarity between a specific response data and each of the remaining multiple response data and generate a similar response data group based thereon.

[0089] Specifically, the electronic device (100) can calculate the similarity of input data pairs by inputting a specific response data included in a data structure and a response data pair among the remaining multiple response data into an LLM-based artificial neural network model. For example, the electronic device (100) can calculate the similarity value of two documents by inputting a query to an LLM-based artificial neural network model along with documents A and B, stating, "Calculate the similarity between the two documents. Respond with 0 if they are completely unrelated documents and 1 if they are identical documents."

[0090] Next, the electronic device (100) can generate a group of similar response data by comparing the calculated similarity value with a predetermined threshold value (i.e., a similarity threshold). For example, let's assume that the similarity threshold is 0.7 and the similarity between document A and document B output by an LLM-based artificial neural network model is 0.9. In this case, the electronic device (100) can determine that document A and document B, which have a similarity greater than or equal to the similarity threshold, are similar response data.

[0091] In a second embodiment regarding the generation of a data structure including a plurality of similar response data groups, the electronic device (100) can generate similar response data groups by calculating a hash value based on similarity for each of the plurality of response data.

[0092] Referring to FIG. 7, the electronic device (100) can generate an embedding vector for each of the plurality of response data included in the data structure (S610). For example, let us assume that document A corresponds to an embedding vector such as [1, 1, 1, 0, 0], document B to [1, 0, 1, 0, 0], and document C to [0, 0, 0, 1, 1].

[0093] Next, the electronic device (100) can generate one or more reference vectors having the same dimensions as the embedding vector (S520). Here, the reference vector may be any vector existing in space having the same dimensions as the embedding vector. For the sake of explanation, let us assume, for example, that there are two reference vectors, and that the first reference vector is [0.5, -0.2, 0.7, 0.1, -0.4] and the second reference vector is [-0.3, 0.6, 0.4, -0.1, 0.2].

[0094] At this time, the electronic device (100) can calculate a hash value for each of the multiple response data based on an embedding vector for each response data and one or more reference vectors (S530). Specifically, the electronic device (100) can perform a dot product operation on each of the embedding vector for a specific response data and one or more reference vectors, and generate a hash value having a length equal to the number of one or more reference vectors according to the result. More specifically, the hash value for a specific response data may be a string having a length equal to the number of reference vectors depending on whether the result of the dot product operation between the embedding vector of the specific response data and the reference vector is positive or negative. At this time, if the result of the dot product operation is positive, 1 may be assigned to the corresponding position of the hash value string, and if it is not positive, 0 may be assigned to the corresponding position of the hash value string.

[0095] For example, the dot product of the embedding vector [1, 1, 1, 0, 0] of document A and the first reference vector [0.5, -0.2, 0.7, 0.1, -0.4] is 1 (= 1*0.5 + 1*(-0.2) + 1*0.7 + 0*0.1 + 0*(-0.4)), which is positive, and the dot product of the embedding vector of document A and the second reference vector [-0.3, 0.6, 0.4, -0.1, 0.2] is 0.7 (= 1*(-0.3) + 1*(0.6) + 1*0.4 + 0*(-0.1) + 0*0.2), which is positive. Therefore, the hash value of document A can be '11'. In addition, the dot product of the embedding vector [1, 0, 1, 0, 0] of document B and the first reference vector [0.5, -0.2, 0.7, 0.1, -0.4] is 1.2 (= 1*0.5 + 0*(-0.2) + 1*0.7 + 0*0.1 + 0*(-0.4)), which is positive, and the dot product of the embedding vector of document B and the second reference vector [-0.3, 0.6, 0.4, -0.1, 0.2] is 0.1 (= 1*(-0.3) + 0*(0.6) + 1*0.4 + 0*(-0.1) + 0*0.2), which is positive. Therefore, the hash value of document B can be '11'. In addition, the dot product of the embedding vector [0, 0, 0, 1, 1] of document C and the first reference vector [0.5, -0.2, 0.7, 0.1, -0.4] is -0.3 (= 0*0.5 + 0*(-0.2) + 0*0.7 + 1*0.1 + 1*(-0.4)), which is negative, and the dot product of the embedding vector of document C and the second reference vector [-0.3, 0.6, 0.4, -0.1, 0.2] is 0.1 (= 0*(-0.3) + 0*(0.6) + 0*0.4 + 1*(-0.1) + 1*0.2), which is positive. In this case, the hash value of document A may be '01'. In other words, the hash value of document A can be determined as '11', the hash value of document B as '11', and the hash value of document C as '01'.

[0096] Next, the electronic device (100) can classify response data having the same hash value into the same similar response data group based on the calculated hash value (S540). That is, documents having the same hash value can be classified as similar documents by belonging to the same similar response data group (or bucket). As described above, the number of reference vectors can determine the maximum number of groups constituting similar documents by determining the length of the hash value, and although two reference vectors were set for the purpose of explanation in this disclosure, this does not limit this disclosure and can be freely set.

[0097] In the data structure according to the present disclosure, each of the plurality of similar response data groups included in the data structure may be stored by matching with corresponding metadata. The metadata may be the folder name in which the documents of the similar response data group are stored. Additionally, the metadata may be recorded and stored in a separate document corresponding to the similar response data group.

[0098] In the present disclosure, metadata may consist of one or more key keywords included in a specific group of similar response data. For example, metadata may be text data such as 'Incheon_Geomdan New City_Geumgang Penterium_4.47_Cheonan_Dujeong Station_Bando Ubora_5.20'.

[0099] In a first embodiment regarding the generation of metadata, the electronic device (100) can generate one or more major keywords by calculating the term frequency within each data in a pair of similar response data pairs and extracting words that exist only in one document and whose term frequency is greater than or equal to a threshold frequency. For the purpose of explanation, let us assume, for example, that the threshold term frequency is 10 times, and the term frequency calculated in document A is equal to {'down payment': 11, 'Cheonan': 12, 'bank': 7}, and the term frequency calculated in document B is equal to {'down payment': 11, 'Dujeong': 13, 'bank': 6}. In this case, among the words in document A, the words that appear at or greater than the threshold term frequency are 'down payment' and 'Cheonan', and when compared with document B, the word that exists only in document A is 'Cheonan', so the electronic device (100) can determine 'Cheonan' as a major keyword regarding document A. Additionally, among the words present in Document B, the words that appear at a threshold frequency are 'down payment' and 'Dujeong', and since the word present only in Document B when compared to Document A is 'Dujeong', the electronic device (100) can determine 'Dujeong' as a major keyword regarding Document B. Accordingly, the electronic device (100) can include 'Cheonan' and 'Dujeong' in the metadata of a similar response data group that includes Document A and Document B.

[0100] Additionally, even for the same document, keywords may be extracted differently depending on the relative document being calculated, and the metadata of the similar response data group may include all keywords extracted differently as main keywords. For example, let us assume that the word occurrence frequency calculated in Document A is {'down payment': 11, 'Cheonan': 12, 'bank': 7} as described above, and the word occurrence frequency calculated in Document C is {'loan': 9, 'apartment': 5, 'penalty': 11}. In this case, the keyword 'down payment' in Document A can be extracted as a main keyword because, when compared with Document C, it does not exist in the keyword set of Document C and appears at a threshold occurrence frequency of 10 or more. Similarly, 'penalty' can be extracted as a main keyword in Document C as described above. Accordingly, the electronic device (100) can additionally include 'down payment' and 'penalty' in the metadata of the similar response data group.

[0101] As a result, the metadata of the similar response data group described in the first embodiment regarding metadata generation may include 'Cheonan', 'Dujeong', 'down payment', and 'penalty' as main keywords.

[0102] In a second embodiment regarding metadata generation, the electronic device (100) can generate key keywords by inputting response data included in a specific similar response data group into an LLM-based artificial neural network model and thereby generate metadata.

[0103] For example, the electronic device (100) can extract key keywords and generate metadata by inputting a query such as “extract keywords that are uniquely present in each document or appear relatively more frequently than other documents” along with documents included in a specific similar response data group.

[0104] FIG. 8 is a diagram illustrating, in an exemplary manner, the process of generating a target response data group according to one embodiment of the present disclosure.

[0105] The electronic device (100) can determine a first response data group corresponding to a first query within a basic response data group included in a data structure (S710).

[0106] In the present disclosure, the basic response data group may be a data group comprising target documents for extracting essentially similar response data for an input query. For example, an electronic device (100) compares the embedding vector of the query with the embedding vector for each document included in the basic response data group and the similarity is given by a threshold value (t d Extract documents that are greater than or equal to ) and form a first response data group (D) composed of the extracted documents default ) can be determined.

[0107] Next, the electronic device (100) can determine a candidate similar response data group for a first query based on the metadata of each of the multiple similar response data groups (S720). Specifically, the electronic device (100) can determine the candidate similar response data group by comparing the embedding vector of the query with the embedding vector of the metadata corresponding to each of the similar response data groups, rather than the embedding vector of the individual document. For example, let us assume that the query is 'How much is the interim payment loan for an apartment in the Cheonan 1-2 redevelopment zone?', there is a first similar response data group having metadata 'Cheonan_Dujeong_Interim_Payment_Loan', and a second similar response data group having metadata 'Gangnam_Daechi_Building_Sale'. In this case, the electronic device (100) can determine the first similar response data group having metadata similar to the query as a candidate similar response data group by comparing the similarity between the embedding vectors.

[0108] Next, the electronic device (100) can determine a second response data group corresponding to the first query within the candidate similar response data group (S730). The electronic device (100) compares the embedding vector of each document included in the candidate similar response data group with the embedding vector of the query to determine a similarity threshold (t b Extract documents that are greater than or equal to ) and a second response data group (D) composed of the extracted documents folder ) can be determined.

[0109] The electronic device (100) is a first response data group (D default ) and the second response data group (D folder A first target response data group including ) can be created (S740). The first target response data group is the union (D) of the first response data group and the second response data group. default UD folder It may be a response data group. The first target response data group is data input to an LLM-based generative model along with the first query, and may be target information for the first query. Through this, the present disclosure has the effect of obtaining more accurate output data for the query.

[0110] FIG. 9 is a diagram illustrating a method for generating output data for a second query that follows a first query according to one embodiment of the present disclosure.

[0111] With reference to FIG. 9, the state in which a first target response data group is created for a preceding first query as described above with reference to FIG. 8 is assumed and explained.

[0112] The electronic device (100) can obtain a second query from the user that follows the first query (S810). The second query may be a question given additionally while maintaining the same context as the first query. For example, if the first query is "How much is available for an interim payment loan for an apartment in the Cheonan 1-2 Redevelopment Zone?", the second query may be text such as "What is the interest rate on the loan?".

[0113] The electronic device (100) can generate a new query by concatenating the first query and the second query (S820). That is, the new query can be text such as "How much is the interim loan for an apartment in the Cheonan 1-2 redevelopment zone? What is the interest rate on the loan?"

[0114] Next, the electronic device (100) can generate a second target response data group corresponding to the new query (S830). The method of generating the second target response data group for the new query can be performed in the same or similar way as the method of generating the first target response data group for the first query. That is, the electronic device (100) generates a third response data group (D') corresponding to the new query. default ) and the fourth response data group (D' folder A second target response data group including ) can be created.

[0115] The electronic device (100) can obtain output data for a new query from an LLM-based generative model based on a new query, a first target response data group, and a second target response data group (S840). Specifically, the electronic device (100) can obtain a document set (D) which is the union of the first target response data group and the second target response data group. default UD folder UD' default UD' folderOutput data for a new query can be generated based on ). In other words, the electronic device (100) can provide both a first target response data group and a second target response data group to an LLM-based generation model to effectively generate answers to questions that additionally appear in the second query while maintaining the context according to the first query for a new query including a first query and a second query.

[0116] In an additional embodiment according to the present disclosure, when a third query is input following a second query, the electronic device (100) may generate input data provided to an LLM-based language model by concatenating the second query and the third query, excluding the first query, to generate a new query again, and by generating a second target response data group and a third target response data group therefrom. This process may be performed repeatedly N (N is a natural number greater than or equal to 1) times or more.

[0117] In conventional methods, if an answer to a second query is generated without considering the context of the first query, an LLM-based artificial neural network model may output a loan interest rate from another document or a general level of loan interest rate in response to a question such as "What is the loan interest rate?" in the example described above. On the other hand, the present disclosure inputs a second query concatenated with the first query as a new query and provides corresponding target information together, thereby enabling the generation of an accurate answer more effectively.

[0118] Furthermore, if multiple subsequent queries are all concatenated, incorrect output data may be obtained when the context changes (e.g., when asking about apartment loan information for a specific region and then asking about apartment loan information for a different region) due to the excessive preservation of the previous context. On the other hand, the present disclosure has the effect of flexibly responding to such changes in context by concatenating only the immediate previous query and the current query.

[0119] In a further embodiment of the present disclosure, the electronic device (100) may receive additional information regarding a user-specified folder from the user. In this case, the electronic device (100) may also extract a similar response data group based on a response data group included in the user-specified folder and utilize it as target information. That is, when the electronic device (100) receives information regarding a user-specified folder, a first response data group (D1) generated based on a basic response data group default ), a second response data group (D) generated based on a similar response data group folder In addition to ), a third response data group (D) generated based on the response data group included in the custom folder. user Target response data group (D) including ) default UD folder UD user Can generate ).

[0120] In an additional embodiment of the present disclosure, if the target response data group calculated for the first query is an empty set, the electronic device (100) may determine that the input first query question is 'Out of Domain'.

[0121] In a further embodiment of the present disclosure, when the target response data group calculated by the first query and the second query is an empty set, the electronic device (100) compares the first query with all documents included in the data structure and the similarity is a threshold value (t b ) The first document set (D) that is greater than or equal to ) all Generate ) and compare the second query with all documents included in the data structure, and the similarity is a threshold value (t b ) The second document set (D') that is greater than or equal to all ) can be generated. At this time, the first document set (D all ) and the second document set (D' all If the union of ) is an empty set, the electronic device (100) may determine that the question consisting of the input first query and the subsequent second query is 'Out of Domain'. In addition, the first document set (D all ) and the second document set (D' all If the union of ) is not an empty set, the electronic device (100) can determine that the question consisting of the input first query and the subsequent second query is ‘Insufficient Information’.

[0122] The above-described flowchart and description are merely examples and may be implemented differently in some embodiments. For instance, in some embodiments, the order of each step may be changed, some steps may be repeated, some steps may be omitted, or some steps may be added.

[0123] The method described above may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Furthermore, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0124] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate such interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functional aspects. Whether such functions are implemented in hardware or in software depends on the design requirements imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0125] In a hardware implementation, the processing units used to perform the techniques may be implemented in one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or a combination thereof.

[0126] Accordingly, the various exemplary logic blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors coupled with a DSP core, or any other combination of configurations.

[0127] In firmware and / or software implementations, techniques may be implemented as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or marked data storage device, etc. The instructions may be executable by one or more processors, and may cause the processor(s) to perform specific aspects of the functions described in this disclosure.

[0128] When implemented in software, the techniques described above may be stored on a computer-readable medium as one or more instructions or code, or transmitted through a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available media accessible by a computer. As a non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium accessible by a computer that can be used to transfer or store desired program code in the form of instructions or data structures. Additionally, any connection is appropriately referred to as a computer-readable medium.

[0129] For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of a medium. As used herein, disks and discs include CDs, laser discs, optical discs, DVDs (digital versatile discs), floppy disks, and Blu-ray discs, wherein disks usually play data magnetically, whereas discs play data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0130] The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other known form of storage medium. An exemplary storage medium may be connected to a processor so that the processor can read information from the storage medium or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may exist within an ASIC. The ASIC may exist within a user terminal. Alternatively, the processor and the storage medium may exist as separate components within the user terminal.

[0131] Although the embodiments described above have been described as utilizing aspects of the subject matter disclosed herein in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or a distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and portable devices.

[0132] Although the present disclosure has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure as understood by a person skilled in the art to which the invention of the present disclosure pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to this specification.

Claims

A method for generating output data of a language model based on target information, performed by at least one processor, A step of obtaining a first query from a user; A step of generating a first target response data group corresponding to the first query within a data structure including a plurality of similar response data groups classified based on similarity; and A step of obtaining output data for the first query from an LLM-based generative model based on the first query and the first target response data group; A method for generating output data of a language model based on target information, including In paragraph 1, It further includes the step of generating the above data structure, and The step of generating the above data structure is, A step of calculating the similarity between a specific response data included in the above data structure and each of the remaining multiple response data; and A step of generating the similar response data group by comparing the calculated similarity with a predetermined threshold; A method for generating output data of a language model based on target information, including In paragraph 2, The step of calculating the above similarity is, A step of calculating an embedding vector for each of the specific response data and the plurality of response data; A step of calculating the similarity between an embedding vector for the specific response data and an embedding vector for each of the plurality of response data; including, Method for generating output data of a language model based on target information. In paragraph 2, The step of calculating the above similarity is, A step of calculating similarity through the output of an artificial neural network model by inputting response data pairs into an LLM-based artificial neural network model; A method for generating output data of a language model based on target information, including In paragraph 1, It further includes the step of generating the above data structure, and The step of generating the above data structure is, A step of generating an embedding vector for each of the plurality of response data included in the above data structure; A step of generating one or more reference vectors having the same dimensions as the above embedding vector; A step of calculating a hash value for each of the plurality of response data based on the embedding vector and the one or more reference vectors; and A step of classifying response data having the same hash value into the same similar response data group based on the calculated hash value; A method for generating output data of a language model based on target information, including In paragraph 5, The step of calculating a hash value for each of the above plurality of response data is, A step of performing a dot product operation between an embedding vector for specific response data and each of the one or more reference vectors; and A step of generating a hash value having a length equal to the number of one or more reference vectors according to the result of the above inner product operation; A method for generating output data of a language model based on target information, including In paragraph 1, A plurality of similar response data groups included in the above data structure are each matched with corresponding metadata and stored, Method for generating output data of a language model based on target information. In Paragraph 7, The above metadata is, Composed of one or more major keywords included in a specific similar response data group, Method for generating output data of a language model based on target information. In paragraph 8, The above one or more major keywords are, Calculate the term frequency within each response data included in the aforementioned specific similar response data group, and the term frequency is composed of words whose calculated term frequency is greater than or equal to a threshold frequency. Method for generating output data of a language model based on target information. In paragraph 8, The above one or more major keywords are, Composed of keywords derived by inputting response data included in the aforementioned specific similar response data group into an LLM-based artificial neural network model, Method for generating output data of a language model based on target information. In paragraph 1, The above data structure further includes a basic response data group, and The step of generating the first target response data group above is, A step of determining a first response data group corresponding to the first query within the above basic response data group; A step of determining a candidate similar response data group for the first query based on the metadata of each of the plurality of similar response data groups; A step of determining a second response data group corresponding to the first query within the above candidate similar response data group; and A step of generating a first target response data group including the first response data group and the second response data group; A method for generating output data of a language model based on target information, including In paragraph 1, A step of obtaining a second query from a user that follows the first query above; A step of generating a new query by concatenating the first query and the second query; A step of generating a second target response data group corresponding to the above new query; A step of obtaining output data for the new query from an LLM-based generative model based on the new query, the first target response data group, and the second target response data group; A method for generating output data of a language model based on target information, including A computer-readable, non-transient recording medium recording instructions for executing the method according to paragraph 1 on a computer. In electronic devices, Memory; and It includes at least one processor connected to the memory and configured to execute at least one computer-readable program contained in the memory, and The above at least one program is, Obtain the first query from the user, and A first target response data group corresponding to the first query is generated within a data structure including a plurality of similar response data groups classified based on similarity, and An electronic device comprising instructions for obtaining output data for the first query from an LLM-based generative model based on the first query and the first target response data group. In Paragraph 14, The above at least one program is, Calculate the similarity between a specific response data included in the above data structure and each of the remaining multiple response data, and By generating the similar response data group by comparing the calculated similarity with a predetermined threshold, An electronic device further comprising instructions for generating the above data structure. In paragraph 15, The above at least one program is, Calculate an embedding vector for each of the above specific response data and the above plurality of response data, and By calculating the similarity between the embedding vector for the specific response data and the embedding vector for each of the plurality of response data, An electronic device further comprising instructions for calculating the above similarity. In Paragraph 14, The above at least one program is, An embedding vector is generated for each of the multiple response data included in the above data structure, and One or more reference vectors having the same dimensions as the above embedding vector are generated, and A hash value for each of the plurality of response data is calculated based on the embedding vector and the one or more reference vectors, and By classifying response data having the same hash value into the same similar response data group based on the calculated hash value, An electronic device further comprising instructions for generating the above data structure. In Paragraph 14, The above at least one program is, Obtain a second query following the first query above from the user, and A new query is generated by concatenating the first query and the second query. A second target response data group corresponding to the above new query is created, and An electronic device further comprising instructions for obtaining output data for the new query from an LLM-based generative model based on the new query, the first target response data group and the second target response data group.

Images