Method and system for generating embedding vector

Abstract

The present disclosure includes a method, performed by at least one processor, for generating an embedding vector associated with a specific domain. The generation method comprises the steps of: acquiring an embedding model pre-trained to generate an embedding vector on the basis of a training document set; receiving a query for a to-be-searched document set associated with a specific domain; calculating search weight coefficients associated with a statistical distribution of the query; and generating, by means of the embedding model, the embedding vector on which the calculated search weight coefficients are reflected.

Description

Method and System for Generating Embedded Vectors

[0001] The present invention relates to a method for generating an embedding vector associated with a specific domain and a system for doing the same.

[0002] In the field of natural language processing technology, semantic search is a search technology that returns relevant results based on semantic similarity between a search query and a document, rather than simply relying on keyword matching.

[0003] To perform semantic-based search, a pre-trained model based on a corpus containing general language data not limited to specific topics can be used, and the pre-trained model can be fine-tuned to fit a specific domain and applied to a specific task.

[0004] However, since the size of the corpus-based pre-trained model is substantial, setting the vast number of parameters requires massive training data and resources, resulting in significant time and cost.

[0005] The information described above disclosed in the background technology of this invention is intended only to enhance understanding of the background of the present invention and may therefore include information that does not constitute prior art.

[0006] The present disclosure provides a method, a system, and a computer program stored on a computer-readable recording medium for generating an embedding vector based on the statistical distribution of queries associated with a specific domain to solve the above-mentioned problems.

[0007] The present disclosure may be implemented in various ways, including a method, an apparatus (system), or a computer program stored on a readable storage medium.

[0008] A method for generating an embedding vector associated with a specific domain, performed by at least one processor according to one embodiment of the present disclosure for solving a technical problem, may include the steps of: obtaining an embedding model that is pre-trained to generate an embedding vector based on a set of training documents; receiving a query for a set of search target documents associated with a specific domain; calculating a search weight coefficient associated with the statistical distribution of the query; and generating an embedding vector reflecting the calculated search weight coefficient through the embedding model.

[0009] According to one embodiment of the present disclosure, the step of calculating a search weight coefficient may include calculating a search weight coefficient based on the total number of search target document sets and the number of search target documents containing words constituting the query.

[0010] According to one embodiment of the present disclosure, the step of calculating a search weight coefficient may include calculating a search weight coefficient based on a value obtained by converting a first ratio of the total number of search target document sets relative to the number of search target documents containing words constituting the query into a log scale, and a value obtained by converting a second ratio of the total number of training document sets relative to the number of training documents containing words constituting the query into a log scale.

[0011] According to one embodiment of the present disclosure, the method for generating an embedding vector may further include the step of calculating a search weight coefficient to lower the importance of the corresponding word constituting the query when the value obtained by converting a first ratio to a log scale is smaller than a preset first threshold and the value obtained by converting a second ratio to a log scale is larger than a preset second threshold.

[0012] According to one embodiment of the present disclosure, the method for generating an embedding vector may further include the step of calculating a search weight coefficient to increase the importance of the corresponding word constituting the query when the value obtained by converting a first ratio to a log scale is greater than a second threshold and the value obtained by converting a second ratio to a log scale is smaller than the first threshold.

[0013] According to one embodiment of the present disclosure, the method for generating an embedding vector may further include the step of calculating a search weight coefficient to lower the importance of the corresponding word constituting the query when the value obtained by converting a first ratio to a log scale is smaller than a first threshold value and the value obtained by subtracting the value obtained by converting a second ratio to a log scale from the value obtained by converting the first ratio to a log scale is greater than a third threshold value.

[0014] According to one embodiment of the present disclosure, the method for generating an embedding vector may further include the step of calculating a search weight coefficient to increase the importance of the corresponding word constituting the query when the value obtained by converting a first ratio to a log scale is greater than a second threshold and the value obtained by subtracting the value obtained by converting the second ratio to a log scale from the value obtained by converting the first ratio to a log scale is greater than a third threshold.

[0015] According to one embodiment of the present disclosure, the method for generating an embedding vector may further include the step of calculating a search weighting coefficient to lower the importance of at least some of the words constituting a query in which the difference between the value obtained by converting a second ratio to a log scale and the value obtained by converting a first ratio to a log scale is greater than or equal to a fourth threshold.

[0016] According to one embodiment of the present disclosure, the step of generating an embedding vector reflecting calculated weight coefficients through an embedding model may include the step of tokenizing a query, the step of replacing each token of the tokenized query with a one-hot vector, and the step of generating an embedding vector by inputting the one-hot vectors multiplied by the calculated weight coefficients into an embedding model.

[0017] According to one embodiment of the present disclosure, the embedding model further includes a Sparse Auto Encoder (SAE), and the step of generating an embedding vector reflecting calculated weight coefficients through the embedding model may include the step of tokenizing a query, the step of inputting the tokenized query into the SAE to calculate a sparse feature vector, the step of masking some tokens among the tokenized query and inputting them into the SAE to specify a location associated with the corresponding token in the sparse feature vector, and the step of generating an embedding vector by multiplying the calculated weight coefficients by the specified location.

[0018] A computer program stored on a computer-readable recording medium may be further provided to execute the embedding vector generation method according to one embodiment of the present disclosure on a computer.

[0019] An information processing system according to one embodiment of the present disclosure comprises a memory and a processor connected to the memory and configured to execute at least one computer-readable program included in the memory, wherein the at least one program may include instructions for obtaining an embedding model that is pre-trained to generate an embedding vector based on a set of documents for training, receiving a query for a set of documents to be searched associated with a specific domain, calculating a search weighting coefficient associated with the statistical distribution of the query, and generating an embedding vector reflecting the calculated weighting coefficient through the embedding model.

[0020] According to various embodiments of the present disclosure, since embedding vectors associated with a specific domain can be effectively generated without performing fine-tuning on a pre-trained, large-scale model, semantic-based search associated with a specific domain can be effectively performed.

[0021] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art to which the present disclosure pertains (referred to as "person skilled in the art") from the description in the claims.

[0022] Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein similar reference numerals indicate similar elements, but are not limited thereto.

[0023] FIG. 1 is a diagram schematically illustrating an embedding vector generation system according to one embodiment of the present disclosure.

[0024] FIG. 2 is a schematic diagram showing a configuration in which an information processing system is connected to communicate with a plurality of user terminals in order to provide an embedding vector generation and semantic-based search service using an embedding model according to one embodiment of the present disclosure.

[0025] FIG. 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to one embodiment of the present disclosure.

[0026] FIG. 4 is a diagram illustrating the calculation of the statistical distribution of a query included in a set of search target documents associated with a specific domain according to one embodiment of the present disclosure.

[0027] FIG. 5 is a diagram illustrating a tokenized query that reflects a calculated search weight coefficient according to one embodiment of the present disclosure.

[0028] FIG. 6 is a diagram illustrating a method for applying calculated search weight coefficients to an embedding model according to one embodiment of the present disclosure.

[0029] FIG. 7 is a sequence diagram illustrating a method for generating an embedding vector associated with a specific domain according to one embodiment of the present disclosure.

[0030] FIG. 8 is a sequence diagram illustrating a method for generating an embedding vector reflecting calculated weighting coefficients according to one embodiment of the present disclosure.

[0031] FIG. 9 is a diagram illustrating a method of applying calculated search weight coefficients to an embedding model according to one embodiment of the present disclosure.

[0032] Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, specific descriptions regarding widely known functions or configurations will be omitted if there is a risk that the gist of the present disclosure may be unnecessarily obscured.

[0033] In the attached drawings, identical or corresponding components are assigned the same reference numerals. Additionally, in the description of the following embodiments, the description of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0034] The advantages and features of the disclosed embodiments and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments provided are merely to make the present disclosure complete and to fully inform those skilled in the art of the scope of the invention.

[0035] The terms used in this specification will be briefly explained, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected to be as generally used as possible, taking into account their functions in this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0036] In this specification, singular expressions include plural expressions unless the context clearly specifies them as singular. Additionally, plural expressions include singular expressions unless the context clearly specifies them as plural. Throughout the specification, when a part is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0037] Additionally, the terms 'module' or 'part' as used in the specification refer to software or hardware components, and the 'module' or 'part' performs certain roles. However, the meaning of 'module' or 'part' is not limited to software or hardware. The 'module' or 'part' may be configured to reside in an addressable storage medium or configured to run on one or more processors. Thus, as an example, the 'module' or 'part' may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. The components and the functions provided within the 'module' or 'part' may be combined into a smaller number of components and 'modules' or 'parts', or further separated into additional components and 'modules' or 'parts'.

[0038] According to one embodiment of the present disclosure, a ‘module’ or ‘part’ may be implemented as a processor and memory. The term ‘processor’ should be broadly interpreted to include a general-purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the term ‘processor’ may refer to an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. The term ‘processor’ may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a DSP core, or any other combination of such configurations. Additionally, the term ‘memory’ should be broadly interpreted to include any electronic component capable of storing electronic information. 'Memory' may refer to various types of processor-readable media, such as Random Access Memory (RAM), Read-Only Memory (ROM), Non-Volatile Random Access Memory (NVRAM), Programmable Read-Only Memory (PROM), Erasable-Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), Flash Memory, Magnetic or Optical Data Storage Devices, Registers, etc. If a processor can read information from memory and / or write information to memory, the memory is said to be in an electronic communication state with the processor. Memory integrated into a processor is in an electronic communication state with the processor.

[0039] In the present disclosure, the 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, the system may be composed of one or more server devices. As another example, the system may be composed of one or more cloud devices. As yet another example, the system may be configured and operated with both a server device and a cloud device.

[0040] In the present disclosure, 'display' may refer to any display device associated with a computing device, for example, any display device capable of displaying any information / data controlled by or provided by the computing device.

[0041] In the present disclosure, 'each of a plurality of A' or 'each of a plurality of A' may refer to each of all components included in a plurality of A, or each of some components included in a plurality of A.

[0042] In the present disclosure, a corpus refers to a collection of texts constructed using a computer for the purpose of language research, or large-scale digitized language data extracted from various sources, and may be in the form of unlabeled large-scale data constructed based on, for example, various dictionaries, books, papers, academic data, etc.

[0043] Various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The size or location of display screens, images, buttons, etc., as illustrated and described in the drawings are exemplary and are not limited thereto. For example, some buttons may be added or omitted, or configured with sizes and locations different from those illustrated. Furthermore, the flowcharts and descriptions illustrated in the drawings are merely examples and may be implemented differently in some embodiments. For example, one or more steps may be omitted, the order of each step may be changed, one or more steps may be performed in overlap, or one or more steps may be performed repeatedly.

[0044] FIG. 1 is a diagram for schematically illustrating an embedding vector generation system (1) according to one embodiment of the present disclosure.

[0045] Referring to FIG. 1, the embedding vector generation system (1) may be a system that generates an embedding vector that converts text data from a high dimension to a low dimension vector space (e.g., a latent space (30)).

[0046] The embedding vector generation system (1) may include a pre-trained embedding model (100). The embedding model (100) may be a model pre-trained to receive a set of training documents (e.g., a Corpus) and position them in a latent space (30). Here, the set of training documents may be a set of text data not limited to a specific topic or domain, but the present disclosure is not limited thereto.

[0047] In the present disclosure, the embedding vector generation system (1) can apply a pre-trained embedding model (100) to a specific domain without fine-tuning. That is, the embedding vector generation system (1) can apply the embedding model (100), which is trained based on a set of training documents, to a specific domain corresponding to a specific untrained task, and can process the specific task without fine-tuning.

[0048] An embedding vector generation system (1) can receive a set of search target documents (10) associated with a specific domain (e.g., a 'finance' domain) and a query (20) associated with the set of search target documents (10). An embedding model (100) can embed the set of search target documents (10) and the query (20) into a latent space (30). The embedding vector generation system (1) can generate an embedding vector based on the statistical distribution of the received query (20) within the set of search target documents (10).

[0049] The embedding vector generation system (1) can determine the similarity between embedding vectors generated through the embedding model (100) (e.g., cosine similarity, similarity based on distance in Euclidean geometry, etc.) and determine one or more documents similar to the query (20).

[0050] In one embodiment, the embedding vector generation system (1) can generate an embedding vector through an embedding model (100) based on the statistical distribution of the received query (20) within the search target document set (10) and / or training document set when calculating the statistical distribution of the received query (20).

[0051] In the present disclosure, the embedding vector generation system (1) is described as generating an embedding vector based on text data, but is not limited thereto, and it is also possible to generate an embedding vector by receiving other types of data.

[0052] FIG. 2 is a schematic diagram showing a configuration in which an information processing system (230) is connected to communicate with a plurality of user terminals (210_1, 210_2, 210_3) to provide an embedding vector generation and semantic-based search service using an embedding model according to one embodiment of the present disclosure.

[0053] Referring to FIG. 2, the information processing system (230) may include system(s) capable of generating a specific domain-based embedding vector. In one embodiment, the information processing system (230) may include one or more server devices and / or databases capable of storing, providing, and executing data and computer-executable programs (e.g., downloadable applications) related to generating a specific domain-based embedding vector, or one or more distributed computing devices and / or distributed databases based on cloud computing services. For example, the information processing system (230) may include separate systems (e.g., servers) for generating embedding vectors.

[0054] The embedding vector generation and semantic-based search service based on the generated embedding vector provided by the information processing system (230) can be provided to the user through applications installed on each of the multiple user terminals (210_1, 210_2, 210_3).

[0055] Multiple user terminals (210_1, 210_2, 210_3) can communicate with an information processing system (230) through a network (220). The network (220) can be configured to enable communication between the multiple user terminals (210_1, 210_2, 210_3) and the information processing system (230). Depending on the installation environment, the network (220) may be configured as a wired network such as Ethernet, Power Line Communication, telephone line communication device and RS-serial communication, a mobile communication network, a Wireless LAN (WLAN), Wi-Fi, Bluetooth and ZigBee, or a combination thereof. The communication method is not limited and may include not only communication methods utilizing communication networks that the network (220) may include (e.g., mobile communication network, wired internet, wireless internet, broadcasting network, satellite network, etc.) but also short-range wireless communication between user terminals (210_1, 210_2, 210_3).

[0056] In FIG. 2, a mobile phone terminal (210_1), a tablet terminal (210_2), and a PC terminal (210_3) are illustrated as examples of user terminals, but are not limited thereto, and the user terminals (210_1, 210_2, 210_3) may be any computing device capable of wired and / or wireless communication and capable of installing and running applications, etc. For example, user terminals may include smartphones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, wearable devices, IoT (Internet of Things) devices, VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc. Additionally, FIG. 2 illustrates three user terminals (210_1, 210_2, 210_3) communicating with an information processing system (230) through a network (220), but is not limited thereto, and may be configured so that a different number of user terminals communicate with an information processing system (230) through a network (220).

[0057] In one embodiment, each of the user terminals (210_1, 210_2, 210_3) can receive information or data from another user terminal or transmit it to another user terminal through the network (220).

[0058] In FIG. 2, the information processing system (230) is shown as an independent device separated from the user terminals (210_1, 210_2, 210_3), but is not limited thereto, and the information processing system (230) may be implemented in an integrated manner with the user terminals (210_1, 210_2, 210_3).

[0059] FIG. 3 is a block diagram showing the internal configuration of a user terminal (210) and an information processing system (230) according to one embodiment of the present disclosure.

[0060] The user terminal (210) may refer to any computing device capable of running applications, web browsers, etc., and capable of wired / wireless communication, and may include, for example, the mobile phone terminal (210_1), tablet terminal (210_2), PC terminal (210_3) of FIG. 2. Referring to FIG. 3, the user terminal (210) may include memory (312), a processor (314), a communication module (316), and an input / output interface (318). Similarly, the information processing system (230) may include memory (332), a processor (334), a communication module (336), and an input / output interface (338). As illustrated in FIG. 3, the user terminal (210) and the information processing system (230) may be configured to communicate information and / or data through a network (220) using their respective communication modules (316, 336). Additionally, the input / output device (320) may be configured to input information and / or data to the user terminal (210) or output information and / or data generated from the user terminal (210) through the input / output interface (318).

[0061] The memory (312, 332) may include any non-transient computer-readable recording medium. According to one embodiment, the memory (312, 332) may include a permanent mass storage device such as ROM (read-only memory), a disk drive, a solid-state drive (SSD), or flash memory. As another example, a permanent mass storage device such as ROM, an SSD, flash memory, or a disk drive may be included in the user terminal (210) or information processing system (230) as a separate permanent storage device distinct from the memory. Additionally, an operating system and at least one program code may be stored in the memory (312, 332).

[0062] These software components may be loaded from a computer-readable recording medium separate from memory (312, 332). This separate computer-readable recording medium may include a recording medium that can be directly connected to the user terminal (210) and the information processing system (230), for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. As another example, the software components may be loaded into memory (312, 332) via a communication module (316, 336) rather than a computer-readable recording medium. For example, at least one program may be loaded into memory (312, 332) based on a computer program installed by files provided through a network (220) by developers or a file distribution system that distributes installation files for the application.

[0063] The processor (314, 334) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (314, 334) by memory (312, 332) or a communication module (316, 336). For example, the processor (314, 334) may be configured to execute instructions received according to program code stored in a recording device such as memory (312, 332).

[0064] The communication module (316, 336) may provide a configuration or function for the user terminal (210) and the information processing system (230) to communicate with each other via the network (220), and may provide a configuration or function for the user terminal (210) and / or the information processing system (230) to communicate with another user terminal or another system (e.g., a separate cloud system). For example, a request or data generated by the processor (314) of the user terminal (210) according to program code stored in a recording device such as memory (312) may be transmitted to the information processing system (230) via the network (220) under the control of the communication module (316). Conversely, a control signal or command provided under the control of the processor (334) of the information processing system (230) may be received by the user terminal (210) via the communication module (316) of the user terminal (210) through the communication module (336) and the network (220).

[0065] The input / output interface (318) may be a means for interfacing with an input / output device (320). As an example, the input device may include a device such as a camera including an audio sensor and / or an image sensor, a keyboard, a microphone, or a mouse, and the output device may include a device such as a display, a speaker, or a haptic feedback device. As another example, the input / output interface (318) may be a means for interfacing with a device in which the configuration or function for performing input and output is integrated into one, such as a touchscreen. For example, when the processor (314) of the user terminal (210) processes instructions of a computer program loaded in memory (312), a service screen configured using information and / or data provided by an information processing system (230) or another user terminal may be displayed on a display through the input / output interface (318). In FIG. 3, the input / output device (320) is depicted as not being included in the user terminal (210), but is not limited thereto and may be configured as a single device with the user terminal (210). Additionally, the input / output interface (338) of the information processing system (230) may be a means for interfacing with a device (not shown) for input or output that is connected to the information processing system (230) or that the information processing system (230) may include. In FIG. 3, the input / output interface (318, 338) is shown as an element configured separately from the processor (314, 334), but is not limited thereto, and the input / output interface (318, 338) may be configured to be included in the processor (314, 334).

[0066] The user terminal (210) and the information processing system (230) may include more components than those of FIG. 3. However, it is not necessary to clearly illustrate most of the prior art components. In one embodiment, the user terminal (210) may be implemented to include at least some of the input / output devices (320) described above. Additionally, the user terminal (210) may further include other components such as a transceiver, a GPS (Global Positioning System) module, a camera, various sensors, a database, etc. For example, if the user terminal (210) is a smartphone, it may include components that are generally included in a smartphone, and may be implemented to include various components such as an accelerometer, a gyroscope, a microphone module, a camera module, various physical buttons, buttons using a touch panel, input / output ports, and a vibrator for vibration.

[0067] While a program for an application including the generation of embedding vectors and a semantic-based search service based on the generated embedding vectors is running, the processor (314) can receive text, images, video, voice and / or actions, etc. that are input or selected through an input device such as a touch screen, keyboard, audio sensor and / or image sensor, camera, microphone, etc. connected to an input / output interface (318), and can store the received text, images, video, voice and / or actions, etc. in memory (312) or provide them to an information processing system (230) through a communication module (316) and a network (220).

[0068] The processor (314) of the user terminal (210) may be configured to manage, process, and / or store information and / or data received from an input / output device (320), another user terminal, an information processing system (230), and / or a plurality of external systems. The information and / or data processed by the processor (314) may be provided to the information processing system (230) through a communication module (316) and a network (220). The processor (314) of the user terminal (210) may transmit information and / or data to the input / output device (320) through an input / output interface (318) and output it. For example, the processor (314) may display the received information and / or data on the screen of the user terminal (210).

[0069] The processor (334) of the information processing system (230) may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals (210) and / or a plurality of external systems. The information and / or data processed by the processor (334) may be provided to the user terminals (210) through a communication module (336) and a network (220). Hereinafter, the processor may refer to the processor of the information processing system or the user terminal.

[0070] FIG. 4 is a diagram illustrating the calculation of the statistical distribution of a query included in a set of search target documents (410) associated with a specific domain according to one embodiment of the present disclosure.

[0071] Referring to FIG. 4, the processor may receive a set of search target documents (410) associated with a specific domain after acquiring a pre-trained embedding model to generate embedding vectors based on a set of training documents. For example, the specific domain may be a domain related to finance, but the present disclosure is not limited thereto.

[0072] The processor may receive a query (420) for a set of search target documents (410) associated with a specific domain. For example, the processor may receive a first query (422, 'What is the fixed deposit interest rate?') and a second query (424, 'What is the credit loan interest rate?'), but the present disclosure is not limited thereto.

[0073] The processor can calculate the statistical distribution of queries (420) for a set of search target documents (410) associated with a specific domain (S432, S434). For example, the processor can calculate the statistical distribution of a first query (422) in a set of search target documents (410) associated with a specific domain (S432) and calculate the statistical distribution of a second query (424) (S434). At this time, the processor can calculate the statistical distribution of the set of search target documents (410) by tokenizing the first query (422) and the second query (424) into units of 'words' or 'syllables'.

[0074] The processor may calculate search weight coefficients when calculating the statistical distribution of queries for a set of search target documents (410) associated with a specific domain. The search weight coefficients may be applied to each token of a tokenized query and may be factors for increasing or decreasing the importance of each token in the tokenized query so that a pre-trained embedding model, which generates embedding vectors according to a general document set, generates embedding vectors tailored to a specific domain. For example, in a general document set, the word 'interest' may be treated as an important word because it has a low statistical frequency, but in a financial domain, the word 'interest' may be treated as a word of low importance because it has a high statistical frequency.

[0075] In one embodiment, the processor may use an Inverse Document Frequency (IDF) value, which is a statistical indicator for evaluating the importance of a specific word in a document set, as a search weighting factor. The processor may calculate the search weighting factor based on [Equation 1] based on the number (TD1) of the entire collection set (D1) of the search target document set and the number of search target documents (DF1) containing a specific word (Qi) that constitutes the query. In one embodiment, 1 may be added to the denominator to prevent the denominator from becoming '0', but the present disclosure is not limited thereto.

[0076] [Formula 1]

[0077] IDF(Qi, D1) = log (TD1 / DF1),

[0078] Here, IDF(Qi, D1) may be a logarithmic scaled value representing the ratio of the total collection set (D1) of the search target documents (TD1) to the number of search target documents (DF1) containing a specific word (Qi) that constitutes the query.

[0079] IDF(Qi, D1) can output a larger value as the number of search target documents (DF1) containing a specific word (Qi) decreases, and a smaller value as the number of search target documents (DF1) containing a specific word (Qi) increases. Accordingly, the processor can determine the importance of a specific word (Qi) as higher as the number of search target documents (DF1) containing the specific word (Qi) decreases within the entire collection set (D1) of the search target documents. As a result, since the importance of a word can be determined in a specific domain, semantic-based search for a specific domain can be performed effectively.

[0080] Below, we will explain how to calculate search weight coefficients by considering both the search target document set and the training document set.

[0081] The processor can calculate search weight coefficients for the search target document set and the training document set used during training of the embedding model. The processor can calculate search weight coefficients based on a logarithmic scale converted value (IDF(Qi, D1)) of the first ratio of the total collection set (D1) of the search target document set to the number of search target documents (DF1) containing a specific word (Qi) constituting the query, and a logarithmic scale converted value (IDF(Qi, D2)) of the second ratio of the total collection set (D2) of the training document set to the number of training documents (DF2) containing a specific word (Qi) constituting the query.

[0082] In the first embodiment, the processor can calculate the search weight coefficient (SWC) based on the first ratio converted to a log scale value (IDF(Qi, D1)) and the second ratio converted to a log scale value (IDF(Qi, D2)) according to [Equation 2] below.

[0083] [Equation 2]

[0084] SWC = IDF(Qi, D1) / IDF(Qi, D2);

[0085] In a second embodiment, the processor may calculate a search weight coefficient to lower the importance of the corresponding word constituting the query when the value obtained by converting the first ratio to a log scale (IDF(Qi, D1)) is smaller than a preset first threshold (TH1) and the value obtained by converting the second ratio to a log scale (IDF(Qi, D2)) is larger than a preset second threshold (TH2). In this case, the first threshold (TH1) is a value smaller than or equal to the second threshold (TH2), and the processor may lower the weight of the corresponding word by multiplying the calculated search weight coefficient by a value less than 1. The value less than 1 may be IDF(Qi, D1) / IDF(Qi, D2), but the present disclosure is not limited thereto.

[0086] Conversely, the processor may calculate a search weighting coefficient to increase the importance of the corresponding word constituting the query when the value of the first ratio converted to a log scale (IDF(Qi, D1)) is greater than the second threshold (TH2) and the value of the second ratio converted to a log scale (IDF(Qi, D2)) is less than the first threshold (TH1). In this case, the weight of the corresponding word may be increased by multiplying by a value greater than 1, but the present disclosure is not limited thereto.

[0087] In the third embodiment, the processor may calculate a search weight coefficient to lower the importance of the word when the value obtained by converting the first ratio to a log scale (IDF(Qi, D1)) is smaller than a preset first threshold (TH1) and the value obtained by subtracting the value obtained by converting the first ratio to a log scale (IDF(Qi, D1)) from the value obtained by converting the second ratio to a log scale (IDF(Qi, D2)) is greater than a third threshold (TH3). At this time, the processor may lower the weight of the word by multiplying the calculated search weight coefficient by a value less than 1. The value less than 1 may be IDF(Qi, D1) / IDF(Qi, D2), but the present disclosure is not limited thereto.

[0088] Conversely, if the value obtained by converting the first ratio to a log scale (IDF(Qi, D1)) is greater than the second threshold (TH2) and the value obtained by subtracting the value obtained by converting the second ratio to a log scale (IDF(Qi, D2)) from the value obtained by converting the first ratio to a log scale (IDF(Qi, D1)) is greater than the third threshold (TH3), the processor may calculate a search weighting coefficient to increase the importance of the corresponding word constituting the query. In this case, the weight of the corresponding word may be increased by multiplying by a value greater than 1, but the present disclosure is not limited thereto.

[0089] In the fourth embodiment, the processor may calculate a search weighting coefficient to lower the importance of at least some of the words constituting a query in which the difference between the second ratio converted to a log scale value (IDF(Qi, D2)) and the first ratio converted to a log scale value (IDF(Qi, D1)) is greater than or equal to the fourth threshold (TH4). In this case, the processor may multiply by a value less than 1 if the difference between the second ratio converted to a log scale value (IDF(Qi, D2)) and the first ratio converted to a log scale value (IDF(Qi, D1)) is greater than 0, and multiply by a value greater than 1 if the difference is less than 0, but the present disclosure is not limited thereto.

[0090] FIGS. 5 and 6 are drawings for explaining the process of generating an embedding vector reflecting the calculated search weight coefficients through an embedding model. FIG. 5 is a drawing (500) for explaining a tokenized query reflecting the calculated search weight coefficients according to one embodiment of the present disclosure.

[0091] Referring to Fig. 5, the processor can generate an embedding vector reflecting the calculated search weight coefficients through an embedding model.

[0092] The processor can tokenize the query (510). Specifically, the processor can divide the query (510) into token units (e.g., words). For example, the processor can generate a first token (522) of the tokenized query and a second token (524) of the tokenized query.

[0093] The processor can replace each token of the tokenized query with a one-hot vector. For example, the processor can replace the first token (522) of the tokenized query with a one-hot vector (532) and the second token (524) of the tokenized query with a one-hot vector (534). A one-hot vector can correspond to a single token using a vector representation method in which only one value is 1 and all other values ​​are 0.

[0094] The processor can generate an embedding vector by inputting the calculated weighting coefficients into the embedding model, multiplying the permuted one-hot vectors. For example, when a tokenized query is found in a set of documents to be searched, the processor can multiply the calculated search weighting coefficients by the 1 values ​​of the one-hot vectors corresponding to each token of the query (e.g., 532A, 534A). Accordingly, the search weighting coefficients can be reflected in the embedding vectors.

[0095] FIG. 6 is a diagram illustrating a method for applying calculated search weight coefficients to an embedding model according to one embodiment of the present disclosure.

[0096] Referring to FIG. 6, the embedding model (600) is a configuration that generates embedding vectors and may include an SAE (AA, Sparse Auto Encoder). The SAE may be a neural network that constrains most of the activation values ​​in the hidden layer of the encoder to be 0.

[0097] The SAE (AA) may include an encoder matrix (620) for the process of receiving an activated vector (610) as input data and mapping it to a latent space. The SAE (AA) may add a bias to the output data of the encoder matrix (620) and apply an activation function (630). The SAE (AA) may include a sparse feature vector (640), and the sparse feature vector (640) may be a feature vector for setting most of the activation values ​​to 0. The SAE (AA) may generate a decoder matrix (650) for restoring the original data and a restored activated vector (660).

[0098] In order for the processor to apply the calculated search weight coefficients to the embedding model (600), the processor may tokenize the query and input the tokenized query into the SAE to produce a sparse feature vector (640). For example, the processor may produce a sparse feature vector (640) associated with a specific tokenized query. The SAE (AA) may perform a decoder matrix (650) to restore the original data to generate a restored activated vector (660).

[0099] Next, the processor can mask a specific token among a query that has already been computed once and input it into the SAE (AA) to identify a location (642, 646) associated with that token in the sparse feature vector. The processor can generate an embedding vector by multiplying the identified location by the calculated weighting factor. That is, the processor can first input the tokenized query into the SAE (AA) to calculate a sparse feature vector, and then mask the specific token (including at least some tokens in the query) among the query and input it into the SAE (AA) to identify the features resulting from the masking in the sparse feature vector (642, 646). That is, the processor can identify a location through the deactivated part of the sparse feature vector (640) obtained by masking the specific token among the activated sparse features in the sparse feature vector of the tokenized query, and the processor can generate a specific domain-specific embedding vector by multiplying the identified location by the search weighting factor.

[0100] FIG. 7 is a sequence diagram illustrating a method (700) for generating an embedding vector associated with a specific domain according to one embodiment of the present disclosure.

[0101] In step S710, the processor can obtain a pre-trained embedding model to generate embedding vectors based on a set of training documents.

[0102] In step S720, the processor can receive a query for a set of search target documents associated with a specific domain.

[0103] In step S730, the processor can calculate search weight coefficients associated with the statistical distribution of the query.

[0104] In step S740, the processor can generate an embedding vector reflecting the calculated search weight coefficients through the embedding model.

[0105] The processor may consider only the target document set, and can calculate search weight coefficients based on the total number of the target document set and the number of target document sets containing words constituting the query. Additionally, the processor may consider both the target document set and the training document set, and a detailed explanation of the above is omitted.

[0106] FIG. 8 is a sequence diagram showing a method (800) for generating an embedding vector reflecting calculated weighting coefficients according to one embodiment of the present disclosure.

[0107] In step S810, the processor can tokenize the query. The processor can divide the query into token units (e.g., words), and the tokenized query may contain multiple tokens.

[0108] In step S820, the processor can replace each token of the tokenized query with a one-hot vector.

[0109] In step S830, the processor can generate an embedding vector by multiplying the substituted one-hot vector by the calculated weighting coefficients and inputting them into the embedding model.

[0110] FIG. 9 is a diagram illustrating a method of applying calculated search weight coefficients to an embedding model according to one embodiment of the present disclosure.

[0111] In step S910, the processor may tokenize the query. Each token of the tokenized query may contain a word, but the present disclosure is not limited thereto.

[0112] In step S920, the processor can input the tokenized query into the SAE to produce a sparse feature vector.

[0113] In step S930, the processor can mask at least some of the tokens in the tokenized query and input them into the SAE to identify the locations associated with those tokens in the sparse feature vector.

[0114] In step S940, the processor can generate an embedding vector by multiplying the calculated weighting coefficients at a specific location.

[0115] The method described above may be provided as a computer program stored on a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Furthermore, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0116] The methods, operations, or techniques of the present disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various exemplary logical blocks, modules, circuits, and algorithmic steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate such interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functional aspects. Whether such functions are implemented in hardware or in software depends on the design requirements imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementations should not be construed as departing from the scope of the present disclosure.

[0117] In a hardware implementation, the processing units used to perform the techniques may be implemented in one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure, computers, or a combination thereof.

[0118] Accordingly, the various exemplary logic blocks, modules, and circuits described in connection with the present disclosure may be implemented or performed by any combination of general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors coupled with a DSP core, or any other combination of configurations.

[0119] In firmware and / or software implementations, techniques may be implemented as instructions stored on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. The instructions may be executable by one or more processors, and the processor(s) may be enabled to perform specific aspects of the functions described in this disclosure.

[0120] Where implemented in software, techniques may be stored on a computer-readable medium as one or more instructions or code, or transmitted through a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available medium accessible by a computer. As a non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium accessible by a computer that can be used to transfer or store desired program code in the form of instructions or data structures. Additionally, any connection is appropriately referred to as a computer-readable medium.

[0121] For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, coaxial cable, fiber optic cable, twisted pair cable, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of a medium. As used herein, disk and disc include CD, laser disc, optical disc, DVD (digital versatile disc), floppy disk, and Blu-ray disc, wherein disks usually play data magnetically, whereas discs play data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0122] The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other known form of storage medium. An exemplary storage medium may be connected to a processor so that the processor can read information from the storage medium or write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may exist within an ASIC. The ASIC may exist within a user terminal. Alternatively, the processor and the storage medium may exist as separate components within the user terminal.

[0123] Although the embodiments described above have been described as utilizing aspects of the subject matter disclosed herein in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or a distributed computing environment. Furthermore, aspects of the subject matter in the present disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across a plurality of devices. Such devices may include PCs, network servers, and portable devices.

[0124] Although the present disclosure has been described in relation to some embodiments, various modifications and changes may be made without departing from the scope of the present disclosure as understood by a person skilled in the art to which the invention of the present disclosure pertains. Furthermore, such modifications and changes should be considered to fall within the scope of the claims appended to this specification.

Claims

1. A method for generating an embedding vector associated with a specific domain, performed by at least one processor, A step of obtaining a pre-trained embedding model to generate embedding vectors based on a set of training documents; A step of receiving a query for a set of search target documents associated with the above-mentioned specific domain; A step of calculating a search weight coefficient associated with the statistical distribution of the above query; and A step of generating an embedding vector reflecting the search weight coefficients calculated above through the embedding model including, Method for generating embedding vectors.

2. In Paragraph 1, The step of calculating the above search weighting coefficient is, A step of calculating the search weight coefficient based on the total number of the set of search target documents and the number of search target documents containing the word constituting the query. including, Method for generating embedding vectors.

3. In Paragraph 1, The step of calculating the above search weighting coefficient is, A step of calculating the search weight coefficient based on a value obtained by converting the first ratio of the total number of the set of search target documents to the number of search target documents containing the word constituting the query into a logarithmic scale, and a value obtained by converting the second ratio of the total number of the set of training documents to the number of training documents containing the word constituting the query into a logarithmic scale. including, Method for generating embedding vectors.

4. In Paragraph 3, A step of calculating the search weight coefficient to lower the importance of the corresponding word constituting the query when the value obtained by converting the first ratio to a log scale is smaller than a preset first threshold and the value obtained by converting the second ratio to a log scale is larger than a preset second threshold. including, Method for generating embedding vectors.

5. In Paragraph 4, If the value obtained by converting the first ratio to a log scale is greater than the second threshold value and the value obtained by converting the second ratio to a log scale is smaller than the first threshold value, the step of calculating the search weight coefficient to increase the importance of the corresponding word constituting the query including, Method for generating embedding vectors.

6. In Paragraph 4, A step of calculating the search weight coefficient to lower the importance of the corresponding word constituting the query when the value obtained by converting the first ratio to a log scale is smaller than the first threshold value, and the value obtained by subtracting the value obtained by converting the first ratio to a log scale from the value obtained by converting the second ratio to a log scale is greater than the third threshold value. including, Method for generating embedding vectors.

7. In Paragraph 4, If the value obtained by converting the first ratio to a log scale is greater than the second threshold value, and the value obtained by subtracting the value obtained by converting the second ratio to a log scale from the value obtained by converting the first ratio to a log scale is greater than the third threshold value, the step of calculating the search weight coefficient to increase the importance of the corresponding word constituting the query. including, Method for generating embedding vectors.

8. In Paragraph 3, A step of calculating the search weighting coefficients to lower the importance of at least some of the words constituting the query, wherein the difference between the value obtained by converting the second ratio to a log scale and the value obtained by converting the first ratio to a log scale is greater than or equal to a fourth threshold. including, Method for generating embedding vectors.

9. In Paragraph 1, The step of generating an embedding vector reflecting the above-calculated weight coefficients through the above-described embedding model is: Step of tokenizing the above query; A step of replacing each token of the above tokenized query with a one-hot vector; and A step of generating an embedding vector by multiplying the above-mentioned substituted one-hot vector by the above-mentioned calculated weight coefficients and inputting them into the above-mentioned embedding model. including, Method for generating embedding vectors.

10. In Paragraph 1, The above embedding model further includes a SAE (Sparse Auto Encoder), and The step of generating an embedding vector reflecting the above-calculated weight coefficients through the above-described embedding model is: Step of tokenizing the above query; A step of inputting the above tokenized query into the above SAE to calculate a sparse feature vector; A step of masking some tokens among the above tokenized queries and inputting them into the above SAE to specify the location associated with the corresponding token in the above sparse feature vector; and A step of generating an embedding vector by multiplying the calculated weight coefficients at the specified location. including, Method for generating embedding vectors.

11. A computer-readable, non-transient recording medium recording instructions for executing the method according to paragraph 1 on a computer.

12. In information processing systems, Memory; and A processor connected to the memory and configured to execute at least one computer-readable program contained in the memory. Includes, The above at least one program is, Obtain a pre-trained embedding model to generate embedding vectors based on a training document set, and Receives a query for a set of search target documents associated with a specific domain, and Calculate search weight coefficients associated with the statistical distribution of the above query, and Instructions for generating an embedding vector reflecting the above-calculated weight coefficients through the above-described embedding model, Information processing system.

Images