Electronic apparatus for comparing similarity between texts on basis of sub-sentence extraction and method for comparing similarity between texts by using same

Abstract

An electronic apparatus for comparing a similarity between texts on the basis of sub-sentence extraction, according to the present disclosure, may comprise: a memory storing at least one instruction; and at least one processor that executes the at least one instruction. The at least one processor may: perform analysis of dependence between a plurality of morphemes included in at least two pieces of text data; generate a dependent relationship graph for the plurality of morphemes, on the basis of the dependence analysis; extract at least one sub-sentence by using the dependent relationship graph; and calculate a similarity between the at least two pieces of text data by cross-comparing the at least one sub-sentence.

Description

Electronic device for comparing text similarity based on sub-sentence extraction and a method for comparing text similarity using the same

[0001] The present disclosure relates to a technology for determining the similarity of text data, and more specifically, to an apparatus and method for determining text similarity by separating a plurality of text data into morpheme units and extracting grammatical or semantic dependency relationships between the separated morphemes.

[0002] Text similarity comparison technology is one of the important research topics in the field of natural language processing (NLP), and various methodologies have been developed. String-based metrics (LCS algorithms, hash algorithms, etc.) and n-gram-based methods composed of word units or morphemes can be primarily used to compare the structural similarity of text. As a method for evaluating semantic similarity, vectorization techniques such as TF-IDF and Doc2Vec, which are based on word statistics or information content, can be widely utilized. These techniques are disclosed in Japanese Patent Publication No. 2022-029273, “Sentence similarity calculation device, training completion model generation device, and distributed representation model.” Recently, deep learning-based embedding models such as BERT have enabled sentence-level similarity calculation that reflects context, thereby increasing the precision of semantic comparison.

[0003] Existing text similarity comparison methods can be sensitive to text length, structural complexity of sentences, and differences in specific expressions. String-based metrics and n-gram-based methods may underestimate similarity when texts are long or complex. Because TF-IDF and Doc2Vec cannot separate key meanings within a sentence at the word level and rely on the overall statistical characteristics of the text, they may not accurately reflect semantic similarity when the representation style differs. While deep learning-based embedding models can calculate similarity while reflecting context, important information may be diluted or dependency relationships may not be clearly expressed when processing long sentences.

[0004] In text similarity comparisons, it may be necessary to separate text into sub-sentences with independent meanings to accurately analyze long sentences and complex structures. For example, the limitations of existing technologies can be overcome by structurally analyzing text based on inter-morpheme dependency relationships and extracting key meanings through graphical visualization. Therefore, there is a growing need for a technology that preserves core text information through the sub-sentence extraction process and can evaluate high similarity even in long sentences or complex structures through cross-comparison.

[0005] One objective of the present disclosure is to provide an electronic device and a method for comparing text similarity using the same, which overcome the limitations of existing technology by more accurately reflecting semantic similarity between texts.

[0006] Another objective of the present disclosure is to provide an electronic device that provides robust performance even against text modification or reconstruction, and a method for comparing text similarity using the same.

[0007] Another objective of the present disclosure is to provide an electronic device that expands application possibilities by simultaneously providing accuracy and efficiency in text comparison, and a text similarity comparison method using the same.

[0008] However, the problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below.

[0009] An electronic device for comparing text similarity based on sub-sentence extraction according to the present disclosure for achieving the technical problem described above may include a memory for storing at least one instruction and at least one processor for executing said at least one instruction. The at least one processor may perform a dependency analysis between a plurality of morphemes included in at least two text data, generate a dependency relationship graph for said plurality of morphemes based on said dependency analysis, extract at least one sub-sentence using said dependency relationship graph, and calculate the similarity of said at least two text data by cross-comparing said at least one sub-sentence.

[0010] In one embodiment, the at least one processor can remove noise data by preprocessing the at least two text data, separate the at least two text data into morpheme units, analyze the parts of speech of the plurality of morphemes, and extract grammatical or semantic dependency relationships between the plurality of morphemes.

[0011] In one embodiment, the at least one processor can distinguish the relationship between the plurality of morphemes as 'Ancestor' and 'Child', and identify a sentence component including at least one of the subject, predicate, and object of the at least two text data.

[0012] In one embodiment, the at least one processor can generate the dependency relationship graph by configuring nodes and edges based on the dependency relationship information between the morphemes, and can hierarchically arrange the dependency relationships between the plurality of morphemes based on the Root node by exploring the connectivity of the nodes and edges included in the dependency relationship graph.

[0013] In one embodiment, the at least one processor can generate a set of words having semantic association by searching a Root-to-Leaf Path in the dependency relationship graph, and can extract the at least one sub-sentence having independent meaning by combining the set of words.

[0014] In one embodiment, the at least one processor may perform duplicate removal or filtering on the at least one extracted sub-sentence and store the at least one sub-sentence on which duplicate removal or filtering has been performed as a set of sub-sentences.

[0015] In one embodiment, the at least one processor can generate an embedding vector by converting the at least one sub-sentence into a vector, and generate a similarity matrix by calculating the similarity between the generated embedding vectors.

[0016] In one embodiment, the at least one processor can calculate the similarity of the at least two text data by cross-comparing the at least one sub-sentence based on at least one value among the maximum value criterion and the average value in the similarity matrix.

[0017] In one embodiment, the at least one processor can adjust the similarity of the at least two text data by assigning a high weight in proportion to the text length of each of the at least two text data and applying a high correction factor inversely proportional to the difference in text lengths.

[0018] Additionally, the text similarity comparison method according to the present disclosure may include the steps of: performing a dependency analysis between a plurality of morphemes included in at least two text data; generating a dependency relationship graph for the plurality of morphemes based on the dependency analysis; extracting at least one sub-sentence using the dependency relationship graph; and calculating the similarity of the at least two text data by cross-comparing the at least one sub-sentence.

[0019] In addition to this, a computer program stored on a computer-readable recording medium for implementing the present disclosure may be further provided.

[0020] In addition to this, a computer-readable recording medium for recording a computer program for implementing the present disclosure may be further provided.

[0021] According to the aforementioned means for solving the problem of the present disclosure, the electronic device of the present disclosure and the text similarity comparison method using the same can improve the precision of text comparison even in complex sentence structures by separating long texts into sub-sentences. By performing cross-comparison at the sub-sentence level, the electronic device of the present disclosure and the text similarity comparison method using the same can enable stable similarity evaluation that does not depend on text length or structure.

[0022] Furthermore, the electronic device of the present disclosure and the text similarity comparison method using the same can flexibly respond to differences in text representation styles through dependency analysis and a graph-based approach. By identifying key components such as subjects, predicates, and objects, and preserving the core meaning of the text through sub-sentences, the electronic device of the present disclosure and the text similarity comparison method using the same can enable consistent similarity calculation even in specific or summarized expressions.

[0023] Furthermore, the electronic device of the present disclosure and the text similarity comparison method using it can be applied to various text lengths or structures through the flexibility of weight application and similarity calculation methods. The generated sub-sentences can be utilized in various fields such as search, recommendation systems, and content analysis, and are particularly suitable for comparing complex texts such as news articles or legal documents. In other words, the electronic device of the present disclosure and the text similarity comparison method using it can expand the possibilities of application by simultaneously providing accuracy and efficiency in text comparison.

[0024] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0025] FIG. 1 is a drawing showing the block configuration of an electronic device of the present disclosure.

[0026] FIG. 2 is a flowchart illustrating the operation of an electronic device of the present disclosure.

[0027] FIG. 3 is an example of the result of morphological analysis of an electronic device of the present disclosure.

[0028] FIG. 4 is an example of a dependency analysis of an electronic device of the present disclosure.

[0029] FIG. 5 is an example of graph configuration, path extraction, and sentence configuration of the electronic device of the present disclosure.

[0030] FIG. 6 is an example of a sub-sentence extraction result of an electronic device of the present disclosure.

[0031] FIG. 7 is an example of calculating a similarity score of an electronic device of the present disclosure.

[0032] FIG. 8 is a graph comparing similarity scores between sentences of the electronic device of the present disclosure.

[0033] FIG. 9 is an example of a heatmap for a similarity matrix of sub-sentences of the present disclosure.

[0034] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' as used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.

[0035] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0036] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0037] Throughout the specification, when it is stated that a component is located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.

[0038] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0039] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0040] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0041] The operating principles and embodiments of the present disclosure will be described below with reference to the attached drawings.

[0042] In this specification, the term "device according to the present disclosure" includes all various devices capable of performing computational processing and providing results to a user. For example, the device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may be in the form of any one of these.

[0043] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0044] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0045] The above portable terminal may include, for example, all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0046] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0047] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a basic artificial intelligence model is trained using a number of training data by a learning algorithm, thereby creating predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0048] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained by the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0049] According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that enables a machine to learn by mimicking human biological neurons. Methodologies of artificial intelligence can be classified according to the learning method into supervised learning, where input and output data are provided together as training data and the solution (output data) to the problem (input data) is predetermined; unsupervised learning, where only input data is provided without output data and the solution (output data) to the problem (input data) is not predetermined; and reinforcement learning, where a reward is given from an external environment whenever an action is taken from the current state, and learning proceeds in a direction that maximizes such reward. In addition, artificial intelligence methodologies can be classified according to the architecture, which is the structure of the learning model. The architectures of widely used deep learning technologies can be classified into Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Transformers, and Generative Adversarial Networks (GAN).

[0050] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0051] FIG. 1 is a drawing showing the block configuration of the electronic device (100) of the present disclosure, and FIG. 2 is a conceptual diagram (200) showing the operation of the electronic device (100) of the present disclosure.

[0052] Referring to FIG. 1, an electronic device (100) according to one embodiment may include a processor (120), a memory (110), and a communication unit (not shown). The configuration shown in FIG. 1 illustrates a configuration for describing an embodiment according to the present disclosure, and other configurations (e.g., a communication unit) for performing the functions of the electronic device may be added in addition to the illustrated configuration, and the illustrated configurations may be omitted.

[0053] A memory (110) according to one embodiment is a storage medium used by an electronic device (100) and can store data such as at least one instruction or setting information corresponding to at least one program. The program may include an operating system (OS) program and various application programs.

[0054] In one embodiment, the memory (110) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (random access memory, RAM), SRAM (static random access memory), ROM (read only memory, ROM), EEPROM (electrically erasable programmable ROM), PROM (programmable ROM), magnetic memory, a magnetic disk, and an optical disk.

[0055] In one embodiment, the communication unit may include a wireless communication unit (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication unit. The communication unit may communicate with an external electronic device through a network (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a wide-area communication network such as a computer network (e.g., a LAN or WAN).

[0056] In one embodiment, the communication unit may support 5G networks and next-generation communication technologies, for example, new radio access technology. NR access technology may support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low-latency (ultra-reliable and low-latency communications (URLLC)). The communication unit may support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The communication unit may support various technologies for securing performance in the high-frequency band, for example, beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large-scale antenna.

[0057] A processor (120) according to one embodiment can, for example, execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of an electronic device (100) connected to the processor (120) and can perform various data processing or operations.

[0058] In one embodiment, the processor (120) may store commands or data received from another component (e.g., a communication unit) in volatile memory as at least part of data processing or computation, process the commands or data stored in volatile memory, and store the result data in non-volatile memory.

[0059] In one embodiment, the processor (120) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), or a communication processor). For example, the processor (120) may include at least one of a CPU (Central Processing Unit), an AP (Application Processor), or a microprocessor.

[0060] A processor (120) according to one embodiment can perform the operation of an electronic device (100) described below through the drawings. For example, the processor (120) can perform a dependency analysis between a plurality of morphemes included in at least two text data, generate a dependency relationship graph for the plurality of morphemes based on the dependency analysis, extract at least one sub-sentence using the dependency relationship graph, and calculate the similarity of the at least two text data by cross-comparing the at least one sub-sentence.

[0061] In one embodiment, the processor (120) can remove noise data by preprocessing the at least two text data, separate the at least two text data into morpheme units, analyze the parts of speech of the plurality of morphemes, and extract grammatical or semantic dependency relationships between the plurality of morphemes.

[0062] In one embodiment, the processor (120) can distinguish the relationship between the plurality of morphemes as 'Ancestor' and 'Child' and identify a sentence component including at least one of the subject, predicate, and object of the at least two text data.

[0063] In one embodiment, the processor (120) can generate the dependency relationship graph by configuring nodes and edges based on the dependency relationship information between the morphemes, and can hierarchically arrange the dependency relationships between the plurality of morphemes based on the Root node by exploring the connectivity of the nodes and edges included in the dependency relationship graph.

[0064] In one embodiment, the processor (120) can generate a set of words having semantic association by searching the Root-to-Leaf Path in the dependency relationship graph, and can extract at least one sub-sentence having independent meaning by combining the set of words.

[0065] In one embodiment, the processor (120) may perform duplicate removal or filtering on the extracted at least one sub-sentence and store the at least one sub-sentence on which duplicate removal or filtering has been performed as a set of sub-sentences.

[0066] In one embodiment, the processor (120) can generate an embedding vector by converting the at least one sub-sentence into a vector, and generate a similarity matrix by calculating the similarity between the generated embedding vectors.

[0067] In one embodiment, the processor (120) can calculate the similarity of the at least two text data by cross-comparing the at least one sub-sentence based on at least one value among the maximum value and the average value in the similarity matrix.

[0068] In one embodiment, the processor (120) can adjust the similarity of the at least two text data by assigning a high weight in proportion to the text length of each of the at least two text data and applying a high correction factor inversely proportional to the difference in text lengths.

[0069] Based on the operation of the above processor (120), the electronic device of the present disclosure and the text similarity comparison method using the same can increase the precision of text comparison even in complex sentence structures by separating long text into sub-sentences. By performing cross-comparison at the sub-sentence level, the electronic device of the present disclosure and the text similarity comparison method using the same can enable stable similarity evaluation that does not depend on text length or structure.

[0070] In addition, based on the operation of the processor (120), the electronic device of the present disclosure and the text similarity comparison method using the same can flexibly respond to differences in text representation methods through dependency analysis and a graph-based approach. The electronic device of the present disclosure and the text similarity comparison method using the same can identify key components such as subjects, predicates, and objects, and by preserving the core meaning of the text through sub-sentences, consistent similarity calculation can be performed even in specific or summarized expressions.

[0071] In addition, based on the operation of the processor (120), the electronic device of the present disclosure and the text similarity comparison method using it can be applied to various text lengths or structures through the flexibility of the weighting application and similarity calculation method. The generated sub-sentences can be utilized in various fields such as search, recommendation systems, and content analysis, and are particularly suitable for comparing complex texts such as news articles or legal documents. That is, the electronic device of the present disclosure and the text similarity comparison method using it can expand the possibilities of application by simultaneously providing accuracy and efficiency in text comparison.

[0072] FIG. 2 is a flowchart showing the operation of the electronic device (100) of the present disclosure.

[0073] Referring to FIG. 2, the electronic device (100) of the present disclosure may perform a dependency analysis between a plurality of morphemes included in at least two text data (operation 210), generate a dependency relationship graph for the plurality of morphemes based on the dependency analysis (operation 220), extract at least one sub-sentence using the dependency relationship graph (operation 230), and calculate the similarity of the at least two text data by cross-comparing the at least one sub-sentence (operation 240). For example, the electronic device (100) may compare the similarity of the text based on the extraction of sub-sentences by using at least one artificial intelligence model.

[0074] Here, the artificial intelligence model used by the electronic device (100) to compare text similarity based on sub-sentence extraction may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. The artificial intelligence model may be composed of a neural network (or artificial neural network) and may include statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network may refer to a model that possesses problem-solving capabilities by changing the strength of synaptic connections through learning, where artificial neurons (nodes) form a network through the connection of synapses. The neurons of the neural network may include a combination of weights or biases. The neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. The neural network constituting the device can infer a result (output) to be predicted from an arbitrary input by changing the weights of the neurons through learning.

[0075] At least one processor included in the electronic device (100) can generate a neural network, train or learn a neural network, perform operations based on received input data, generate an information signal based on the results of the operation, or retrain the neural network. The neural network models may include, but are not limited to, various types of models such as Convolutional Neural Network (CNN), Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restricted Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory Network (LSTM), and Classification Network, such as GoogleNet, AlexNet, and VGG Network. The processor may include one or more processors for performing operations according to the neural network models. For example, neural networks can include deep neural networks.

[0076] Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Deep Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning Machine), ESN (Echo A person skilled in the art will understand that any neural network may be included, but is not limited to, State Network, Deep Residual Network, Differential Neural Computer, Neural Turing Machine, Capsule Network, Kohonen Network, and Attention Network.

[0077] According to an exemplary embodiment of the present disclosure, the processor comprises a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restricted Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory (LSTM) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4 for Natural Language Processing, Visual Analytics, Visual Understanding, Video Synthesis for Vision Processing, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, Recommendation for ResNet Data Intelligence, Various artificial intelligence structures and algorithms, such as data creation, may be used, but are not limited thereto.

[0078] According to one example, in operation 210, the electronic device (100) can perform dependency analysis between multiple morphemes included in at least two text data. For example, the electronic device (100) can analyze the structural semantics of the text data through the process of operation 210 and provide basic data that can be used in subsequent steps. Operation 210 can play an important role in simplifying the complex structure of long sentences and evaluating semantic similarity between texts. Hereinafter, operation 210 of the electronic device (100) will be described in detail with reference to FIGS. 3 and 4.

[0079] The electronic device (100) can receive at least two text data inputs and perform a preprocessing process. During the preprocessing process, the electronic device (100) can remove unnecessary symbols, spaces, emojis, HTML tags, or other noise data within the input text data. Depending on the linguistic characteristics of the text data, the electronic device (100) can perform case conversion, standardization, or punctuation removal.

[0080] FIG. 3 is an example of a morphological analysis result (300) of an electronic device (100) of the present disclosure.

[0081] Referring to FIG. 3, the electronic device (100) can separate preprocessed text data into morpheme units. The electronic device (100) can use a morpheme analyzer to divide text data into minimum semantic units and analyze the part of speech of each morpheme. For example, the electronic device (100) can morphologically analyze the text data “Today, countries around the world signed an agreement to reduce carbon emissions” and separate it into morpheme units such as “today,” “world,” “countries,” “carbon,” “emission,” “reduction,” “for,” “agreement,” and “signed.”

[0082] The electronic device (100) can extract part-of-speech information of each morpheme through morpheme analysis. The electronic device (100) can identify the grammatical role of each morpheme, such as noun, verb, adjective, and particle. For example, the electronic device (100) can identify “carbon” and “emission” as nouns, and “decrease” and “concluded” as an object and a predicate, respectively.

[0083] FIG. 4 is an example of a dependency analysis (400) of an electronic device (100) of the present disclosure.

[0084] Referring to FIG. 4, the electronic device (100) can extract grammatical or semantic dependency relationships between morphemes. The electronic device (100) can analyze the relationships between morphemes to determine the semantic influence that a specific morpheme has on another morpheme. The electronic device (100) can distinguish the relationships between morphemes as 'Ancestor' and 'Child'. For example, the electronic device (100) can analyze the relationship between “each country” and “concluded” as a “subject (nsubj)”, and the relationship between “agreement” and “concluded” as an “object (obj)”.

[0085] The electronic device (100) can identify major grammatical relationships based on the components of a sentence. The electronic device (100) can analyze sentence components including at least one of a subject, a predicate, and an object, and store the relationships between these elements as data. For example, the electronic device (100) can identify “each country” as the subject, “concluded” as the predicate, and “agreement” as the object.

[0086] The electronic device (100) can define the directionality of the relationship between morphemes. The electronic device (100) can determine whether a specific morpheme plays a central role in a sentence by setting the direction of the dependency relationship. Through this, the electronic device (100) can more clearly understand the structural intent of the sentence.

[0087] The electronic device (100) can store the results of dependency analysis in a hierarchical data structure. The electronic device (100) can convert the analyzed morphemes and their relationships into data in the form of a tree or graph, and prepare them for use in subsequent processes.

[0088] According to one example, in operation 220, the electronic device (100) can generate a dependency relationship graph for the plurality of morphemes based on the dependency analysis. For example, the electronic device (100) can systematically represent the grammatical and semantic relationships of text data through the process of operation 220 and provide a basis that can be utilized in subsequent steps of sub-sentence extraction and similarity calculation. Through the graph generation process, the electronic device (100) can structurally simplify the complex relationships of long sentences and effectively compare semantic similarities between texts. Hereinafter, operation 220 of the electronic device (100) will be described in detail with reference to FIG. 5.

[0089] The electronic device (100) can generate a dependency relationship graph based on information regarding dependency relationships between morphemes. The electronic device (100) receives the results of morpheme analysis and dependency analysis as input and can define each morpheme as a node of the graph. The electronic device (100) can represent the dependency relationships between morphemes as edges and give directionality to the edges to clearly indicate the semantic flow of the relationship.

[0090] The electronic device (100) can determine the connectivity of each morpheme according to dependency relationship information during the graph construction process. For example, the electronic device (100) can create connections between nodes by setting "today → each country" as a "compound" relationship and "each country → concluded" as an "nsubj" relationship. The electronic device (100) can set the type of relationship as a tag so that the meaning represented by each edge can be intuitively understood.

[0091] The electronic device (100) can explore the connectivity of each node and edge within the generated graph. The electronic device (100) can form a structural hierarchy of sentences by arranging the graph hierarchically based on the ROOT node. For example, the electronic device (100) can set "concluded" as the ROOT and construct a hierarchical structure by connecting sub-nodes such as "each country," "agreement," and "carbon." The electronic device (100) can visually represent the semantic flow and dependency relationships of sentences through the hierarchical arrangement.

[0092] FIG. 5 is an example of graph configuration, path extraction, and sentence configuration (500) of the electronic device (100) of the present disclosure.

[0093] Referring to FIG. 5, the electronic device (100) can set the graph configuration method to a tree form or extend it into an acyclic graph that allows multiple paths. The electronic device (100) can concisely express the relationships between key components within a sentence through the tree form. Additionally, the electronic device (100) can accurately reflect cases where the same word is involved in multiple dependency relationships through a graph that includes multiple paths.

[0094] The electronic device (100) can highlight semantically important nodes and edges within the graph. The electronic device (100) can be configured to give high weight to specific relationships (e.g., relationships related to subject, predicate, and object) so that those relationships are more clearly revealed in the graph. For example, the electronic device (100) can give priority to the subject relationship between "each country" and "concluded".

[0095] The electronic device (100) can correct or supplement incomplete paths while exploring the connectivity of the graph. The electronic device (100) can improve the accuracy and completeness of the graph by inferring missing relationships or removing incomplete nodes. Through this, the electronic device (100) can lay the foundation for accurately extracting sub-sentences in a subsequent step.

[0096] The electronic device (100) can store the generated dependency relationship graph. The electronic device (100) can store the graph in a database or memory so that it can be utilized later in the sub-sentence extraction or text similarity calculation steps. The electronic device (100) can convert the stored graph into a suitable data structure so that it can be visualized or utilized in other processes.

[0097] According to one example, in operation 230, the electronic device (100) can extract at least one sub-sentence using the dependency relationship graph. For example, the electronic device (100) can simplify long text data through the sub-sentence extraction process performed in operation 230 and provide a basis for independently analyzing semantically important sentences. The electronic device (100) can provide an efficient data structure for comparison between texts and reduce the structural complexity of the text through the sub-sentence extraction process. Hereinafter, operation 230 of the electronic device (100) will be described in detail with reference to FIG. 6.

[0098] The electronic device (100) can generate a set of words with semantic associations by traversing a Root-to-Leaf Path in a dependency relationship graph. The electronic device (100) can traverse a path starting from the Root node of the graph to each Leaf node and collect all nodes (morphemes) included in the traversed path. For example, the electronic device (100) can generate a set of words such as "carbon," "emission," "reduction," "for," "agreement," and "concluded" by traversing the path "ROOT → carbon → emission → reduction → concluded."

[0099] FIG. 6 is an example of a sub-sentence extraction result (600) of an electronic device (100) of the present disclosure.

[0100] Referring to FIG. 6, the electronic device (100) can generate a sub-sentence having independent meaning by combining a set of words collected from a searched path. The electronic device (100) can generate a grammatically complete sentence by maintaining the order of words along the path of the graph and adding particles or conjunctions as needed. For example, the electronic device (100) can generate a sub-sentence "concluded an agreement for carbon emission reduction" by combining the set of words "carbon," "emission," "reduction," "for," "agreement," and "concluded."

[0101] The electronic device (100) can be configured so that the generated sub-sentence contains contextually important information. The electronic device (100) can search for a path that prioritizes the inclusion of key sentence components (e.g., subject, predicate, object) based on a hierarchical structure of dependencies. For example, the electronic device (100) can generate a sub-sentence "today, each country signed" by combining "today," "each country," and "signed" in the path "ROOT → today → each country → signed".

[0102] The electronic device (100) can remove duplicate or unnecessary paths during the search process. The electronic device (100) can filter or merge duplicate sub-sentences when similar sets of words are generated in multiple paths that have the same meaning. For example, the electronic device (100) can compare similar sentences, such as "concluded an agreement for carbon emission reduction" and "concluded a carbon emission agreement," and select a more concise or key sentence.

[0103] The electronic device (100) can refine the generated sub-sentences and store them as a set of sub-sentences. The electronic device (100) can check for grammatical completeness and set the priority of the sub-sentences according to specific criteria. For example, the electronic device (100) can store sentences such as "concluded an agreement to reduce carbon emissions" and "countries concluded it today" respectively, and use them in the similarity evaluation stage.

[0104] The electronic device (100) may store metadata of the sub-sentences together with the sub-sentences during the process of storing a set of sub-sentences. The electronic device (100) may include the source path of each sub-sentence, location information within the text, and the importance evaluation score of the sub-sentences as metadata. This process may be useful for considering contextual information of the sub-sentences in a subsequent similarity calculation step.

[0105] According to one example, in operation 240, the electronic device (100) can calculate the similarity of the at least two text data by cross-comparing the at least one sub-sentence. For example, the electronic device (100) can receive a set of sub-sentences generated in operation 230 and convert each sub-sentence into an embedding vector. The electronic device (100) can numerically express the meaning of each sub-sentence by utilizing a sentence embedding model (e.g., Sentence-BERT). The electronic device (100) can quantitatively evaluate the similarity between sub-sentences based on the converted embedding vectors. Hereinafter, operation 240 of the electronic device (100) will be described in detail with reference to FIGS. 7 to 9.

[0106] The electronic device (100) can generate an embedding vector by converting at least one sub-sentence into a vector. The electronic device (100) can utilize a sentence embedding model (e.g., Sentence-BERT, Universal Sentence Encoder, etc.) to numerically represent the semantic characteristics of the sub-sentence. The electronic device (100) can receive each sub-sentence as input and generate an embedding vector representing the position of the corresponding sentence in a high-dimensional vector space. For example, the electronic device (100) can analyze the sub-sentence "signed an agreement to reduce carbon emissions" and generate an embedding vector such as [0.123, -0.456, 0.789, ...].

[0107] FIG. 7 is an example of a similarity score calculation (700) of an electronic device (100) of the present disclosure.

[0108] Referring to FIG. 7, the electronic device (100) can generate a similarity matrix by calculating the similarity between generated embedding vectors. The electronic device (100) can evaluate the similarity between sub-sentences using algorithms such as cosine similarity, Euclidean distance, or dot product. The electronic device (100) can generate a matrix that quantitatively expresses the similarity of each sub-sentence by calculating the similarity between a set of n sub-sentences and m combinations (nxm). For example, the electronic device (100) can generate a similarity matrix of size 3 x 4 by comparing three sub-sentences of text data 1 with four sub-sentences of text data 2.

[0109] FIG. 8 is a diagram (800) that cross-compars similarity scores between sub-sentences of the electronic device (100) of the present disclosure.

[0110] Referring to FIG. 8, the electronic device (100) can calculate the final similarity between two text data based on a generated similarity matrix. The electronic device (100) can calculate the similarity by applying a maximum value criterion or an average value criterion to the similarity matrix. The maximum value criterion is a method of highlighting the most similar parts between the two text data and may be suitable when a specific sentence has high relevance. The average value criterion is a method of evaluating the overall similarity of the two text data and may be suitable for calculating the consistent similarity of the entire document. For example, if the maximum value in the similarity matrix is ​​0.92 and the average value is 0.49, the electronic device (100) can calculate the final similarity as 0.71 by combining the two criteria.

[0111] The electronic device (100) can adjust similarity by applying weights during the similarity calculation process. The electronic device (100) can assign higher weights to specific sub-sentences by considering the length, structure, or contextual importance of the text data. For example, the electronic device (100) can be configured to apply weights to sub-sentences close to the ROOT or sub-sentences containing key keywords so that those sentences have a greater influence on the final similarity calculation. The electronic device (100) can also reduce similarity distortion that may occur in comparison between long and short documents by applying a correction factor inversely proportional to the difference in text length.

[0112] FIG. 9 is an example of a heatmap (900) for a similarity matrix of sub-sentences of the present disclosure.

[0113] Referring to FIG. 9, the electronic device (100) can visually represent the results of a similarity calculation. The electronic device (100) can convert the similarity matrix into a heatmap form to represent the similarity between each sub-sentence as color intensity. Through the visualized results, the electronic device (100) can clearly identify pairs of sentences that have a high semantic association between text data. For example, the electronic device (100) can confirm from the heatmap that "concluded an agreement to reduce carbon emissions" and "concluded an agreement to reduce carbon emissions" have a high similarity.

[0114] The electronic device (100) can identify key and important associations even in long text data and can utilize the generated similarity results in various applications such as search, recommendation, and content analysis. The electronic device (100) can function as a powerful tool capable of overcoming structural differences in text data and evaluating semantic consistency based on the similarity calculation results.

[0115] In this way, the electronic device (100) of the present disclosure and the text similarity comparison method using the same can increase the precision of text comparison even in complex sentence structures by separating long text into sub-sentences. By performing cross-comparison at the sub-sentence level, the electronic device (100) of the present disclosure and the text similarity comparison method using the same can enable stable similarity evaluation that does not depend on text length or structure.

[0116] In addition, the electronic device (100) of the present disclosure and the text similarity comparison method using the same can flexibly respond to differences in text expression methods through dependency analysis and a graph-based approach. The electronic device (100) of the present disclosure and the text similarity comparison method using the same can identify key components such as subjects, predicates, and objects, and by preserving the core meaning of the text through sub-sentences, consistent similarity calculation can be performed even in specific or summarized expressions.

[0117] In addition, the electronic device (100) of the present disclosure and the text similarity comparison method using it can be applied to various text lengths or structures through the flexibility of the weighting application and similarity calculation method. The generated sub-sentences can be utilized in various fields such as search, recommendation systems, and content analysis, and are particularly suitable for comparing complex texts such as news articles or legal documents. That is, the electronic device (100) of the present disclosure and the text similarity comparison method using it can expand the possibilities of application by simultaneously providing accuracy and efficiency in text comparison.

[0118] However, as this has been explained above, a redundant explanation thereof will be omitted.

[0119] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0120] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0121] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below are also within the scope of the claims.

Claims

1. In an electronic device for comparing text similarity based on sub-sentence extraction, Memory for storing at least one instruction; and At least one processor executing the above at least one instruction; comprising, The above-mentioned at least one processor is, Perform dependency analysis between multiple morphemes included in at least two text data, and Based on the above dependency analysis, a dependency relationship graph for the above plurality of morphemes is generated, and Using the above dependency relationship graph, at least one sub-sentence is extracted, and Calculating the similarity of the at least two text data by cross-comparing the at least one sub-sentence, Electronic device.

2. In Paragraph 1, The above-mentioned at least one processor is, By preprocessing the above at least two text data, noise data is removed, and The above at least two text data are separated into morpheme units, and Analyze the parts of speech of the above multiple morphemes, and Extracting grammatical or semantic dependency relationships between the above multiple morphemes, Electronic device.

3. In Paragraph 1, The above-mentioned at least one processor is, The relationship between the above multiple morphemes is classified as 'Ancestor' and 'Child', and Identifying a sentence component comprising at least one of the subject, predicate, and object of the above at least two text data, Electronic device.

4. In Paragraph 1, The above-mentioned at least one processor is, By configuring nodes and edges based on the above-mentioned inter-morpheme dependency information, the above-mentioned dependency relationship graph is generated, and By exploring the connectivity of the nodes and edges included in the dependency relationship graph, the dependency relationships between the plurality of morphemes are arranged hierarchically based on the Root node. Electronic device.

5. In Paragraph 1, The above-mentioned at least one processor is, By traversing the Root-to-Leaf Path in the above dependency graph, a set of words with semantic associations is generated, and Extracting at least one sub-sentence having an independent meaning by combining the above set of words, Electronic device.

6. In Paragraph 5, The above-mentioned at least one processor is, Among the above at least two text data, n first sub-sentences are extracted by traversing a Root-to-Leaf Path in a first dependency relationship graph for the first text data (where n is a natural number greater than or equal to 1), and m second sub-sentences are extracted by traversing a Root-to-Leaf Path in a second dependency relationship graph for the second text data (where m is a natural number greater than or equal to 1), Electronic device.

7. In Paragraph 1, The above-mentioned at least one processor is, Duplicate removal or filtering is performed on the extracted at least one sub-sentence, and Storing the at least one sub-sentence, on which duplicate removal or filtering has been performed, as a sub-sentence set, Electronic device.

8. In Paragraph 1, The above-mentioned at least one processor is, By converting the above at least one sub-sentence into a vector, an embedding vector is generated, and By calculating the similarity between the generated embedding vectors, a similarity matrix is ​​generated. Electronic device.

9. In Paragraph 8, The above-mentioned at least one processor is, Calculating the similarity of the at least two text data by cross-comparing the at least one sub-sentence based on at least one value among the maximum value criterion and the average value in the similarity matrix, Electronic device.

10. In paragraph 8, the above at least one sub-sentence is, Among the above at least two text data, it includes n first sub-sentences (where n is a natural number greater than or equal to 1) extracted by traversing a Root-to-Leaf Path in a first dependency relationship graph for the first text data, and m second sub-sentences (where m is a natural number greater than or equal to 1) extracted by traversing a Root-to-Leaf Path in a second dependency relationship graph for the second text data, The above-mentioned at least one processor is, Generating an n*m similarity matrix based on embedding vectors generated for each of the n first sub-sentences and the m second sub-sentences, Electronic device.

11. In Paragraph 10, The above-mentioned at least one processor is, When generating the above n*m ​​size similarity matrix, a first weight set in advance is applied to a first sub-sentence adjacent to the Root in the first dependency relationship graph and a second sub-sentence adjacent to the Root in the second dependency relationship graph, and a second weight set in advance is applied to a first sub-sentence and a second sub-sentence containing a core keyword. Electronic device.

12. In Paragraph 11, The above-mentioned at least one processor is, For each of the n*m ​​combinations of the n first sub-sentences and the m second sub-sentences, a correction factor inversely proportional to the difference in length between sentences is applied. Electronic device.

13. In Paragraph 1, The above-mentioned at least one processor is, Adjusting the similarity of the at least two text data by assigning a high weight proportional to the text length of each of the at least two text data and applying a high correction coefficient inversely proportional to the difference in text lengths. Electronic device.

14. A method performed by a processor of an electronic device, A step of performing dependency analysis between multiple morphemes included in at least two text data; A step of generating a dependency relationship graph for the plurality of morphemes based on the above dependency analysis; A step of extracting at least one sub-sentence using the above dependency relationship graph; and A step of calculating the similarity of the at least two text data by cross-comparing the at least one sub-sentence; comprising Text similarity comparison method.

Images