Method, device, and computer-readable recording medium for segmenting document into semantic units on basis of semantic chunking
A natural language processing method segments documents into semantic units based on semantic similarity, addressing contextual discontinuity and redundancy, improving document understanding and search accuracy.
Patent Information
- Application Number
- PCT/KR2026/096690
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-07
- Filing Date
- 2026-05-07
- Publication Date
- 2026-11-12
AI Technical Summary
Conventional document segmentation methods fail to reflect actual topic flow or semantic turning points, leading to contextual discontinuity, redundant information, and distortion of semantic units, making it difficult to utilize documents in intelligent systems for search, summarization, and question answering.
A natural language processing-based method that segments sentences based on semantic similarity, clusters sentences with the same topic, adjusts paragraph length, and calculates comprehensive vector values for each semantic paragraph, allowing users to select relevant paragraphs for queries.
Improves document understanding and processing accuracy by structuring documents based on semantic criteria, enhancing context preservation, search accuracy, and logical connectivity in information systems.
Smart Images

Figure KR2026096690_12112026_PF_FP_ABST
Abstract
Description
Method, device, and computer-readable recording medium for splitting documents into semantic units based on semantic chunking
[0001] The present invention relates to a method for dividing documents into semantic units based on semantic chunking. Specifically, it relates to a natural language processing-based semantic unit document segmentation technology configured to divide sentences included in a digital document based on semantics, generate semantic paragraphs by clustering sentences that share the same topic through an analysis of the semantic similarity of each sentence, perform correction processing such as length correction between paragraphs and inclusion of duplicate sentences to preserve context, calculate a comprehensive vector value for each of the thus generated corrected semantic paragraphs and store it in a database, and subsequently allow the user to select and refer to relevant paragraphs based on the corresponding vector value when providing a response to a user's query.
[0002]
[0003] [National R&D projects that supported this invention]
[0004] [Project ID] 2710006727
[0005] [Project No.] RS-2024-00337489
[0006] [Ministry Name] Ministry of Science and ICT
[0007] [Project Management (Specialized) Agency Name] Korea Institute of Information & Communications Technology Planning & Evaluation
[0008] [Research Project Name] Development of Core Technologies for the SW Computing Industry
[0009] [Research Project Title] Data Drift Management to Overcome Analysis Model Performance Degradation
[0010] Technology development
[0011] [Name of Project Performing Organization] Daiquest Co., Ltd.
[0012] [Research Period] April 1, 2024 ~ December 31, 2027
[0013] Conventionally, techniques have primarily been used to physically divide the text constituting a document based on specific keywords, punctuation marks, or statistical word distributions, or to split documents into paragraphs using simple rule-based segmentation methods or machine learning-based classification models. However, these methods have limitations, as they fail to reflect the actual topic flow or semantic turning points of the document and cause problems such as contextual discontinuity, the generation of redundant information, and distortion of semantic units. Furthermore, conventional techniques fail to structurally define reference relationships or semantic connectivity between divided paragraphs, making it difficult to utilize documents in intelligent systems for search, summarization, and question answering.
[0014] Accordingly, the industry is developing various technologies to solve these problems.
[0015] As an example, Korean registered patent 10-1938212 (Subject-based automatic document classification system considering meaning and context) discloses a technology that analyzes morphemes from various document sets to filter stop words, transforms documents into vectors and extracts features, and automatically classifies documents based on a recurrent neural network.
[0016] However, the aforementioned prior art discloses only a technology that extracts similar documents by considering the meaning and context of words included in a document and classifies them by topic. It does not disclose a natural language processing-based semantic unit document segmentation technology that divides sentences included in digital documents based on meaning, generates semantic paragraphs by clustering sentences sharing the same topic through analysis of the semantic similarity of each sentence, performs correction processing such as length correction between paragraphs and inclusion of duplicate sentences to preserve context, calculates a comprehensive vector value for each of the thus generated corrected semantic paragraphs and stores it in a database, and subsequently configures the system to allow the user to select and refer to relevant paragraphs based on the corresponding vector value when providing a response to a user query. Consequently, there is a growing need for a technology capable of addressing this issue.
[0017] Accordingly, the present invention is designed to solve the problems of the aforementioned existing technologies. It generates semantic paragraphs by segmenting sentences included in digital documents based on meaning and clustering sentences that share the same topic by analyzing the semantic similarity of each sentence. It then performs correction processing, such as adjusting the length between paragraphs and including duplicate sentences to preserve context. Finally, it calculates a comprehensive vector value for each of the generated corrected semantic paragraphs and stores it in a database. Subsequently, when providing a response to a user's query, it is configured to allow the user to select and refer to relevant paragraphs based on the corresponding vector value. This configuration makes it applicable not only to news and blogs but also to unstructured documents such as technical documents, manuals, reports, and theses, and aims to enable high-speed matching of highly relevant paragraphs in a question-and-answer system.
[0018] A method for dividing documents into semantic units based on semantic chunking, implemented by a computing device comprising one or more processors and one or more memories for storing instructions executable by said processors according to an embodiment of the present invention, wherein when a division process for a plurality of document information stored in a storage is started, a sentence corresponding to each of said plurality of document information is analyzed, said sentences are divided according to a predetermined division criterion, and a similarity analysis process is started for said sentences divided according to said predetermined division criterion; when said similarity analysis process is started, a composite vector value for each of said divided sentences is calculated, and said sentences are classified and grouped into same-topic groups sharing the same topic based on the result of comparing the similarity between said calculated sentence vector value and a similarity threshold value specified by topic; and when the function of said same-topic grouping step is completed, a corrected semantic paragraph generation step is performed to generate a semantic paragraph by integrating the sentences included in each of said same-topic groups into a single paragraph, and then performing a correction process on said semantic paragraph to adjust the length of the paragraph to a predetermined paragraph length to generate a corrected semantic paragraph. The method is characterized by including a semantic paragraph storage step in which, once the generation of the correction semantic paragraph is completed, a composite vector value for the correction semantic paragraph is calculated and the calculated composite vector value is stored in a database so that it can be utilized by a system to select and provide an answer to a subsequently entered query from the correction semantic paragraph.
[0019] The process start step preferably includes: a sentence identification step in which, when a division process for multiple document information stored in a repository is started, the multiple document information is analyzed through a stored natural language processing algorithm to identify a sentence corresponding to each of the multiple document information; a sentence division step in which, when the function of the sentence identification step is completed, the identified sentence is divided according to a pre-set division criterion in which the sentence's ending, punctuation, and newline character are used as division criteria for the sentence; and a similarity analysis process start step in which, when the division of the sentence is completed according to the pre-set division criterion, a similarity analysis process is started to compare the similarity of the divided sentences.
[0020] The above-mentioned same-topic grouping step preferably includes: a tokenization execution step in which, when the similarity analysis process starts, a tokenization process is performed on each of the sentences using a stored natural language processing algorithm to tokenize each of the sentences; a vector value calculation step in which, when the tokenization of the sentences is completed, a vectorizing process is performed on the tokens using the stored natural language processing algorithm to calculate a vector value for each token quantified based on the position of each token within the sentence, thereby calculating a sentence vector value for each of the sentences; and a grouping completion step in which, when the calculation of the sentence vector value for each of the sentences is completed by performing the function of the vector value calculation step, the calculated sentence vector values are compared to group sentences that exceed a similarity threshold designated by topic into a same-topic group sharing the same topic.
[0021] The above correction semantic paragraph generation step preferably includes: a semantic paragraph generation step that generates a semantic paragraph by integrating sentences included in each of the same topic groups into a single paragraph when the function of the same topic grouping step is completed; a correction process start determination step that determines whether to start an adjustment process based on the result of checking whether the total length of sentences having the same topic included in the semantic paragraph is included within a preset paragraph length range when the generation of the semantic paragraph is completed; and a semantic paragraph correction step that completes the generation of the correction semantic paragraph when the correction process is started, by dividing the semantic paragraph exceeding the preset paragraph length range into multiple paragraphs or by overlapping a predetermined sentence located at the end of a previous semantic paragraph located in the order preceding the semantic paragraph less than the preset paragraph length range into the semantic paragraph.
[0022] The above-mentioned corrected semantic paragraph is a paragraph corrected based on the above-mentioned pre-set paragraph length range by clustering semantic paragraphs that share the same topic among the sentences included in each of the above-mentioned multiple document informations, and it is possible to have a semantic paragraph corrected to maintain contextual continuity and prevent information loss.
[0023] The above correction semantic paragraph generation step may include: a paragraph structuring step, which, when the generation of the correction semantic paragraph is completed, structures the sentences included in the correction semantic paragraph based on the title, section, paragraph, and line structure of the reference document information serving as the standard for the plurality of document information, thereby finally completing the generation of the correction semantic paragraph; and a relationship graph generation step, which, when the function of the paragraph structuring step is completed, predicts the reference relationships between the structured correction semantic paragraphs through a neural network classifier that classifies the presence or absence of reference relationships based on a pre-trained language model of a stored natural language processing algorithm, determines whether there is a connection according to the reference relationships between the correction semantic paragraphs through the prediction results, and generates the reference relationships between the correction semantic paragraphs as a relationship graph in the form of a graph.
[0024] The above relationship graph is information that implements the reference relationships between structured correction semantic paragraphs in the form of a visualized graph, and it is possible for the above information to expand into a knowledge graph that utilizes sentences included in other correction semantic paragraphs forming a reference relationship with the selected correction semantic paragraph when an answer to a subsequently input query is selected and provided from the structured correction semantic paragraphs after the semantic paragraph storage step is completed.
[0025] The above semantic paragraph storage step preferably includes: a comprehensive vector value calculation step, which calculates a comprehensive vector value for a corrected semantic paragraph by performing a tokenization process and a vectorizing process on the corrected semantic paragraph through a previously stored natural language processing algorithm when the generation of the corrected semantic paragraph is completed; and a query correspondence vector storage step, which saves the calculated comprehensive vector value in a database when the function of the comprehensive vector value calculation step is completed, so that it can be utilized by a system for selecting and providing an answer to a subsequently input query from the corrected semantic paragraph based on the comprehensive vector value.
[0026] A device for dividing documents into semantic units based on semantic chunking, implemented as a computing device comprising one or more processors and one or more memories for storing instructions executable by said processors according to an embodiment of the present invention, wherein when a division process for a plurality of document information stored in a storage is started, a process starter unit analyzes a sentence corresponding to each of said plurality of document information, divides said sentences according to a pre-set division criterion, and then starts a similarity analysis process for said sentences divided according to said pre-set division criterion; when the similarity analysis process is started, a same-topic grouping unit calculates a composite vector value for each of said divided sentences, and classifies and groups said sentences into same-topic groups sharing the same topic based on the result of comparing the similarity between said calculated sentence vector value and a similarity threshold value specified by topic; and when the function of said same-topic grouping unit is completed, a corrected semantic paragraph generation unit integrates the sentences included in each same-topic group into a single paragraph to generate a semantic paragraph, and then performs a correction process on the generated semantic paragraph to adjust the length of the paragraph to a pre-set paragraph length to generate a corrected semantic paragraph. The invention is characterized by including a semantic paragraph storage unit that, when the generation of the correction semantic paragraph is completed, calculates a composite vector value for the correction semantic paragraph and stores the calculated composite vector value in a database so that it can be utilized in a system for selecting and providing an answer to a subsequently entered query from the correction semantic paragraph.
[0027] A computer-readable recording medium according to an embodiment of the present invention, wherein the computer-readable recording medium stores instructions for a computing device to perform the following steps, the steps comprising: a process start step in which, when a partitioning process for a plurality of document information stored in a storage is started, a sentence corresponding to each of the plurality of document information is analyzed, the sentence is partitioned according to a pre-set partitioning criterion, and a similarity analysis process is started for the sentences partitioned according to the pre-set partitioning criterion; a subject grouping step in which, when the similarity analysis process is started, a composite vector value for each of the partitioned sentences is calculated, and the sentences are classified and grouped into subject groups sharing the same subject based on the result of comparing the similarity between the calculated sentence vector value and a similarity threshold value designated by subject; and a corrected semantic paragraph generation step in which, when the function of the subject grouping step is completed, the sentences included in each subject group are integrated into a single paragraph to generate a semantic paragraph, and a correction process is performed on the generated semantic paragraph to adjust the length of the paragraph to a pre-set paragraph length to generate a corrected semantic paragraph. The method is characterized by including a semantic paragraph storage step in which, once the generation of the correction semantic paragraph is completed, a composite vector value for the correction semantic paragraph is calculated and the calculated composite vector value is stored in a database so that it can be utilized by a system to select and provide an answer to a subsequently entered query from the correction semantic paragraph.
[0028] Through the method of dividing documents into semantic units based on semantic chunking according to the present invention, document structuring based on actual semantic criteria rather than arbitrary physical division is realized through division based on semantic paragraph similarity and context flow, thereby improving the accuracy of document understanding and processing in semantic units.
[0029] In addition, by integrating document segmentation, semantic analysis, determination of reference relationships, and vector-based search into a series of flows, it is possible to provide the effect of simultaneously improving context preservation, search accuracy, and logical connectivity of responses in document-based information systems.
[0030] FIG. 1 is a flowchart illustrating a method for dividing documents into semantic units based on semantic chunking according to an embodiment of the present invention.
[0031] FIG. 2 is a flowchart illustrating the process start step of a method for dividing documents into semantic units based on semantic chunking according to an embodiment of the present invention.
[0032] FIG. 3 is a block diagram illustrating a same-subject grouping unit of a device that divides documents into semantic units based on semantic chunking according to an embodiment of the present invention.
[0033] FIG. 4 is a flowchart illustrating the step of generating corrected semantic paragraphs in a method for dividing a document into semantic units based on semantic chunking according to an embodiment of the present invention.
[0034] FIG. 5 is another flowchart illustrating the step of generating corrected semantic paragraphs in a method for dividing documents into semantic units based on semantic chunking according to one embodiment of the present invention.
[0035] FIG. 6 is another flowchart illustrating the semantic paragraph storage step of a method for dividing a document into semantic units based on semantic chunking according to one embodiment of the present invention.
[0036] FIG. 7 is a drawing for explaining an example of the internal configuration of a computing device according to an embodiment of the present invention.
[0037] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.
[0038] As used herein, terms such as "examples," "examples," "aspects," "examples," etc., may not be interpreted as implying that any aspect or design described is better or more advantageous than other aspects or designs.
[0039] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that the relevant feature and / or component is present, but not to exclude the presence or addition of one or more other features, components and / or groups thereof.
[0040] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.
[0041] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.
[0042] FIG. 1 is a flowchart illustrating a method for dividing documents into semantic units based on semantic chunking according to an embodiment of the present invention.
[0043] Referring to FIG. 1, a method for dividing a document into semantic units based on semantic chunking, implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed on said processors, may include a process start step (step S101), a same-topic grouping step (step S103), a corrected semantic paragraph generation step (step S105), and a semantic paragraph storage step (step S107).
[0044] In the following description, it will be understood that the method of dividing a document into semantic units based on semantic chunking according to each embodiment of the present invention is performed by the device for dividing a document into semantic units based on semantic chunking according to each embodiment of the present invention illustrated in FIG. 3 (hereinafter referred to as the "device of the present invention") and / or the computing device illustrated in FIG. 7. That is, it will be understood that the device of the present invention is implemented by combining one or more of the computing devices of FIG. 7.
[0045] In step S101, when a partitioning process for multiple document information stored in a storage is started, the one or more processors (hereinafter referred to as processors) may analyze a sentence corresponding to each of the multiple document information, partition the sentence according to a pre-set partitioning criterion, and then start a similarity analysis process for the sentences partitioned according to the pre-set partitioning criterion.
[0046] According to one embodiment, when a partitioning process for a plurality of document information stored in a storage is started, the processor can identify a sentence corresponding to each of the plurality of document information through a stored natural language processing algorithm.
[0047] In relation to the above, when the processor completes the identification of sentences within a document corresponding to each of the plurality of document informations, it can divide the identified sentences according to pre-set division criteria. More specifically, when the processor completes the identification of text within the document, it can divide the continuous sentences into individual sentences according to pre-set division criteria to identify individual sentence boundaries within the identified continuous text.
[0048] At this time, the processor can divide the sentence according to a pre-set division criterion that uses terminal endings (e.g., ~da, ~yo, ~kka?), punctuation marks (e.g., period, question mark, exclamation mark), and newline characters (e.g., newline symbol ( / n)) as division criteria for the sentence.
[0049] According to one embodiment, when the processor completes the division of a continuous sentence into individual sentences, it can start a similarity analysis process for the divided sentences.
[0050] According to one embodiment, when the similarity analysis process starts, the processor may perform the same subject grouping step (step S103).
[0051] In step S103, when the similarity analysis process is initiated, the processor can calculate a composite vector value for each of the divided sentences and, based on the result of comparing the similarity between the calculated sentence vector value and the similarity threshold value specified for each topic, classify and group the sentences into a same topic group that shares the same topic.
[0052] According to one embodiment, when the similarity analysis process starts, the processor can perform a tokenization process and a vectorizing process for each of the divided sentences through a stored natural language processing algorithm to calculate a sentence vector value for each of the sentences.
[0053] According to one embodiment, when the processor completes the calculation of sentence vector values for each of the sentences, it can compare similarity between similarity threshold values specified by topic.
[0054] In relation to the above, the similarity threshold specified by topic serves as a criterion for classifying sentences belonging to a specific topic category into the same topic group when comparing semantic similarity between divided sentences within a document, and can be set differently depending on the characteristics of the topic and the degree of diversity of linguistic expressions.
[0055] According to one embodiment, the processor can complete semantic chunking for clustering semantically related sentences by classifying the sentences into same-topic groups that share the same topic and then completing the grouping.
[0056] According to one embodiment, when the processor completes grouping by classifying sentences into the same subject group, it may include a step of generating a corrected semantic paragraph (step S105).
[0057] In step S105, when the function of the same topic grouping step is completed, the processor can generate a semantic paragraph by integrating the sentences included in each same topic group into a single paragraph, and then perform a correction process on the generated semantic paragraph to adjust the length of the paragraph to a preset paragraph length to generate a corrected semantic paragraph.
[0058] According to one embodiment, when sentences are classified and grouped by same subject group, the processor can integrate the sentences included in each of the same subject group into a single paragraph to generate a semantic paragraph.
[0059] In relation to the above, a semantic paragraph is a text unit formed by clustering sentences that are semantically similar or share the same topic among multiple sentences within a document, and each semantic paragraph is a paragraph that constitutes an independent semantic unit separated based on a topic turning point, and may be composed of sentences that are subject to subsequent processing such as vectorization, hierarchical structuring, or reference relationship analysis.
[0060] According to one embodiment, when the processor completes the generation of the semantic paragraph, it can perform a correction process on the semantic paragraph.
[0061] According to one embodiment, the correction process may not end with grouping semantic sentences into the same subject group, but may be a process performed to refine and improve the document into a structure suitable for actual use.
[0062] In relation to the above, since the lengths (number of sentences, amount of text, etc.) of semantic paragraphs generated during the semantic chunking process may vary, there may be problems where paragraphs that are too long result in complex contexts, while paragraphs that are too short result in insufficient semantic information or low query matching rates.
[0063] Accordingly, the processor can perform a correction process on the semantic paragraph (sentences included in the semantic paragraph) to adjust the length of the semantic paragraph to a preset paragraph length and generate a corrected semantic paragraph.
[0064] In addition, the processor can finally generate a corrected semantic paragraph by structuring and aligning the sentences included in the corrected semantic paragraph based on the title, section, paragraph, and line structure of the reference document information after generating the corrected semantic paragraph. That is, the semantic paragraph saving step (S107) performed after the corrected semantic paragraph generation step (S105) may be a step performed on the finally generated corrected semantic paragraph.
[0065] According to one embodiment, when the generation of the correction semantic paragraph is completed, the processor may perform a semantic paragraph storage step (step S107).
[0066] In step S107, when the generation of the correction semantic paragraph is completed, the processor calculates a composite vector value for the correction semantic paragraph and stores the calculated composite vector value in a database so that it can be utilized in a system for selecting and providing an answer to a subsequently input query from the correction semantic paragraph.
[0067] According to one embodiment, when the generation of a corrected semantic paragraph is completed, the processor can calculate a composite vector value, which is an embedding vector value for the corrected semantic paragraph, through the result of analyzing the corrected semantic paragraph using a stored embedding model of a natural language processing algorithm (e.g., BGE-M3, SBERT).
[0068] According to one embodiment, the composite vector value may be stored in a database as an expression representing the semantic content of the corresponding correction semantic paragraph, so that it can be utilized for comparison of similarity with a user's query, search, or matching in a question-and-answer system.
[0069] Accordingly, the composite vector value stored in the database can be used as reference information for selection as a response candidate paragraph through a comparison of similarity with a user query entered thereafter.
[0070] FIG. 2 is a flowchart illustrating the process start step of a method for dividing documents into semantic units based on semantic chunking according to an embodiment of the present invention.
[0071] Referring to FIG. 2, a method for dividing documents into semantic units based on semantic chunking, implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed on said processors, may include a process start step (e.g., the process start step (step S101) of FIG. 1).
[0072] According to one embodiment, the process start step may be a step of, when a splitting process for a plurality of document information stored in a repository is started, analyzing a sentence corresponding to each of the plurality of document information, splitting the sentence according to a preset splitting criterion, and then starting a similarity analysis process for the sentences split according to the preset splitting criterion.
[0073] According to one embodiment, the process start step may include a sentence identification step (step S201), a sentence segmentation step (step S203), and a similarity analysis process start step (step S205) as detailed steps for performing the above-described function.
[0074] In step S201, when a partitioning process for multiple document information stored in a storage is initiated, the processor can analyze the multiple document information through a stored natural language processing algorithm to identify a sentence corresponding to each of the multiple document information.
[0075] According to one embodiment, the processor can preprocess the plurality of document information through a stored natural language processing algorithm to extract actual analyzable text data from various document formats.
[0076] Accordingly, the processor can complete the identification of sentences based on text within the document corresponding to each of the plurality of document information.
[0077] According to one embodiment, when the processor completes the identification of a sentence, it may perform a sentence segmentation step (step S203).
[0078] In step S203, when the function of the sentence identification step (step S201) is completed, the processor can divide the identified sentence according to a pre-set division criterion in which sentence endings, punctuation marks, and newline characters are used as division criteria for the sentence.
[0079] According to one embodiment, the processor can identify terminal endings, punctuation marks, and newline characters in consecutive sentences based on the preset division criteria, and divide the consecutive sentences into individual sentences.
[0080] According to one embodiment, when the processor completes the division of consecutive sentences into individual sentences, it can perform a similarity analysis process start step (step S205).
[0081] In step S205, when the division of sentences is completed according to the preset division criteria, the processor may start a similarity analysis process to compare similarities between the divided sentences.
[0082] According to one embodiment, when the processor completes dividing the consecutive sentences into individual sentences, it may start a similarity analysis process to compare the similarity of the divided sentences.
[0083] In relation to the above, the similarity analysis process may be a process that calculates the sentence vector value of the divided sentence and compares it with a specified similarity threshold value (topic-specific reference vector value) to determine the degree of similarity of the divided sentence according to the topic.
[0084] FIG. 3 is a block diagram illustrating a same-subject grouping unit of a device that divides documents into semantic units based on semantic chunking according to an embodiment of the present invention.
[0085] Referring to FIG. 3, a device for dividing documents into semantic units based on semantic chunking (e.g., performing the same function as the method for dividing documents into semantic units based on semantic chunking of FIG. 1), which is implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed on said processors, may include a same subject grouping unit (300) (e.g., performing the same function as the same subject grouping step (step S103) of FIG. 1).
[0086] According to one embodiment, the same subject grouping unit (300) can calculate a composite vector value for each of the divided sentences (301a) when a similarity analysis process is started, and classify and group the sentences into same subject groups that share the same subject based on the result of comparing the similarity between the calculated sentence vector value and a similarity threshold value specified for each subject.
[0087] According to one embodiment, the same subject grouping unit (300) may include a tokenization performing unit (301), a vector value calculation unit (303), and a grouping completion unit (305) as detailed configurations for performing the above-described function.
[0088] According to one embodiment, the tokenization performing unit (301) can tokenize each of the sentences by performing a tokenization process on each of the sentences through a stored natural language processing algorithm when the similarity analysis process starts.
[0089] According to one embodiment, the tokenization unit (301) can separate each sentence into a certain unit of word, morpheme, or subword token by performing a tokenization process on each of the sentences through the stored natural language processing algorithm.
[0090] The above tokenization process may be performed by applying at least one of subword-based methods such as token-by-space, morpheme-based analysis, or BPE (Byte Pair Encoding), WordPiece, and SentencePiece, depending on the grammatical characteristics of the language and the purpose of analysis.
[0091] The token sequence generated in this way is used as an input value for calculating sentence embedding vectors in subsequent steps and can serve as key foundational information for calculating semantic similarity between sentences.
[0092] According to one embodiment, the stored natural language processing algorithm may be an algorithm having a structure that includes a sentence splitter, a morpheme-based tokenizer, a sentence embedding model (e.g., BERT or KoBERT), a similarity estimator, a reference classifier, etc. as a language model.
[0093] According to one embodiment, when the tokenization of the sentences is completed, the vector value calculation unit (303) can perform a vectorizing process on the tokens through the stored natural language processing algorithm to calculate a vector value for each token based on the position of each token in the sentence, thereby calculating a sentence vector value for each sentence.
[0094] According to one embodiment, when the tokenization of the sentences is completed, the vector value calculation unit (303) can perform a vectorizing process on the tokenized tokens through a stored natural language processing algorithm.
[0095] According to one embodiment, the vector value calculation unit (303) can generate an embedding vector that reflects semantic features for each token by inputting a token sequence (token) constituting each sentence into a pre-trained language model (BERT, SBERT, BGE-M3, etc.) when the vectorization process starts, and generate a sentence vector value that is a sentence embedding representing the entire sentence by mathematically integrating each calculated token vector (e.g., average, weighted sum, etc.).
[0096] In relation to the above, the vector value calculation unit (303) calculates a vector in token units based on contextualized token embeddings provided in the output layer of the language model included in the previously stored natural language processing algorithm, and the sentence vector considers the position information of each token vector, or directly extracts a vector corresponding to the [CLS] token,
[0097] Alternatively, it can be generated through operations such as mean pooling and max pooling.
[0098] According to one embodiment, when the grouping completion unit (305) completes the calculation of sentence vector values for each of the sentences by performing the function of the vector value calculation unit (303), it can compare the calculated sentence vector values and group sentences that exceed a similarity threshold value designated by topic into a same topic group that shares the same topic.
[0099] According to one embodiment, when the calculation of sentence vector values for each of the sentences is completed by the function of the vector value calculation unit (303), the grouping completion unit (305) calculates the semantic similarity between sentences by comparing the calculated sentence vector values with each other.
[0100] Sentences whose similarity value exceeds a similarity threshold specified for each topic can be clustered into same-topic groups that share the same topic.
[0101] In relation to the above, the grouping completion unit (305) may calculate semantic similarity using at least one of mathematical similarity formulas such as Cosine Similarity, Euclidean Distance, or vector dot product between each sentence vector, and if the similarity exceeds a predetermined similarity threshold value (e.g., 0.85), it may be determined that the two sentences are within the same subject category.
[0102] In addition, the grouping completion unit (305) can be configured to enable flexible division by setting multiple thresholds differently for each subject according to the genre, subject classification, or clustering density criteria of the document, in addition to using a specific threshold as a single criterion.
[0103] In this way, grouped sentences with the same subject are integrated into a single semantic unit during the subsequent semantic paragraph generation stage and can be utilized as semantic chunks divided according to the topic flow within the document.
[0104] FIG. 4 is a flowchart illustrating the step of generating corrected semantic paragraphs in a method for dividing a document into semantic units based on semantic chunking according to an embodiment of the present invention.
[0105] Referring to FIG. 4, a method for dividing a document into semantic units based on semantic chunking, implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed on said processors, may include a corrected semantic paragraph generation step (e.g., the corrected semantic paragraph generation step (step S105) of FIG. 1).
[0106] According to one embodiment, the step of generating a corrected semantic paragraph may be a step of generating a semantic paragraph by integrating sentences included in each of the same subject groups into one paragraph after the function of the same subject grouping step (e.g., the same subject grouping step (step S103) of FIG. 1) is completed, and then performing a correction process on the generated semantic paragraph to adjust the length of the paragraph to a preset paragraph length to generate a corrected semantic paragraph.
[0107] According to one embodiment, the correction semantic paragraph generation step may include a detailed step for performing the above-described function, a semantic paragraph generation step (step S401), a correction process start determination step (step S403), and a semantic paragraph correction step (step S405).
[0108] In step S401, when the processor completes the function of the same topic grouping step, it can integrate the sentences included in each same topic group into a single paragraph to generate a semantic paragraph.
[0109] According to one embodiment, when the grouping of same subject groups is completed by performing the function of the same subject grouping step, the processor can integrate the sentences included in each of the same subject groups into a single paragraph to generate a semantic paragraph in which the sentences included in the same subject groups share a single topic.
[0110] According to one embodiment, when the generation of the semantic paragraph is completed, the processor may perform a correction process start determination step (step S403).
[0111] In step S403, when the generation of the semantic paragraph is completed, the processor may determine whether to start the adjustment process based on the result of checking whether the total length of sentences having the same subject included in the semantic paragraph falls within a preset paragraph length range.
[0112] According to one embodiment, when the generation of semantic paragraphs is completed, the processor can check the total length of the generated semantic paragraphs and then check whether the total length of the checked semantic paragraphs is included within a preset paragraph length range.
[0113] More specifically, the processor can determine whether the semantic paragraph falls within a preset allowed paragraph length range (e.g., at least 3 sentences, at most 7 sentences) based on the number of sentences included in the semantic paragraph or the total text length.
[0114] In relation to the above, the processor checks the number of sentences, total number of words, total number of tokens, or byte-based text length included in each semantic paragraph and compares it with the pre-set paragraph length range, and if it is confirmed through the comparison result that the semantic paragraph is not included within the paragraph length range (is shorter than the range or longer than the range), the adjustment process may be initiated.
[0115] In addition, if the total length of the semantic paragraphs falls within a preset paragraph length range, the processor may confirm the paragraphs as corrected semantic paragraphs without correction.
[0116] In relation to the above, the pre-set allowable paragraph length range can be dynamically set by considering document type (news, reports, technical documents, etc.), subject complexity, average length between paragraphs, etc., in addition to being based on simple numerical values.
[0117] According to one embodiment, if the processor determines the start of the adjustment process by performing the function of the correction process start determination step (step S403), it may perform the semantic short-circuit correction step (step S405).
[0118] In step S405, when the correction process starts, the processor can complete the generation of the corrected semantic paragraph by dividing the semantic paragraph exceeding the preset paragraph length range into multiple paragraphs, or by overlapping the semantic paragraph with a predetermined sentence located at the end of the previous semantic paragraph located in the order preceding the semantic paragraph less than the preset paragraph length range.
[0119] According to one embodiment, the processor may divide a semantic paragraph into one or more sub-paragraphs when the total length of the semantic paragraph exceeds a preset allowable paragraph length range (e.g., exceeds the maximum number of sentences or tokens). In this case, the division criteria may be set based on the number of sentences, the number of tokens, or the distribution of similarity between sentences within the paragraph, and the paragraphs may be configured to be separated at natural topic boundaries or points of abrupt change in similarity.
[0120] For example, if a semantic paragraph consisting of 10 sentences exceeds a pre-set allowable paragraph length range of up to 7 sentences, the processor can analyze the similarity distribution or connectivity between sentences and divide it into a 6-sentence + 4-sentence paragraph at the most appropriate location.
[0121] More specifically, when the length of a semantic paragraph exceeds a preset allowable paragraph length range, the processor can generate a similarity distribution curve by calculating the semantic similarity between each adjacent sentence within the paragraph and detect a section where a similarity value decreases rapidly in the similarity distribution, or a point where it drops below a preset similarity threshold (e.g., 0.65), thereby detecting a section where a break in context or a change in topic occurs.
[0122] The above processor can divide a single semantic paragraph into two or more sub-paragraphs based on the discontinuity point of the distribution curve, thereby satisfying the length condition of the corrected semantic paragraph while maintaining the continuity of the topic and logical flow.
[0123] According to one embodiment, if the total length of the semantic paragraphs is less than a preset allowable paragraph length range, the processor can compensate for the paragraph length and contextual continuity by overlapping some of the last sentences of the previous semantic paragraphs located in the order preceding the semantic paragraphs at the beginning of the current semantic paragraphs.
[0124] At this time, the processor may include duplicate sentences by setting the number of duplicate sentences to a fixed value (e.g., 1 to 2 sentences), or may include duplicate sentences based on the contextual similarity between the previous paragraph and the current paragraph. Accordingly, the processor can minimize contextual interruption and compensate for the problem of insufficient information in short paragraphs.
[0125] More specifically, if the length of the semantic paragraph is less than a preset allowable paragraph length range, the processor can calculate the semantic similarity between the starting sentence of the current semantic paragraph and the last sentence of the adjacent previous semantic paragraph, and if the similarity value is greater than or equal to a preset standard (e.g., 0.80), perform a correction process to strengthen contextual connectivity and compensate for a lack of information by including the last sentence of the previous semantic paragraph redundantly in the beginning of the current semantic paragraph.
[0126] That is, the above correction process may be a process performed to maintain the balance of information within a paragraph and to utilize it as a preprocessing step to improve consistency and search efficiency in subsequent embedding and query-answer processing.
[0127] According to one embodiment, when the processor completes the correction process, it can generate a corrected semantic paragraph as a corrected semantic paragraph based on the result of the correction process.
[0128] In relation to the above, the corrected semantic paragraph is a paragraph corrected based on the pre-set paragraph length range by clustering semantic paragraphs that share the same topic among the sentences included in each of the plurality of document information, and may be a semantic paragraph corrected to maintain contextual continuity and prevent information loss.
[0129] FIG. 5 is another flowchart illustrating the step of generating corrected semantic paragraphs in a method for dividing documents into semantic units based on semantic chunking according to one embodiment of the present invention.
[0130] Referring to FIG. 5, a method for dividing a document into semantic units based on semantic chunking, implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed on said processors, may include a corrected semantic paragraph generation step (e.g., the corrected semantic paragraph generation step (step S105) of FIG. 1).
[0131] According to one embodiment, the step of generating a corrected semantic paragraph may be a step of generating a semantic paragraph by integrating sentences included in each of the same subject groups into one paragraph after the function of the same subject grouping step (e.g., the same subject grouping step (step S103) of FIG. 1) is completed, and then performing a correction process on the generated semantic paragraph to adjust the length of the paragraph to a preset paragraph length to generate a corrected semantic paragraph.
[0132] According to one embodiment, the correction meaning paragraph generation step may include a paragraph structuring step (step S501) and a relationship graph generation step (step S503) as detailed steps for performing other functions in addition to the functions described above.
[0133] In step S501, when the generation of the correction semantic paragraph is completed, the processor can structure the sentences included in the correction semantic paragraph based on the title, section, paragraph, and line structure of the reference document information that serves as the standard for the plurality of document information, thereby finally completing the generation of the correction semantic paragraph.
[0134] According to one embodiment, when the generation of the correction semantic paragraph is completed, the processor can verify the title, section, paragraph, and line structure of the reference document information that serves as the standard for the plurality of document information.
[0135] In relation to the above, the corrected semantic paragraph generated in the semantic paragraph correction step (e.g., the semantic paragraph correction step of FIG. 3 (step S305)) may have parts of Chapter 1 and Chapter 3 of the document mixed together to form a single chunk as the semantic paragraphs were previously divided based on similarity. Therefore, a process of rearranging the sentences to fit the document format is required so that the original document flow (order of chapters, sections, and paragraphs) remains consistent in context, in order for the user to reread the document or for the system to utilize the summary / search function. That is, the paragraph structuring step (step S501) is a process to solve the aforementioned problem.
[0136] Accordingly, when the processor completes the verification of the title, section, paragraph, and line structure of the reference document information that serves as the standard for multiple document information, it can structure the sentences included in the correction semantic paragraph based on the verified title, section, paragraph, and line structure of the reference document information to finally complete the generation of the correction semantic paragraph.
[0137] According to one embodiment, when the processor completes the function of the short-circuit structuring step (step S501), it may perform the relationship graph generation step (step S503).
[0138] In step S503, when the function of the paragraph structuring step (step S501) is completed, the processor can predict the reference relationships between the structured corrected semantic paragraphs through a neural network classifier that classifies the presence or absence of reference relationships based on a pre-trained language model of a stored natural language processing algorithm, and determine whether there is a connection based on the reference relationships between the corrected semantic paragraphs through the prediction results, thereby generating the reference relationships between the corrected semantic paragraphs as a relationship graph in the form of a graph (a graph in the form of nodes and edges, for example, B->AB is a graph describing a relationship based on A).
[0139] More specifically, the processor takes sentences between corrective semantic paragraphs as input and analyzes the contextual association of each pair of paragraphs, expressions of causality ("therefore," "for example," "due to this," "likewise," etc.), and the presence of demonstratives ("he," "this," "previously mentioned," etc.) to predict whether a specific corrective semantic paragraph refers to another paragraph or provides a dependent explanation.
[0140] In relation to the above, the processor can perform the above function through a pre-trained language model (BERT, RoBERTa) of a stored natural language processing algorithm.
[0141] For example, the processor may have identified that the last sentence of correction semantic paragraph A contains the sentence "Such technology is based on the previously presented system structure," and that correction semantic paragraph B contains the explanation "The relevant system structure is as follows." Accordingly, the processor determines through a neural network classifier based on the language model that paragraph B is referenced in paragraph A, and can establish a reference relationship having the directionality "B->A".
[0142] In relation to the above, if such a reference relationship is predicted, the processor may generate a relationship graph that represents the reference relationships between correction semantic paragraphs as a graph in the form of nodes and edges. Here, each correction semantic paragraph is represented as a node, and if a specific paragraph refers to another paragraph, it is represented by a connecting line having the corresponding direction (edge).
[0143] For example, the processor can generate a graph that visually represents the relationship between correction semantic paragraphs A, B, and C as follows: "B refers to A = Edge: "B -> A", "C is independent of A = "No edge", "C refers to B = Edge: C -> B".
[0144] According to one embodiment, the relationship graph is information that expresses the reference relationships between structured correction semantic paragraphs in the form of a visualized graph, and can be utilized as key basic data in subsequent processing steps such as summarizing, reconstructing, or semantic searching of documents.
[0145] In particular, after the semantic paragraph storage step (e.g., the semantic paragraph storage step of FIG. 1 (step S107)), when a specific query is entered by a user, a corrected semantic paragraph corresponding to the query can be selected and utilized to generate a response. At this time, sentences of other corrected semantic paragraphs that have a reference relationship with the selected corrected semantic paragraph are considered together, thereby enabling the provision of a richer and more contextually accurate response.
[0146] Accordingly, the above relationship graph is not limited to the visualization of simple reference relationships but can provide foundational information that can be expanded into a Knowledge Graph usable in question-answering systems, automatic summarization, knowledge search, etc.
[0147] FIG. 6 is another flowchart illustrating the semantic paragraph storage step of a method for dividing a document into semantic units based on semantic chunking according to one embodiment of the present invention.
[0148] Referring to FIG. 6, a method for dividing a document into semantic units based on semantic chunking, implemented as a computing device comprising one or more processors and one or more memories for storing instructions that can be executed on said processors, may include a semantic paragraph storage step (e.g., the semantic paragraph storage step (step S107) of FIG. 1).
[0149] According to one embodiment, the semantic paragraph storage step may be a step of calculating a composite vector value for the corrected semantic paragraph when the creation of the corrected semantic paragraph is completed, and storing the calculated composite vector value in a database so that it can be utilized in a system for selecting and providing an answer to a subsequently input query from the corrected semantic paragraph.
[0150] According to one embodiment, the semantic paragraph storage step may include a comprehensive vector value calculation step (step S601) and a query corresponding vector storage step (step S603) as detailed steps for performing the above-described function.
[0151] In step S601, when the generation of the correction semantic paragraph is completed, the processor can perform a tokenization process and a vectorizing process on the correction semantic paragraph through a stored natural language processing algorithm to produce a composite vector value for the correction semantic paragraph.
[0152] According to one embodiment, when the generation of the correction semantic paragraph is completed, the processor can calculate a composite vector value for the correction semantic paragraph by utilizing a pre-trained language model (BERT, RoBERTa, SBERT, etc.) included in a stored natural language processing algorithm.
[0153] More specifically, the processor can perform a tokenization process on the sentences of the correction semantic paragraph through a pre-stored natural language processing algorithm. At this time, position information and segment information are assigned to the tokens divided into individual words or subwords, so that they can be preprocessed into a form suitable for the model input format.
[0154] Subsequently, the above tokenized paragraph is input into the encoder layer of a pre-trained language model, and a multidimensional representation vector can be output for each token.
[0155] According to one embodiment, the processor may optionally utilize mean pooling, max pooling, or [CLS] token-based representations for these token vectors to produce a global sentence / document embedding representing the entire corrected semantic paragraph.
[0156] For example, the SBERT model efficiently generates vectors used for comparing semantic similarity between two paragraphs or for semantic-based search, and the generated composite vectors can be composed of real vectors of fixed dimensions (e.g., 768 dimensions, 1024 dimensions).
[0157] The composite vector value calculated as above is stored in a form comparable to the query vector in the subsequent query correspondence vector storage step (step S603), and can be used as reference information for selecting correction semantic paragraphs during subsequent query-response processing.
[0158] According to one embodiment, when the function of the comprehensive vector value calculation step (step S601) is completed, the processor may perform the query corresponding vector storage step (step S603).
[0159] In step S603, when the processor completes the function of the comprehensive vector value calculation step (step S601), the calculated comprehensive vector value can be stored in a database so that it can be utilized in a system for selecting and providing a corrected semantic paragraph based on the comprehensive vector value as an answer to a query entered thereafter.
[0160] According to one embodiment, the processor can store the composite vector value of the correction semantic paragraph calculated in the composite vector value calculation step (step S601) in a database in an indexable format that enables query response.
[0161] More specifically, the processor may generate a structure for each corrected semantic paragraph that includes a unique identifier (ID) of the paragraph, the original text (Text) of the paragraph, a composite vector value corresponding to the paragraph (e.g., 768 dimensions or more), and meta information (title, section, paragraph, line, etc.) of the reference document to which the paragraph belongs, and store this in a vector-based search system (e.g., Faiss, ElasticSearch's dense vector index, Weaviate, etc.).
[0162] This storage is intended to efficiently search for the most similar semantic paragraph by generating a query embedding through the same language model when a specific query is input by a user in the future, and by comparing it with the composite vectors of the stored corrected semantic paragraphs based on cosine similarity or dot product.
[0163] For example, in a system designed to select and provide an answer to a subsequently entered query from a correction semantic paragraph based on the aforementioned composite vector value, if the query "How can a method be applied to AI document summarization?" is entered, the query can be vectorized through the same language model (SBERT, RoBERTa, etc.).
[0164] Subsequently, similarity calculations are performed with the thousands of corrected semantic paragraph vectors stored above, and the top n similar paragraphs (ID and original text) are retrieved and can be used to generate a response.
[0165] That is, the above-mentioned query correspondence vector storage step (step S603) may be a key step for establishing a real-time response basis to connect the structured correction semantic paragraph with the actual user's natural language query.
[0166] FIG. 7 is a drawing for explaining an example of the internal configuration of a computing device according to an embodiment of the present invention.
[0167] FIG. 7 illustrates an example of the internal configuration of a computing device according to an embodiment of the present invention. In the following description, descriptions of unnecessary embodiments that overlap with the descriptions of FIG. 1 to 6 described above will be omitted.
[0168] As illustrated in FIG. 7, the computing device (10000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (11400), power circuit (11500), and communication circuit (11600). In this case, the computing device (10000) may correspond to a user terminal (A) connected to a haptic interface device or the aforementioned computing device (B).
[0169] The memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (10000).
[0170] At this time, access to memory (11200) from other components, such as the processor (11100) or peripheral device interface (11300), can be controlled by the processor (11100).
[0171] The peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (10000) to the processor (11100) and memory (11200). The processor (11100) can perform various functions for the computing device (10000) and process data by executing software modules or instruction sets stored in the memory (11200).
[0172] The input / output subsystem (11400) can connect various input / output peripherals to the peripheral interface (11300). For example, the input / output subsystem (11400) may include a controller for connecting peripherals such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral interface (11300). According to another aspect, input / output peripherals may be connected to the peripheral interface (11300) without passing through the input / output subsystem (11400).
[0173] The power circuit (11500) can supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.
[0174] The communication circuit (11600) can enable communication with another computing device using at least one external port.
[0175] Alternatively, as described above, the communication circuit (11600) may enable communication with other computing devices by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal.
[0176] The embodiment of FIG. 7 is merely an example of a computing device (10000), and the computing device (11000) may have some components shown in FIG. 7 omitted, additional components not shown in FIG. 7 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include a touchscreen or sensors, etc., in addition to the components shown in FIG. 6, and the communication circuit (1160) may include a circuit for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (10000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.
[0177] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file upon a request from the user terminal.
[0178] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.
[0179] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed across networked computing devices and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.
[0180] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.
[0181] Although the embodiments have been described above with reference to limited embodiments and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below also fall within the scope of the claims.
Claims
1. A method for dividing a document into semantic units based on semantic chunking, implemented by a computing device comprising one or more processors and one or more memories storing instructions executable on said processors, A process start step in which, when a partitioning process for multiple document information stored in a repository is started, a sentence corresponding to each of the multiple document information is analyzed, the sentence is partitioned according to a preset partitioning criterion, and a similarity analysis process is started for the sentences partitioned according to the preset partitioning criterion; A same-topic grouping step in which, when the above similarity analysis process is initiated, a composite vector value for each of the above-described sentences is calculated, and the sentences are classified and grouped into same-topic groups sharing the same topic based on the result of comparing the similarity between the calculated sentence vector value and a similarity threshold value designated by topic; A corrected semantic paragraph generation step, wherein when the function of the above-mentioned same-topic grouping step is completed, sentences included in each same-topic group are integrated into a single paragraph to generate a semantic paragraph, and a correction process is performed on the generated semantic paragraph to adjust the paragraph length to a preset paragraph length to generate a corrected semantic paragraph; and A method for dividing documents into semantic units based on semantic chunking, characterized by including: a semantic paragraph storage step in which, when the generation of the above-mentioned correction semantic paragraph is completed, a composite vector value for the above-mentioned correction semantic paragraph is calculated and the calculated composite vector value is stored in a database so that it can be utilized in a system for selecting and providing an answer to a subsequently input query from the above-mentioned correction semantic paragraph.
2. In Paragraph 1, The above process start step is, When a partitioning process for multiple document information stored in a repository is initiated, a sentence identification step that analyzes the multiple document information through a previously stored natural language processing algorithm to identify a sentence corresponding to each of the multiple document information; When the function of the above sentence identification step is completed, a sentence division step that divides the identified sentence according to a pre-set division criterion in which sentence endings, punctuation marks, and newline characters are used as division criteria for the sentence; and A method for dividing documents into semantic units based on semantic chunking, characterized by including a similarity analysis process start step for starting a similarity analysis process to compare the similarity of divided sentences when the division of sentences is completed according to the previously set division criteria.
3. In Paragraph 1, The above step of grouping identical topics is, When the above similarity analysis process is initiated, a tokenization execution step of performing a tokenization process on each of the above sentences through a pre-stored natural language processing algorithm to tokenize each of the above sentences; A vector value calculation step in which, once the tokenization of the above sentences is completed, a vectorizing process is performed on the tokens through the above-stored natural language processing algorithm to calculate a vector value for each token quantified based on the position of each token within the sentence, thereby calculating a sentence vector value for each sentence; and A method for dividing documents into semantic units based on semantic chunking, characterized by including: a grouping completion step in which, upon completing the calculation of sentence vector values for each of the sentences by performing the function of the vector value calculation step, sentences exceeding a similarity threshold specified by topic are grouped into a same topic group that shares the same topic by comparing the calculated sentence vector values.
4. In Paragraph 1, The above correction meaning paragraph generation step is, When the function of the above-mentioned same-topic grouping step is completed, a semantic paragraph generation step that integrates the sentences included in each same-topic group into a single paragraph to generate a semantic paragraph; A correction process start determination step that determines whether to start an adjustment process based on the result of checking whether the total length of sentences having the same subject included in the semantic paragraph is within a preset paragraph length range when the generation of the semantic paragraph is completed; and A method for dividing a document into semantic units based on semantic chunking, characterized by including: a semantic paragraph correction step, wherein when the above correction process is started, the generation of the corrected semantic paragraph is completed by dividing a semantic paragraph exceeding the above-set paragraph length range into multiple paragraphs, or by overlapping a predetermined sentence located at the end of a previous semantic paragraph located in the order preceding a semantic paragraph less than the above-set paragraph length range into the semantic paragraph.
5. In Paragraph 4, The above correction meaning paragraph is, A method for dividing documents into semantic units based on semantic chunking, characterized in that semantic paragraphs formed by clustering sentences sharing the same topic among the sentences included in each of the plurality of document information are corrected based on the aforementioned pre-set paragraph length range, and are corrected semantic paragraphs to ensure contextual continuity and prevent information loss.
6. In Paragraph 4, The above correction meaning paragraph generation step is, A paragraph structuring step for finally completing the generation of the correction semantic paragraph by structuring the sentences included in the correction semantic paragraph based on the title, section, paragraph, and line structure of the reference document information that serves as the standard for the plurality of document information when the generation of the correction semantic paragraph is completed; and A method for dividing documents into semantic units based on semantic chunking, characterized by including: a relationship graph generation step in which, when the function of the paragraph structuring step described above is completed, the reference relationships between structured corrected semantic paragraphs are predicted through a neural network classifier that classifies the presence or absence of reference relationships based on a pre-trained language model of a stored natural language processing algorithm, and the connection status according to the reference relationships between corrected semantic paragraphs is determined through the prediction result, and the reference relationships between corrected semantic paragraphs are generated as a relationship graph in the form of a graph.
7. In Paragraph 6, The above relationship graph is, A method for dividing a document into semantic units based on semantic chunking, characterized by information that visualizes the reference relationships between structured corrected semantic paragraphs in the form of a graph, and which expands into a knowledge graph that utilizes sentences included in other corrected semantic paragraphs forming a reference relationship with the selected corrected semantic paragraph when an answer to a subsequently input query is selected and provided from the structured corrected semantic paragraphs after the semantic paragraph storage step is completed.
8. In Paragraph 1, The above meaning paragraph saving step is, A comprehensive vector value calculation step in which, when the generation of the above-mentioned corrected semantic paragraph is completed, a tokenization process and a vectorizing process for the corrected semantic paragraph are performed through a previously stored natural language processing algorithm to calculate a comprehensive vector value for the corrected semantic paragraph; and A method for dividing documents into semantic units based on semantic chunking, characterized by including: a query correspondence vector storage step in which, when the function of the above-mentioned comprehensive vector value calculation step is completed, the calculated comprehensive vector value is stored in a database so that it can be utilized in a system for selecting and providing a corrected semantic paragraph based on the above-mentioned comprehensive vector value as an answer to a subsequently input query.
9. A device for dividing documents into semantic units based on semantic chunking, implemented as a computing device comprising one or more processors and one or more memories storing instructions executable on said processors, When a splitting process for multiple document information stored in a repository is started, a process starter analyzes a sentence corresponding to each of the multiple document information, splits the sentence according to a preset splitting criterion, and then starts a similarity analysis process for the sentences split according to the preset splitting criterion; A same-topic grouping unit that, when the above similarity analysis process is initiated, calculates a composite vector value for each of the above-described sentences and, based on the result of comparing the similarity between the calculated sentence vector value and a similarity threshold value designated by topic, classifies and groups the sentences into same-topic groups that share the same topic; A corrected semantic paragraph generation unit that, when the function of the above-mentioned same-topic grouping unit is completed, integrates the sentences included in each same-topic group into a single paragraph to generate a semantic paragraph, and then performs a correction process on the generated semantic paragraph to adjust the paragraph length to a preset paragraph length to generate a corrected semantic paragraph; and A device for dividing documents into semantic units based on semantic chunking, characterized by including: a semantic paragraph storage unit that, when the generation of the above-mentioned correction semantic paragraph is completed, calculates a composite vector value for the above-mentioned correction semantic paragraph and stores the calculated composite vector value in a database so that it can be utilized in a system for selecting and providing an answer to a subsequently input query from the above-mentioned correction semantic paragraph.
10. As a computer-readable recording medium, The above computer-readable recording medium stores instructions that cause a computing device to perform the following steps, said steps being: A process start step in which, when a partitioning process for multiple document information stored in a repository is started, a sentence corresponding to each of the multiple document information is analyzed, the sentence is partitioned according to a preset partitioning criterion, and a similarity analysis process is started for the sentences partitioned according to the preset partitioning criterion; A same-topic grouping step in which, when the above similarity analysis process is initiated, a composite vector value for each of the above-described sentences is calculated, and the sentences are classified and grouped into same-topic groups sharing the same topic based on the result of comparing the similarity between the calculated sentence vector value and a similarity threshold value designated by topic; A corrected semantic paragraph generation step, wherein when the function of the above-mentioned same-topic grouping step is completed, sentences included in each same-topic group are integrated into a single paragraph to generate a semantic paragraph, and a correction process is performed on the generated semantic paragraph to adjust the paragraph length to a preset paragraph length to generate a corrected semantic paragraph; and A computer-readable recording medium characterized by including: a semantic paragraph storage step in which, when the generation of the above-mentioned correction semantic paragraph is completed, a composite vector value for the above-mentioned correction semantic paragraph is calculated and the calculated composite vector value is stored in a database so that it can be utilized in a system for selecting and providing an answer to a subsequently input query from the above-mentioned correction semantic paragraph.
