Document Chunking Using Section Delimiters and Sentence Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing document chunking methods fail to consider context and structure, leading to loss of accurate responses and reduced accuracy in query answering due to improper chunking of documents stored in external databases.
Innovation Solution
A method and system for chunking documents by combining structural and contextual analysis, using a two-step process involving section chunking based on delimiters and sub-chunking based on sentence similarity, with threshold determination through vector similarity analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If documents are chunked into fixed units without considering context, then the chunking process is simple and fast, but sentences containing correct answers may be lost or damaged and retrieval accuracy is reduced
Solution Approach 1:
The patent divides document chunking into two distinct phases: structural chunking (dividing by section delimiters like chapter titles) and semantic chunking (dividing by sentence similarity thresholds). This segmentation allows each phase to optimize for its specific purpose while collectively achieving both efficiency and accuracy.
Solution Approach 2:
The patent performs structural chunking as a preliminary step before semantic chunking. By first organizing documents into sections based on delimiters, the system creates a coarse-grained structure that reduces the overall processing load for the subsequent semantic chunking phase, improving overall efficiency.
2Measurement precision
If semantic chunking is performed on all documents using sentence similarity comparison, then retrieval accuracy is improved, but the frequency of embedding model invocation increases and total processing time increases
Solution Approach 1:
The patent segments the chunking process into structural and semantic phases, applying semantic chunking only to sections that exceed a size threshold after structural chunking. This reduces the total number of sentences requiring embedding model invocation while maintaining retrieval accuracy for relevant portions.
Solution Approach 2:
The patent applies different chunking strategies to different parts of the document based on local characteristics. Sections exceeding the size threshold undergo semantic chunking with similarity comparison, while smaller sections are handled more efficiently, optimizing resource allocation based on local document properties.
3Measurement precision
If a two-step chunking process is used (structural then semantic), then retrieval accuracy is improved and processing efficiency is optimized, but the system complexity increases
Solution Approach 1:
The patent divides the chunking system into two independent modules: a structural chunking module that handles delimiter-based division and a semantic chunking module that handles similarity-based division. This modular segmentation makes the system more manageable and allows each module to be optimized independently.
Solution Approach 2:
The structural chunking module performs preliminary organization of documents into sections before the semantic chunking module processes them. This preliminary action simplifies the input for the second module, reducing its complexity while maintaining overall system effectiveness.
4Manufacturing precision
If section delimiters are determined by multiple criteria (occurrence frequency, text size, identification order), then chunking quality is improved, but the determination process becomes more complex
Solution Approach 1:
The patent uses multiple parameters (occurrence frequency, text size, identification order) to determine section delimiters, changing the evaluation criteria based on document characteristics. This multi-parameter approach improves delimiter accuracy while the systematic evaluation process manages the complexity through clear prioritization rules.
Data Source
AI summary
A method and system for chunking a document are provided. The method according to some embodiments may include chunking a document including a plurality of sentences into a plurality of section chunks based on sentences including a section delimiter, determining whether a size of each of the plurality of section chunks exceeds a preset first threshold and chunking a section chunk, a size of the section chunk among the plurality of section chunks exceeds the preset first threshold, into a plurality of sub-chunks based on whether a similarity between sentences included in the section chunk is equal to or greater than a second threshold. The second threshold may be determined based on a similarity distribution of a query for the document, calculated by comparing a query generated from the document using a generative model with the document.