Semantic Text Segmentation Using Sliding-Window Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text segmentation methods are simplistic and contextually unaware, failing to produce semantically coherent segments, which hampers effective information retrieval and generative processes in Retrieval-Augmented Generation (RAG) pipelines.
Innovation Solution
A method and system that employs a sliding window technique with pre-trained language models to embed sentences into dense vector representations, calculate cosine similarity between mean vector representations of sentence groups, identify peaks in boundary scores, and segment text based on these peaks to create semantically coherent segments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If max token length splitting is used, then text segmentation is simple and fast, but semantic coherence is lost
Solution Approach 1:
The patent replaces the mechanical token-length-based splitting approach with a semantic-aware method using sentence embeddings and cosine similarity. Instead of arbitrarily cutting text at fixed token lengths, the system uses vector representations to identify semantically appropriate boundaries, substituting a naive mechanical process with an intelligent semantic analysis process.
Solution Approach 2:
The patent changes the segmentation parameter from fixed token length to dynamic semantic boundaries. By computing cosine similarity between consecutive sentence embeddings and identifying peaks in the similarity scores, the system adapts segmentation points based on actual semantic content rather than predetermined length constraints, allowing segments to vary in length while maintaining coherence.
2Reliability
If markup or headline-based splitting is used, then structural segments are obtained, but applicability to unstructured text is limited
Solution Approach 1:
The patent creates a universal segmentation method that works across both structured and unstructured text formats. By using sentence embeddings and semantic similarity analysis, the system can process texts with markup/headings as well as plain unstructured text, making the segmentation approach multi-functional and format-agnostic while maintaining semantic coherence.
Solution Approach 2:
The patent replaces dependency on explicit structural markers (headings, markup) with a semantic-based segmentation mechanism. Instead of requiring predefined structural cues, the system uses cosine similarity of sentence embeddings to identify boundaries, enabling it to handle unstructured text effectively while still respecting structural divisions when present.
3Reliability
If statistical word frequency algorithms are used, then segmentation is more sophisticated, but computational complexity increases
Solution Approach 1:
The patent replaces complex statistical word frequency analysis with a more elegant embedding-based approach. Instead of analyzing word frequency patterns and transitions, the system uses pre-trained sentence embeddings and simple cosine similarity computation, achieving sophisticated semantic-aware segmentation with reduced algorithmic complexity by leveraging existing language model representations.
Data Source
Figure 1~2
Figure 3
AI summary
The method is based on embedding (4), by a pre-trained language model, each sentence contained in an input text into a dense vector representation and thereby forming a sequence of sentence embeddings, processing (5) the sequence of sentence embeddings by sequentially traversing a sliding window through the sequence of sentence embeddings, wherein a window position is defined by a location of a center of the window in the sequence of sentence embeddings, and computing, for each window position, a first mean vector representation for sentence embeddings in a first half of the window and a second mean vector representation for sentence embeddings in a second half of the window, calculating (6) a boundary score for each window position by computing the cosine similarity between the corresponding first mean vector representation and second mean vector representation, identifying (7) peaks in the boundary scores, segmenting (8) the input text at the identified peaks into a sequence of text segments, and outputting (9) the sequence of text segments. The algorithm provides contextually relevant and semantically coherent segments, thus enhancing the performance of RAG pipelines, leading to more accurate and contextually appropriate outputs. The algorithm is also highly efficient.