Semantic Text Segmentation Using ML Split Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for segmenting text into segments based on rules often result in segments that are not semantically meaningful, leading to suboptimal processing results when input to large language models, and existing methods for document tagging are time-consuming and limited to specific document types.
Innovation Solution
A system that uses machine learning models to identify and generate semantically relevant segments by dividing text into sentence fragments, determining split scores, and assigning split positions to combine fragments into self-contained segments, suitable for input to large language models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional rule-based systems are used to segment text, then the segmentation process is simple and fast, but the resulting segments are not semantically meaningful
Solution Approach 1:
The patent replaces conventional rule-based mechanical segmentation systems with machine learning-based semantic segmentation. Instead of using fixed rules to divide text, the system employs transformer models that understand semantic relationships between words and sentences, enabling meaningful segmentation based on content rather than form.
Solution Approach 2:
The patent changes the segmentation approach from static rule-based parameters to dynamic ML-based parameters. The system uses attention mechanisms and transformer architectures that adapt to different text structures and semantic contexts, allowing the segmentation to be optimized for meaning rather than fixed formatting rules.
2Adaptability or versatility
If existing document tagging methods are used, then the process can identify document types, but it is time-consuming and limited to specific document types
Solution Approach 1:
The patent creates a universal segmentation system that can handle various document types and languages simultaneously. The transformer model is trained on diverse datasets and can automatically adapt to different document structures, legal contexts, and languages without requiring type-specific processing rules, making the system both versatile and efficient.
Solution Approach 2:
The system performs self-optimization by automatically learning document structures and segmentation patterns from the input data itself. The transformer model adapts to different document types through its attention mechanisms, eliminating the need for manual configuration or pre-programming for specific document categories, thus reducing processing time and increasing versatility.
3Manufacturing precision
If text is divided into segments for processing, then the input size for machine learning models is reduced, but the segmentation boundaries may not align with semantic meaning
Solution Approach 1:
The patent replaces mechanical rule-based segmentation with semantic-aware transformer models that understand the meaning and context of text. The model uses attention mechanisms to identify semantically meaningful boundaries based on content relationships rather than fixed formatting rules, ensuring alignment between segment boundaries and semantic meaning while maintaining processing efficiency.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining segments from a sequence of text. One of the methods includes obtaining data representing a sequence of text; dividing the sequence of text into a plurality of sentence fragments; determining split scores comprising determining a split score for each of a plurality of pairs of sentence fragments formed from the plurality of sentence fragments; assigning one or more split positions based on the split scores; and combining the plurality of sentence fragments back into at least two segments, with a boundary of at least one of the at least two segments being identified by one of the one or more split positions, and wherein each segment comprises one or more sentence fragments.