ML Text Segmentation Using Sentence Fragments and Split Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for segmenting text into segments based on rules often result in segments that are not semantically meaningful, leading to suboptimal processing results when used with large language models, and systems that rely on document tags are limited and time-consuming.
Innovation Solution
A system that uses a machine learning model to determine classification scores for pairs of sentence fragments, assigning split positions based on these scores to create semantically relevant and self-contained segments, which are then combined into segments that are processed by downstream systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional rule-based systems are used to segment text, then the segmentation process is simple and fast, but the resulting segments are not semantically meaningful
Solution Approach 1:
The patent replaces mechanical rule-based segmentation with a machine learning model that uses semantic understanding to determine segment boundaries. The model processes sentence fragments and outputs classification scores based on semantic similarity, automatically identifying optimal split positions without relying on predefined linguistic rules.
Solution Approach 2:
The patent introduces sentence fragments as an intermediary representation between the input text and final segments. The machine learning model processes these fragments to generate classification scores, which then guide the segmentation process. This intermediary layer enables semantic-aware segmentation while maintaining system modularity.
2Reliability
If document tags are used to identify segments, then segment identification is straightforward, but the process is time-consuming and limited in applicability
Solution Approach 1:
The patent enables the text segmentation system to automatically identify segment boundaries without requiring external document tags or manual annotation. The machine learning model self-determines optimal split positions by analyzing semantic relationships between sentence fragments, eliminating the time-consuming process of tag-based identification.
Solution Approach 2:
The patent changes the fundamental parameter for segment identification from the presence of document tags to semantic similarity scores generated by the machine learning model. This parameter change allows the system to work on untagged documents and significantly reduces processing time while improving identification accuracy.
3Measurement precision
If text is divided into many small sentence fragments for machine learning processing, then semantic accuracy improves, but computing time and resources increase
Solution Approach 1:
The patent divides the input text into sentence fragments as intermediate units for machine learning processing. This segmentation enables the model to analyze semantic relationships at an appropriate granularity level, improving classification accuracy while managing computational complexity through structured processing of smaller units.
Solution Approach 2:
The patent combines adjacent sentence fragments into final segments based on classification scores. By merging fragments that have high semantic similarity, the system reduces the total number of output segments and minimizes computing resources required for downstream processing while preserving semantic accuracy.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining segments from a sequence of text. One of the methods includes obtaining data representing a sequence of text; dividing the sequence of text into a plurality of sentence fragments; determining classification scores comprising determining a classification score using a machine learning model for each of a plurality of pairs of sentence fragments formed from the plurality of sentence fragments; assigning one or more split positions based on the classification scores; and combining the plurality of sentence fragments back into at least two segments, with a boundary of at least one of the at least two segments being identified by one of the one or more split positions, and wherein each segment comprises one or more sentence fragments.