Multilingual Text Chunking for Context-Preserving ML Input
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in processing input text beyond a fixed maximum length, leading to loss of context and inaccurate predictions due to random chunking, and struggle with mixed-language inputs, compromising downstream applications like intent prediction and sentiment analysis.
Innovation Solution
A system for logically chunking input text based on linguistic characteristics and language detection, splitting into segments and merging them to form chunks within the maximum length, preserving context and identifying language for each chunk.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If input text is chunked randomly based on maximum length, then the input text can be processed by machine learning model, but context is lost and prediction accuracy deteriorates
Solution Approach 1:
The patent divides the input text into multiple segments or chunks, each within the maximum length limit of the machine learning model. This segmentation allows the model to process longer texts that would otherwise exceed its capacity, improving productivity while maintaining context through careful chunking strategies.
Solution Approach 2:
The system performs preliminary actions by detecting the language of the input text before chunking, and by strategically determining chunk boundaries to preserve contextual integrity. This preliminary language detection and context-aware segmentation ensures that subsequent processing maintains accuracy while enabling extended text processing.
2Loss of time
If existing chunking solutions are used, then processing time is reduced, but context preservation and language identification are compromised
Solution Approach 1:
The system performs language detection as a preliminary action before chunking the text. This allows the chunking process to be tailored to the specific language characteristics, preserving context while enabling efficient processing. The preliminary language identification prevents information loss that would occur with generic chunking approaches.
Solution Approach 2:
The patent applies language-specific chunking strategies tailored to the detected language. Each language may have different contextual boundaries and structural characteristics, and the system adapts its chunking approach locally to match these characteristics, preserving context while maintaining processing efficiency.
3Adaptability or versatility
If machine learning model processes text beyond maximum length, then longer input capability is achieved, but context is lost due to random splitting
Solution Approach 1:
The patent implements segmentation of long texts into multiple chunks, each within the model's maximum length limit. This enables the model to process texts of any length by breaking them into manageable pieces, achieving versatility in handling different text lengths while maintaining context through intelligent segmentation boundaries.
Solution Approach 2:
Before dividing the text into chunks, the system performs preliminary language detection and analyzes the text structure to identify appropriate chunk boundaries. This preliminary analysis ensures that segments are created in a way that preserves contextual relationships, allowing the model to maintain reliability even when processing texts beyond its original length capacity.
Data Source
AI summary
There are provided systems and methods for building a chunked text as input for a machine learning model. An example system may receive an input text in at least one language. The system may determine a character limit corresponding to the language for deriving a segment from the input text, a character number of the segment does not exceed the character limit. The system may divide the input text into one or more segments sequentially based on a set of characteristics of input text, such as conjunction and punctuation. The system may generate a list of segments including the one or more segments by appending segment sequentially. The system may generate a list of combined segments as the input for the machine learning model by appending each segment in the list of segments sequentially.