Long Text Chunking for Pre-trained Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pre-trained language models for chatbots face challenges in handling long text, as they typically support a maximum length of text, leading to truncated data and reduced performance.
Innovation Solution
The technique involves dividing long text into overlapping chunks of a predefined size, assigning labels to each chunk, and merging the confidence scores from overlapping token pieces to determine the final label for each token piece.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the maximum length of text is limited to support pre-trained language models, then the model can process text efficiently, but long text inputs are truncated leading to loss of information and reduced accuracy
Solution Approach 1:
The patent divides long text inputs into multiple overlapping chunks, each within the maximum length limit of the pre-trained language model. Each chunk is processed separately and the results are aggregated to produce the final output for the entire long text, thereby avoiding truncation while maintaining processing efficiency
Solution Approach 2:
The patent introduces a temporal dimension by processing text in sequential overlapping chunks rather than attempting to process the entire long text at once. This transforms a single large processing task into multiple smaller sequential tasks, allowing the model to handle long text without exceeding its length constraints
2Productivity
If long text is divided into multiple non-overlapping chunks, then processing time is reduced, but information at chunk boundaries is lost
Solution Approach 1:
The patent segments long text into multiple chunks while introducing overlap between adjacent chunks. The overlap region contains tokens that appear in both chunks, ensuring that contextual information at boundaries is preserved and can be consistently labeled across chunks
Solution Approach 2:
The patent applies different processing strategies to different regions of the text: most tokens are labeled by their primary chunk, but tokens in overlapping regions are labeled by both chunks and their labels are aggregated. This local differentiation ensures boundary information is preserved without significantly increasing overall processing time
3Reliability
If overlapping chunks are used to process long text, then information accuracy is maintained, but training and inferencing time increases
Solution Approach 1:
The patent segments text into overlapping chunks but processes them in an optimized manner where each token is labeled by its first occurrence chunk, with boundary tokens also considered in overlapping chunks. This segmentation approach maintains accuracy while controlling processing time through efficient label aggregation
Solution Approach 2:
The patent applies partial overlap between chunks rather than full redundancy. The overlap is sufficient to preserve boundary information but limited in size to minimize the number of duplicate processing operations, striking a balance between accuracy and efficiency
Data Source
AI summary
In some aspects, a computing device may receive, at a data processing system, a set of utterances for training or inferencing with a named entity recognizer to assign a label to each token piece from the set of utterances. The computing device may determine a length of each utterance in the set and when the length of the utterance exceeds a pre-determined threshold of token pieces: dividing the utterance into a plurality of overlapping chunks of token pieces; assigning a label together with a confidence score for each token piece in a chunk; determining a final label and an associated confidence score for each chunk of token pieces by merging two confidence scores; determining a final annotated label for the utterance based at least on the merging the two confidence scores; and storing the final annotated label in a memory.