Language Model Context Expansion Using Summarization Tokens
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for expanding the context window size of language models, such as position interpolation and Randomized Positional encodings (RandPos), face limitations in computational complexity and continuity, leading to potential loss of important information.
Innovation Solution
A method involving a summarization token sequence is used to extract and train a pre-trained language model, adding position information and segmenting input data into chunks to generate a summarization token sequence, which includes main tokens with position information, effectively expanding the context window while maintaining computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of moving object
If position interpolation is used to adjust position information to the context window size, then the context window size can be expanded, but computational complexity increases
Solution Approach 1:
The patent segments the input sequence into multiple chunks and processes each chunk independently through the summarization model. This segmentation allows the model to handle longer contexts by dividing the computational task into smaller, manageable units, thereby expanding the context window size while controlling computational complexity through parallel processing of chunks.
Solution Approach 2:
The patent extracts and selects only the most important tokens from each chunk using the summarization model, creating a compressed representation that retains essential information. This extraction process reduces the amount of data that needs to be processed while maintaining the semantic meaning, thus expanding context capacity without proportionally increasing computational complexity.
2Volume of moving object
If Randomized Positional encodings (RandPos) are used to expand context window size, then position information can be adjusted, but continuity between adjacent tokens deteriorates
Solution Approach 1:
The patent performs preliminary processing by segmenting the input sequence into chunks and generating summaries for each chunk before final assembly. This preliminary action ensures that position information is established and continuous relationships are maintained within each chunk, preventing the discontinuity problem that arises from random position reassignment in RandPos.
Solution Approach 2:
The summarization model acts as an intermediary that processes each chunk and generates a condensed representation preserving positional relationships. This intermediary step maintains continuity between adjacent tokens by ensuring that the summary of each chunk properly reflects its position relative to other chunks, thus avoiding the randomization effect of RandPos.
3Stability of the object's composition
If Positional Skip-wisE (PoSE) is used to segment input sequence into chunks, then continuity can be maintained, but important information may be lost due to random skipping
Solution Approach 1:
The patent employs the summarization model to generate summaries that feedback the most important information from each chunk. The summarization process inherently selects and emphasizes critical tokens while discarding redundant ones, ensuring that continuity is maintained through meaningful semantic connections rather than arbitrary positional connections, thus preventing information loss.
Solution Approach 2:
The patent changes the parameter of token selection from random skipping to informed selection based on the summarization model's output. By using the summarization model to determine which tokens to retain from each chunk, the system dynamically adjusts the selection criteria to preserve important information while maintaining continuity, avoiding the fixed random skipping mechanism of PoSE.
Data Source
AI summary
Provided is a method performed by at least one computing device. The method may comprise extracting a summarization token sequence including a plurality of tokens from input data, the plurality of tokens including position information, a length of the summarization token sequence being within a reference length; and additionally training a pre-trained language model using the summarization token sequence, wherein the reference length corresponds to an initial context length of the pre-trained language model.