Abstract Token Compression for Long-Context Dialogue Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The execution of language models requires significant processing and memory resources due to the size of the dialogue history, leading to resource constraints on local computing devices, and truncation of dialogue history negatively affects response quality.
Innovation Solution
A machine-trained model generates abstract token information to summarize dialogue history, reducing prompt size and resource consumption, allowing local devices to implement the model efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the full dialogue history is included in the prompt, then the response quality is maintained, but the processing and memory resources required increase significantly
Solution Approach 1:
The patent extracts only the essential information from the full dialogue history and represents it using abstract tokens. Instead of including the complete dialogue history in the prompt, the system identifies and extracts key semantic elements, transforming them into compact abstract token representations that preserve meaning while dramatically reducing token count and associated computational resources.
Solution Approach 2:
The patent changes the parameter of information representation from detailed text tokens to compressed abstract tokens. This parameter transformation allows the same semantic information to be conveyed with fewer tokens, reducing the computational burden on the language model while maintaining response quality.
2Reliability
If the full dialogue history is included in the prompt, then the response quality is maintained, but the memory resources required increase significantly
Solution Approach 1:
The system extracts essential semantic content from the dialogue history and represents it using abstract tokens. This extraction process removes redundant and less important information, keeping only the critical elements needed for generating accurate responses, thereby reducing memory consumption.
Solution Approach 2:
Instead of storing and processing the full dialogue history, the system creates abstract token copies that represent the essential meaning of the dialogue. These abstract tokens serve as compressed representations that occupy minimal memory space while preserving the necessary information for response generation.
3Use of energy by moving object
If the prompt size is limited to a predetermined limit, then resource consumption is reduced, but relevant parts of the dialogue history are eliminated
Solution Approach 1:
The patent changes the representation parameter from detailed text to abstract tokens, allowing the same amount of information to be conveyed in fewer tokens. This parameter transformation enables the prompt to stay within size limits while preserving all relevant dialogue information through efficient compression.
Solution Approach 2:
The system performs preliminary processing of the dialogue history to generate abstract tokens before constructing the prompt. This preliminary action of creating compressed representations ensures that all relevant information is captured in an efficient format before being included in the prompt, preventing any loss of important dialogue context.
4Reliability
If the prompt size increases with each dialogue turn, then the complete dialogue history is maintained, but the scalability is reduced and local devices cannot implement the model
Solution Approach 1:
The patent applies parameter change by transforming dialogue representations into abstract tokens, which dramatically reduces the growth rate of prompt size. Instead of adding numerous text tokens with each dialogue turn, the system adds a small number of abstract tokens that capture the essence of new dialogue content, enabling scalability to resource-constrained local devices.
Solution Approach 2:
The system creates abstract token copies that represent dialogue history efficiently. These compressed copies allow the model to maintain complete dialogue context while keeping prompt sizes manageable, enabling deployment on local devices with limited computational resources.
Data Source
AI summary
A technique uses a machine-trained model to generate a response based on a prompt which expresses current input information and abstract token information. The abstract token information summarizes a full dialogue history of a dialogue, and is generated by the model itself. The technique reduces the size of the prompt by incorporating the abstract summary information in lieu of the full dialogue history. A training system trains the machine-trained model by successively improving the predictive accuracy of the machine-trained model, while rewarding the machine-trained model based on an extent to which the machine-trained model compresses instances of abstract token information.