Attention Output-Based Token Eviction for LLM Cache Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, particularly large models like LLMs and LVMs, incur substantial computational expense due to large caches of intermediate values that exceed available memory, especially on resource-constrained devices, and existing token eviction methods fail to effectively manage cache size and memory usage.
Innovation Solution
Implement retention-score-based token eviction, where the change in attention output is calculated using both key and value tensors to determine which tokens to evict, ensuring efficient cache management and reduced memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If caching intermediate values during inferencing is implemented, then computational expense is reduced, but cache size rapidly becomes unacceptably large and exceeds available memory space
Solution Approach 1:
The patent extracts and evicts key-value pairs from the cache based on retention scores. When the cache reaches a size threshold, the system calculates retention scores for all cached key-value pairs and evicts those with the lowest scores, thereby removing unnecessary data while preserving important intermediate values for later use.
Solution Approach 2:
The patent introduces retention scores as a new parameter to evaluate and manage cache contents. By calculating retention scores based on attention weights and temporal patterns, the system dynamically adjusts which key-value pairs remain in cache and which are evicted, enabling intelligent cache management that balances memory usage with computational efficiency.
2Quantity of substance
If existing token eviction methods are used, then cache size is managed, but memory usage is not effectively controlled and model performance deteriorates
Solution Approach 1:
The patent implements a feedback mechanism where retention scores are continuously calculated and used to guide cache eviction decisions. The system monitors memory usage and dynamically adjusts eviction strategies based on real-time retention score calculations, ensuring that cache management actions align with actual model performance requirements.
Solution Approach 2:
The patent makes the cache eviction strategy dynamic by calculating retention scores based on current attention weights and temporal patterns. Rather than using static eviction rules, the system adapts its eviction decisions in real-time based on the specific computational context, allowing flexible management of cache contents that evolves with the inferencing process.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for machine learning. In an example method, an input prompt comprising a set of tokens is accessed as input to a generative machine learning model. A first key tensor and a first value tensor are generated for a first token of the set of tokens, and the first key tensor and the first value tensor are stored in a memory. A first retention score is generated, for the first token, based on the first key tensor, the first value tensor, and a second token of the set of tokens. The first key tensor and the first value tensor are evicted from the memory in response to determining that the first retention score is a lowest retention score of the memory.