Attention Cache Token Eviction for Memory-Limited LLM Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, particularly large models like LLMs and LVMs, incur substantial computational expense due to large caches of intermediate attention data that exceed available memory, especially on resource-constrained devices, and existing token eviction methods fail to effectively manage cache size and memory usage.
Innovation Solution
Implement retention-score-based token eviction, where the change in attention output is calculated using both key and value tensors to determine which tokens to evict, ensuring efficient cache management and reduced memory footprint.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If caching intermediate attention data is used to reduce computational expense, then inferencing efficiency is improved, but cache size grows unacceptably large and exceeds available memory space
Solution Approach 1:
The patent implements token eviction mechanisms that discard least-recently-used or lowest-impact key-value pairs from the cache when memory capacity is reached. This allows the system to continuously reuse cache space by removing obsolete data and adding new intermediate attention data, resolving the contradiction between maintaining large caches for efficiency and limited memory availability.
Solution Approach 2:
The patent dynamically adjusts cache management parameters such as cache capacity thresholds, eviction policies, and retention criteria based on available memory space and computational needs. By changing these parameters adaptively, the system optimizes the balance between cache size and inferencing efficiency under different memory constraints.
2Reliability
If larger caches are used to maintain more tokens, then model accuracy is improved, but memory usage exceeds available space on resource-constrained devices
Solution Approach 1:
The patent applies different retention priorities and eviction strategies to different portions of the cache based on their importance and usage patterns. High-impact key-value pairs are retained longer with higher priority, while low-impact pairs are evicted more readily. This localized quality differentiation allows the system to maintain model accuracy by preserving critical data while reducing overall memory usage through selective discarding.
3Quantity of substance
If cache size is reduced to fit memory constraints, then memory usage is controlled, but inferencing performance and model accuracy deteriorate
Solution Approach 1:
The patent implements self-managing cache systems where the cache automatically monitors its own usage, identifies candidates for eviction based on usage patterns and impact metrics, and performs evictions without external intervention. This self-service mechanism ensures that the cache continuously optimizes its contents to maintain maximum inferencing performance within the constrained memory capacity, preventing performance deterioration despite size limitations.
Data Source
AI summary
Certain aspects of the present disclosure provide techniques and apparatus for machine learning. In an example method, an input prompt comprising a set of tokens is accessed as input to a generative machine learning model. A first key tensor and a first value tensor are generated for a first token of the set of tokens, and the first key tensor and the first value tensor are stored in a memory. A first retention score is generated, for the first token, based on the first key tensor, the first value tensor, and a second token of the set of tokens. The first key tensor and the first value tensor are evicted from the memory in response to determining that the first retention score is a lowest retention score of the memory.