LLM Key-Value Cache Optimization With Cross-Layer Attention Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The key value cache in large language model (LLM) inference becomes a bottleneck as memory usage scales with sequence length, leading to increased latency and decreased throughput due to caching of more tokens and reduced parallel requests fitting in memory.
Innovation Solution
Implementing multi-query attention, cross-layer KV sharing, and hybrid attention mechanisms to optimize key value cache size, including shared key and value vectors across transformer layers and balancing local and global attention to reduce memory usage and computation time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the LLM maintains a cache of all previously computed keys and values to generate tokens one at a time, then the model can accurately infer outputs based on input data, but memory usage scales with sequence length causing latency to increase and throughput to drop
Solution Approach 1:
The patent extracts only the necessary key-value pairs from the full cache by using query-aware filtering. Instead of caching all previous tokens, the system identifies and stores only the key-value pairs that are actually needed for accurate inference, thereby reducing memory usage while maintaining inference accuracy
Solution Approach 2:
The patent applies different caching strategies to different parts of the sequence based on their importance. Recent tokens are cached with higher priority while older tokens are filtered out, creating a quality-based local structure in the cache that balances accuracy requirements with memory constraints
2Reliability
If the LLM caches more tokens to maintain inference accuracy, then output quality is preserved, but latency increases due to larger cache size
Solution Approach 1:
The system extracts and stores only the essential key-value pairs needed for accurate inference rather than caching all tokens. This selective extraction reduces the cache size and associated latency while preserving output quality through query-aware filtering that identifies necessary historical information
3Reliability
If the LLM uses standard attention mechanisms to process sequences, then the model can accurately process input data, but memory usage and computation time increase with sequence length
Solution Approach 1:
The patent extracts and stores only the necessary key-value pairs from the full attention cache. By using query-aware filtering, the system identifies which historical key-value pairs are actually needed for accurate processing and stores only those, significantly reducing memory usage while maintaining processing accuracy
Solution Approach 2:
Instead of caching all possible key-value pairs (excessive action), the system caches only the necessary subset (partial action) determined by query awareness. This partial caching approach reduces memory complexity while maintaining sufficient processing accuracy for the given tasks
Data Source
AI summary
An input sequence is received from a client device. Large language model inference is performed by processing the input sequence through a series of transformer layers to generate one or more tokens including by performing hybrid attention, multi-query attention, and cross-layer key value sharing. The one or more generated tokens are provided to the client device.