Key-Value Vector Caching for Repeated LLM Prompt Prefixes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face high computational costs due to expensive attention computations, particularly for key-value vectors generated from repeated data patterns in prompts, leading to inefficient inferencing.
Innovation Solution
Implement a caching policy that selectively stores key-value vectors for frequently occurring prompt prefixes based on a temporal moving average, using a smart encoding policy and machine learning models to predict token generation, reducing the need for repeated computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If attention computations are performed for every prompt to generate key-value vectors, then response accuracy is maintained, but computational cost and time consumption increase significantly
Solution Approach 1:
The system performs attention computations and generates key-value vectors in advance for prompt prefixes that meet the temporal moving average threshold. These pre-computed key-value vectors are stored in a cache, allowing subsequent prompts with the same prefix to reuse the cached results without repeating the expensive attention computations, thus improving inferencing speed while maintaining accuracy
Solution Approach 2:
The system creates copies of key-value vectors generated from prompt prefixes and stores them in a cache. When a new prompt arrives, the system checks if a matching prefix exists in the cache and reuses the copied key-value vectors instead of regenerating them, significantly reducing computational overhead while preserving the accuracy that would be achieved through full recomputation
2Productivity
If key-value vectors are cached for all prompt prefixes, then inferencing speed improves, but memory consumption increases
Solution Approach 1:
The system changes the parameter of caching eligibility by introducing a temporal moving average threshold. Only prompt prefixes that exceed this threshold (indicating frequent occurrence) are eligible for caching. This selective approach ensures that cache memory is allocated to the most beneficial cases, improving inferencing speed for common prefixes while limiting memory consumption by excluding infrequent prefixes
3Productivity
If temporal moving average threshold is set low, then more key-value vectors are cached improving speed, but memory consumption increases
Solution Approach 1:
The temporal moving average threshold serves as a controllable parameter that balances cache size and inferencing speed. By adjusting this threshold, the system can optimize the trade-off: a lower threshold caches more key-value vectors improving speed but increasing cache size, while a higher threshold reduces cache size but caches fewer vectors. The threshold allows flexible adaptation to available memory resources
Data Source
AI summary
One or more systems, devices, computer program products and/or computer-implemented methods of use provided herein relate to caching key-value vectors for fixed prefixes according to a caching policy. A system can comprise a memory that can store computer-executable components. The system can further comprise a processor that can execute the computer-executable components stored in the memory, wherein the computer-executable components can comprise an accessing component that can access a request comprising a fixed prefix and variable data. The system can further comprise a storage component that can selectively store in a cache, one or more key-value vectors generated via processing of the fixed prefix by a large language model (LLM), based on a temporal moving average of the fixed prefix being greater than a first defined threshold, according to a smart encoding policy.