LLM Query-Response Cache Replacement Using Semantic Diversity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Caching mechanisms for large language models (LLMs) face challenges in managing cache size and latency due to the storage of semantically similar queries, leading to inefficient resource utilization and increased latency.
Innovation Solution
Implement a cache replacement mechanism that prioritizes semantic diversity by selecting query-response pairs with minimal semantic distance for removal when the cache exceeds a threshold, using utility scores that consider frequency, cost, latency, and size to optimize cache content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the cache stores all query-response pairs, then the cache hit rate is improved, but the cache size grows to an unwieldy size and memory consumption increases
Solution Approach 1:
The patent changes the parameter of cache selection from simple frequency-based to semantic diversity-based. By computing semantic similarity scores between queries and selecting for eviction the query with minimum semantic diversity, the system maintains a cache that is both size-constrained and high-performance, resolving the contradiction between cache size and hit rate.
2Reliability
If the cache size is increased to store more query-response pairs, then the cache hit rate is improved, but the latency in performing cache searches increases
Solution Approach 1:
The patent changes the cache management parameter from unbounded growth to bounded size with semantic diversity constraints. By maintaining a fixed maximum size and using semantic similarity metrics for eviction decisions, the system achieves high cache hit rates without the linearly increasing search latency that would result from unbounded cache growth.
3Quantity of substance
If the cache prunes query-response pairs frequently to maintain size constraints, then memory consumption is reduced, but the cache hit rate deteriorates
Solution Approach 1:
The patent changes the eviction parameter from simple LRU or random selection to semantic diversity-based selection. By computing semantic similarity between queries and evicting only those with minimum semantic diversity to the existing cache, the system maintains memory consumption within constraints while preserving high cache hit rates through intelligent, semantics-aware eviction.
Solution Approach 2:
The patent implements feedback through semantic similarity computation. Before evicting a query, the system evaluates its semantic relationship with remaining cache entries, ensuring that eviction decisions are informed by the actual content relationships. This feedback mechanism allows the system to maintain high hit rates while adhering to memory constraints.
4Quantity of substance
If the cache stores semantically similar queries, then the cache size is reduced, but the semantic diversity and usefulness of the cache deteriorates
Solution Approach 1:
The patent changes the eviction parameter to explicitly optimize for semantic diversity. By computing semantic similarity scores and selecting for eviction the query with the minimum semantic diversity to existing entries, the system ensures that cache size reduction does not compromise semantic diversity. Instead, the cache maintains a diverse set of queries that maximize adaptability and usefulness.
Data Source
AI summary
In one implementation, a device stores a plurality of query-response pairs of queries issued to a language model and their corresponding answers from the language model in a cache. The device determines that the cache should be pruned based on a size of the cache exceeding a threshold size. The device selects a particular query-response pair from amongst the query-response pairs based on that pair having a minimal semantic distance to another query-response pair in the plurality of query-response pairs. The device prunes the particular query-response pair from the cache.