Attention Cache Reuse for Large Language Model Dialogs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) experience latency in responding to user inputs due to computationally expensive operations like attention, which can be time-intensive and memory-bound, especially in systems requiring multiple dialog turns.
Innovation Solution
Implement attention caching in a stateful manner by preserving encoded representations of prompts across dialog turns and storing them on multiple GPUs, allowing retrieval and reuse for subsequent dialog turns, thereby reducing the need for repeated computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If attention operations are performed for each dialog turn, then processing accuracy is maintained, but latency increases and computational resources are consumed
Solution Approach 1:
The patent applies preliminary action by encoding and caching prompt representations before they are needed for generation. The system encodes the prompt once and stores the encoded representation in a cache, so that during subsequent dialog turns, the pre-encoded prompt can be retrieved without re-computation, significantly reducing latency while maintaining processing accuracy
Solution Approach 2:
The patent implements copying by creating and storing encoded copies of prompt representations in a cache. Instead of re-encoding the same prompt multiple times across dialog turns, the system makes copies of the encoded prompt data and stores them for rapid retrieval, eliminating redundant computational operations while preserving the original processing quality
2Reliability
If attention operations are performed for each dialog turn, then processing completeness is ensured, but computational resources are excessively consumed
Solution Approach 1:
The system performs the computationally intensive prompt encoding operation in advance and caches the results. This preliminary action ensures that all necessary processing is completed beforehand, maintaining processing completeness while avoiding repeated execution of the same operations that would consume excessive computational resources during actual dialog turns
Solution Approach 2:
The patent creates cached copies of encoded prompt representations that can be reused across multiple dialog turns. This copying approach ensures processing completeness is maintained (all necessary computations are done) while dramatically reducing computational resource consumption by eliminating redundant encoding operations
3Speed
If cached data is stored on multiple GPUs, then retrieval speed is improved, but system complexity increases
Solution Approach 1:
The patent segments the cached prompt data across multiple GPUs, dividing the storage burden and enabling parallel retrieval operations. Each GPU can independently store and provide data, allowing the system to achieve faster retrieval speeds through parallel access while managing complexity through clear segmentation of data responsibilities across devices
Data Source
AI summary
Techniques for cache management for reducing latency in LLM inferencing are described. In some embodiments, a system caches encoded data of portions of a prompt so that the encoded data is available for use by the LLM across dialog turns of a dialog session. Within a dialog session, a portion of the LLM prompt may be the same across dialog turns, and instead of recomputing the attention/encodings for such portions, the cached encodings can be used by the LLM during processing. In some embodiments, user inputs for the dialog session may be routed to the same LLM container and encoded data for the dialog session may be stored at the same cache associated with the LLM. In some embodiments, the system enables asynchronous prompt encoding while performing ASR processing.