Self-Attention Key-Value Caching for Long-Sequence Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer neural networks face increased processing time as the length of input sequences increases due to the accumulation of generated key-value pairs, leading to inefficiencies in output generation.
Innovation Solution
The implementation of a self-attention layer that stores and utilizes pre-generated key-value pairs from a previous input sequence, utilizing approximate nearest neighbor search to identify relevant pairs for subsequent input sequences, and storing this information in a non-volatile memory like an SSD to facilitate faster processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If the transformer neural network processes longer input sequences by accumulating key-value pairs, then the model can capture more contextual information, but the processing time increases
Solution Approach 1:
The patent pre-generates and stores key-value pairs for common input sequences in an SSD before they are needed for inference. When processing a new input sequence, the system first checks if pre-computed key-value pairs exist and can be reused, avoiding the need to re-generate them and significantly reducing processing time while preserving contextual information
Solution Approach 2:
The patent uses approximate nearest neighbor search to identify and retrieve only the most relevant pre-computed key-value pairs that match the current input sequence, rather than using all stored pairs. This selective retrieval maintains high-quality contextual information while minimizing the amount of data processing required
2Speed
If pre-generated key-value pairs are stored in volatile memory, then fast access is achieved, but data loss occurs when power is interrupted
Solution Approach 1:
The patent introduces an SSD as an intermediary storage device between the volatile memory and permanent storage. The SSD acts as a buffer that maintains pre-generated key-value pairs with fast access characteristics while providing non-volatile persistence, ensuring data is preserved even when power is interrupted
Solution Approach 2:
The system pre-loads key-value pairs into the SSD storage before they are needed for inference tasks. This preliminary action ensures that the data is already in a readily accessible state in non-volatile memory, providing both fast access speeds and data persistence without requiring volatile memory
3Reliability
If all key-value pairs are processed for each input sequence, then complete attention is achieved, but processing efficiency decreases
Solution Approach 1:
The patent extracts and retrieves only the relevant subset of pre-computed key-value pairs that correspond to the current input sequence using approximate nearest neighbor search. This extraction process removes unnecessary data from the processing pipeline, maintaining complete and accurate attention for relevant information while eliminating waste of computational resources on irrelevant pairs
Solution Approach 2:
Instead of processing all possible key-value pairs, the system performs partial action by retrieving and processing only the necessary subset that is relevant to the current input sequence. This partial processing approach maintains attention completeness for relevant data while significantly improving processing efficiency by avoiding excessive computation on unrelated pairs
Data Source
AI summary
According to one embodiment, a first key and a first value are stored in a memory device. Each time a self-attention input is input, a self-attention layer executes the following processing. The self-attention layer generates a query, a second key, and a second value based on the self-attention input. The self-attention layer stores the generated second key and second value into the memory device. The self-attention layer executes a first calculation to acquire an attention score by an inner product of the query and a key matrix including the first key and the second key stored in the memory device. The self-attention layer executes a second calculation to calculate an inner product of the attention score and a value matrix including the first value and the second value stored in the memory device. The self-attention layer outputs a result of the second calculation.