Transformer KV Cache Memory Striding for Faster Token Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer acceleration devices face reduced operation efficiency due to suboptimal storage and retrieval of key-value vectors during multiple iterations, leading to increased processing loads.
Innovation Solution
Incorporation of a memory striding circuit that optimizes access to memory blocks, allowing sequential reading of key-value vectors stored in separate memory blocks, thereby reducing the processing load on the transformer acceleration device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the transformer stores and reads key-value vectors in memory during multiple iterations, then the transformer can reuse computed key-value vectors to reduce calculation load, but the operation efficiency is reduced due to suboptimal storage and retrieval schemes
Solution Approach 1:
The memory device is divided into multiple memory blocks, each storing key-value vectors for specific tokens. This segmentation allows the memory striding circuit to access only the relevant blocks needed for current computation, reducing unnecessary memory access time and improving overall operation efficiency.
Solution Approach 2:
Key-value vectors are pre-computed and stored in memory blocks during previous iterations or preprocessing. The memory striding circuit retrieves these pre-stored vectors during current iterations, avoiding redundant computations and reducing processing time for repeated operations.
2Power
If the transformer reuses key-value vectors computed in previous iterations, then calculation load is reduced, but processing load increases due to inefficient key-value vector retrieval
Solution Approach 1:
The memory striding circuit acts as an intermediary between the transformer computation units and the memory storage blocks. It manages the retrieval process by sequentially accessing memory blocks in an optimized manner, simplifying the complexity of retrieving reused key-value vectors while maintaining efficient calculation load management.
3Productivity
If key-value vectors are stored in separate memory blocks, then memory access can be optimized for specific token sequences, but the complexity of managing multiple memory blocks increases
Solution Approach 1:
The memory striding circuit autonomously manages the sequential access to multiple memory blocks based on token sequence requirements. It automatically determines which blocks to access and in what order, eliminating the need for complex external memory management control and reducing overall system complexity while maintaining optimized memory access patterns.
Data Source
AI summary
A transformer acceleration device may include a memory device including first and second memory blocks respectively storing first and second plurality of cache vectors for first and second plurality of tokens, and a memory striding circuit accessing first and second memory blocks. The memory striding circuit may include a memory block address management circuit storing first and second memory block base addresses for the first and second memory blocks, a target address generation circuit calculating a first target address of the first memory block based on the first memory block base address and a first subblock offset and calculating a second target address of the second memory block based on the second memory block base address and the first subblock offset; and a command issue circuit issuing first and second plurality of memory access commands for first and second target subblock of the first and second target address respectively.