Transformer Memory Striding for Faster Key-Value Vector Reuse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer acceleration devices face reduced operation efficiency due to inefficient storage and retrieval of key-value vectors during multiple iterations, which affects the overall performance.
Innovation Solution
Incorporation of a memory striding circuit that optimizes the reading of key-value vectors by accessing multiple memory blocks sequentially through a striding request, minimizing direct processing load on the processing circuit.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If key-value vectors are stored and read using a conventional scheme, then the transformer can execute iterations, but the operation efficiency is reduced due to redundant storing and reading operations
Solution Approach 1:
The memory striding circuit pre-calculates and stores base addresses of memory blocks in advance, so that during iteration the circuit can directly access cached key-value vectors without performing redundant address calculation and memory access operations, thereby improving operation efficiency and reducing time loss
Solution Approach 2:
The system caches key-value vectors in memory blocks and uses the memory striding circuit to read from the cache rather than re-accessing original computation sources, creating an efficient copy mechanism that eliminates redundant operations while maintaining data integrity
2Loss of energy
If the transformer reuses key-value vectors from previous iterations, then computational redundancy is reduced, but inefficient access schemes increase operation time
Solution Approach 1:
The memory system is divided into multiple memory blocks, each capable of storing key-value vectors. The memory striding circuit sequentially accesses these segmented blocks, enabling efficient reuse of cached vectors from previous iterations while minimizing access time through organized data distribution
Solution Approach 2:
The memory striding circuit continuously reads key-value vectors from cached memory blocks across multiple iterations without interruption or redundant processing, maintaining continuous useful action that reduces both computational redundancy and access time
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A transformer acceleration device may include a memory device including first and second memory blocks respectively storing first and second plurality of cache vectors for first and second plurality of tokens, and a memory striding circuit accessing first and second memory blocks. The memory striding circuit may include a memory block address management circuit storing first and second memory block base addresses for the first and second memory blocks, a target address generation circuit calculating a first target address of the first memory block based on the first memory block base address and a first subblock offset and calculating a second target address of the second memory block based on the second memory block base address and the first subblock offset; and a command issue circuit issuing first and second plurality of memory access commands for first and second target subblock of the first and second target address respectively.