Transformer Acceleration With Segmented Memory Banks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transformer acceleration device experiences bottlenecks due to inefficient data storage and retrieval methods, leading to suboptimal performance in generating output tokens based on input tokens.
Innovation Solution
The device employs a memory bank array system to store weight matrices and key-value vector pairs for decoder layers, allowing for optimized data reuse across iterations, reducing calculation redundancy and minimizing bank collisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the transformer stores and reuses data from previous iterations in a conventional manner, then data can be accessed for iterative operations, but a bottleneck phenomenon occurs due to inefficient storing and reading of data to be reused
Solution Approach 1:
The memory system is segmented into multiple independent memory banks (first memory bank, second memory bank, third memory bank) that can operate simultaneously. Different types of data (weight matrices, key-value vector pairs, activation vectors) are stored in different banks, allowing parallel access without bottlenecks. This segmentation resolves the contradiction by enabling multiple data retrieval operations to occur concurrently, thus improving productivity while minimizing time loss.
Solution Approach 2:
The patent introduces a new dimension of parallelism by organizing memory access across multiple banks rather than sequential access within a single bank. By distributing data across multiple memory banks and enabling simultaneous read/write operations on different banks, the system achieves dimensional expansion in the memory access space, thereby improving operational speed without creating bottlenecks.
2Loss of energy
If the transformer reuses data from previous iterations, then calculation redundancy is reduced, but data access conflicts and bank collisions increase without optimized storage
Solution Approach 1:
The memory system is divided into specialized segments: first memory bank for weight matrices, second memory bank for key-value vector pairs, and third memory bank for activation vectors. This segmentation allows each bank to be optimized for its specific data type, reducing access conflicts while enabling efficient data reuse. The segmentation reduces calculation redundancy by making frequently accessed data readily available without increasing overall system complexity.
Solution Approach 2:
The patent introduces memory banks as intermediary structures between the processor and the data. These intermediaries buffer and organize data in optimal formats, reducing the need for repeated calculations. The memory banks act as mediators that manage data flow efficiently, allowing the processor to reuse data without directly managing the complexity of data storage and retrieval, thus reducing calculation redundancy without proportionally increasing device complexity.
3Reliability
If multiple weight matrices and key-value vector pairs are stored for multiple decoder layers, then transformer functionality is maintained, but memory access conflicts increase
Solution Approach 1:
The patent segments the storage of weight matrices and key-value vector pairs across multiple specialized memory banks. The first memory bank stores weight matrices for all decoder layers, while the second memory bank stores key-value vector pairs. This segmentation maintains complete transformer functionality by ensuring all necessary data is stored, while simultaneously improving data access efficiency by allowing parallel access to different banks without conflicts.
Solution Approach 2:
The patent implements preliminary organization of data in memory banks before processing. Weight matrices and key-value vector pairs are pre-stored in optimized locations in the first and second memory banks respectively. This preliminary action ensures that when the transformer needs to access this data during operation, it can do so efficiently without encountering access conflicts, thus maintaining reliability while improving productivity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A transformer acceleration device configured to execute a transformer comprising a first plurality of decoder layers and a second plurality of decoder layers is disclosed. The transformer acceleration device comprises a first memory bank array configured to store a first plurality of weight matrices corresponding to the first plurality of decoder layers and a first plurality of key-value vector pairs corresponding to the second plurality of decoder layers, and a second memory bank array configured to store a second plurality of weight matrices corresponding to the second plurality of decoder layers, and a second plurality of key-value vector pairs corresponding to the first plurality of decoder layers.