Weight Memory Mapping for Low-Latency Transformer Matrix Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative artificial intelligence models, particularly those with a transformer structure, suffer from high latency due to limited parallelism of operations, necessitating an efficient weight memory mapping method to reduce hardware latency.
Innovation Solution
A weight memory mapping system and method that includes a weight memory, input register, and hardware operators for matrix multiplication operations, utilizing lane-level final sums and partial sum reuse to accelerate processing, and preprocessing operations using computed final sums to reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transformer structure is used for generative AI models, then model performance is improved, but hardware latency increases due to limited parallelism of operations
Solution Approach 1:
The patent segments the weight matrix into multiple column groups and processes them through multiple hardware operators in parallel. Each hardware operator handles a specific column group, allowing simultaneous computation of multiple partial sums. This segmentation enables the system to maintain transformer model performance while reducing hardware latency by increasing operational parallelism through divided computation paths.
2Device complexity
If conventional weight mapping method is used, then hardware structure is simple, but computational throughput is low and resource efficiency is poor
Solution Approach 1:
The patent implements preliminary action by pre-processing weight data before it enters the hardware operators. The weight data is organized into column groups and prepared in advance, allowing the hardware operators to immediately begin computation without waiting for data preparation. This preliminary organization of weight matrices enables higher computational throughput while maintaining efficient resource utilization, as the hardware structure remains relatively simple but operates at optimized speeds.
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
Provided are a weight memory mapping method and a system for a streaming operation of massively generated artificial intelligence hardware. A weight memory mapping system according to an embodiment may comprise: a weight memory for storing a weight matrix for a pre-trained artificial intelligence model; an input register for storing a plurality of pieces of input data; a first hardware operator for processing a matrix multiplication operation between the plurality of pieces of input data and the weight matrix, and calculating a final sum in lane units during the matrix multiplication operation by reusing a partial sum of the matrix multiplication operation; and a second hardware operator for processing, in advance, a next matrix multiplication operation during the matrix multiplication operation by using the final sum.