AI Model Dequantization Layout for GPU Bank Conflict Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face challenges in being uploaded to graphic processing units (GPUs) due to high memory requirements, leading to increased resource consumption and delays when multiple processors are used, and quantized weight values cause bank conflicts in shared memory, slowing down storage speed.
Innovation Solution
A method and apparatus for efficiently storing quantized weights by rearranging them in a specific pattern that avoids bank conflicts in shared memory, using a dequantization process to convert quantized weight values to a higher bit precision and rearranging their order to minimize memory access conflicts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If quantized weight values are stored in shared memory, then memory usage is reduced, but bank conflicts occur and storage speed decreases
Solution Approach 1:
The patent applies preliminary action by pre-rearranging quantized weight values according to a dequantization pattern before storing them in shared memory. This advance preparation ensures that when weights are loaded into registers during matrix multiplication, they can be directly dequantized without causing bank conflicts, thus maintaining both memory efficiency and storage speed.
Solution Approach 2:
The patent changes the dimensional organization of weight values by applying a dequantization pattern that rearranges elements across different dimensions (rows and columns). This dimensional transformation allows weights to be stored in a pattern that avoids bank conflicts while maintaining the benefits of quantization for reduced memory usage.
2Power
If multiple processors are used to handle large language models, then processing capability is improved, but resource consumption increases and delays occur
Solution Approach 1:
The patent changes the parameter of weight representation from high-precision to quantized low-precision format, which reduces the memory bandwidth required when transferring weights between processors and shared memory. This parameter change allows multiple processors to work on large language models with reduced resource consumption and fewer synchronization delays.
3Quantity of substance
If quantized weight values are used directly, then memory efficiency is improved, but dequantization operations become complex
Solution Approach 1:
The patent applies preliminary action by pre-organizing quantized weight values according to the dequantization pattern before they are loaded into registers. This advance arrangement simplifies the actual dequantization operation during matrix multiplication, as the weights are already positioned correctly and can be dequantized in a straightforward manner without complex runtime operations.
Data Source
AI summary
The disclosure relates to a method and an apparatus for lightweighting of artificial intelligence models, and a method of performing matrix multiplication of weight values and input values of artificial intelligence models includes copying quantized weight values stored in a global memory to a register, dequantizing the quantized weight values, copying an input value matrix to the register, and performing matrix multiplication between a dequantized weight value matrix and the input value matrix.