Sparse Dictionary Execution for Low-Latency ML Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative artificial intelligence models are computationally expensive due to the need for multiple passes through large language models, leading to increased computational resource usage and latency, particularly in memory-bandwidth-limited environments.
Innovation Solution
Decompose the weight matrix of machine learning models into a dictionary and coefficient matrix, where the dictionary is stored in on-processor memory and coefficient matrices are swapped in as needed, reducing the amount of data transferred between memory types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple passes through large language models are performed to generate each token in a response, then the accuracy and quality of the generated response is improved, but the computational resource usage and latency increase significantly
Solution Approach 1:
The weight matrix W is segmented into two separate matrices: a dictionary matrix D and a coefficient matrix A. This segmentation allows the system to pre-compute and store the dictionary matrix in on-processor memory, while only loading the smaller coefficient matrix from off-processor memory during inference. The segmentation reduces memory access latency and computational overhead while maintaining the accuracy of the original model.
2Adaptability or versatility
If the number of parameters within the large language model is increased to improve model capability, then the model's intelligence and performance are improved, but the computational resource expense and memory requirements increase
Solution Approach 1:
The invention extracts the dictionary matrix D from the full weight matrix W and stores it permanently in on-processor memory. During inference, only the coefficient matrix A needs to be loaded from off-processor memory, effectively 'taking out' the frequently accessed portion of the model parameters and keeping them readily available. This reduces the computational resource expense associated with repeated memory access while preserving the full model capability.
3Productivity
If coefficient matrices are frequently swapped in and out of memory during inference, then the model can process different portions of the input, but the data transfer overhead between memory types increases latency
Solution Approach 1:
The dictionary matrix D is pre-computed and loaded into on-processor memory before the inference process begins. This preliminary action ensures that the most frequently accessed portion of the model parameters is already in fast memory, eliminating the need for repeated data transfer between memory types during inference. The preliminary loading of D significantly reduces memory access latency and improves overall throughput.
Data Source
AI summary
Techniques and apparatus for efficiently executing inferencing operations using machine learning models are provided. An example method generally includes loading, into a first memory, a dictionary associated with at least a portion of a machine learning model and a first coefficient matrix associated with a first portion of the machine learning model. A first intermediate output associated with the first portion of the machine learning model is generated based on an input into the at least the portion of the machine learning model, the dictionary, and the first coefficient matrix. The first and second coefficient matrices are swapped between the first and second memories. A second intermediate output associated with the second portion of the machine learning model is generated based on the input into the at least the portion of the machine learning model, the first intermediate output, the dictionary, and the second coefficient matrix.