ML Weight Rounding with Expanded Quantization Degrees of Freedom
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing ML models, particularly large language models, face challenges in reducing memory footprint while maintaining accuracy due to the use of 32-bit floating-point values, and existing rounding techniques like AdaRound and LoRA-Rounding are limited in expressivity and resource-intensive.
Innovation Solution
An enhanced rounding technique that utilizes a set of learned matrices to generate expanded matrices constrained to 0 and 1, which are then multiplied by positive and negative factors to create an output matrix with values ranging from -N to N, allowing for larger degrees of freedom in quantization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 32-bit floating-point values are used in ML models, then computational accuracy is maintained, but memory footprint becomes large
Solution Approach 1:
The patent changes the parameter precision from 32-bit floating-point to 8-bit quantized values, fundamentally altering the data representation to reduce memory requirements while maintaining acceptable computational accuracy through carefully designed quantization strategies
Solution Approach 2:
The patent segments the weight matrix into multiple components (weight matrix W, rounding matrix V, and correction matrix E) to enable fine-grained control over quantization, allowing different parts of the model to be quantized at different precision levels
2Quantity of substance
If rounding is applied to reduce precision, then memory footprint is reduced, but accuracy/precision is lost
Solution Approach 1:
The patent performs preliminary rounding to create a quantized weight matrix, then applies a correction mechanism using the rounding matrix and correction matrix to compensate for the accuracy loss, effectively preparing the model for quantization while mitigating its immediate impact on precision
Solution Approach 2:
The patent implements a feedback mechanism where the rounding matrix and correction matrix are trained to predict and compensate for quantization errors, allowing the model to learn from the accuracy loss and adjust subsequent computations to maintain precision
3Ease of manufacture
If existing rounding techniques (AdaRound, LoRA-Rounding) are used, then quantization is enabled, but expressivity is limited and resources are intensive
Solution Approach 1:
The patent introduces dynamic rounding control where the rounding matrix and correction matrix can be adjusted based on the specific computational task and hardware constraints, enabling the model to adapt its quantization strategy to maximize expressivity for different applications
Solution Approach 2:
The patent adds new dimensions to the quantization process by introducing separate rounding and correction matrices that operate in addition to the weight matrix, creating a multi-component representation that significantly increases the model's expressivity compared to traditional single-matrix approaches
Data Source
AI summary
Systems and techniques are described herein for adjusting weights of a machine learning (ML) model. For instance, a process can include generating a first matrix of quantized weight values by rounding values of an input matrix of weight values for the ML model; applying an activation function to a second matrix, the second matrix generated based on a third matrix and a fourth matrix of a first matrix pair; applying the activation function to a fifth matrix, the fifth matrix based on a sixth matrix and seventh matrix of a second matrix pair; generating a positive second matrix by applying a positive factor to the second matrix; generating a negative fifth matrix by applying a negative factor to the fifth matrix; and summing the first matrix of quantized weight values with the positive second matrix and the negative fifth matrix to generate an output matrix of quantized weight values.