Large language model weight binarization method
By classifying LLM weight columns based on average activation values, the method addresses the performance gap in conventional binarization, enabling efficient deployment of LLMs on mobile devices with reduced memory requirements.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- KOREA ELECTRONICS TECH INST
- Filing Date
- 2024-12-30
- Publication Date
- 2026-07-09
AI Technical Summary
Conventional Large Language Model (LLM) binarization methods using the Hessian matrix do not consider input activations, leading to significant performance differences compared to full-precision models, and existing binarization techniques fail to effectively reduce GPU resource requirements for inference, making it difficult to deploy LLMs on memory-constrained devices like mobile devices.
A method that binarizes LLM weights by using the average activation value as a measure to classify columns into salient and non-salient columns, minimizing performance degradation by reducing weight memory requirements, allowing LLMs to be deployed on mobile devices.
The proposed method minimizes performance degradation and reduces memory footprint, enabling LLMs to operate efficiently on mobile devices while maintaining improved performance compared to existing binarization methods.
Smart Images
Figure KR2024021377_09072026_PF_FP_ABST
Abstract
Description
Large-scale language model weight binarization method
[0001] The present invention relates to the optimization of a Large Language Model (LLM), and more specifically, to a method for binarizing, that is, 1-bit quantizing, the weight matrix of an LLM.
[0002] Since LLM still requires significant GPU resources for inference even with 2-bit quantization applied, a more aggressive quantization technique called binarization is necessary. Binarized LLM enables services in memory-constrained environments.
[0003] Conventional LLM binarization utilizes the Hessian matrix. The Hessian matrix is the second derivative of the weights with respect to the loss, and it captures only the influence of the weights.
[0004] However, LLM response generation consists of the repeated process of multiplying input activations by weights; conventional LLM binarization using Hessian matrices is a method that does not consider input activations, which leads to a problem where there is a significant performance difference compared to full-precision models.
[0005] The present invention has been devised to solve the above-mentioned problems, and the objective of the present invention is to provide a method for binarizing LLM weights using the average activation value as a measure to minimize performance degradation of the LLM while lightweighting the LLM so that it can be run on mobile devices for user information security, etc.
[0006] An LLM weighted binarization method according to an embodiment of the present invention for achieving the above objective comprises: a step of generating a row matrix by listing channel-specific statistical values in an input activation matrix to be operated with a weight matrix in each layer of a Large Language Model (LM); a step of classifying columns of a weight matrix into salient columns and non-salient columns based on the row matrix; and a step of binarizing the non-salient columns in the weight matrix in which the columns are classified.
[0007] The in-channel dimension of the weight matrix can be the same as the in-channel dimension of the input activation matrix.
[0008] The row matrix generation step may involve calculating and listing statistical values of activation values in the in-channel direction from the input activation matrix.
[0009] The channel-specific statistical value can be the absolute average of the channel-specific activation values.
[0010] The classification step may include: a step of dividing the row matrix into blocks of a specified size to generate activation scale matrices; and a first classification step of classifying the columns of the weight matrix into salient columns and non-salient columns using the activation scale matrices.
[0011] The classification step further includes a step of assigning an order according to the magnitude of the matrix values for each generated activation scale matrix; and the first classification step may include a step of matching each activation scale matrix to the corresponding columns of the weight matrix to assign an order of the activation scale matrix values matched to the corresponding columns of the weight matrix; and a second classification step of classifying the columns of the weight matrix into salient columns and non-salient columns based on the order of the columns of the weight matrix.
[0012] The second classification step may be to binarize the non-salient columns and classify the columns of the weight matrix into salient columns and non-salient columns such that the calculated binarization error is minimized.
[0013] The second classification step may be to sequentially classify the columns of the weight matrix into salient columns according to the order assigned to the columns of the weight matrix.
[0014] The number of classifiable salient columns may be limited to a fixed number or less.
[0015] According to another aspect of the present invention, an LLM weight binarization system is provided, comprising: a processor that, in each layer of a Large Language Model (LM), generates a row matrix by listing channel-specific statistical values in an input activation matrix to be operated on with a weight matrix, classifies columns of a weight matrix into salient columns and non-salient columns based on the row matrix, and binarizes non-salient columns in the weight matrix in which columns are classified; and a storage unit that provides storage space required for the processor.
[0016] According to another aspect of the present invention, an LLM weight binarization method is provided, comprising: a step of generating statistical data for an input activation matrix to be operated on with a weight matrix in each layer of a Large Language Model (LM); a step of assigning order to the columns of the weight matrix based on the generated statistical data; a step of classifying the columns of the weight matrix into salient columns and non-salient columns based on the order of the columns of the weight matrix; and a step of binarizing the non-salient columns in the weight matrix in which the columns are classified.
[0017] According to another aspect of the present invention, an LLM weight binarization system is provided, comprising: a processor that, in each layer of a Large Language Model (LM), generates statistical data for an input activation matrix to be operated on with a weight matrix, assigns order to the columns of the weight matrix based on the generated statistical data, classifies the columns of the weight matrix into salient columns and non-salient columns based on the order of the columns of the weight matrix, and binarizes the non-salient columns in the weight matrix in which the columns are classified; and a storage unit that provides storage space required for the processor.
[0018] As described above, according to the embodiments of the present invention, by binarizing the LLM weights using the activation average value, the performance degradation of the LLM can be minimized when lightweighting the LLM so that it can be run on a mobile device for user information security, etc.
[0019] Figure 1. 2-bit weight-only quantization results of OmniQuant (ICLR 2024)
[0020] Fig. 2. Performance of binarization algorithms PB-LLM and BiLL
[0021] Fig. 3. Input activation matrix distribution of the query weight matrix of LLM
[0022] FIG. 4. LLM weighted binarization method according to an embodiment of the present invention
[0023] FIGS. 5-6. Further explanation of FIG. 4
[0024] FIG. 7-8. Simulation results of the LLM weighted binarization method according to an embodiment of the present invention
[0025] FIG. 9. LLM weighting binarization system according to another embodiment of the present invention
[0026] The present invention will be described in more detail below with reference to the drawings.
[0027] For LLM services, user response speed and model performance are critical. In particular, since performance and model size are proportional in LLM, a large model is required for a good service.
[0028] With the recent rise in importance of personal information security, attempts are being made to integrate LLM into mobile devices. However, mobile devices have the disadvantage of limited available memory compared to servers and are unsuitable for the real-time processing of large-scale models.
[0029] When LLM performs inference, reading weights from memory accounts for the largest portion of the inference time. Therefore, if weight memory is reduced to minimize performance degradation, LLM can be installed on mobile devices and provided in real-time.
[0030] The LLM quantization technique for this purpose exhibits a performance collapse at levels below 2-bit, as shown in Fig. 1. At levels below 2-bit, the perplexity is very high compared to FP16 performance, and a lower perplexity is a better value. Fig. 1 shows the 2-bit weight-only quantization results of OmniQuant (ICLR 2024) (https: / arxiv.org / pdf / 2308.13137).
[0031] Therefore, a dedicated algorithm is required for binarization, which is ultra-low bit (less than 2-bit) quantization. Existing binarization algorithms, such as PB-LLM (Partially Binarized Large Language Models) and BiLLM (Pusing the Limit of Post-Training Quantization for LLMs (ICML 2024)), utilize only the Hessian matrix, which is the second derivative of the weights with respect to the loss, but as shown in Figure 2, there is still a significant gap compared to full-precision performance.
[0032] The biggest challenge when lightweighting LLMs is the presence of outliers in the input activation matrix. Compared to CNNs, LLMs have a high distribution of outliers, and the variance between outliers and non-outliers is very large.
[0033] In the input activation matrix of Figure 3, white represents values with large absolute values, and lines are drawn in the in-channel direction. Additionally, it can be seen that the variance of values within the same in-channel is small, while the variance between in-channels is very large.
[0034] In an embodiment of the present invention, taking into account these characteristics, the saliency for each column of the LLM weight matrix is evaluated by referring to the average value of the in-channel direction in the input activation matrix, thereby minimizing the performance degradation of the LLM while lightweighting the LLM and presenting improved performance compared to existing binarization methods.
[0035] FIG. 4 is a diagram illustrating the flow of an LLM weighted binarization method according to an embodiment of the present invention.
[0036] For LLM weight binarization, first, for each layer of the LLM, the absolute average of the activation magnitudes in the in-channel direction is calculated from the input activation matrix, and the calculated absolute averages are arranged to create a row matrix (a matrix consisting of one row) (S110).
[0037] The input activation matrix is the output of the previous layer and becomes the input of that layer, which is then multiplied by the weight matrix of that layer. The input activation matrix is shown in the upper left of Fig. 5, and the row matrix generated in step S110 is shown in the lower left.
[0038] 1) The first matrix value of the row matrix, 0.8, is the absolute average of the activation values of the first channel (first column) of the input activation matrix, and
[0039] 2) The second matrix value of the row matrix, 6.7, is the absolute average of the activation values of the second channel (second column) of the input activation matrix, and
[0040] 3) The third matrix value of the row matrix, 1.4, is the absolute average of the activation values of the third channel (third column) of the input activation matrix, and
[0041] ...
[0042] 9) The ninth matrix value of the row matrix, 1.5, is the absolute average of the activation values of the ninth channel (ninth column) of the input activation matrix.
[0043] As such, the row matrix generated in step S110 can be described as a matrix listing channel-specific statistical values (absolute average in the embodiment of the present invention) from the input activation matrix.
[0044] Next, the row matrix is divided into specified block sizes to generate activation scale matrices (S120). According to the bottom of Fig. 5, the row matrix is divided into blocks of size 3, in which case the following three activation scale matrices are generated.
[0045] [0.8, 6.7, 1.4], [1.2, 0.4, 0.7], [5.9, 0.3, 1.5]
[0046] The generated activation scale matrices are processed one by one, and the first activation scale matrix to be processed first [0.8, 6.7, 1.4] is shown in the lower right corner of Fig. 5.
[0047] Subsequently, for each activation scale matrix generated in step S120, an order is assigned according to the magnitude of the matrix values (S130). In the lower right of Fig. 5, for the first activation scale matrix [0.8, 6.7, 1.4],
[0048] 1) Since the size of the second matrix value is the largest, order ① is indicated below that matrix value, and
[0049] 2) Since the size of the third matrix value is the second largest, order ② is indicated below the third column of that matrix value, and
[0050] 3) Since the size of the first matrix value is the smallest, order ③ is indicated below that matrix value.
[0051] Meanwhile, as shown in FIG. 5, the in-channel dimension of the weight matrix is the same as the in-channel dimension of the input activation matrix. Therefore, the activation scale matrices obtained by dividing the row matrix generated from the input activation matrix can be matched to the columns of the weight matrix.
[0052] Accordingly, each activation scale matrix is matched to the corresponding columns of the weight matrix, and the order of the activation scale matrix values matched to the corresponding columns of the weight matrix is assigned as is (S140), and based on the order of the columns of the weight matrix, the columns of the weight matrix are classified into salient columns and non-salient columns (S150).
[0053] On the right side of Fig. 5, the result of matching the first activation scale matrix to the 1st, 2nd, and 3rd columns of the weight matrix and the result of assigning the same order accordingly is shown.
[0054] 1) Below the second column of the weight matrix, the order ① of the second matrix value of the corresponding activation scale matrix is indicated, and
[0055] 2) Below the third column of the weight matrix, the order ② of the third matrix value of the corresponding activation scale matrix is indicated, and
[0056] 3) Below the first column of the weight matrix, the order ③ of the first matrix value of the corresponding activation scale matrix is indicated.
[0057] Meanwhile, in the upper part of Fig. 6, only the first to third columns of the weight matrix are shown separately, and the assigned order is indicated immediately below.
[0058] Classification of the columns of the weight matrix is, of Fig. 6<Case 1> As shown in Fig. 6, first, the column with the earliest assigned order among the columns of the weight matrix is classified as a salient column (①), and the rest are classified as non-salient columns (②, ③). Then, the non-salient columns (②, ③) are binarized, and the binarization error of the columns of the weight matrix is calculated, assuming that the binarization error is calculated to be 0.03 in Fig. 6.
[0059] The following Fig. 6<Case 2> As shown in Fig. 6, the two columns of the weight matrix with the earliest and second earliest assigned order are classified as salient columns (①, ②), and the rest are classified as non-salient columns (③). Then, the non-salient columns (③) are binarized, and the binarization error of the columns of the weight matrix is calculated, assuming that the binarization error is calculated to be 0.23 in Fig. 6.
[0060] Meanwhile, in Fig. 6, the number of salient columns is limited to 2, which is the limit value determined by the assigned order, so only up to 2 columns of the weight matrix can be classified as salient columns.
[0061] The above 2 types that can be classified thereafter<Case 1,2> Among them, the one with the smallest binarization error<Case 1> Select a classification based on the order. That is, classify only the column with the earliest assigned order as the salient column (①) and the rest as non-salient columns (②,③).
[0062] When classification is completed through this process, only the columns of the weight matrix classified as non-salient columns are binarized (S160).
[0063] Figures 7 and 8 are simulation results of the LLM weighted binarization method according to an embodiment of the present invention. As shown, the LLM weighted binarization method according to an embodiment of the present invention utilizes the activation average value, which reflects the activation outlier characteristics raised through Figure 3, as a criterion for determining saliency of the weight columns in the weight matrix, thereby confirming the performance improvement of the LLaMA2 model in the C4, PTB, and WikiText2 datasets.
[0064] FIG. 9 is a diagram illustrating the configuration of an LLM weighting binarization system according to another embodiment of the present invention. The LLM weighting binarization system according to an embodiment of the present invention can be implemented as a computing system comprising a communication unit (210), an output unit (220), a processor (230), an input unit (240), and a storage unit (250) as illustrated.
[0065] The communication unit (210) is a communication interface for connecting to an external network or external device, and establishes a communication connection with a mobile device equipped with a binary LLM. The output unit (220) is an output means for displaying the result of an operation performed by the processor (230), and the input unit (240) is a user interface that receives user commands and transmits them to the processor (230).
[0066] The processor (230) binarizes the weight matrix for each layer of the LLM according to the procedure illustrated in FIG. 4 above. The storage unit (250) provides the storage space necessary for the processor (230) to function and operate.
[0067] Up to now, the LLM weighted binarization method has been described in detail with reference to preferred embodiments.
[0068] In the above embodiment, by binarizing the LLM weights using the average activation value, the performance degradation of the LLM can be minimized when lightweighting the LLM to enable operation on mobile devices for user information security, etc.
[0069] Meanwhile, it goes without saying that the technical concept of the present invention may also be applied to a computer-readable recording medium containing a computer program that enables the device and method according to the present embodiment to perform their functions. Furthermore, the technical concept according to various embodiments of the present invention may be implemented in the form of computer-readable code recorded on a computer-readable recording medium. A computer-readable recording medium may be any data storage device that can be read by a computer and store data. For example, a computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, etc. Additionally, computer-readable code or a program stored on a computer-readable recording medium may be transmitted through a network connected between computers.
[0070] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.
Claims
1. A step of generating a row matrix in each layer of the LLM (Large Language Model) by listing channel-specific statistical values in the input activation matrix to be operated with the weight matrix; A step of classifying the columns of a weight matrix into salient columns and non-salient columns based on the row matrix; and An LLM weighted binarization method characterized by including the step of binarizing non-salient columns in a weight matrix in which columns are classified.
2. In Claim 1, The in-channel dimension of the weight matrix is, An LLM weighted binarization method characterized by being identical to the in-channel dimension of the input activation matrix.
3. In Claim 1, The row matrix generation step is, An LLM weighted binarization method characterized by calculating and listing statistical values of activation values in the in-channel direction in an input activation matrix.
4. In Claim 3, The statistical values by channel are, An LLM weighted binarization method characterized by being the absolute average of activation values per channel.
5. In Claim 1, The classification stage is, A step of dividing a row matrix into blocks of a specified size to generate activation scale matrices; An LLM weighted binarization method characterized by including a first classification step of classifying columns of a weight matrix into salient columns and non-salient columns using activation scale matrices.
6. In Claim 5, The classification stage is, For each generated activation scale matrix, the method further includes the step of assigning an order according to the magnitude of the matrix values; The first classification stage is, A step of matching each activation scale matrix to the corresponding columns of the weight matrix and assigning an order to the activation scale matrix values matched to the corresponding columns of the weight matrix; An LLM weighted binarization method characterized by including a second classification step of classifying the columns of a weight matrix into salient columns and non-salient columns based on the order of the columns of the weight matrix.
7. In Claim 6, The second classification stage is, An LLM weighted binarization method characterized by binarizing non-salient columns and classifying columns of a weight matrix into salient columns and non-salient columns such that the calculated binarization error is minimized.
8. In Claim 7, The second classification stage is, An LLM weighted binarization method characterized by sequentially classifying the columns of a weight matrix into salient columns according to the order assigned to the columns of the weight matrix.
9. In Claim 8, The number of classifiable salient columns is, LLM weighted binarization method characterized by being limited to a fixed number or less.
10. A processor that, in each layer of the LLM (Large Language Model), generates a row matrix by listing channel-specific statistical values from the input activation matrix to be operated on with the weight matrix, classifies the columns of the weight matrix into salient columns and non-salient columns based on the row matrix, and binarizes the non-salient columns in the weight matrix in which the columns are classified; and An LLM weighted binarization system characterized by including a storage unit that provides storage space required for a processor.
11. A step of generating statistical data for the input activation matrix to be operated on with the weight matrix in each layer of the LLM (Large Language Model); A step of assigning order to the columns of a weight matrix based on generated statistical data; A step of classifying the columns of a weight matrix into salient columns and non-salient columns based on the order of the columns of the weight matrix; An LLM weighted binarization method characterized by including the step of binarizing non-salient columns in a weight matrix in which columns are classified.
12. A processor that, in each layer of the LLM (Large Language Model), generates statistical data for an input activation matrix to be operated on with a weight matrix, assigns order to the columns of the weight matrix based on the generated statistical data, classifies the columns of the weight matrix into salient columns and non-salient columns based on the order of the columns of the weight matrix, and binarizes the non-salient columns in the weight matrix in which the columns are classified; and An LLM weighted binarization system characterized by including a storage unit that provides storage space required for a processor.