Weighted Low-Rank Factorization for Machine Learning Model Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models, such as deep neural networks, with numerous parameters face challenges in deployment on resource-constrained devices due to high computational complexities, memory usage, and power consumption, making them unsuitable for smartphones, smartwatches, and other devices.
Innovation Solution
The method employs weighted low-rank factorization to compress machine learning models by generating factorized matrices based on importance values, reducing the size of linear layers and generating a compressed model with improved accuracy, which can be deployed on resource-constrained devices without significant loss of performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models with numerous parameters are used, then model accuracy is improved, but computational complexity and memory usage increase
Solution Approach 1:
The patent segments the original large parameter matrix into multiple smaller factor matrices through low-rank factorization. Specifically, a parameter matrix of size M×N is decomposed into two factor matrices of sizes M×K and K×N where K<M and K<N, effectively breaking down the complex model into simpler components that can be processed more efficiently on resource-constrained devices
Solution Approach 2:
The patent changes the parameter representation from dense full-rank matrices to sparse low-rank factorized form. By representing parameters as products of factor matrices with reduced rank, the model maintains its functional capability while significantly reducing the number of parameters that need to be stored and computed, thus lowering computational complexity
2Measurement precision
If machine learning models with numerous parameters are used, then model accuracy is improved, but memory usage increases
Solution Approach 1:
The parameter matrix is segmented into multiple factor matrices that can be stored separately. Instead of storing one large M×N matrix requiring MN memory units, the system stores two smaller matrices of sizes M×K and K×N, requiring MK+KN memory units, which is significantly less when K is much smaller than M and N
Solution Approach 2:
The patent transforms the parameter storage format from dense to sparse low-rank representation. This parameter change enables the model to maintain accuracy while using fewer memory resources by exploiting the low-rank structure inherent in many machine learning parameter matrices
3Measurement precision
If machine learning models with numerous parameters are used, then model accuracy is improved, but power consumption increases
Solution Approach 1:
By segmenting the computation into matrix multiplication of smaller factor matrices, the patent reduces the total number of computational operations required. The computational burden is distributed across smaller, more efficient operations that consume less power, making the model suitable for mobile and embedded devices
Solution Approach 2:
The parameter representation change to low-rank factorization fundamentally reduces the computational complexity from O(MN) to O(MK+KN), which directly translates to lower power consumption during inference, especially when K is chosen to be much smaller than M and N
Data Source
AI summary
A method includes obtaining a parameter matrix associated with a linear layer of a first machine learning model and containing parameter values for parameters of the linear layer. The method also includes determining importance values corresponding to the parameter values. The method further includes generating factorized matrices such that a product of the importance values and factorized matrices contains approximated parameter values for the parameters of the linear layer. In addition, the method includes generating a second machine learning model representing a compressed version of the first machine learning model. The second machine learning model has first and second linear layers containing parameter values based on the importance values and the factorized matrices. The factorized matrices are generated based on weighted errors between the parameter values for the parameters of the linear layer and the approximated parameter values. Weights associated with the weighted errors are based on the importance values.