Machine-Learning Model Compression With Static Projection Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing compression techniques for large language models (LLMs) face challenges such as reduced accuracy, hardware dependency, and inefficient execution due to quantization and pruning methods, particularly with unstructured pruning leading to sparse structures and irregular computation patterns.
Innovation Solution
A method involving executing a trained machine learning model to identify activation tensors, generating projection matrices based on fidelity metrics, and retraining weight matrices to create a compressed model that maintains projection matrices static, allowing for dimensionality reduction and efficient execution on standard processors without specialized hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If quantization techniques are used to reduce precision of parameter values, then model size and computational complexity are reduced, but model accuracy deteriorates
Solution Approach 1:
The patent segments the model compression process into distinct phases: identification of redundancy, application of compression techniques (quantization and/or pruning), and retraining. This segmentation allows selective application of compression to different model components while preserving critical accuracy through targeted retraining on identified redundant portions.
Solution Approach 2:
The patent employs dynamic adjustment of compression parameters during the retraining phase. The compression ratio, quantization precision, and pruning thresholds are dynamically optimized based on performance metrics, allowing the system to adaptively balance between model size reduction and accuracy preservation.
2Quantity of substance
If unstructured pruning is used to reduce number of parameters, then model size is reduced, but execution efficiency deteriorates due to sparse structures
Solution Approach 1:
The patent extracts and removes identified redundant portions of the model through systematic pruning. By taking out only the truly redundant components rather than applying uniform pruning, the method reduces parameter count while minimizing the creation of sparse structures that would harm execution efficiency.
Solution Approach 2:
The patent changes the structural parameters of the model by applying structured pruning techniques that maintain regular computation patterns. The pruning process modifies connectivity parameters in a way that preserves hardware-friendly computation structures, enabling efficient execution despite reduced parameter count.
3Device complexity
If structured pruning is used to reduce parameters systematically, then computational complexity is reduced, but model accuracy deteriorates due to removal of important information
Solution Approach 1:
The patent implements feedback mechanisms where model performance is continuously monitored during and after pruning. The retraining phase uses performance feedback to identify which pruned portions truly impacted accuracy, allowing iterative refinement of the pruning strategy to achieve better complexity reduction without sacrificing critical accuracy.
Solution Approach 2:
The patent performs preliminary identification of redundant model portions before applying compression techniques. This preliminary analysis creates a roadmap for selective pruning that targets truly redundant components first, preserving important information while still achieving computational complexity reduction through systematic removal of identified redundancies.
Data Source
AI summary
Techniques for compressing a machine learning mode include executing a first trained machine learning model on training data to identify one or more activation tensors associated with at least one layer of the trained machine learning model; for each pairing of a first activation tensor included in the one or more activation tensors and a different fidelity metric included in a plurality of fidelity metrics, generating a corresponding partially compressed machine learning model; identifying a first projection matrix corresponding to the first activation tensor based on the plurality of corresponding partially compressed machine learning models; generating a compressed machine learning model by at least multiplying the first projection matrix and a corresponding weight matrix; and generating a retrained compressed machine learning model by at least retraining the corresponding weight matrix while keeping the first projection matrix static.