AI Model Pruning Using Mutual Information and Subunit Ranking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern machine learning (ML) models have grown large in size due to complexity, requiring excessive resources and computation time, necessitating a reduction in size without significantly impacting performance.
Innovation Solution
The ML model is split into subunits, ranked using stochastic independence (SI) or mutual information (MI) scores, and the lowest-scoring subunits are pruned, followed by retraining to ensure acceptable accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the ML model complexity is increased to handle complex patterns, then the model accuracy is improved, but the model size and resource consumption increase
Solution Approach 1:
The patent divides the ML model into discrete subunits (layers, neurons, or filters) that can be independently evaluated and selectively removed. This segmentation enables the pruning process to identify and eliminate specific subunits that contribute least to model accuracy, thereby reducing model size while maintaining performance.
Solution Approach 2:
The patent changes the parameter being optimized from pure accuracy to a balanced metric that considers both accuracy and model size. By introducing stochastic independence scores and mutual information metrics as pruning criteria, the system transforms the optimization problem to find the optimal trade-off point between model complexity and performance.
2Measurement precision
If the ML model size is increased to capture complex patterns, then the model accuracy is improved, but the computation time increases
Solution Approach 1:
The patent extracts and removes unnecessary subunits from the ML model based on their stochastic independence scores. By taking out these redundant components, the model computation time is reduced while the essential functionality and accuracy are preserved through the careful selection of what to remove.
Solution Approach 2:
The patent applies partial pruning by removing only the least important subunits rather than eliminating entire model components. This partial action allows the model to retain sufficient complexity for accurate pattern recognition while removing enough redundancy to reduce computation time significantly.
3Measurement precision
If the ML model size is increased to handle complex datasets, then the model accuracy is improved, but the resource consumption increases
Solution Approach 1:
The patent discards redundant subunits from the model architecture and recovers computational resources by eliminating unnecessary calculations. The stochastic independence analysis identifies which subunits can be safely discarded without impacting model performance, thereby recovering valuable computational resources.
Solution Approach 2:
By segmenting the model into evaluatable subunits, the patent enables selective removal of resource-intensive components that contribute minimally to accuracy. This segmentation allows the system to reduce overall resource consumption by pruning specific high-cost subunits while maintaining essential model functionality.
Data Source
AI summary
Described herein are techniques for intelligently pruning a machine learning or generative AI model. The model may first be split up into subunits. Each subunit may be analyzed to calculate a suitable measure such as a stochastic independence score or mutual information score. The subunits may in turn be ranked by their associated score and the lowest ranked subunit or subunits may be pruned from the model. The pruned model is then retrained, and accuracy of the pruned model is evaluated. A determination is then made whether to prune more or to return the pruned model.