Neural Network Weight Unification for DNN Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Network (DNN) models require efficient compression to reduce storage and computation costs, especially for deployment on mobile devices and chips, as existing methods like network pruning and quantization are limited in maintaining performance and efficiency.
Innovation Solution
A method involving weight unification regularization and iterative retraining to optimize weight coefficients, focusing on compression rate, unification distortion, and computation speed loss, which reorders indices and unifies weights within a neural network model to enhance compression efficiency and inference computation speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If network pruning and quantization are used to compress DNN models, then model size is reduced, but performance and efficiency are compromised
Solution Approach 1:
The patent changes the parameter distribution of weight coefficients by applying unification regularization that transforms weights into unified patterns (e.g., sign patterns, magnitude patterns). This parameter transformation enables more aggressive compression through quantization and pruning while preserving performance, as the unified parameters maintain the essential functional characteristics of the original weights.
Solution Approach 2:
The patent applies local quality by differentiating between important and less important weight coefficients through unification regularization. By identifying and preserving critical weight patterns locally within the network while allowing less critical weights to be unified or pruned, the method maintains overall performance while achieving compression.
2Volume of stationary object
If network pruning and quantization are used to compress DNN models, then model size is reduced, but computation efficiency deteriorates
Solution Approach 1:
The unification regularization transforms weight parameters into unified patterns that are more amenable to efficient computation. By organizing weights into consistent magnitude and sign patterns, the method enables faster inference through simplified computation operations while maintaining model size reduction benefits.
3Volume of stationary object
If weight coefficients are unified through reordering indices, then compression rate improves, but model complexity increases
Solution Approach 1:
The patent applies preliminary action by performing index reordering and weight unification during the training phase before deployment. This preliminary transformation of the weight matrix into unified patterns simplifies subsequent compression operations and reduces the complexity of the deployment model, as the unification structure is already in place.
Data Source
AI summary
A method, computer program, and computer system is provided for compressing a neural network model. One or more indices corresponding to a multi-dimensional tensor associated with a neural network are reordered. A set of weight coefficients associated with the one or more reordered indices is unified. A model of the neural network is compressed based on the unified set of weight coefficients.