Neural Network Compression with Quantizability Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of efficiently compressing Deep Neural Network (DNN) models for deployment on mobile devices and chips, given the need for reducing storage size and computation requirements, is not adequately addressed by existing methods that focus on sparsity-promoting regularization and subsequent quantization.
Innovation Solution
A method utilizing quantizability regularization in an iterative network retraining framework to optimize weight coefficients for improved quantization efficiency, involving reshaping tensors, identifying subsets of coefficients, and iteratively refining the network to maintain performance and facilitate further compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If sparsity-promoting regularization and quantization are used to compress DNN models, then model size is reduced, but performance degradation occurs
Solution Approach 1:
The patent applies quantizability regularization during the training phase (before quantization) to pre-optimize weight coefficients for quantization. This preliminary action ensures that when quantization is subsequently applied, the model maintains performance while achieving compression, resolving the contradiction between model size reduction and performance preservation
Solution Approach 2:
The patent modifies the training objective by adding a quantizability regularization term that changes the parameters (weight coefficients) to be more amenable to quantization. This parameter transformation allows the model to achieve both compression and performance retention by optimizing weights for low-precision representation before actual quantization occurs
2Manufacturing precision
If iterative network retraining with quantizability regularization is applied, then quantization efficiency is improved, but training time increases
Solution Approach 1:
The patent employs iterative retraining where quantizability regularization is continuously applied across multiple training epochs. This continuous optimization process gradually refines weight coefficients for quantization, achieving high quantization efficiency that justifies the extended training time through progressively improving compression quality
Data Source
AI summary
A method, computer program, and computer system is provided for compressing a neural network model. A multi-dimensional tensor corresponding to a set of weight coefficients associated with a neural network is reshaped. A subset of weight coefficients is identified from among the set of weight coefficients. A model of the neural network is compressed based on the identified subset of weight coefficients.