Neural Network Quantization via Hessian Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant memory resources for storage, making it difficult to deploy them on devices with limited storage, such as mobile devices, due to their large number of network parameters.
Innovation Solution
The method involves determining the diagonals of a second-order partial derivative matrix (Hessian matrix) of a loss function to weight network parameters during quantization, and using first and second moment estimates of gradients to optimize parameter weighting for reduced storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If deep neural networks are deployed on devices with limited storage, then accessibility and usability improve, but memory storage requirements cannot be met due to large number of network parameters
Solution Approach 1:
The patent extracts only the essential information from full-precision network parameters by representing them in a compressed format using codebooks and index mappings. This extraction process removes redundant data while preserving the functional capabilities of the neural network, enabling deployment on devices with limited storage.
Solution Approach 2:
The patent changes the precision parameter of network parameters from high precision (32-bit floating point) to low precision (8-bit integers or binary values). This parameter transformation significantly reduces memory storage requirements while maintaining acceptable model performance through quantization techniques.
2Quantity of substance
If network parameters are quantized to reduce storage, then memory requirements decrease, but model performance may degrade
Solution Approach 1:
The patent performs preliminary training of the neural network in high precision before quantization. This preliminary action allows the model to learn optimal weight distributions that are more amenable to quantization, reducing performance degradation when transitioning to lower precision formats.
Solution Approach 2:
The patent implements fine-tuning procedures after quantization where the quantized model is retrained or adjusted using feedback from performance metrics. This feedback loop allows the model to adapt to the quantized representation and recover performance losses incurred during the quantization process.
Data Source
AI summary
Apparatuses and methods of manufacturing same, systems, and methods for performing network parameter quantization in deep neural networks are described. In one aspect, diagonals of a second-order partial derivative matrix (a Hessian matrix) of a loss function of network parameters of a neural network are determined and then used to weight (Hessian-weighting) the network parameters as part of quantizing the network parameters. In another aspect, the neural network is trained using first and second moment estimates of gradients of the network parameters and then the second moment estimates are used to weight the network parameters as part of quantizing the network parameters. In yet another aspect, network parameter quantization is performed by using an entropy-constrained scalar quantization (ECSQ) iterative algorithm. In yet another aspect, network parameter quantization is performed by quantizing the network parameters of all layers of a deep neural network together at once.