Neural Network Weight Compression Using Lossless Exponents
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern deep neural networks (DNNs) are hindered by the large number of parameters, which require significant processing resources and time for training due to their representation in floating-point format, leading to inefficiencies in training and potential bottlenecks in communication.
Innovation Solution
A system that compresses floating-point parameters by separating mantissa and exponent values, using fixed-point arithmetic for mantissa compression and lossless encoding of exponents, reducing the number of values to transmit and processing complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point format is used to represent parameters, then precision is maintained, but processing time and resource consumption increase significantly
Solution Approach 1:
The patent segments floating-point parameters into two distinct components: exponent values and mantissa values. This segmentation allows independent processing of each component - exponents are quantized with higher precision while mantissas use lower precision, resolving the contradiction between maintaining overall parameter precision and reducing processing complexity.
Solution Approach 2:
The patent applies different precision levels to different parts of the floating-point representation. Specifically, exponent values are quantized with higher precision (using more bits) while mantissa values use lower precision (using fewer bits). This local differentiation optimizes the balance between precision preservation and processing efficiency.
2Measurement precision
If full precision floating-point parameters are transmitted, then accuracy is maintained, but communication bandwidth and time increase
Solution Approach 1:
The patent extracts and separately processes the exponent and mantissa components of floating-point parameters. By taking out the exponent values and quantizing them independently with higher precision, the system maintains accuracy for the significant part of the parameter while reducing the bit-width requirements for the mantissa, thereby reducing transmission time.
3Reliability
If exponent values are quantized with high precision, then numerical stability is improved, but compression ratio decreases
Solution Approach 1:
The patent applies different quantization precision levels to different components: high precision quantization for exponent values (using more bits) to maintain numerical stability, and low precision quantization for mantissa values (using fewer bits) to achieve better compression. This local differentiation resolves the contradiction between reliability and compression ratio.
Data Source
Figure 1~2
Figure 3
Figure 4~5
AI summary
Systems, methods, and apparatuses are provided for compressing values. A plurality of parameters may be obtained from a memory, each parameter comprising a floating-point number that is used in a relationship between artificial neurons or nodes in a model. A mantissa value and an exponent value may be extracted from each floating-point number to generate a set of mantissa values and a set of exponent values. The set of mantissa values may be compressed to generate a mantissa lookup table (LUT) and a plurality of mantissa LUT index values. The set of exponent values may be encoded to generate an exponent LUT and a plurality of exponent LUT index values. The mantissa LUT, mantissa LUT index values, exponent LUT, and exponent LUT index values may be provided to one or more processing entities to train the model.