Logarithmic Number System for Low-Precision Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Low-precision training for deep neural networks is challenging due to complex interactions between learning algorithms and low-precision number systems, particularly in energy-constrained edge devices, where high-precision training methods are computationally costly and inefficient.
Innovation Solution
An end-to-end low-precision training framework using a multi-base logarithmic number system (LNS) and a multiplicative weight update algorithm (LNS-Madam) is implemented, which expresses high dynamic range and computational energy efficiency, allowing for reduced precision in weight updates while maintaining accuracy through logarithmic representation and approximation techniques.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high-precision floating-point (32-bit or 16-bit) is used for training deep neural networks, then model accuracy is maintained, but computational energy cost and memory footprint increase significantly
Solution Approach 1:
The patent changes the number system parameters from traditional floating-point to logarithmic number system (LNS), where weights are represented as logarithms. This parameter change enables low-precision training (using fewer bits) while maintaining model accuracy, as the logarithmic representation naturally handles the dynamic range of weights and gradients more efficiently, reducing both memory footprint and computational energy cost.
Solution Approach 2:
The patent substitutes the traditional floating-point arithmetic mechanism with logarithmic arithmetic operations. By representing weights and gradients in logarithmic form, multiplication operations become addition operations, and division becomes subtraction, fundamentally changing the computational mechanism. This substitution reduces the complexity of low-precision arithmetic and maintains accuracy while lowering energy consumption.
2Use of energy by moving object
If low-precision number systems are used for training, then computational energy efficiency improves, but training becomes challenging due to complex interactions between learning algorithms and low-precision representations
Solution Approach 1:
The patent modifies the learning algorithm parameters to work naturally with logarithmic representations. The multiplicative update rule is adapted to operate in the logarithmic domain, where weight updates are performed by adding logarithmic gradients to logarithmic weights. This parameter adaptation simplifies the training process by eliminating the need for complex conversions between floating-point and low-precision formats, reducing training algorithm complexity while maintaining energy efficiency.
3Loss of substance
If quantization is applied to floating-point numbers for training and inference, then redundancy is reduced and accuracy is maintained, but the interaction with learning algorithms becomes more complex
Solution Approach 1:
The patent substitutes quantized floating-point representations with logarithmic number system representations. In LNS, numbers are represented as logarithms with a sign bit, exponent, and mantissa. This substitution eliminates the need for complex quantization schemes while maintaining low redundancy, as the logarithmic representation naturally compresses the dynamic range of neural network weights and gradients, reducing the number of bits needed while simplifying algorithm interactions.
Data Source
AI summary
An end-to-end low-precision training system based on a multi-base logarithmic number system and a multiplicative weight update algorithm. The multi-base logarithmic number system is applied to update weights of the neural network, with different bases of the multi-base logarithmic number system utilized between calculation of weight updates, calculation of feed-forward signals, and calculation of feedback signals. The LNS expresses a high dynamic range and computational energy efficiency, making it advantageous for on-board training in energy-constrained edge devices.