Floating-Point Signed Digit Weight Representation for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) face significant computation complexity and high hardware power consumption during training due to the use of 32-bit standard floating-point numbers for representing neural network weights, which becomes particularly cumbersome with larger network sizes.
Innovation Solution
The method employs floating-point signed digit (FloatSD) representation for neural network weights, where the mantissa is divided into multiple signed digit groups and the exponent into a single group, allowing for reduced non-zero digits and simplifying weight adjustment and inference computations by using shift-and-add circuits instead of multipliers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If 32-bit standard floating-point numbers are used to represent neural network weights, then precision is maintained, but computation complexity and hardware power consumption increase significantly
Solution Approach 1:
The patent segments the floating-point number representation by dividing the mantissa into multiple signed-digit groups (e.g., 4-bit groups with at most one non-zero digit). This segmentation allows each group to be processed independently using simple shift-and-add operations instead of full multiplication, significantly reducing computation complexity while maintaining acceptable precision for neural network weights.
Solution Approach 2:
The patent changes the parameter representation from standard binary floating-point to a customized floating-point format with signed-digit groups in the mantissa. By modifying how weights are represented (using groups with limited non-zero digits), the computation complexity is reduced while preserving sufficient precision for deep neural network training and inference.
2Measurement precision
If 32-bit standard floating-point numbers are used for weight representation, then accuracy is preserved, but hardware power consumption increases
Solution Approach 1:
By segmenting the weight representation into signed-digit groups with at most one non-zero digit per group, the patent enables the use of simple shift-and-add circuits instead of power-hungry multipliers. This segmentation maintains sufficient accuracy for neural network inference while dramatically reducing hardware power consumption.
Solution Approach 2:
The patent replaces expensive, complex floating-point multipliers with cheaper, simpler shift-and-add circuitry. The simplified computation units consume significantly less power and can be implemented more efficiently in hardware, making the system more energy-efficient while maintaining acceptable inference accuracy.
3Adaptability or versatility
If larger neural network models are deployed, then model capability improves, but computation complexity and memory access requirements explode
Solution Approach 1:
The patent applies segmentation to weight representation by dividing mantissas into signed-digit groups. This allows larger neural network models to be deployed with reduced computation complexity, as each weight can be processed using simple shift-and-add operations rather than full multiplication, making large-scale models more feasible.
Solution Approach 2:
By changing the weight representation parameters to use floating-point signed-digit groups, the patent reduces the computational burden of processing large neural networks. This parameter change enables the deployment of more capable, larger models without the computation complexity and memory access requirements exploding.
4Measurement precision
If standard floating-point multiplication is used for weight adjustments, then precision is maintained, but the number of operations increases
Solution Approach 1:
The patent segments weight adjustment computations into operations on signed-digit groups. Instead of performing full floating-point multiplication, the system uses shift-and-add operations on segmented groups, reducing the number of operations required while maintaining sufficient precision for effective weight adjustment during training.
Solution Approach 2:
By changing the computational parameters from standard floating-point multiplication to shift-and-add operations on signed-digit groups, the patent increases training productivity. The reduced number of operations accelerates weight adjustment while preserving the precision needed for effective neural network training.
Data Source
AI summary
A method of training a neural network including multiple neural network weights and multiple neurons, and the method includes using floating-point signed digit numbers to represent each of the multiple neural network weights, wherein a mantissa of each of the multiple neural network weights is represented by multiple mantissa signed digit groups and an exponent of each of the multiple neural network weights is represented by an exponent digit group; and using the exponent digit group and at least one of the multiple mantissa signed digit groups to perform weight adjustment computation and neural network inference computation.