Quantized Neural Network Training Using Fixed-Point Arithmetic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) face challenges in being efficiently trained and deployed on low-power devices due to high computational and energy requirements, with existing methods struggling to balance training precision and power efficiency.
Innovation Solution
The method involves training neural networks using quantized activation and connection weight functions, which output binary values, allowing for reduced precision during both training and inference phases, and utilizing bit-shift operations to minimize computational complexity and energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are trained using conventional floating-point arithmetic on GPUs, then training precision and convergence are improved, but computational complexity and energy consumption increase significantly
Solution Approach 1:
The patent applies parameter changes by transitioning from floating-point arithmetic to fixed-point arithmetic with reduced precision (e.g., 8-bit or 16-bit integers). This changes the representation parameters of weights and activations, enabling lower computational precision while maintaining acceptable network performance, thereby reducing energy consumption on mobile devices without sacrificing training convergence
Solution Approach 2:
The patent replaces the mechanical system of floating-point arithmetic with fixed-point arithmetic. This substitution eliminates the need for complex floating-point units in hardware, allowing implementation on resource-constrained mobile devices while maintaining the essential computational functionality for neural network training and inference
2Measurement precision
If deep neural networks use high-precision floating-point arithmetic, then computational accuracy is improved, but device complexity and hardware requirements increase
Solution Approach 1:
The patent changes the numerical representation parameters from floating-point to fixed-point formats with limited precision. This parameter transformation reduces the hardware requirements by eliminating the need for complex floating-point units, FPU, and associated precision control mechanisms, while still achieving sufficient computational accuracy for neural network operations on mobile devices
3Quantity of substance
If neural networks are trained with quantized binary values, then memory usage and power consumption are reduced, but training stability and convergence may deteriorate
Solution Approach 1:
The patent applies parameter changes by using quantized fixed-point representations (e.g., 8-bit or 16-bit integers) instead of binary values. This provides sufficient precision to maintain training stability and convergence while dramatically reducing memory usage and computational complexity compared to full floating-point precision, achieving a balance between reliability and resource efficiency
Solution Approach 2:
The patent applies partial quantization rather than full binary quantization. By using fixed-point arithmetic with 8-16 bit precision, the patent applies just enough quantization to reduce memory and computational requirements while retaining sufficient precision to ensure training stability, avoiding the excessive quantization that would cause convergence failures
Data Source
AI summary
Training neural networks by constructing a neural network model having neurons each associated with a quantized activation function adapted to output a quantized activation value. The neurons are arranged in layers and connected by connections associated quantized connection weight functions adapted to output quantized connection weight values. During a training process a plurality of weight gradients are calculated during backpropagation sub-processes by computing neuron gradients, each of an output of a respective the quantized activation function in one layer with respect to an input of the respective quantized activation function. Each neuron gradient is calculated such that when an absolute value of the input is smaller than a positive constant threshold value, the respective neuron gradient is set as a positive constant output value and when the absolute value of the input is smaller than the positive constant threshold value the neuron gradient is set to zero.