Stochastic Dynamic Fixed-Point Training for Low-Bit Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural network training requires high bit precision, especially in the error propagation stage, leading to power consumption and memory access issues, making it unsuitable for mobile devices, and existing methods for determining optimal precision are time-consuming and inefficient, relying on trial and error.
Innovation Solution
The introduction of a Stochastic Dynamic Fixed-Point (SDFXP) system with a Layer-wise Adaptive Precision Scaling (LAPS) method that dynamically adjusts precision for each layer by comparing low-bit and high-bit operations during training, using stochastic rounding and thresholding to determine optimal precision, allowing for low-bit precision training while maintaining accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high bit precision (16 bits or more) is used for deep neural network training, then training accuracy is maintained, but power consumption and memory access increase significantly
Solution Approach 1:
The patent applies dynamic precision adjustment by determining optimal bit precision for each layer individually during training. The system dynamically selects precision levels (e.g., 8-bit, 16-bit, or mixed precision) based on layer-specific requirements rather than using uniform high precision throughout the network, thereby reducing overall power consumption while maintaining training accuracy.
Solution Approach 2:
The patent implements layer-wise precision determination where different bit precision levels are assigned to different layers of the neural network based on their specific needs. This local optimization allows critical layers to use higher precision while less sensitive layers use lower precision, reducing total power consumption without sacrificing overall training accuracy.
2Measurement precision
If high bit precision is used for training, then training accuracy is maintained, but memory access requirements increase
Solution Approach 1:
The system dynamically determines the bit precision required for each layer during the training process. By adjusting precision levels adaptively rather than using fixed high precision, the patent reduces the memory bandwidth and storage requirements while maintaining the accuracy needed for effective training.
Solution Approach 2:
Different memory resources are allocated to different layers based on their precision requirements. Layers requiring higher precision receive appropriate memory resources, while layers that can function with lower precision use less memory, thereby optimizing overall memory access efficiency.
3Measurement precision
If conventional trial and error methods are used to determine optimal precision, then precision requirements are identified, but training time increases significantly
Solution Approach 1:
The patent implements a self-determining mechanism where the system automatically identifies optimal precision requirements for each layer during training without requiring external trial-and-error intervention. The layer-wise precision determination algorithm autonomously analyzes training data and network characteristics to select appropriate precision levels, eliminating time-consuming manual optimization processes.
Solution Approach 2:
The system incorporates feedback mechanisms that monitor training progress and performance metrics to dynamically adjust precision settings. By continuously evaluating training outcomes and adapting precision levels accordingly, the patent efficiently identifies optimal precision requirements without extensive trial and error, significantly reducing training time.
Data Source
AI summary
Disclosed herein are an apparatus and method for training a low-bit-precision deep neural network. The apparatus includes an input unit configured to receive training data to train the deep neural network, and a training unit configured to train the deep neural network using training data, wherein the training unit includes a training module configured to perform training using first precision, a representation form determination module configured to determine a representation form for internal data generated during an operation procedure for the training and determine a position of a decimal point of the internal data so that a permissible overflow bit in a dynamic fixed-point system varies randomly, and a layer-wise precision determination module configured to determine precision of each layer during an operation in each of a feed-forward stage and an error propagation stage and automatically change the precision of a corresponding layer based on the result of determination.