Block Floating-Point Activation Storage for Faster Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning and artificial intelligence techniques face challenges in performing feature extraction in real-time due to the computational expense of training and using neural networks on general-purpose processors.
Innovation Solution
The proposed solution involves storing activation values from a neural network in a compressed format, specifically using a quantized or block floating-point format, to facilitate efficient forward and backward propagation during training. This compression allows for reduced precision representations of activation values, which can be stored and retrieved efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If activation values are stored in full precision block floating-point format during forward propagation, then computational accuracy is maintained, but memory storage requirements and processing time increase
Solution Approach 1:
The patent segments the activation value representation into two parts: a shared exponent that applies to all values in a block, and individual mantissas that can be compressed. This segmentation allows the system to store and process only the mantissa portions individually while sharing the exponent information, reducing overall storage requirements and processing time while maintaining the precision needed for computational accuracy.
2Quantity of substance
If activation values are compressed to reduced precision format for storage in bulk memory, then memory storage requirements decrease, but retrieval and processing complexity increases
Solution Approach 1:
The patent merges the exponent information into a shared block-level parameter rather than storing it with each individual activation value. This combining of the exponent into a common block descriptor reduces the per-value storage requirement to only the mantissa portion, significantly decreasing memory storage requirements while simplifying the retrieval process since the exponent is readily available for the entire block.
3Quantity of substance
If non-uniform mantissas are used for compressed values, then storage efficiency improves, but computational overhead during backward propagation increases
Solution Approach 1:
The patent changes the parameter representation by using non-uniform mantissa encoding schemes that allocate different precision levels to different ranges of activation values. This parameter change allows the system to achieve better storage efficiency by using fewer bits for values that require less precision while maintaining higher precision for values that need it, thereby reducing overall storage requirements without proportionally increasing computational energy during backward propagation.
Data Source
AI summary
Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format having lossy or non-uniform mantissas for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a non-uniform and/or lossy mantissa. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.