Neural Network Activation Compression with Outlier BFP Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning and artificial intelligence techniques, particularly in neural networks, face challenges with computationally expensive training and inference processes, which are not feasible in real-time using general-purpose processors due to the high computational demands of floating-point operations.
Innovation Solution
The use of block floating-point (BFP) formats, where activation values are stored in a compressed format with a shared exponent and optional additional mantissa for outlier values, allowing for efficient storage and retrieval during forward and backward propagation in neural networks, leveraging both general-purpose and specialized processors like FPGAs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If activation values are stored in standard floating-point format during forward propagation, then computational accuracy is maintained, but memory bandwidth requirements and storage space increase significantly
Solution Approach 1:
The floating-point representation is segmented into two parts: a shared exponent common to all activation values in a block, and individual mantissas for each value. This segmentation allows the exponent to be stored once for the entire block rather than with each individual value, reducing memory bandwidth requirements while maintaining the precision needed for accurate computation.
Solution Approach 2:
The patent changes the parameter representation from full floating-point format to a compressed format where the exponent parameter is shared across multiple values. This parameter change reduces the amount of data that needs to be read from and written to memory, directly addressing the memory bandwidth requirement issue while preserving computational accuracy through proper handling of the shared exponent during operations.
2Quantity of substance
If all activation values use the same exponent in block floating-point format, then memory storage efficiency improves, but precision is lost for outlier values with significantly different magnitudes
Solution Approach 1:
The system dynamically adjusts the exponent assignment based on the characteristics of the data. For most activation values, a shared exponent is used to maximize storage efficiency. However, when outlier values with significantly different magnitudes are detected, the system dynamically assigns individual exponents to these outliers, allowing them to maintain their precision while the majority of values continue to benefit from the shared exponent compression.
3Quantity of substance
If compressed block floating-point format is used for storing activation values, then memory bandwidth requirements decrease, but computational complexity increases due to format conversion and outlier handling
Solution Approach 1:
The system performs preliminary action by identifying outlier values and assigning them individual exponents during the forward propagation phase, before the backward propagation begins. This preliminary classification and marking of outliers allows the backward propagation to efficiently handle the mixed format by simply reading the pre-assigned exponents for outliers, rather than performing complex format conversions during the computationally intensive backward propagation phase.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Apparatus and methods for training a neural network accelerator using quantized precision data formats having outlier values are disclosed, and in particular for storing activation values from a neural network in a compressed format for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a narrower numerical precision than the first block floating-point format. Outlier values, comprising additional bits of mantissa and/or exponent are stored in ancillary storage for subset of the activation values. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.