Activation Compression With Outlier Block Floating-Point Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning and artificial intelligence techniques face challenges in performing real-time feature extraction due to the computational expense of training and using neural networks, which is not efficiently handled by general-purpose processors.
Innovation Solution
The use of a compressed format for storing activation values from neural networks, employing block floating-point representation with shared and outlier exponents, allows for efficient storage and retrieval during forward and backward propagation, utilizing general-purpose and special-purpose processors, and memory optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If activation values are stored in a compressed block floating-point format, then memory requirements are reduced, but retrieval and processing complexity increases due to outlier handling
Solution Approach 1:
The patent segments activation values into two groups: normal values stored in compressed block floating-point format and outlier values stored in a separate format. This segmentation allows the system to reduce memory requirements for the majority of values while maintaining high precision for outliers, resolving the contradiction between compression and processing complexity by handling different value types differently.
Solution Approach 2:
The patent applies different storage qualities to different portions of activation values. Normal values receive aggressive compression while outlier values maintain higher precision representation. This local quality differentiation optimizes the balance between memory efficiency and processing accuracy, addressing the contradiction by tailoring the storage approach to the specific characteristics of each value type.
2Speed
If all activation values use the same compressed format, then processing speed improves, but precision is lost for outlier values
Solution Approach 1:
The patent implements a dynamic storage strategy where the system adaptively selects between compressed block floating-point format and full precision format based on the characteristics of each activation value. This dynamic approach allows the system to maintain high processing speed for normal values while ensuring precision is preserved for outliers, resolving the contradiction between speed and precision.
Solution Approach 2:
The patent changes the precision parameter for outlier values while maintaining compression for normal values. By adjusting the representation format based on value characteristics, the system achieves both fast processing and high precision where needed, addressing the contradiction between processing speed and measurement precision.
3Device complexity
If general-purpose processors are used for neural network training, then system simplicity is maintained, but computational efficiency and energy consumption are insufficient
Solution Approach 1:
The patent changes the numerical representation parameters of activation values to a compressed block floating-point format that is optimized for hardware implementation. This parameter change enables general-purpose processors to achieve better computational efficiency and lower energy consumption while maintaining system simplicity, resolving the contradiction between device complexity and productivity.
Data Source
AI summary
Apparatus and methods for training a neural network accelerator using quantized precision data formats having outlier values are disclosed, and in particular for storing activation values from a neural network in a compressed format for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a narrower numerical precision than the first block floating-point format. Outlier values, comprising additional bits of mantissa and/or exponent are stored in ancillary storage for subset of the activation values. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.