Activation Compression With Non-Uniform Mantissas for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current machine learning and artificial intelligence techniques, particularly in neural networks, face challenges in real-time feature extraction due to the computationally expensive nature of training and using models, which is not efficiently handled by general-purpose processors.
Innovation Solution
The use of compressed activation values in a quantized or block floating-point format, specifically during forward and backward propagation in neural networks, allows for efficient storage and retrieval, reducing computational complexity and memory bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If activation values are stored in full precision format during forward propagation, then computational accuracy is maintained, but memory bandwidth requirements and storage overhead increase
Solution Approach 1:
The patent applies parameter changes by transitioning activation values from full precision floating-point format to compressed block floating-point format with non-uniform mantissas. This format transformation reduces the number of bits required to represent each activation value while maintaining sufficient precision for backward propagation computations, thereby reducing memory bandwidth requirements and storage overhead.
Solution Approach 2:
The patent employs asymmetric mantissa representation where different mantissa widths are assigned to different blocks of activation values based on their magnitude ranges. This asymmetric approach allows optimal precision allocation - using wider mantissas for blocks requiring higher precision and narrower mantissas for blocks where lower precision suffices, thus balancing accuracy requirements with memory efficiency.
2Quantity of substance
If compressed activation values with non-uniform mantissas are used, then memory overhead is reduced, but computational complexity of decompression increases
Solution Approach 1:
The patent segments activation values into multiple blocks, each with its own exponent and non-uniform mantissa configuration. This segmentation allows independent compression and decompression of each block, enabling parallel processing during decompression and reducing the computational burden on any single processing unit while maintaining overall memory efficiency.
Solution Approach 2:
The patent performs preliminary organization of activation values into blocks with predetermined exponent and mantissa configurations during the forward propagation phase. This preliminary action prepares the data in a structured format that facilitates efficient decompression during backward propagation, reducing the computational complexity of the decompression operation.
3Quantity of substance
If block floating-point format with non-uniform mantissas is used for compression, then storage efficiency improves, but hardware design complexity increases
Solution Approach 1:
The patent designs a universal block floating-point compression architecture that can handle multiple precision requirements using the same hardware structure. The non-uniform mantissa system uses a standardized encoding scheme that can represent different precision levels within a single hardware framework, allowing the same hardware to efficiently compress and decompress activation values with varying precision requirements without requiring multiple specialized units.
Data Source
AI summary
Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format having lossy or non-uniform mantissas for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a non-uniform and/or lossy mantissa. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.