Activation Compression With Non-Uniform Mantissas for Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models, particularly neural networks, are computationally expensive and inefficient for real-time processing due to their reliance on general-purpose processors, necessitating improvements in hardware and software to enhance computational efficiency.
Innovation Solution
Implementing neural networks with activation values stored in a compressed format using block floating-point representation with non-uniform mantissas, allowing for efficient storage and retrieval during forward and backward propagation, utilizing general-purpose processors, neural network accelerators, or reconfigurable logic devices like FPGAs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If activation values are stored in full precision format during forward propagation, then computational accuracy is maintained, but memory overhead and storage requirements increase
Solution Approach 1:
The patent changes the precision parameter of activation values from full precision to compressed precision formats. During forward propagation, activation values are stored in a compressed format with reduced precision, and during backward propagation, they are restored to higher precision. This parameter transformation resolves the contradiction by allowing low memory overhead during forward pass while maintaining high computational accuracy during backward pass.
Solution Approach 2:
The patent implements dynamic precision switching for activation values based on the operational phase. The system dynamically adjusts the precision level of stored activation values - using compressed low-precision format during forward propagation and restored high-precision format during backward propagation. This dynamic adaptation resolves the contradiction between memory efficiency and computational accuracy by optimizing precision requirements for each phase.
2Ease of operation
If neural networks are executed on general-purpose processors, then flexibility and ease of implementation are maintained, but computational speed and energy efficiency deteriorate
Solution Approach 1:
The patent changes the data format parameter of activation values from standard floating-point to compressed block floating-point formats. This parameter transformation enables general-purpose processors to achieve higher computational efficiency by reducing the number of bits processed and stored, while maintaining compatibility with standard processor architectures. The compression reduces memory bandwidth requirements and improves cache utilization, thereby increasing computational speed without requiring specialized hardware.
3Productivity
If activation values are compressed during storage, then memory bandwidth and storage efficiency improve, but retrieval and decompression complexity increase
Solution Approach 1:
The patent changes the numerical representation parameter of activation values to block floating-point format with shared exponents. This parameter transformation enables efficient compression by grouping multiple activation values that share common exponent bits, thereby reducing storage requirements and improving memory bandwidth efficiency. The block floating-point format maintains relatively simple compression and decompression operations compared to other compression schemes.
Solution Approach 2:
The patent merges multiple activation values into block floating-point representations where common exponent bits are shared across multiple values. By combining the exponent information for groups of activation values, the system reduces redundant data storage and improves memory efficiency. This merging approach simplifies the compression process by identifying and consolidating common patterns in the exponent bits of adjacent activation values.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and in particular for storing activation values from a neural network in a compressed format having lossy or non-uniform mantissas for use during forward and backward propagation training of the neural network. In certain examples of the disclosed technology, a computing system includes processors, memory, and a compressor in communication with the memory. The computing system is configured to perform forward propagation for a layer of a neural network to produced first activation values in a first block floating-point format. In some examples, activation values generated by forward propagation are converted by the compressor to a second block floating-point format having a non-uniform and/or lossy mantissa. The compressed activation values are stored in the memory, where they can be retrieved for use during back propagation.