Floating-Point Exponent Compression for DNN Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural network (DNN) models face challenges with memory capacity and bandwidth limitations, particularly in accelerators like TPUs and GPUs, which are exacerbated by the need for larger models, and existing data compression techniques impact model quality or require model modifications.
Innovation Solution
A lossless hardware compression (LHC) method for floating-point tensors that compresses exponent bits, allowing transparent decompression without affecting model quality, and supports various floating-point formats, enabling efficient memory and communication bandwidth utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If lossy compression formats are used to reduce memory and bandwidth requirements, then memory capacity and bandwidth are improved, but model quality deteriorates
Solution Approach 1:
The patent changes the parameter of data representation by compressing only the exponent bits of floating-point numbers while preserving the full precision of the significand and sign bits. This selective parameter compression reduces memory and bandwidth requirements while maintaining the numerical precision necessary for model quality.
Solution Approach 2:
The patent segments the floating-point number into its three components (sign, exponent, significand) and applies compression only to the exponent portion. This segmentation allows differential treatment of data components, compressing only the redundant part while preserving the critical precision-carrying components.
2Quantity of substance
If quantization techniques are applied to reduce data representation size, then memory capacity and bandwidth are improved, but model quality deteriorates
Solution Approach 1:
Instead of quantizing the entire floating-point number, the patent changes only the representation parameter of the exponent by replacing repeated exponent values with indices. This maintains full floating-point precision for computation while reducing the storage parameter size.
3Quantity of substance
If sparsity techniques are applied to reduce data representation size, then memory capacity and bandwidth are improved, but model quality deteriorates
Solution Approach 1:
The patent applies parameter changes specifically to the exponent field by identifying and compressing repeated values, rather than applying sparsity techniques that would zero out or approximate data values. This preserves all original data values while reducing representation size.
4Quantity of substance
If simple lossless compression techniques like run-length encoding are used, then memory capacity and bandwidth are improved, but compression effectiveness deteriorates
Solution Approach 1:
The patent applies local quality by tailoring the compression approach specifically to the statistical properties of exponent bits in floating-point data. Instead of using generic compression, it exploits the local characteristic that exponent values are highly repetitive within batches, achieving superior compression ratios.
Solution Approach 2:
The patent achieves high compression effectiveness by changing the representation parameter of exponent values from full precision to compact indices, leveraging the high repetition rate of exponent values in neural network computations to achieve compression ratios exceeding 5:1.
5Device complexity
If processor software decompression is used instead of hardware decompression, then device complexity is reduced, but productivity deteriorates
Solution Approach 1:
The patent replaces the mechanical/software-based decompression process with a dedicated hardware decompression unit that operates in parallel with the processor. This substitution of the decompression mechanism enables simultaneous compression and decompression operations, eliminating the performance penalty of software processing.
6Manufacturing precision
If larger models are deployed to improve AI performance, then model accuracy is improved, but memory capacity and bandwidth requirements worsen
Solution Approach 1:
The patent enables deployment of larger, more accurate models by changing the storage parameter efficiency through exponent compression. By reducing the parameter size of individual floating-point numbers while maintaining full precision, the overall memory capacity required for large models is significantly reduced.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for compression and decompression of floating point numbers. A floating point number is compressed by accessing a dictionary that associates exponent values with respective exponent indices to determine whether or not the exponent value for the floating point number is present in the dictionary; when the exponent value is present in the dictionary, generating a compressed floating point number for the floating point number by substituting for the exponent value the exponent index corresponding to the exponent value; and when the exponent value is not present in the dictionary, generating a compressed floating point number for the floating point number by substituting for the exponent value an unencoded indicator indicating that the exponent value is located apart from the compressed floating point number.