Floating-Point Exponent Compression for DNN Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural network (DNN) models face challenges with memory capacity and bandwidth limitations, particularly in accelerators like TPUs and GPUs, which are exacerbated by the need for larger models, and existing data compression techniques impact model quality or require model modifications.
Innovation Solution
A lossless hardware compression (LHC) method for floating-point tensors that compresses and decompresses exponent bits transparently, supporting various floating-point formats without affecting model quality, using a dictionary-based approach for efficient data handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data compression techniques are applied to reduce memory capacity and bandwidth requirements, then memory capacity and bandwidth efficiency are improved, but model quality is degraded
Solution Approach 1:
The patent changes the parameter representation by compressing only the exponent field of floating-point numbers while maintaining the full precision of the mantissa. This selective parameter compression reduces memory capacity requirements and bandwidth consumption while preserving the critical precision needed for model quality, thereby resolving the contradiction between compression efficiency and model accuracy.
2Quantity of substance
If lossy compression formats are used to reduce memory and bandwidth requirements, then memory capacity and bandwidth efficiency are improved, but model quality is degraded
Solution Approach 1:
The patent applies local quality by treating different parts of the floating-point representation differently - compressing the exponent field while preserving the mantissa in full precision. This localized compression approach targets the less critical exponent portion for compression while maintaining high precision in the mantissa, achieving both memory efficiency and model quality preservation without using lossy compression.
3Quantity of substance
If quantization techniques are applied to reduce bits required for data representation, then memory capacity and bandwidth requirements are reduced, but model quality is impacted
Solution Approach 1:
The patent segments the floating-point number into its exponent and mantissa components, applying compression only to the exponent segment while preserving the mantissa segment in full precision. This segmentation allows selective compression of the less critical exponent portion without quantizing the mantissa, thereby reducing bits required for representation while maintaining model quality.
4Quantity of substance
If sparsity techniques are applied to reduce the number of bits required for data representation, then memory capacity and bandwidth requirements are reduced, but model quality is impacted
Solution Approach 1:
The patent changes the representation parameter of the exponent by encoding it in a compressed format that reduces the number of bits required, while maintaining the full precision of the mantissa. This parameter change in the exponent representation achieves bit reduction without the quality degradation associated with sparsity techniques that modify the actual data values.
5Quantity of substance
If software-based decompression is used to decompress data on the processor, then compression ratio is improved, but processor performance is reduced due to software operations and memory bandwidth consumption
Solution Approach 1:
The patent introduces an intermediary hardware decompression unit positioned between the compressed memory storage and the processor. This intermediary performs decompression in hardware, eliminating the need for software-based decompression operations and associated memory bandwidth consumption, thereby maintaining both high compression ratios and processor performance.
6Ease of manufacture
If simple lossless compression techniques like run-length encoding are used, then implementation simplicity is improved, but compression ratio is limited
Solution Approach 1:
The patent applies parameter changes by exploiting the statistical properties of exponent values in floating-point numbers, which often contain repeated patterns. By encoding these exponent patterns with variable-length codes based on their frequency of occurrence, the patent achieves superior compression ratios compared to simple techniques like run-length encoding, while maintaining implementation simplicity through dictionary-based encoding.
Data Source
AI summary
Systems and methods for compression and decompression of floating point numbers. A floating point number is compressed by accessing a dictionary that associates exponent values with respective exponent indices to determine whether or not the exponent value for the floating point number is present in the dictionary; when the exponent value is present in the dictionary, generating a compressed floating point number for the floating point number by substituting for the exponent value the exponent index corresponding to the exponent value; and when the exponent value is not present in the dictionary, generating a compressed floating point number for the floating point number by substituting for the exponent value an unencoded indicator indicating that the exponent value is located apart from the compressed floating point number.