Floating-Point Model Compression for Mixed-Compressibility Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression techniques are ineffective for mixed compressibility data, particularly floating-point data in machine-learned models, as they either result in significant information loss or limited memory reduction due to the bulk of the mantissa component being uncompressible.
Innovation Solution
A method that separates data into compressible and uncompressible portions, applying predictive techniques to the former and Kashin Decomposition followed by uniform quantization to the latter, spreading quantization error evenly across the data to minimize impact on machine-learned models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression techniques are applied to floating-point data in machine-learned models, then memory reduction is achieved, but significant information loss occurs in the mantissa component
Solution Approach 1:
The patent segments floating-point data into three distinct components: sign bits, exponent bits, and mantissa bits. Each component is then processed using different compression techniques tailored to its specific compressibility characteristics. The sign and exponent portions are compressed using predictive techniques, while the mantissa portion undergoes Kashin Decomposition followed by uniform quantization, thereby achieving memory reduction while preserving critical information.
Solution Approach 2:
The patent applies different compression strategies to different portions of the floating-point data based on their local compressibility properties. The sign and exponent fields, which exhibit higher compressibility, receive predictive compression treatment. The mantissa field, which is less compressible, receives Kashin Decomposition and uniform quantization. This localized approach ensures optimal compression efficiency for each data portion while minimizing overall information loss.
2Quantity of substance
If compression techniques are applied to reduce memory usage, then storage efficiency improves, but quantization error adversely affects model parameters
Solution Approach 1:
The patent transforms the mantissa portion of floating-point data through Kashin Decomposition, which changes the parameter representation from the original floating-point format to a decomposed form with different statistical properties. This transformation spreads the quantization error more evenly across the data, reducing its adverse impact on model parameters. The uniform quantization is then applied to this transformed data, achieving storage efficiency while preserving model accuracy.
3Quantity of substance
If uniform quantization is applied to compressed data, then storage bandwidth is reduced, but quantization error spreads unevenly across the data
Solution Approach 1:
The patent performs Kashin Decomposition on the mantissa portion before applying uniform quantization. This preliminary transformation restructures the data in a way that makes subsequent uniform quantization produce more evenly distributed errors. By preparing the data through decomposition first, the patent ensures that the quantization step achieves both storage bandwidth reduction and favorable error distribution characteristics.
Data Source
AI summary
Systems and methods for compression of data that exhibits mixed compressibility, such as floating-point data, are provided. As one example, aspects of the present disclosure can be used to compress floating-point data that represents the values of parameters of a machine-learned model. Therefore, aspects of the present disclosure can be used to compress machine-learned models (e.g., for reducing storage requirements associated with the model, reducing the bandwidth expended to transmit the model, etc.).