Variable-Precision ML Data Compression for Edge Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently compressing large ML model variables such as weights and activations, particularly for edge devices, due to non-optimal numerical data formats, complex hardware requirements, and inefficient sharing across processing units, leading to increased bandwidth and memory access.
Innovation Solution
The invention employs a method to form coding pairs consisting of a code and additional data for various numerical data formats, including integer, posit, and floating-point formats, using entropy coding to compress and decompress model variables efficiently, allowing for flexible and efficient sharing across multiple processing units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression is implemented to reduce storage and memory requirements, then storage and memory requirements are reduced, but hardware complexity increases due to complex circuits with large memory buffers
Solution Approach 1:
The compression method segments model variables by magnitude, separating them into different groups based on their value ranges. This allows each group to be compressed with appropriate precision, reducing the need for complex hardware that would need to handle all values uniformly.
Solution Approach 2:
The system dynamically adjusts the number of bits used to represent different model variables based on their magnitude. Smaller variables use fewer bits while larger variables use more bits, creating a flexible compression scheme that reduces storage requirements without requiring fixed complex hardware architecture.
2Adaptability or versatility
If multiple numerical data formats are supported to optimize for different applications, then adaptability improves, but device complexity increases due to needing to handle various formats
Solution Approach 1:
The compression method provides a universal approach that can handle multiple numerical data formats (int8, bfloat16, fp8, etc.) through a single unified algorithm. The system automatically detects and processes different formats without requiring separate specialized hardware for each format, thereby supporting versatility while controlling complexity.
3Device complexity
If a fixed number of bits is used to represent all model variables, then hardware simplicity is maintained, but performance deteriorates due to outliers forcing worst-case bit allocation
Solution Approach 1:
The system applies different bit allocations to different groups of model variables based on their local characteristics (magnitude). Instead of using a uniform fixed bit depth for all variables, the method assigns appropriate precision levels to each group, optimizing the balance between hardware simplicity and representation efficiency.
4Productivity
If ML model variables are not shared efficiently across processing units, then each unit can operate independently, but bandwidth requirements increase due to redundant memory access
Solution Approach 1:
The compression method enables efficient sharing of model variables across processing units by creating a compact unified representation. Compressed variables can be shared among multiple units, reducing redundant memory accesses and bandwidth requirements while maintaining the ability for each unit to operate independently when needed.
Data Source
AI summary
One aspect of the invention includes an apparatus comprising a compressor accepting a model variable such as a weight or activation of an ML model such as an LLM, the compressor converting the model variable to a coding pair, the coding pair consisting of a code and additional data. The model variable may be one of a variety of formats including an integer format, a posit format, part of a binary code group, part of a ternary code group, and one of a plurality of floating-pint formats. Another aspect is an interface apparatus for compressing internal floating-point numbers in memory to a data stream of compressed model variables. Another aspect is an interface apparatus for decompressing a data stream of compressed model variables to internal floating-point numbers in memory.