Selective Data Compression for Machine Learning Model Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models face significant overhead and performance costs due to large data sizes, leading to increased latency, compute resource consumption, and bandwidth usage during inference, as they often require extensive storage and retrieval of weights, biases, and input data, which can exceed on-chip memory limits and necessitate off-chip storage access.
Innovation Solution
Implementing an efficient data compression scheme that compresses machine learning model data using a compression analyzer to generate compressed data, which is stored with a lookup table for decompression, reducing memory footprint, compute resources, and latency by allowing data to be processed on fewer chips and minimizing off-chip memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models store and process large amounts of data (weights, biases, input data) in uncompressed format, then the model can maintain high computational accuracy, but the memory footprint, bandwidth usage, and latency increase significantly
Solution Approach 1:
The patent applies parameter changes by modifying the data representation format through compression schemes. Different compression schemes transform the original data parameters into compressed forms that occupy less memory space while preserving the essential information needed for accurate computation. The system selects and applies appropriate compression parameters based on the specific data characteristics and computational requirements.
Solution Approach 2:
The patent introduces compression and decompression circuits as intermediary components between the data storage and the processing elements. These intermediary circuits compress data before it enters the processing pipeline and decompress it when needed, acting as mediators that enable efficient data transmission and storage without compromising the computational accuracy required by the machine learning model.
2Quantity of substance
If machine learning models access data from off-chip memory, then the model can operate with larger datasets, but the latency and compute resource consumption increase
Solution Approach 1:
The patent uses parameter changes through compression to reduce the volume of data that needs to be transferred between off-chip and on-chip memory. By compressing the dataset before storage and decompressing during processing, the system maintains the ability to work with large datasets while significantly reducing the transfer time and latency associated with off-chip memory access.
Solution Approach 2:
The patent applies preliminary action by pre-compressing the data before it is stored in memory. This compression is performed in advance of the actual processing operations, so that when the data needs to be accessed during model execution, it is already in a compact form, reducing the time required for data retrieval and transfer from off-chip memory.
3Productivity
If machine learning models use more compute resources for data processing, then the model can process larger datasets faster, but the power consumption and operational costs increase
Solution Approach 1:
The patent applies parameter changes by using compressed data representation to reduce the amount of data that needs to be processed and transmitted. This compression parameter change decreases the computational workload and power consumption while maintaining processing throughput. The decompression is performed efficiently using dedicated circuits that minimize the energy cost of restoring the original data format.
Solution Approach 2:
The patent uses copying by creating compressed versions of the original data that can be stored and processed more efficiently. Instead of repeatedly accessing and processing large uncompressed datasets, the system works with compact compressed copies, reducing the computational resources and power needed for each processing operation while maintaining the ability to reconstruct the original data when necessary.
Data Source
AI summary
Systems, apparatuses and methods provide technology that compresses first data based on a first compression scheme to generate second data, where the first data is associated with a first machine learning model. The technology stores the second data into a memory, adjusts a first entry of a lookup table to correspond to the first compression scheme based on the first data being compressed based on the first compression scheme, provide the second data from the memory to processing elements of a processing array during execution of the first machine learning model, and decompresses, at the processing array, the second data based on the lookup table to obtain the first data.