Embedding Table Compression for Independent Lookup and Decompression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models require significant memory and computational resources due to large embedding tables, which are challenging to scale and run on devices with limited hardware capabilities, especially mobile devices.
Innovation Solution
Individual compression and decompression of embeddings in embedding tables are performed independently, allowing efficient storage and lookup without loading the entire table into memory, using compression operations that are not reliant on underlying machine-learning platforms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If embedding tables are stored in full size to maintain model accuracy, then model performance is improved, but memory usage and computational resources increase significantly
Solution Approach 1:
The embedding table is divided into multiple shards or partitions that can be stored separately. Only the necessary shards containing embeddings for current input tokens need to be loaded into memory, while other shards remain on disk or external storage. This segmentation allows the model to maintain full embedding table capacity for accuracy while using minimal memory at any given time.
Solution Approach 2:
Embeddings are pre-computed and stored in compressed or sparse formats on external storage before runtime. During inference, pre-processed embedding data is quickly retrieved and decompressed only for the specific tokens needed, rather than loading and processing the entire embedding table in memory.
2Speed
If large embedding tables are loaded into memory to enable fast lookup, then lookup speed is improved, but power consumption increases
Solution Approach 1:
The system implements lazy loading where embedding shards are loaded into memory periodically or on-demand based on the input tokens encountered. Rather than continuously holding the entire embedding table in memory, the system loads only necessary portions, processes them, and allows memory to be freed when not needed, reducing overall power consumption while maintaining fast lookup when data is in memory.
3Quantity of substance
If compression is applied to reduce embedding table size, then storage efficiency is improved, but decompression complexity increases
Solution Approach 1:
Different compression techniques are applied to different portions of the embedding table based on local characteristics. Frequently accessed or important embeddings may use lighter compression schemes for faster decompression, while less critical embeddings use more aggressive compression. This localized approach optimizes the balance between storage efficiency and decompression complexity for different parts of the data.
Data Source
AI summary
The present disclosure provides systems, methods, and computer program products for providing efficient embedding table storage and lookup in machine-learning models. A computer-implemented method may include obtaining an embedding table comprising a plurality of embeddings respectively associated with a corresponding index of the embedding table, compressing each particular embedding of the embedding table individually allowing each respective embedding of the embedding table to be decompressed independent of any other embedding in the embedding table, packing the embedding table comprising individually compressed embeddings with a machine-learning model, receiving an input to use for locating an embedding in the embedding table, determining a lookup value based on the input to search indexes of the embedding table, locating the embedding based on searching the indexes of the embedding table for the determined lookup value, and decompressing the located embedding independent of any other embedding in the embedding table.