Compressed Embedding Matrix Memory Architecture for Low-Capacity Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large memory requirements and overhead in processing language models due to the size of embedding matrices, which are difficult to handle in devices with limited memory capacity, leading to repeated data access and inefficient arithmetic operations.
Innovation Solution
An electronic device with separate memories operating at different speeds, storing compressed embedding matrices and scaling factors, where frequently used vectors and factors are stored in faster, smaller-capacity memories, optimizing access and processing through a neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a language model with a large embedding matrix is used to handle at least 10,000 words, then the language model can process a wide vocabulary, but the memory capacity required becomes extremely large and difficult to use in devices with small memory capacity
Solution Approach 1:
The patent divides the embedding matrix into multiple blocks, where each block corresponds to a subset of the vocabulary (e.g., 10,000 words per block). Instead of loading the entire embedding matrix into memory at once, the system selectively loads only the required block corresponding to the input word's frequency range. This segmentation allows the system to handle large vocabularies while using minimal memory capacity at any given time.
Solution Approach 2:
The patent applies different storage strategies to different parts of the embedding matrix based on word frequency. Frequently used words (high frequency) are stored in fast memory (e.g., SRAM or cache), while less frequently used words are stored in slower memory (e.g., DRAM or flash memory). This local quality differentiation optimizes both memory usage and access speed, allowing the system to efficiently handle the full vocabulary without requiring uniform high-speed memory for all words.
2Reliability
If all data of the embedding matrix is read during arithmetic operation, then complete data is available for processing, but the memory bandwidth required becomes excessively large and slows down processing speed
Solution Approach 1:
The patent extracts and stores only the frequently accessed portions of the embedding matrix (corresponding to high-frequency words) in fast memory, separating them from the less frequently accessed data. During arithmetic operations, the system reads only the necessary block from fast memory rather than reading the entire embedding matrix from slow memory. This extraction approach ensures that the most critical data is readily available, maintaining processing reliability while dramatically reducing memory bandwidth requirements and improving processing speed.
3Quantity of substance
If some data of the embedding matrix is read multiple times and arithmetic operation is repeatedly performed, then memory capacity requirements are reduced, but processing overhead increases significantly
Solution Approach 1:
The patent performs preliminary organization of the embedding matrix data by sorting words based on their frequency of use and pre-grouping them into blocks. Frequently used words are identified in advance and placed in fast memory blocks, while less frequent words are placed in slower memory blocks. This preliminary action ensures that during actual processing, the system can directly access the required data from fast memory without needing to repeatedly read and reprocess data from slow memory, thereby reducing processing overhead while maintaining efficient memory usage.
Data Source
AI summary
An electronic device is provided. The electronic device includes a first memory configured to operate at a first speed and store compressed vectors corresponding to words, and scaling factors corresponding to the compressed vectors; a second memory that is faster than the first memory and is configured to store a first group of the compressed vectors, and store a first group of the scaling factors; and a processor configured to obtain a first compressed vector and a first scaling factor corresponding to an input word from the first memory or the second memory and process the obtained first compressed vector and the obtained first scaling factor by using a neural network.