Near-Memory Embedding Pooling With Compressed Far-Memory Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing memory devices face challenges in efficiently managing large embedding tables due to high data size and latency issues, which impact the performance of neural networks in applications like deep learning recommendation models.
Innovation Solution
A memory device architecture comprising a near-memory device and a far-memory device, where the far-memory device stores compressed embedding vectors, and the near-memory device performs quantization and compression of frequently accessed vectors, utilizing a buffer, address calculation circuit, decoding circuit, and pooling circuit to optimize data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If embedding tables are stored in full size in memory, then data completeness is maintained, but memory usage and access time increase significantly
Solution Approach 1:
The patent divides the embedding table into two segments: frequently accessed embedding vectors are stored in a buffer (near-memory), while less frequently accessed vectors are stored in far-memory. This segmentation allows hot data to be accessed quickly while maintaining complete data storage, resolving the contradiction between data completeness and access speed.
Solution Approach 2:
The system performs preliminary action by pre-loading frequently accessed embedding vectors into the buffer before they are needed. The buffer stores embedding vectors corresponding to frequently accessed feature columns, so when lookup requests occur, the data is already in place for immediate access, reducing access time without requiring full embedding tables in fast memory.
2Speed
If all embedding vectors are stored in the buffer, then access speed improves, but buffer memory requirements become prohibitive
Solution Approach 1:
The patent applies local quality by giving different storage locations to different embedding vectors based on their access characteristics. Frequently accessed vectors (local hot spots) are placed in the buffer with fast access, while other vectors remain in far-memory. This selective placement optimizes access speed for critical data without requiring the entire buffer to be large enough to hold all embedding vectors.
Solution Approach 2:
The system performs preliminary action by pre-loading only the frequently accessed embedding vectors into the buffer before they are needed. The buffer stores embedding vectors corresponding to frequently accessed feature columns, so when lookup requests occur, the data is already in place for immediate access, reducing access time without requiring full embedding tables in fast memory.
3Quantity of substance
If embedding vectors are compressed to reduce data size, then storage efficiency improves, but decoding complexity and time increase
Solution Approach 1:
The patent changes the parameter of embedding vector representation by storing compressed versions in far-memory and using quantization techniques. The embedding vectors are compressed by reducing precision (e.g., from 32-bit floating point to lower precision formats), which improves storage efficiency. The decoding complexity is managed by using simple quantization operations rather than complex decompression algorithms.
Data Source
AI summary
A memory device including a near-memory device and a far-memory device, wherein the far-memory device stores compressed embedding vectors, the near-memory device including: a buffer to receive a request for pooling an embedding vector and output a hit or miss signal in response to the request, wherein the hit signal includes the embedding vector corresponding to the request; an address calculation circuit to calculate a starting memory address and memory size of the embedding vector in response to the miss signal; a decoding circuit to obtain a compressed embedding vector corresponding to the request from the far-memory device based on the starting memory address and memory size, and output the embedding vector corresponding to the request by decoding the compressed embedding vector; and a pooling circuit to output a pooled embedding vector by performing a pooling operation on the embedding vector.