Storage-Side Sparse Embedding to Relieve Bandwidth Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage devices face bandwidth bottlenecks during memory-intensive embedding operations, particularly in recommendation systems using deep learning, due to the large size of one-hot vectors required for embedding operations.
Innovation Solution
The storage device generates vector data based on non-zero data and indices provided by the host, performing multiplication operations with stored matrix data to calculate embedding vectors, thereby reducing bandwidth requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If one-hot vectors are used for embedding operations in recommendation systems, then the embedding operation can be performed, but bandwidth bottleneck occurs due to large vector size
Solution Approach 1:
The patent extracts only the non-zero elements from the one-hot vector and transmits them along with their indices to the storage device. Instead of transmitting the entire large-sized one-hot vector, only the essential non-zero data portions are extracted and transmitted, significantly reducing bandwidth requirements while enabling the embedding operation to proceed.
Solution Approach 2:
The patent inverts the traditional approach by having the storage device generate the complete vector data locally using the received non-zero data and indices, rather than having the host device transmit the complete vector data. This inversion shifts the data generation responsibility to the storage device, reducing bandwidth requirements.
2Measurement precision
If complete one-hot vectors are transmitted to storage device, then embedding calculation can be performed accurately, but transmission bandwidth increases significantly
Solution Approach 1:
The patent extracts only the non-zero elements from the one-hot vector for transmission. By identifying and transmitting only the non-zero data portions along with their positions (indices), the patent maintains the ability to accurately reconstruct the complete vector at the storage device while minimizing transmission volume.
Solution Approach 2:
The patent changes the representation parameters of the vector data by using sparse representation (storing only non-zero elements and their indices) instead of dense representation (storing all elements). This parameter change enables accurate embedding calculation while significantly reducing data transmission volume.
3Productivity
If host device transmits full one-hot vector, then storage device can perform embedding operation, but host device bandwidth capacity is exceeded
Solution Approach 1:
The patent extracts and transmits only the necessary non-zero data portions rather than the complete one-hot vector. This extraction approach enables the embedding operation to be performed with minimal data transmission, avoiding the bandwidth bottleneck that would occur with full vector transmission.
Solution Approach 2:
The patent introduces an intermediary representation format (sparse format with non-zero data and indices) that mediates between the host device and storage device. This intermediary representation enables the embedding operation to proceed while reducing the bandwidth burden on the host device.
Data Source
AI summary
Methods, systems, and devices for alleviating a bandwidth bottleneck during an embedding operation are described. An example storage device, based on the disclosed technology, includes a memory device configured to store matrix data, a memory controller, coupled to the memory device, configured to receive, from a host, non-zero data and the index of the non-zero data, and generate vector data based on the non-zero data and the index, and an operating component, coupled to the memory device and the memory controller, configured to perform a multiplication operation between the matrix data and the vector data.