Computational Storage Embedding Vector Calculation for GPU Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network training operations are computationally burdensome, particularly when processing categorical inputs, which require large storage and slow down the processing time due to the need for embedding operations.
Innovation Solution
A system comprising a GPU cluster and a computational storage system with embedding tables, where the computational storage system calculates embedded vectors speculatively without waiting for updates, allowing for reduced processing time by updating vectors based on gradients calculated during the preceding pass.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If embedding tables are stored in GPU memory for fast access, then processing speed improves, but memory capacity is insufficient for large embedding tables
Solution Approach 1:
The system divides the embedding table storage across multiple computational storage devices (CSDs), each holding a portion of the embedding table. This segmentation allows the system to handle large embedding tables that exceed single-GPU memory capacity while maintaining fast access through parallel operations across multiple devices.
Solution Approach 2:
Computational storage devices act as intermediaries between GPU memory and persistent storage. The CSDs cache embedding tables in their local memory, providing fast access to GPUs while managing data transfer and synchronization, thus resolving the conflict between speed and capacity requirements.
2Measurement precision
If the system waits for embedding table updates before processing, then data accuracy is maintained, but processing time increases
Solution Approach 1:
The system performs preliminary actions by pre-calculating and caching embedding tables in computational storage devices before they are needed for training. This allows GPUs to immediately access pre-processed embedding data without waiting for real-time updates or computations during the training pipeline.
Solution Approach 2:
The system dynamically manages embedding table updates by allowing multiple versions to coexist in computational storage devices. Different GPUs can access different versions simultaneously, enabling parallel training operations without mutual blocking, thus maintaining both accuracy and speed.
3Reliability
If data is transferred between storage and GPU frequently, then data freshness is improved, but processing efficiency decreases
Solution Approach 1:
Computational storage devices maintain continuous caching of embedding tables in their local memory, eliminating the need for repeated data transfers between persistent storage and GPUs. The CSDs handle multiple read requests from different GPUs simultaneously, ensuring continuous data availability without interrupting the training workflow.
Data Source
AI summary
A system and method for training a neural network. In some embodiments, the system includes: a graphics processing unit cluster; and a computational storage cluster connected to the graphics processing unit cluster by a cache-coherent system interconnect. The graphics processing unit cluster may include one or more graphics processing units. The computational storage cluster may include one or more computational storage devices. A first computational storage device of the one or more computational storage devices may be configured to (i) store an embedding table, (ii) receive an index vector including a first index and a second index; and (iii) calculate an embedded vector based on: a first row of the embedding table, corresponding to the first index, and a second row of the embedding table, corresponding to the second index.