Computational Storage AI Training Architecture for Bandwidth Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The bandwidth bottleneck in storage-offload training for large language models due to the lower bandwidth of storage devices compared to GPUs is a significant challenge.
Innovation Solution
An electronic device architecture that includes a host with processors, a computational storage device (CSD) with storage and accelerators, and an interconnect for direct communication between the host and CSD, allowing for the update of gradients and optimizer states through inner-paths, reducing data traffic and bandwidth constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If storage-offload training is used to train large language models, then memory capacity is improved, but bandwidth bottleneck occurs due to lower storage bandwidth compared to GPU devices
Solution Approach 1:
The system segments the training workflow into distinct phases: forward propagation, backward propagation, and parameter update. During forward propagation, parameters are loaded from storage to GPU. During backward propagation, gradients are computed and stored back to storage. This segmentation allows independent optimization of each phase, reducing the impact of storage bandwidth limitations on overall training performance.
Solution Approach 2:
The system performs preliminary actions by pre-loading parameters into GPU memory before training begins, and by organizing gradient computations to minimize storage access frequency. The gradient checkpointing technique is used to selectively store intermediate activations, reducing the need for frequent storage reads during backward propagation.
2Quantity of substance
If multiple CSDs are integrated to increase memory capacity, then model training capability is improved, but system complexity increases
Solution Approach 1:
The CSD architecture implements a universal interface and standardized communication protocol that allows multiple CSD devices to be integrated into the system. Each CSD can independently serve as both a storage device and a computation unit, providing multi-functionality that reduces the need for additional specialized components and simplifies system integration.
Solution Approach 2:
The system employs a hierarchical memory architecture where CSD devices are nested within the host system's memory hierarchy. The CSDs appear as extended memory to the host processor, allowing seamless integration without requiring complex external interface management. This nesting approach simplifies the system architecture by presenting a unified memory interface to the application layer.
Data Source
AI summary
An electronic device includes a host including a host memory, a first processor, and a second processor. The first processor updates gradients based on parameters stored in the host memory. The electronic device further includes a computational storage device (CSD) including a storage device storing parameters of an artificial intelligence learning model, gradients, and optimizer states (OSs) and an accelerator configured to transmit and receive the parameters, the gradients, and the OSs to and from the storage device through an inner-path. The electronic device also includes an interconnect configured to connect the host to the CSD and transmit the gradients and the parameters between the host and the CSD. The second processor controls the accelerator to update the OSs and the parameters based on the gradients and the OSs.