Distributed Memory Boards for Large Model Training Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large model training tasks impose high demands on processor performance, memory capacity, storage systems, and network connections, leading to increased host load, slow training efficiency, and risks of data loss due to power failures.
Innovation Solution
A data processing system with multiple hosts, each equipped with acceleration devices and memory boards, distributes model training tasks in parallel, utilizing non-volatile storage modules for data management and synchronization through protocols like CXL and RDMA to reduce host load and improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If model training tasks are executed using host memory to store data, intermediate results, and weights, then the training can be performed, but the host load increases and training efficiency slows down
Solution Approach 1:
The system segments the monolithic host memory system into distributed memory boards across multiple hosts. Each memory board independently stores portions of training data, intermediate results, and weights. This segmentation distributes the storage burden, reducing individual host load while maintaining adequate memory capacity for efficient parallel training operations.
Solution Approach 2:
The system transitions from a single-host memory architecture to a multi-host distributed memory architecture, adding spatial dimensionality to the memory system. Memory boards are distributed across multiple physical hosts, creating a three-dimensional memory topology that increases total capacity and bandwidth without overloading any single host.
2Reliability
If complete model weights are saved in host memory when training goes wrong, then recovery is possible, but it takes a lot of memory access time
Solution Approach 1:
The system performs preliminary actions by continuously maintaining up-to-date copies of model weights in the distributed memory board system during training. Before failures can occur, the latest weight data is already persisted across the distributed memory infrastructure, eliminating the need for time-consuming emergency saves and enabling immediate recovery.
Solution Approach 2:
The system creates and maintains multiple copies of weight data across distributed memory boards. Instead of relying on a single host memory location, weight data is replicated across the distributed memory system, allowing any node to access the latest weights without requiring time-consuming transfers from the original source.
3Ease of operation
If model weights are stored in host memory, then they are readily accessible, but they are at risk of power failure and loss
Solution Approach 1:
The system merges the functionality of host memory with persistent storage by integrating memory boards that combine volatile and non-volatile memory technologies. This hybrid approach maintains the fast accessibility of volatile memory while incorporating the data persistence of non-volatile memory, protecting against power failures.
Solution Approach 2:
The system implements beforehand cushioning by using non-volatile memory components in the memory boards to create a protective layer against power failures. This persistent storage layer acts as a cushion that preserves weight data even when power is lost, preventing data loss while maintaining quick recovery capabilities.
Data Source
AI summary
The present application discloses a data processing system and method, and a medium in the technical field of model training. According to the present application, a model training task is executed using acceleration devices in a host; and train data, intermediate results and weight data are stored using a memory board in the host, whereby the host does not need to execute the task or store the data.