AI Platform Data Acquisition via Distributed RDMA Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence clusters face inefficiencies in resource utilization due to a one-to-many storage design, leading to high network and magnetic-disk burdens on the master storage node, which results in wasted resources and decreased performance.
Innovation Solution
Implement a shared storage network using remote direct memory access technology to distribute data operations across compute nodes, optimizing data transmission based on task burden and availability, and constructing a fully connected network file system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a one-to-many storage design mode is used where one shared storage is loaded to each compute node, then the storage structure is simple and easy to implement, but the network burden and magnetic-disk I/O burden are concentrated on one node resulting in low resource usage efficiency
Solution Approach 1:
The patent segments the centralized storage burden by enabling each compute node to store local copies of datasets. Instead of one master storage node handling all data operations, the storage functionality is distributed across multiple nodes, with each node capable of serving as a storage source for others. This segmentation reduces the concentration of network and I/O burden on a single node.
Solution Approach 2:
The patent implements a copying mechanism where datasets are replicated across multiple compute nodes. When a node needs a dataset, it can obtain local copies from other nodes that already have them stored, rather than all nodes accessing a single master storage node. This copying approach reduces network traffic and I/O operations on any single node.
2Ease of operation
If the data set is placed on the master storage node for user directory transfer or local caching, then data access is centralized and simple to manage, but the resources of the artificial intelligence cluster are wasted and storage and network resources are not sufficiently utilized
Solution Approach 1:
The patent merges the storage capabilities of multiple compute nodes into a distributed storage system. Instead of relying solely on the master storage node, the system combines the storage resources of all compute nodes to create a collective storage pool. This allows any node to contribute its stored datasets to the cluster, improving resource utilization and reducing waste.
Solution Approach 2:
The patent enables compute nodes to serve themselves and each other by storing and retrieving datasets locally or from peer nodes. Each node maintains its own dataset cache and can provide datasets to other nodes that need them, reducing the need for centralized data management and minimizing network traffic to the master storage node.
3Productivity
If a mass quantity of file operations are performed in the artificial intelligence cluster, then the training tasks can be executed, but the performance of the artificial intelligence cluster is affected due to high network and storage burdens
Solution Approach 1:
The patent implements preliminary action by pre-distributing datasets to multiple compute nodes before they are needed for training tasks. Nodes proactively store and cache datasets locally or on peer nodes in advance, so that when training tasks require these datasets, the data is already available locally, reducing network traffic and I/O operations during actual training execution.
Data Source
AI summary
The present application discloses a method for acquiring datum of artificial intelligence platform, device, apparatus and medium, which includes: acquiring datum operation request initiated by target node of the artificial intelligence cluster aiming at target datum; counting current datum operation burden of each of the other compute nodes; according to order of the current datum operation task burden from lower to higher, traversing all of the other compute nodes sequentially, and in the traversal process, judging whether the compute node that is currently being traversed has already stored the target datum; under the condition it has already stored the target datum, by shared storage network that is pre-constructed between the different nodes of the artificial intelligence cluster based on remote direct datum access technology, transmitting the target datum of the compute node that is currently being traversed to the target node.