Distributed file system adapted to GPU cluster training, creation method, electronic device, and storage medium

Abstract

The present disclosure relates to a distributed file system adapted to GPU cluster training, a creation method, an electronic device, and a storage medium. The system comprises: a management module, used for, after allocating a target GPU cluster to a target training task, creating service ends for a plurality of first computer nodes comprised in the target GPU cluster and creating clients for a plurality of second computer nodes comprised in the target GPU cluster, at least one hardware GPU being deployed on each computer node comprised in the target GPU cluster; the service ends used for using preset local memory spaces deployed on corresponding first computer nodes to construct a GPU cluster distributed file system adapted to the target training task; and the clients used for executing an IO operation on the GPU cluster distributed file system in the process of using the target GPU cluster to execute the target training task. Embodiments of the present disclosure can effectively meet the IO requirements in a GPU cluster training scenario.

Description

A distributed file system adapted for GPU cluster training, its creation method, electronic device, and storage medium.

[0001] This application claims priority to Chinese Patent Application No. 202411640610.8, filed on November 15, 2024, entitled "A Distributed File System Adapted to GPU Cluster Training and its Creation Method, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of computer technology, and in particular to a distributed file system adapted for GPU cluster training and its creation method, electronic devices, and storage media. Background Technology

[0003] In scenarios involving large-scale model training using GPU clusters, file system usage primarily involves three types of input / output (IO) operations: dataset reading, checkpoint data writing, and checkpoint data reading. Currently, GPU cluster training typically utilizes externally configured storage systems, such as general-purpose network file systems, parallel file systems, and object storage. Data read performance depends on the performance of this storage system and may not be suitable for the data read performance requirements of GPU cluster training. Summary of the Invention

[0004] This disclosure proposes a technical solution for a distributed file system adapted for GPU cluster training, a method for creating such a file system, an electronic device, and a storage medium.

[0005] According to one aspect of this disclosure, a distributed file system adapted for GPU cluster training is provided, comprising: a management module, a server, and a client; the management module is configured to, after allocating a target GPU cluster for a target training task, create a server for a plurality of first computer nodes included in the target GPU cluster and a client for a plurality of second computer nodes included in the target GPU cluster, wherein each computer node included in the target GPU cluster has at least one hardware GPU deployed on it; the server is configured to, using a preset local memory space deployed on the corresponding first computer node, construct a GPU cluster distributed file system adapted for the target training task; the client is configured to, during the execution of the target training task using the target GPU cluster, perform I / O operations on the GPU cluster distributed file system.

[0006] In one possible implementation, the preset local memory space deployed on each first computer node has the same memory size.

[0007] In one possible implementation, the GPU cluster distributed file system is used to store the dataset required for the target training task, as well as checkpoint data during the training process.

[0008] In one possible implementation, the client includes a system call interception module and a client communication module; the server includes a server communication module; the system call interception module is used to intercept target file system calls in the second computer node where the client is located, and to convert the target file system calls into IO requests, wherein the target file system calls are used to access the GPU cluster distributed file system; the client communication module is used to send the IO requests to the server communication module.

[0009] In one possible implementation, the client communication module includes a Remote Procedure Call (RPC) client module; the server communication module includes an RPC server module; the RPC client module is used to send the IO request to the RPC server module using a non-blocking RPC network communication mode.

[0010] In one possible implementation, the client communication module includes an IPC client module; the server communication module includes an IPC server module; the IPC client module is configured to send the IO request to the IPC server module when the first computing point where the server is located and the second computer node where the client is located are the same computer node.

[0011] In one possible implementation, the client includes: a file graph; the file graph is used to store the client's access records to the GPU cluster's distributed file system.

[0012] In one possible implementation, the IO request includes a metadata IO request; the client includes a data segmentation module; the server includes a metadata database; the data segmentation module is used to segment the target data to be written to the GPU cluster distributed file system into blocks, determine multiple data blocks corresponding to the target data, and metadata information for each data block, wherein the metadata information corresponding to each data block is used to indicate the target server to which the data block needs to be written; the client communication module is used to send the metadata IO request to each server communication module, wherein the metadata IO request is used to request that the metadata information of each data block corresponding to the target data be written to each server; the metadata database in each server is used to respond to the metadata IO request and store the metadata information of each data block corresponding to the target data.

[0013] In one possible implementation, the IO request includes a data IO request; the server includes a persistence module; the communication client module is used to send data IO requests to multiple target servers, wherein the data IO request corresponding to each target server is used to request that a specified data block corresponding to the target data be written to the target server; the persistence module in each target server is used to respond to the received data IO request and, according to the metadata IO request stored in the metadata database of the target server, store the specified data block to be written to the target server into the preset local memory space corresponding to the target server.

[0014] In one possible implementation, the persistence module in each target server stores each specified data block that needs to be written to the target server into a chunk file in the preset local memory space corresponding to the target server.

[0015] In one possible implementation, the number of specified data blocks corresponding to the target data written to each target server is the same.

[0016] In one possible implementation, the data I / O request corresponding to each target server includes client memory region information corresponding to a specified data block that needs to be written to the target server; the persistence module in each target server, based on the client memory region information included in the received data I / O request, reads the specified data block from the memory space specified by the client memory region information through Remote Direct Memory Access (RDMA).

[0017] In one possible implementation, the GPU cluster distributed file system is further configured to asynchronously send all or part of the checkpoint data to a PFS outside the target GPU cluster.

[0018] In one possible implementation, the management module is configured to delete the client and the server after the target training task has been completed using the target GPU cluster.

[0019] According to one aspect of this disclosure, a method for creating a distributed file system adapted for GPU cluster training is provided, comprising: after allocating a target GPU cluster for a target training task, creating a server for a plurality of first computer nodes included in the target GPU cluster, and creating a client for a plurality of second computer nodes included in the target GPU cluster, wherein each computer node included in the target GPU cluster has at least one hardware GPU deployed on it; based on the server, constructing a GPU cluster distributed file system adapted for the target training task using a preset local memory space deployed on the corresponding first computer node; and based on the client, performing I / O operations on the GPU cluster distributed file system during the execution of the target training task using the target GPU cluster.

[0020] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0021] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0022] In this embodiment, a distributed file system adapted for GPU cluster training is designed, including a management module, a server, and a client. After allocating a target GPU cluster for a target training task, the management module creates a server for multiple first computer nodes included in the target GPU cluster and a client for multiple second computer nodes included in the target GPU cluster according to the data read and write requirements of the target training task. This enables the server to flexibly construct a GPU cluster distributed file system adapted for the target training task using the preset local memory space deployed on the corresponding first computer node. Furthermore, during the execution of the target training task using the target GPU cluster, the client can perform IO operations on the GPU cluster distributed file system to effectively meet the IO requirements in the GPU cluster training scenario.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0025] Figure 1 shows a block diagram of a distributed file system adapted for GPU cluster training according to an embodiment of the present disclosure.

[0026] Figure 2 illustrates a schematic diagram of the interaction between a client and a server according to an embodiment of the present disclosure.

[0027] Figure 3 illustrates a schematic diagram of the transmission of metadata information and data between the client and the server according to an embodiment of the present disclosure.

[0028] Figure 4 illustrates a schematic diagram of a client writing a file to a GPU cluster distributed file system according to an embodiment of the present disclosure.

[0029] Figure 5 shows a flowchart of a method for creating a distributed file system adapted for GPU cluster training according to an embodiment of the present disclosure.

[0030] Figure 6 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0031] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0032] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0033] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0034] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0035] In scenarios involving large-scale model training based on Graphics Processing Unit (GPU) clusters, the use of the file system mainly includes three types of input / output (IO) operations: dataset reading, checkpoint data writing, and checkpoint data reading.

[0036] Dataset reading: At the beginning of each training iteration on a GPU cluster, the training framework (e.g., PyTorch) uses a data loader to load the dataset. Dataset reading is characterized by a large amount of random I / O, primarily 4k random reads. Tools can also be used to pre-convert the dataset's format, allowing random reads to be converted into sequential read operations.

[0037] Checkpoint data writing: After a certain number of iterations of training on a GPU cluster, the training framework needs to persistently save the output of the training process, i.e., the checkpoint data. Saving checkpoint data serves two main purposes: firstly, it can be directly output for fine-tuning the model and continuing training; secondly, in the event of a training failure, training can resume based on the checkpoint data after recovery. Checkpoint data is typically very large; for example, the checkpoint data for a 70B model is approximately 980GB. Therefore, checkpoint data is usually sliced ​​and distributed for storage and retrieval. The characteristic of checkpoint data writing is a large number of sequential write operations, primarily 1MB sequential writes.

[0038] Checkpoint data reading: During large-scale model training based on large-scale GPU clusters, software or hardware failures inevitably occur, causing training interruptions. After the failure is resolved, training needs to resume, which requires reading previously saved checkpoint data. Checkpoint data reading is characterized by a large number of sequential read operations, primarily 1MB sequential reads.

[0039] The storage systems used for reading datasets and checking point data during GPU cluster training can primarily fall into three categories: 1. General-purpose network file systems or distributed file systems, such as NFS and CephFS. These ensure data reliability but require independent deployment and maintenance, and their performance is relatively poor. 2. Parallel file systems, such as Lustrefs, gpfs, and beegfs. These offer good performance but also require independent deployment and maintenance. Physically, they are mainly divided into integrated (single storage device) and distributed (multiple storage servers) architectures. 3. Object storage offers good scalability but poor performance. In some scenarios, distributed caching, such as JuiceFS and Alluxio, can be used for acceleration, but this requires pre-downloading the data to the local file system within the GPU cluster, or the GPU cluster needs to provide a client file system similar to S3FS to offer POSIX semantic access.

[0040] The current storage systems used for dataset reading and checkpoint data reading / writing in GPU cluster training are large and complex, and most of their functions are not required for the three core IO business scenarios in GPU cluster training. Therefore, this disclosure provides a distributed file system adapted for GPU cluster training, which can adapt to the three core IO business scenarios in GPU cluster training, and is simple to operate and maintain, saving costs while improving performance. The distributed file system adapted for GPU cluster training provided in this disclosure is described in detail below.

[0041] Figure 1 shows a block diagram of a distributed file system adapted for GPU cluster training according to an embodiment of the present disclosure. As shown in Figure 1, the system includes a management module, a server, and a client.

[0042] The management module is used to create servers for multiple first computer nodes included in the target GPU cluster and clients for multiple second computer nodes included in the target GPU cluster after allocating the target GPU cluster for the target training task, wherein each computer node included in the target GPU cluster has at least one hardware GPU deployed on it.

[0043] A GPU cluster is a computer cluster consisting of multiple computer nodes. Each computer node is equipped with at least one hardware GPU, as well as single-core CPUs, multi-core CPUs, or even multi-CPU computing resources. GPU clusters utilize the computing power of the hardware GPUs to perform very fast computational tasks.

[0044] For a specific training task, a target GPU cluster can be allocated based on the Kubernetes platform, according to the task requirements. The number of computer nodes included in the target GPU cluster depends on the task requirements of the target training task, and this disclosure does not impose a specific limitation on this.

[0045] After allocating a target GPU cluster for the target training task and before starting the execution of the target training task, the management module can create servers for multiple first computer nodes included in the target GPU cluster according to the actual IO requirements of the target training task.

[0046] The management module can reuse the operator mechanism for cluster management in the Kubernetes platform to create and distribute the server to multiple first computer nodes in the target GPU cluster.

[0047] The server-side component is used to construct a GPU cluster distributed file system adapted to the target training task by utilizing the preset local memory space deployed on the corresponding first computer node.

[0048] Multiple servers can communicate via Remote Direct Memory Access (RDMA) network, enabling the pre-defined local memory spaces deployed on multiple primary computer nodes to serve as the backend of the file system, jointly constructing a GPU cluster distributed file system adapted to the target training task.

[0049] The number of first computer nodes can be determined based on the actual I / O requirements of the target training task and the size of the preset local memory space deployed on each first computer node. The key is to ensure that the total size of the preset local memory space deployed on all first computer nodes, i.e., the storage space of the GPU cluster distributed file system constructed above, can meet the actual I / O requirements of the target training task. The first computer nodes can be all the computer nodes in the target GPU cluster, or only a portion of the computer nodes in the target GPU cluster; this disclosure does not impose specific limitations on this.

[0050] As shown in Figure 1, only some of the computer nodes in the target GPU cluster serve as the first computer nodes, with servers deployed there.

[0051] In one possible implementation, the pre-defined local memory space deployed on each first computer node has the same memory size.

[0052] To ensure the balance of the distributed file system of the GPU cluster constructed above, the memory of the preset local memory space deployed on each first computer node can be set to the same size.

[0053] In one example, the preset local memory space deployed on the first computer node can be a local NVME (Non-Volatile Memory Express) disk space deployed on the first computer node, or a shared memory space deployed on the first computer node (e.g., shared memory / dev / shm in the Linux operating system), or other local memory spaces deployed on the first computer node. This disclosure does not specifically limit the specific type of local memory space deployed on the first computer node.

[0054] For example, a target GPU cluster is allocated for a target training task. The target GPU cluster includes N computer nodes. Before the target training task starts, the management module creates a server for each computer node in the target GPU cluster, meaning each computer node in the target GPU cluster serves as a first computer node. The preset local memory space deployed on each first computer node is 15.36TB. At this point, the N servers can communicate through an RDMA network to jointly construct a GPU cluster distributed file system adapted to the target training task, with a storage space size of 15.36TB×N.

[0055] The management module is also used to create clients for multiple secondary computer nodes included in the target GPU cluster.

[0056] The number of second computer nodes can be determined based on the actual data read requirements of each computer node in the target GPU cluster. During the execution of the target training task in the target GPU cluster, each computer node in the target GPU cluster executes a different training process. Based on the training process executed by each computer node, it can be determined whether that computer node has a data read requirement. The management module can create clients only for multiple second computer nodes in the target GPU cluster that have a data read requirement, without needing to create clients for computer nodes in the target GPU cluster that do not have a data read requirement. The second computer nodes can be all the computer nodes in the target GPU cluster, or only a portion of the computer nodes in the target GPU cluster; this disclosure does not specifically limit this.

[0057] As shown in Figure 1, only some of the computer nodes in the target GPU cluster serve as secondary computer nodes, with servers deployed there.

[0058] Furthermore, certain computer nodes in the target GPU cluster can serve as both first and second computer nodes, meaning that both the server and the client can be deployed on these computer nodes.

[0059] As shown in Figure 1, some computer nodes in the target GPU cluster have both server-side and client-side deployments.

[0060] The client is used to perform I / O operations on the distributed file system of the GPU cluster during the execution of the target training task using the target GPU cluster.

[0061] The client is integrated into the target training task process so that the client can access the GPU cluster distributed file system adapted to the target training task. In this way, during the execution of the target training task using the target GPU cluster, the client can perform IO operations on the GPU cluster distributed file system.

[0062] In one possible implementation, a distributed file system for the GPU cluster is used to store the dataset required for the target training task, as well as checkpoint data during the training process.

[0063] Since the GPU cluster distributed file system can store the dataset required for the target training task, as well as the checkpoint data during the training process, the client can perform I / O operations such as dataset reading, checkpoint data writing, and checkpoint data reading on the GPU cluster distributed file system, adapting to the three core I / O business scenarios in the GPU cluster training scenario.

[0064] In this embodiment, a distributed file system adapted for GPU cluster training is designed, including a management module, a server, and a client. After allocating a target GPU cluster for a target training task, the management module creates a server for multiple first computer nodes included in the target GPU cluster and a client for multiple second computer nodes included in the target GPU cluster according to the data read and write requirements of the target training task. This enables the server to flexibly construct a GPU cluster distributed file system adapted for the target training task using the preset local memory space deployed on the corresponding first computer node. Furthermore, during the execution of the target training task using the target GPU cluster, the client can perform IO operations on the GPU cluster distributed file system to effectively meet the IO requirements in the GPU cluster training scenario.

[0065] In one possible implementation, the client includes a system call interception module and a client communication module; the server includes a server communication module; the system call interception module is used to intercept the target file system calls in the second computer node where the client is located, and to convert the target file system calls into IO requests, wherein the target file system calls are used to access the distributed file system of the GPU cluster; the client communication module is used to send the IO requests to the server communication module.

[0066] Figure 2 illustrates the interaction between a client and a server according to an embodiment of the present disclosure. As shown in Figure 2, each client includes a system call interception module.

[0067] When the client, located on the second computer node, needs to read data during the training process, it will initiate a target file system call to the GPU cluster distributed file system constructed above. The system call interception module included in the client can intercept the target file system call, and then convert the target file system call into an IO request through random hashing. The client communication module will then send the IO request to the server communication module included in the server.

[0068] In one possible implementation, the client communication module includes a Remote Procedure Call (RPC) client module; the server communication module includes an RPC server module; and the RPC client module is used to send IO requests to the RPC server module in a non-blocking RPC network communication mode.

[0069] The client and server can communicate across computer nodes via a high-performance RPC network. In addition, to further improve performance, the client and server use a non-blocking RPC network communication mode.

[0070] Non-blocking RPC network communication mode refers to a mode where, after an RPC client module initiates a remote call to an RPC server module, it does not wait for the remote call to complete but immediately returns and continues executing subsequent code. Once the remote call process is complete, the RPC client obtains the result through some means (e.g., callback functions, event notifications, etc.).

[0071] The RPC client module in this embodiment can use a non-blocking RPC network communication mode to communicate with any RPC server module.

[0072] As shown in Figure 2, each client includes an RPC client module, and each server includes an RPC server module. Each RPC client module can communicate with each RPC server module based on a high-performance network (e.g., an RPC network).

[0073] In one possible implementation, the client communication module includes an IPC client module; the server communication module includes an IPC server module; the IPC client module is used to send IO requests to the IPC server module when the first computing point where the server is located and the second computer node where the client is located are the same computer node.

[0074] When the client and server are located on the same computer node, in order to improve performance, inter-process communication (IPC) can be used within the computer node.

[0075] As shown in Figure 2, the client includes an IPC client module, and the server includes an IPC server module. For clients and servers within the same computer node, communication within the computer node can be achieved based on the IPC client module and the IPC server module.

[0076] In one possible implementation, the client includes: a file graph; the file graph is used to store the client's access records to the GPU cluster's distributed file system.

[0077] The file graph in the client stores the client's access history to the GPU cluster's distributed file system, for example, recording which files in the GPU cluster's distributed file system the client has currently accessed.

[0078] In one example, the file graph can store data in a map data structure.

[0079] In this embodiment of the disclosure, the high-performance network is abstracted and divided into an RPC layer and a bulk layer. Figure 3 shows a schematic diagram of the transmission of metadata information and data between the client and the server according to an embodiment of the disclosure. As shown in Figure 3, the client and server use the RPC layer to transmit metadata information point-to-point, and the bulk layer to transmit the actual data. The bulk layer can use RDMA to transmit the actual data.

[0080] In one example, the underlying implementation of RPC transmission could involve calling the OFED interface via the Fabric API, then calling the Verbs or TCP interface, and finally transmitting data through a high-performance network card driver. The specific transmission process can be found in related RPC transmission techniques; this disclosure does not impose specific limitations on it.

[0081] In one example, data transmission is performed based on the RPC layer, and a non-blocking RPC communication mode can be used to improve data transmission performance.

[0082] In one example, a thread pool can be pre-built, and then the threads in the thread pool can be used to implement data transmission between the client and the server to improve data transmission performance.

[0083] The following section provides a detailed description of the data transmission between the client and the server.

[0084] In one possible implementation, the IO request includes: a metadata IO request; the client includes: a data segmentation module; the server includes: a metadata database; the data segmentation module is used to segment the target data that needs to be written to the GPU cluster's distributed file system into blocks, determine multiple data blocks corresponding to the target data, and metadata information for each data block, wherein the metadata information corresponding to each data block is used to indicate the target server to which the data block needs to be written; the client communication module is used to send the metadata IO request to each server communication module, wherein the metadata IO request is used to request that the metadata information of each data block corresponding to the target data be written to each server; the metadata database in each server is used to respond to the metadata IO request and store the metadata information of each data block corresponding to the target data.

[0085] In this embodiment of the disclosure, the GPU cluster distributed file system is a fully distributed system, that is, each client can communicate with each server in the GPU cluster distributed file system, so that each client can independently parse each server in the GPU cluster distributed file system without needing a central data structure that can track metadata information or the location of data.

[0086] To achieve balanced data distribution for large files, before requesting to write the target data to the GPU cluster's distributed file system for storage, the client can use a data segmentation module to divide the target data into multiple data blocks of the same size and determine the metadata information of each data block. The metadata information corresponding to each data block is used to indicate the target server to which the data block needs to be written.

[0087] As shown in Figure 2, each server includes a metadata database for storing metadata information.

[0088] In one possible implementation, the number of specified data blocks corresponding to the target data written to each target server is the same.

[0089] To achieve balanced data distribution for large files, the number of specified data blocks corresponding to the target data written to each target server is the same.

[0090] In one example, for any given client, it is possible to choose to distribute all data blocks corresponding to the target data evenly across all servers, or to distribute all data blocks corresponding to the target data evenly across some servers.

[0091] For example, a GPU cluster distributed file system has 10 servers. The client distributes all data blocks corresponding to the target data evenly across all servers, meaning all 10 servers are target clients. Furthermore, the client can use a data partitioning module to divide the target data into 20 equal-sized data blocks, and subsequently store 2 data blocks on each server.

[0092] For example, a GPU cluster distributed file system has 10 servers. The client distributes all data blocks corresponding to the target data evenly across 8 servers, meaning it selects 8 servers from the 10 (e.g., randomly, or based on configuration information) as the target client. Furthermore, the client can use a data partitioning module to divide the target data into 8 equal-sized data blocks, and subsequently store one data block on each server.

[0093] After determining the metadata information of each data block obtained from the target data segmentation, the client can generate a metadata I / O request and send it to each server using the client communication module and server communication module. This request requests the writing of the metadata information for each data block corresponding to the target data to each server. In other words, the metadata information for each data block corresponding to the target data is fully stored on each server within the GPU cluster's distributed file system, making the GPU cluster's distributed file system a fully distributed system.

[0094] Since the GPU cluster distributed file system provided in this embodiment can only support the three core IO business scenarios in the GPU cluster training scenario, the size of metadata information can be reduced, thereby enabling high-performance processing of metadata information even in scenarios with hundreds or even thousands of computer nodes.

[0095] In one possible implementation, the IO request includes: a data IO request; the server includes: a persistence module; a communication client module, used to send data IO requests to multiple target servers, wherein the data IO request corresponding to each target server is used to request that a specified data block corresponding to the target data be written to the target server; the persistence module in each target server is used to respond to the received data IO request, and according to the metadata IO request stored in the metadata database of the target server, store the specified data block that needs to be written to the target server into the preset local memory space corresponding to the target server.

[0096] After the metadata information of each data block corresponding to the target data is fully stored in each server in the distributed file system of the GPU cluster, the client generates a data IO request based on the metadata information of each data block corresponding to the target data, and sends the data IO request to each target server using the client communication module and the server communication module. This allows the specified data block that needs to be written to the target server to be persistently stored in the preset local memory space corresponding to the target server based on the metadata IO request stored in the metadata database of each target server.

[0097] Figure 4 illustrates a schematic diagram of a client writing a file to a GPU cluster distributed file system according to an embodiment of the present disclosure. As shown in Figure 4, when the training process on the computer node where the client resides needs to write a file to the GPU cluster distributed file system, it first writes the target data to the cache in the client. Then, it uses a data segmentation module to segment the target data into six data blocks of the same size: data block 0 to data block 5. Data block 0 and data block 5 are then packaged and sent to a preset local memory space in server 1 via thread 1. Data block 2 and data block 4 are packaged and sent to a preset local memory space in server 2 via thread 2. Data block 1 and data block 3 are packaged and sent to a preset local memory space in server 3 via thread 3.

[0098] In one possible implementation, the data I / O request corresponding to each target server includes client memory region information corresponding to the specified data block to be written to the target server; the persistence module in each target server reads the specified data block from the memory space specified by the client memory region information through RDMA based on the client memory region information included in the received data I / O request.

[0099] With the support of the underlying network architecture protocol, the client can expose the client memory region information corresponding to the specified data block to be written to the target server through a data I / O request. This allows the target server to directly read the specified data block from the memory space specified by the client memory region information via RDMA, thereby improving data transmission performance.

[0100] In one possible implementation, the persistence module in each target server stores each specified data block that needs to be written to the target server into a chunk file in the preset local memory space corresponding to the target server.

[0101] The persistence module in each target server will store each specified data block that needs to be written to the target server in a chunk file in the preset local memory space of the target server, and store it in the underlying node structure domain.

[0102] In one possible implementation, the GPU cluster distributed file system is also used to asynchronously send all or part of the checkpoint data to a parallel file storage (PFS) outside the target GPU cluster.

[0103] In practical use, the GPU cluster distributed file system can be used only for temporary data storage. During or after the execution of the target training task, all or part of the checkpoint data temporarily stored in the GPU cluster distributed file system can be asynchronously sent to the PFS (e.g., GPFS, Lustre, etc.) outside the target GPU cluster so that the data will not be affected by the failure of the target GPU cluster.

[0104] In one possible implementation, a management module is used to delete the client and server after the target training task has been completed using the target GPU cluster.

[0105] After completing the target training task using the target GPU cluster, the management module can delete the client and server to free up resources.

[0106] In one example, the management module can reuse the operator mechanism for cluster management in the Kubernetes platform to create and delete servers on computer nodes based on the target training task.

[0107] In one example, the management module is also used to store configuration information for each server, as well as monitoring information during the training process, to ensure that the target training task is successfully executed based on the target GPU cluster.

[0108] In this embodiment, a distributed file system adapted for GPU cluster training is designed, including a management module, a server, and a client. After allocating a target GPU cluster for a target training task, the management module creates a server for multiple first computer nodes included in the target GPU cluster and a client for multiple second computer nodes included in the target GPU cluster according to the data read and write requirements of the target training task. This enables the server to flexibly construct a GPU cluster distributed file system adapted for the target training task using the preset local memory space deployed on the corresponding first computer node. Furthermore, during the execution of the target training task using the target GPU cluster, the client can perform IO operations on the GPU cluster distributed file system to effectively meet the IO requirements in the GPU cluster training scenario.

[0109] Compared to traditional parallel file systems using main memory or localfs, the distributed file system adapted for GPU cluster training provided in this disclosure offers unified storage space, larger capacity, and linearly increasing throughput. Furthermore, the distributed file system adapted for GPU cluster training provided in this disclosure can be adapted only to file system calls in GPU cluster training scenarios, discarding other capabilities of traditional file systems and traditional parallel file systems, simplifying the call chain, thereby enabling GPU cluster training to achieve 6-7 times or more performance, while also being very easy to maintain. In addition, because the distributed file system adapted for GPU cluster training provided in this disclosure can support shared memory on each computer node in the GPU cluster as the backend storage medium, it can achieve more than 10 times the performance.

[0110] The distributed file system adapted for GPU cluster training provided in this disclosure can flexibly create and deploy GPU cluster distributed file systems that meet the requirements of the task as training tasks are started and deleted. It makes full use of the CPU, storage, PCIe bandwidth and network resources of the computing nodes in the GPU cluster, saving costs while improving performance.

[0111] Figure 5 shows a flowchart of a method for creating a distributed file system adapted for GPU cluster training according to an embodiment of the present disclosure. As shown in Figure 5, the method includes:

[0112] In step S51, after allocating a target GPU cluster for the target training task, a server is created for the multiple first computer nodes included in the target GPU cluster, and a client is created for the multiple second computer nodes included in the target GPU cluster, wherein each computer node included in the target GPU cluster has at least one hardware GPU deployed on it.

[0113] In step S52, based on the server, a GPU cluster distributed file system adapted to the target training task is constructed using the preset local memory space deployed on the corresponding first computer node.

[0114] In step S53, based on the client, during the execution of the target training task using the target GPU cluster, I / O operations are performed on the GPU cluster's distributed file system.

[0115] The specific process of the client performing IO operations on the GPU cluster distributed file system can be referred to the relevant descriptions of the embodiments shown in Figures 1 to 5 above, and will not be repeated here.

[0116] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0117] In addition, this disclosure also provides electronic devices, computer-readable storage media, and programs, all of which can be used to implement any of the methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding records in the method section and will not be repeated here.

[0118] This method is specifically technically related to the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects that improve the internal performance of computer systems in accordance with natural laws.

[0119] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0120] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0121] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0122] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0123] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0124] Figure 6 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Referring to Figure 6, the electronic device 1900 may be provided as a server or a terminal device. Referring to Figure 6, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions executable by the processing component 1922, such as application programs. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0125] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OS X) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0126] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0127] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0128] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0129] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0130] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0131] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0135] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0136] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0137] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0138] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0139] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A distributed file system adapted for GPU cluster training, characterized in that, include: Management module, server, and client; The management module is used to create a server for multiple first computer nodes included in the target GPU cluster and a client for multiple second computer nodes included in the target GPU cluster after allocating a target GPU cluster for the target training task, wherein each computer node included in the target GPU cluster has at least one hardware GPU deployed on it. The server is used to construct a GPU cluster distributed file system adapted to the target training task by utilizing the preset local memory space deployed on the corresponding first computer node. The client is used to perform input / output I / O operations on the GPU cluster distributed file system during the execution of the target training task using the target GPU cluster.

2. The system of claim 1, wherein, The preset local memory space deployed on each first computer node has the same memory size.

3. The system of claim 1, wherein, The GPU cluster distributed file system is used to store the dataset required for the target training task, as well as checkpoint data during the training process.

4. The system of claim 1, wherein, The client includes: a system call interception module and a client communication module; the server includes: a server communication module. The system call interception module is used to intercept the target file system call in the second computer node where the client is located, and to convert the target file system call into an IO request, wherein the target file system call is used to access the GPU cluster distributed file system; The client communication module is used to send the IO request to the server communication module.

5. The system of claim 4, wherein, The client communication module includes a Remote Procedure Call (RPC) client module; the server communication module includes an RPC server module. The RPC client module is used to send the IO request to the RPC server module using a non-blocking RPC network communication mode.

6. The system of claim 4, wherein, The client communication module includes an IPC client module; the server communication module includes an IPC server module. The IPC client module is configured to send the IO request to the IPC server module when the first computing point where the server is located and the second computer node where the client is located are the same computer node.

7. The system of any one of claims 1 to 6, wherein, The client includes: a file graph; The file graph is used to store the client's access records to the GPU cluster's distributed file system.

8. The system of claim 4, wherein, The IO request includes: a metadata IO request; the client includes: a data segmentation module; the server includes: a metadata database; The data segmentation module is used to divide the target data that needs to be written to the GPU cluster distributed file system into blocks, determine multiple data blocks corresponding to the target data, and metadata information of each data block, wherein the metadata information corresponding to each data block is used to indicate the target server to which the data block needs to be written. The client communication module is used to send the metadata IO request to each server communication module, wherein the metadata IO request is used to request that the metadata information of each data block corresponding to the target data be written to each server. Each server-side metadata database is used to respond to the metadata IO request and store metadata information for each data block corresponding to the target data.

9. The system of claim 8, wherein, The IO request includes: a data IO request; the server includes: a persistence module; The client communication module is used to send data I / O requests to multiple target servers, wherein the data I / O request corresponding to each target server is used to request that a specified data block corresponding to the target data be written to the target server. The persistence module in each target server is used to respond to the received data IO request, and according to the metadata IO request stored in the metadata database of the target server, store the specified data block that needs to be written to the target server into the preset local memory space corresponding to the target server.

10. The system of claim 9, wherein, The persistence module in each target server stores each specified data block that needs to be written to the target server into a chunk file in the preset local memory space corresponding to the target server.

11. The system of any one of claims 8 to 10, wherein, The number of specified data blocks corresponding to the target data written to each target server is the same.

12. The system of claim 9 or 10, wherein, Each data I / O request for the target server includes information about the client memory region corresponding to the specified data block that needs to be written to the target server; The persistence module in each target server reads a specified data block from the memory space specified by the client memory region information through Remote Direct Memory Access (RDMA) based on the client memory region information included in the received data I / O request.

13. The system of claim 3, wherein, The GPU cluster distributed file system is also used to asynchronously send all or part of the checkpoint data to the parallel file system PFS outside the target GPU cluster.

14. The system of claim 1, wherein, The management module is used to delete the client and the server after the target training task has been completed using the target GPU cluster.

15. A method for creating a distributed file system adapted for GPU cluster training, the method comprising: include: After allocating a target GPU cluster for the target training task, a server is created for the multiple first computer nodes included in the target GPU cluster, and a client is created for the multiple second computer nodes included in the target GPU cluster, wherein each computer node included in the target GPU cluster has at least one hardware GPU deployed on it. Based on the server, a GPU cluster distributed file system adapted to the target training task is constructed using the preset local memory space deployed on the corresponding first computer node. Based on the client, during the execution of the target training task using the target GPU cluster, I / O operations are performed on the GPU cluster's distributed file system.

16. An electronic device, comprising: include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method of claim 15.

17. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of claim 15.

Images