Cluster, ai task execution method, and related device

Abstract

A cluster, an AI task execution method, and a related device, which relate to the technical field of artificial intelligence. The cluster comprises a management node, a plurality of super nodes and a backup resource pool. The management node is used for: determining, during the execution of an AI task by the plurality of super nodes in the cluster, that a first computing node in a target super node has failed; and adjusting an optical path configuration of an optical switching node used for connecting computing nodes in the backup resource pool to the plurality of super nodes, such that a second computing node in the backup resource pool communicates, on the basis of the optical switching node after the optical path configuration adjustment, with computing nodes in the target super node that have not failed, the second computing node being used for replacing the first computing node to continue to participate in executing the AI task. In this way, a relatively small number of computing nodes in a backup resource pool can provide redundant backup for a relatively large number of super nodes, thereby effectively reducing hardware costs, and also improving the efficiency of replacing a failed first computing node with a second computing node.

Description

Clusters, AI task execution methods and related equipment

[0001] This application claims priority to Chinese patent application filed on January 23, 2025, with application number 202510113119.8 and entitled "Cluster, AI Task Execution Method and Related Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to a cluster, an AI task execution method, and related equipment. Background Technology

[0003] With the development of artificial intelligence (AI) technology, the parameter scale of AI models is also gradually increasing. For example, the parameter scale of a single AI model can reach hundreds of billions. Correspondingly, the scale of clusters used to train AI models or support AI model inference is also constantly expanding. For example, the number of computing nodes (such as accelerators) used to train AI models in a cluster may reach 100,000.

[0004] As cluster size increases, so does the failure rate. In scenarios such as model training or inference, if some compute nodes in the cluster fail, the training or inference process of the AI ​​model can be interrupted. Currently, when a compute node fails, the cluster pauses the AI ​​model training or inference tasks, and technicians perform fault recovery on the failed compute node, such as restarting the node or replacing its hardware. After the compute node is recovered, the cluster resumes the AI ​​model training or inference tasks.

[0005] However, this fault recovery method results in a large delay in fault recovery, thereby reducing the overall efficiency of the cluster in executing AI tasks. Summary of the Invention

[0006] This application provides a cluster to reduce fault recovery latency and improve the efficiency of cluster execution of AI tasks. Furthermore, this application also provides a corresponding AI task execution method, management node, computing device, computer-readable storage medium, and computer program product.

[0007] In a first aspect, this application provides a cluster comprising a management node, multiple supernodes, and a backup resource pool. Each of the multiple supernodes includes multiple compute nodes (such as an NPU), and the backup resource pool includes at least one compute node. Furthermore, different compute nodes in the cluster are connected via communication connections (e.g., through multiple switching nodes). The at least one compute node in the backup resource pool is connected to the multiple supernodes via an optical switching node. The multiple supernodes are used to perform AI (artificial intelligence) tasks, such as training AI models. The management node is used to determine if a first compute node in a target supernode fails during the execution of AI tasks by the multiple supernodes. The management node is also used to adjust the optical path configuration of the optical switching node used when connecting the compute node in the backup resource pool to the multiple supernodes, so that a second compute node in the backup resource pool communicates with a non-faulty compute node in the target supernode based on the optical switching node with the adjusted optical path configuration. This second compute node replaces the first compute node in continuing to participate in the execution of AI tasks.

[0008] Once a failure is detected in the first compute node of the target supernode, the management node adds a second compute node from the backup resource pool to replace it. This allows multiple supernodes to continue executing AI tasks immediately after the replacement. Compared to manual fault recovery, this significantly reduces recovery latency and improves the overall efficiency of the cluster in executing AI tasks. In real-world applications, the recovery latency for large-scale clusters can be reduced to the minute level.

[0009] Furthermore, compute nodes in the backup resource pool can be dynamically added to faulty supernodes, eliminating the need to allocate a separate compute node for each supernode for redundant backup. This allows a smaller number of compute nodes in the backup resource pool to provide redundant backups for a larger number of supernodes in real-world applications, thereby ensuring cluster reliability while effectively reducing the number of backup resources required to be deployed in the cluster and lowering hardware costs.

[0010] Meanwhile, by adjusting the optical path configuration of the optical switching node, the management node can not only improve the efficiency of replacing the faulty first computing node with the second computing node, but also ensure that the communication speed between the second computing node and the fault-free computing nodes in the target supernode via the optical switching node is usually higher than the communication speed between the second computing node and the fault-free computing nodes in the target supernode via the electrical switching node. This reduces the impact of the communication speed between the second computing node and the computing nodes in the target supernode on the overall execution efficiency of the AI ​​task.

[0011] In one possible implementation, different computing nodes in the cluster communicate through multiple switching nodes, each of which can be an electrical switching node (such as an electrical switch) and / or an optical switching node (such as an optical switch). Furthermore, the management node is used to adjust the network topology of the multiple switching nodes based on a load balancing strategy for the multiple supernodes replaced by computing nodes. In this way, the traffic generated by AI services can be relatively evenly distributed across the network topology, allowing the adjusted network topology to adapt to the traffic distribution requirements of the AI ​​services. This effectively improves the efficiency of traffic forwarding by the multiple switching nodes in the cluster, further enhancing the overall performance of the cluster in executing AI tasks.

[0012] In one possible implementation, the management node is further configured to acquire the network topology of multiple switching nodes and the traffic characteristics generated by multiple computing nodes participating in the execution of AI tasks, thereby generating a load balancing strategy based on the network topology and traffic characteristics. In this way, the management node generates a load balancing strategy based on network topology and traffic characteristic information, enabling the load balancing strategy to optimize traffic across the network topology. This helps improve the efficiency of traffic forwarding by multiple switching nodes in the cluster, thereby further improving the overall performance of the cluster in executing AI tasks.

[0013] In one possible implementation, the AI ​​task is a training task for an AI model. The management node is further configured to: after the first computing node completes fault recovery, wait for the AI ​​model to complete the Nth round of training, where N is a positive integer; and after the AI ​​model completes the Nth round of training, replace the second computing node with the first computing node to continue training the AI ​​model, and then use multiple supernodes to continue training the AI ​​model for the N+1th round. Thus, by reusing the first computing node to continue training the AI ​​model after the first round of training is completed, it avoids immediately activating the first computing node to continue training the AI ​​model, thus avoiding wasting the training process already performed on the AI ​​model and preventing an increase in the overall latency of training the AI ​​model.

[0014] In one possible implementation, the AI ​​task includes multiple subtasks. Before the failure, the first computing node runs the target subtask among these subtasks. When the management node adjusts the optical path configuration of the optical switch node, it can specifically determine, based on the target subtask running by the first computing node, a second computing node from the backup resource pool that has loaded the execution resources corresponding to the target subtask and is running a process for executing the target subtask. Then, for the second computing node, the optical path configuration of the optical switch node is adjusted. Thus, when replacing the failed first computing node with a computing node from the backup resource pool, there is no need to temporarily start running processes and load execution resources on the computing nodes in the backup resource pool, thereby shortening the fault recovery latency for the failed first computing node.

[0015] Secondly, this application provides an AI task execution method. This method is applied to a cluster, which includes a management node, multiple supernodes, and a backup resource pool. Each supernode includes multiple compute nodes, and the backup resource pool includes at least one compute node. Communication connections exist between different compute nodes in the cluster. At least one compute node in the backup resource pool is connected to the multiple supernodes via an optical switching node. The multiple supernodes are used to execute AI tasks. The method includes: during the execution of AI tasks by the multiple supernodes, the management node determines that a first compute node in a target supernode has failed; the management node adjusts the optical path configuration of the optical switching node so that a second compute node in the backup resource pool communicates with a non-failed compute node in the target supernode based on the optical switching node with the adjusted optical path configuration. The second compute node replaces the first compute node to continue participating in the execution of the AI ​​task.

[0016] In one possible implementation, different compute nodes in the cluster communicate with each other through multiple switching nodes; the method further includes: the management node adjusting the network topology of multiple switching nodes for multiple supernodes that have been replaced by compute nodes based on a load balancing strategy.

[0017] In one possible implementation, the method further includes: the management node acquiring the network topology of multiple switching nodes; the management node acquiring the traffic characteristics generated by multiple computing nodes participating in the execution of AI tasks; and the management node generating a load balancing strategy based on the network topology and traffic characteristics.

[0018] In one possible implementation, the AI ​​task is a training task for an AI model; the method further includes: after the first computing node completes fault recovery, the management node waits for the AI ​​model to complete the Nth round of training, where N is a positive integer; after the AI ​​model completes the Nth round of training, the management node uses the first computing node to replace the second computing node to continue participating in the training of the AI ​​model; the management node uses multiple supernodes to continue training the AI ​​model for the N+1th round.

[0019] In one possible implementation, the AI ​​task includes multiple subtasks. The first computing node runs the target subtask among the multiple subtasks before the failure. The management node adjusts the optical path configuration of the optical switching node, including: the management node determines, based on the target subtask run by the first computing node, a second computing node loaded with the execution resources corresponding to the target subtask and running a process for executing the target subtask from the backup resource pool; and the management node adjusts the optical path configuration of the optical switching node for the second computing node.

[0020] The AI ​​task execution method provided in the second aspect corresponds to the cluster provided in the first aspect. Therefore, the technical effects of any implementation method provided in the second aspect can be seen in the technical effects of the corresponding implementation method in the first aspect above, and will not be elaborated here.

[0021] Thirdly, this application provides a management node, which includes: a fault determination module, used to determine, during the execution of AI tasks by multiple supernodes in a cluster, that a first computing node in a target supernode has failed; the cluster also includes a management node and a backup resource pool; each of the multiple supernodes includes multiple computing nodes; the backup resource pool includes at least one computing node; and the at least one computing node in the backup resource pool is connected to the multiple supernodes through an optical switching node; and an adjustment module, used to adjust the optical path configuration of the optical switching node so that a second computing node in the backup resource pool communicates with the non-faulty computing node in the target supernode based on the optical switching node whose optical path configuration has been adjusted; the second computing node is used to replace the first computing node to continue participating in the execution of the AI ​​task.

[0022] In one possible implementation, different compute nodes in the cluster communicate with each other through multiple switching nodes; the adjustment module is used to adjust the network topology of multiple switching nodes for multiple supernodes that have been replaced by compute nodes based on a load balancing strategy.

[0023] In one possible implementation, the management node further includes: an acquisition module for acquiring the network topology of multiple switching nodes and acquiring the traffic characteristics generated by multiple computing nodes participating in the execution of AI tasks; and a generation module for generating a load balancing strategy based on the network topology and traffic characteristics.

[0024] In one possible implementation, the AI ​​task is a training task for an AI model; the management node also includes a replacement module, used to: wait for the AI ​​model to complete the Nth round of training after the first computing node completes fault recovery, where N is a positive integer; after the AI ​​model completes the Nth round of training, use the first computing node to replace the second computing node to continue participating in the training of the AI ​​model; and use multiple supernodes to continue training the AI ​​model for the N+1th round.

[0025] In one possible implementation, the AI ​​task includes multiple subtasks, and the first computing node runs the target subtask among the multiple subtasks before the failure; the adjustment module is used to: determine, based on the target subtask run by the first computing node, a second computing node loaded with the execution resources corresponding to the target subtask and running a process for executing the target subtask from the backup resource pool; and adjust the optical path configuration of the optical switching node for the second computing node.

[0026] The management node provided by the third aspect corresponds to the cluster provided by the first aspect. Therefore, the technical effects of any implementation method provided by the third aspect can be seen in the technical effects of the corresponding implementation method in the first aspect mentioned above, and will not be elaborated here.

[0027] Fourthly, this application provides a computing device, which includes a processor and a memory. The processor and the memory communicate with each other. The processor executes instructions stored in the memory to cause the computing device to perform an AI task execution method as described in any implementation of the second aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The computing device may also include a bus. The processor is connected to the memory via the bus. The memory may include readable storage and random access memory.

[0028] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computing device, cause the computing device to perform the operational steps of the AI ​​task execution method described in any implementation of the second aspect above.

[0029] Sixthly, this application provides a computer program product containing instructions that, when run on a computing device, causes the computing device to perform the operational steps of the AI ​​task execution method described in any implementation of the second aspect above.

[0030] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0031] Figure 1 is a schematic diagram of the structure of an exemplary cluster provided in this application.

[0032] Figure 2 is a schematic diagram of the structure of another exemplary cluster provided in this application.

[0033] Figure 3 is a flowchart illustrating an AI task execution method provided in this application;

[0034] Figure 4 is a flowchart illustrating another AI task execution method provided in this application;

[0035] Figure 5 is a schematic diagram of the structure of a management node provided in this application;

[0036] Figure 6 is a schematic diagram of the hardware structure of a computing device provided in this application. Detailed Implementation

[0037] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a method of distinction used in describing objects with the same attributes in the embodiments of this application.

[0038] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0039] Referring to Figure 1, which is a schematic diagram of the structure of an exemplary cluster 10. As shown in Figure 1, cluster 10 includes a management node 100, multiple supernodes, multiple exchange nodes, and a backup resource pool 400. For ease of understanding, Figure 1 illustrates a cluster 10 including 2 supernodes (supernode 210 and supernode 220) and 15 exchange nodes (exchange node 301 and exchange node 315) as an example.

[0040] A supernode is a collection of multiple computing nodes. That is, each supernode includes multiple computing nodes. As shown in Figure 1, supernode 210 includes four computing nodes, namely computing nodes 211 to 214.

[0041] The backup resource pool 400 may include multiple compute nodes. Figure 1 illustrates this by using compute nodes 401 to 404 as an example. In this embodiment, the compute nodes in the backup resource pool 400 can be used to replace a failed compute node in supernode 210 or supernode 220.

[0042] For example, a computing node in cluster 10 refers to a node with data computing capabilities, which can be implemented by a processor or by a computing device including a processor, such as a server. The processor can be any type of processor or any combination thereof, such as a central processing unit (CPU), an accelerator, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system-on-chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, or a data processing unit (DPU). An accelerator can be, for example, a graphics processing unit (GPU), a neural network processing unit (NPU), or a tensor processing unit (TPU).

[0043] Furthermore, the multiple compute nodes in each supernode can be divided into multiple working frames, as shown in Figure 1. Compute nodes 211 and 212 in supernode 210 are assigned to working frame 1, and compute nodes 213 and 214 are assigned to working frame 2. In practical applications, each working frame can correspond to a rack, so multiple compute nodes in each working frame can be deployed in the same rack. Similarly, the multiple compute nodes in the backup resource pool 400 can also be divided into multiple working frames (or backup frames), as shown in working frames 5 and 6 in Figure 1. Different compute nodes within the same working frame can communicate through a switching node (such as a network interface card or switching chip) deployed within that working frame, or through an interconnect bus. There are no limitations on this; Figure 1 illustrates this by showing multiple compute nodes within the same working frame communicating through a built-in switching node.

[0044] The cluster 10 shown in Figure 1 can run AI tasks, and these AI tasks can be executed in parallel by multiple computing nodes in the cluster 10. For example, the AI ​​task can be a training task or an inference task for an AI model. When the AI ​​model has a large number of parameters, the cluster 10 can use multiple computing nodes to perform distributed training on the AI ​​model, or use multiple computing nodes to run the AI ​​model to provide corresponding inference services. For example, the AI ​​model can be a large language model (LLM), a bidirectional encoder representations from transformers (BERT) model, or a generative pre-trained Transformer 3 (GPT-3) model, or other types of models such as GPT-4, etc., without limitation.

[0045] During the execution of AI tasks, different computing nodes in cluster 10 can communicate data through node exchange. For example, when the AI ​​task is specifically a training task for an AI model, during the backpropagation phase of training the AI ​​model, multiple computing nodes can exchange the gradient data they have calculated through node exchange, so as to update the parameter values ​​in the AI ​​model using the gradient data calculated by multiple computing nodes respectively.

[0046] In this context, a switching node refers to a node with data switching capabilities, such as an electrical switching node or an optical switching node. An electrical switching node uses electrical signals for data exchange, which can be transmitted via twisted-pair or coaxial cables, such as electrical switches, network interface cards (NICs), and routers. An optical switching node uses optical signals for data exchange, such as optical cross-connect (OXC) devices, which can be transmitted via optical fiber. For example, assuming that compute node 211 in supernode 210 communicates with compute node 221 in supernode 220, compute node 211 can forward communication data to compute node 221 through switching nodes 301, 307, 309, and 305. In practical applications, switching nodes can also be other types of devices, such as NICs or high-performance NICs; there is no limitation on this.

[0047] For example, some of the switching nodes in cluster 10 can be deployed in multiple layers. Figure 1 shows an example of switching nodes 301 to 311 deployed in three layers. Switching nodes 301 to 306 can act as leaf nodes, deployed in the first layer (L1), and switching nodes 307 to 309 can act as spine nodes, deployed in the second layer (L2). Different leaf nodes can interact with each other through spine nodes. Furthermore, switching nodes 310 and 311 can be deployed in the third layer (L3), and different spine nodes can interact with each other through switching nodes in the third layer. In practical applications, the switching nodes in the first and second layers can be electrical switching nodes, and the switching nodes in the third layer can be optical switching nodes (such as OXC devices). In AI model training scenarios, the network formed by switching nodes 301 to 311 can be an AI parameter plane network, used for multiple computing nodes to interact with the model parameters trained in each iteration of training.

[0048] In cluster 10 shown in Figure 1, each supernode can deploy two sets of exchange nodes. Compute nodes in different supernodes can communicate through one set of exchange nodes, while different compute nodes within the same supernode can communicate through the other set of exchange nodes. For example, compute nodes in supernode 210 can communicate with compute nodes in supernode 220 through exchange nodes 301 and 302; compute nodes 211 to 214 in supernode 210 can communicate through exchange nodes 312 and 313. In practical applications, the data exchange capacity of exchange nodes 312 and 313 can be higher than that of exchange nodes 301 and 302, thereby improving the communication efficiency between different compute nodes within the same supernode.

[0049] The compute nodes in the backup resource pool 400 can access the network through switching nodes (such as electrical switching nodes) to communicate with the compute nodes in each supernode. Furthermore, the compute nodes in the backup resource pool 400 can also connect to the compute nodes in multiple supernodes through optical switching nodes (such as OXC devices), as shown in optical switching nodes 316 and 317 in Figure 1. This allows for the subsequent addition of compute nodes from the backup resource pool 400 to supernodes 210 or 220 by adjusting the optical path configuration of the optical switching nodes.

[0050] During the execution of AI tasks, some compute nodes in cluster 10 may inevitably fail. For example, compute nodes in supernodes 210 or 220 may experience program errors or hardware damage while participating in the execution of AI tasks, causing the AI ​​tasks to be suspended due to the failure of these compute nodes. If technicians manually repair the compute nodes during the fault recovery process of cluster 10 and then continue to use them to participate in the AI ​​tasks after the fault recovery is completed, the overall execution efficiency of the AI ​​tasks may be low due to excessively long fault recovery delays (such as up to 1 hour).

[0051] Based on this, in the cluster 10 provided in this application, the management node 100 utilizes the computing nodes in the backup resource pool 400 to achieve rapid fault recovery of computing nodes that fail in supernode 210 or supernode 220, thereby improving the overall efficiency of executing AI tasks. The management node 100 can connect to the various switching nodes and computing nodes in the cluster 10 through a management network to manage the switching nodes and computing nodes.

[0052] In specific implementation, during the execution of AI tasks using supernodes 210 and 220, after determining that compute node 211 in supernode 210 has failed, management node 100 can adjust the optical path configuration of optical switching node 316 so that compute node 401 in backup resource pool 400 can communicate with the non-failed compute node in supernode 210 based on the optical switching node 316 after the optical path configuration adjustment. That is, compute node 401 in backup resource pool 400 is added to supernode 210, thereby using compute node 401 to replace the failed compute node 211 and continue to participate in the execution of AI tasks. In other words, cluster 10 can subsequently use supernode 210 after the compute node replacement (specifically, using compute nodes 212 to 214 in supernode 210 and compute node 401 added to supernode 210) and non-failed supernode 220 to continue to execute AI tasks. Similarly, when a compute node in supernode 220 fails, management node 100 can replace the failed compute node in supernode 220 with a compute node from backup resource pool 400, in the same manner described above.

[0053] Therefore, compared to manually recovering from a fault in compute node 211, quickly replacing compute node 211 with compute node 401 from the backup resource pool 400 can effectively reduce the latency of fault recovery, thereby improving the overall efficiency of cluster 10 in executing AI tasks. In practical applications, even if the scale of compute nodes in cluster 10 is large (e.g., including hundreds of thousands of compute nodes), the fault recovery latency of cluster 10 can be reduced to the minute level, improving the availability of the cluster.

[0054] Furthermore, the compute nodes in the backup resource pool 400 can be dynamically added to the faulty supernodes without having to allocate a separate compute node for each supernode for redundant backup. This allows a smaller number of compute nodes in the backup resource pool 400 to provide redundant backups for a larger number of supernodes in real-world applications. This ensures the reliability of cluster 10 while effectively reducing the number of backup resources required to be deployed in cluster 10 and lowering hardware costs.

[0055] Meanwhile, by adjusting the optical path configuration of the optical switching node 316, the management node 100 can not only improve the efficiency of replacing the faulty computing node 211 with the computing node 401, but also ensure that the communication speed between the computing node 401 and the fault-free computing nodes in the target supernode via the optical switching node 316 is generally higher than the communication speed between the computing node 401 and the fault-free computing nodes in the target supernode via the electrical switching node. This ensures the communication performance between the computing node 401 and different computing nodes in the target supernode, thereby reducing the impact of the communication speed between the computing node 401 and the computing nodes in the target supernode on the overall execution efficiency of the AI ​​task.

[0056] For example, the management node 100 can be implemented by software or hardware. When implemented by software, the management node 100 can be an application running on hardware, such as a process running on a computing node / server. When implemented by hardware, the management node 100 can be a processor, or a computing device including a processor, or it can be implemented by multiple computing devices, etc., without limitation.

[0057] It is worth noting that the cluster 10 shown in Figure 1 above is only an illustrative example and is not intended to be limiting.

[0058] For example, in other possible implementations, cluster 10 may also include a greater number of compute nodes, switching nodes, and supernodes. For instance, when cluster 10 includes 500,000 compute nodes, the number of supernodes may be 4,000 or 8,000. Alternatively, cluster 10 may include a scheduling node that can schedule AI tasks to multiple compute nodes in cluster 10 so that the AI ​​tasks can be executed using the scheduled compute nodes.

[0059] For example, in cluster 10 shown in Figure 1 above, the computing nodes in the backup resource pool 400 can communicate with the computing nodes in the supernode through the optical switching nodes and the switching nodes in the supernode. In other possible clusters, the computing nodes in the backup resource pool 400 can also communicate with the computing nodes in each supernode through the switching nodes in the AI ​​parameter plane network (such as the switching nodes in layers L1 to L3).

[0060] For example, in cluster 10 shown in Figure 1, management node 100 can replace faulty supernodes at the compute node level, or it can replace faulty work frames at the work frame level, meaning it can replace multiple compute nodes within a faulty work frame as a whole. In cluster 20 shown in Figure 2, compute nodes within different work frames of the backup resource pool 400 can be connected via electrical switching nodes. Furthermore, the electrical switching nodes in the backup resource pool 400 can be connected to optical switching nodes. Therefore, management node 100 can adjust the optical path configuration of the optical switching nodes to replace faulty supernodes at the supernode level using multiple compute nodes in the backup resource pool 400.

[0061] For ease of understanding, embodiments of the AI ​​task execution method provided in this application will be described below with reference to the accompanying drawings.

[0062] Referring to Figure 3, which is a flowchart illustrating an AI task execution method provided in an embodiment of this application, this method can be applied to cluster 10 shown in Figure 1 or cluster 20 shown in Figure 2, or to other applicable clusters. For ease of explanation, this embodiment uses cluster 10 shown in Figure 1 as an example for illustrative purposes.

[0063] The AI ​​task execution method shown in Figure 3 may specifically include the following steps.

[0064] S301: Supernodes 210 and 220 in cluster 10 execute AI tasks.

[0065] Among them, supernodes executing AI tasks specifically refers to multiple computing nodes in a supernode executing AI tasks.

[0066] For example, an AI task can specifically be a training task for an AI model, such as a task that uses multiple computing nodes in cluster 10 to perform distributed training on an AI model. Alternatively, an AI task can specifically be an inference task for an AI model. For example, after deploying an AI task to multiple computing nodes, these multiple computing nodes can be used to support the AI ​​model inference, so as to provide corresponding inference services based on the AI ​​model, such as intelligent question answering services, image recognition services, etc. In this embodiment, the example of using multiple computing nodes in supernode 210 and supernode 220 to jointly execute the AI ​​task is used for illustration.

[0067] S302: During the execution of AI tasks by multiple supernodes, management node 100 determines that computing node 211 in supernode 210 has failed.

[0068] In real-world applications, the compute nodes in Cluster 10 are prone to failure during operation. These failures can be software-related, such as program errors, or hardware-related, such as physical damage to the bus, memory, or some processor cores within the compute nodes. Typically, because AI tasks are executed collaboratively across multiple compute nodes, the failure of some compute nodes may force the AI ​​task to pause.

[0069] During the execution of AI tasks, management node 100 can detect a target supernode that has a fault among multiple supernodes, and further detect the computing node that has a fault in the target supernode. For ease of description, this embodiment takes the failure of computing node 211 in supernode 210 as an example (that is, the target supernode is specifically supernode 210).

[0070] In one possible implementation, management software can be configured in working frame 1 of supernode 210. This management software can run on some of the compute nodes in working frame 1, or it can run on a CPU, which can be deployed separately in supernode 210. Furthermore, the management software can periodically receive heartbeat messages from each compute node in working frame 1. Specifically, if the management software receives a heartbeat message from a compute node within each detection period, it can determine that the compute node is not faulty; conversely, if the management software does not receive a heartbeat message from the compute node within the detection period, it can determine that the compute node has failed. Alternatively, when a compute node fails, it can proactively report the fault to the management software. This embodiment does not limit the implementation method of the management software sensing faulty compute nodes.

[0071] Then, the management software can generate fault information and send it to the management node 100. This fault information may include the identifier of the supernode 210 and the identifier of the compute node 211 (and may also include the identifier of the work frame 1), etc. In this way, the management node 100 can detect that a compute node 211 in the supernode 210 has failed based on the received fault information.

[0072] The above-described implementation method of the management node 100 to determine the faulty computing node is only an example. In other embodiments, the management node 100 may also locate the faulty computing node based on other methods, and there is no limitation on this.

[0073] If a failure is detected in compute node 211, management node 100 can further suspend the processes in the remaining compute nodes that are used to execute subtasks (i.e., the subtasks included in the aforementioned AI tasks), and terminate the process executing subtasks in compute node 211, so as to control cluster 10 to suspend the execution of AI tasks.

[0074] S303: Management node 100 adjusts the optical path configuration of optical switching node 316 so that computing node 401 in backup resource pool 400 can communicate with the non-faulty computing node in super node 210 based on optical switching node 316 after optical path configuration adjustment. Computing node 401 is used to replace computing node 211 to continue to participate in the execution of AI tasks.

[0075] In this embodiment, a backup resource pool 400 is configured in the cluster 10, and when a computing node in a supernode fails, the management node 100 can dynamically replace the failed computing node using computing nodes in the backup resource pool 400.

[0076] In one possible implementation, after determining that compute node 211 has failed, management node 100 can select a corresponding number of compute nodes from backup resource pool 400 to replace the failed compute node. The compute nodes in backup resource pool 400 can replace the failed compute node using warm backup or cold backup.

[0077] In the case of a warm standby fault replacement, the compute node 401 in the backup resource pool 400 can pre-start and run a process to execute the subtasks performed by compute node 211. These subtasks are the ones executed by compute node 211 among multiple subtasks included in the AI ​​task. For ease of distinction and description, this subtask will be referred to as the target subtask. Furthermore, compute node 401 can pre-load the execution resources corresponding to the target subtask (i.e., the resources required to execute the target subtask). For example, in a model training scenario, the pre-loaded execution resources of compute node 401 could be the model structure corresponding to the model parameters deployed on compute node 211. Therefore, in the event of a failure of compute node 211, management node 100 can directly add the process already running the target subtask and compute node 401 with loaded execution resources from the backup resource pool 400 to supernode 210, so that compute node 401 can replace compute node 211. In this way, the compute node 401 added to supernode 210 no longer needs to temporarily start running processes and load execution resources, thereby shortening the fault recovery latency for compute node 211.

[0078] When using cold standby for fault replacement, if compute node 211 fails, management node 100 can randomly select a compute node from backup resource pool 400 and add the selected compute node to supernode 210 to replace the failed compute node 211. Assuming compute node 401 is added to supernode 210, after being added, compute node 401 can create and start a process to execute a target subtask, and load the execution resources required for that subtask. This allows it to replace the failed compute node 211 after successfully running the process and loading the execution resources.

[0079] In practical applications, supernodes 210 and 220 contain a large number of compute nodes, and different compute nodes can execute different subtasks based on different resources. However, the number of processes that a single compute node in the backup resource pool 400 can run and the amount of resources it can load are limited. Therefore, the backup resource pool 400 can use a combination of warm and cold backup to replace failed compute nodes. For example, the management node 100 can pre-sort the subtasks executed by the compute nodes in supernodes 210 and 220, such as sorting them according to their importance, and for the first K subtasks (K being a positive integer greater than 1), run the corresponding processes and load resources on multiple compute nodes in the backup resource pool 400. Each compute node can run the corresponding processes and load resources for some of the K subtasks. In this way, when a compute node executing any of the first K subtasks fails, the management node 100 can use the compute nodes in the backup resource pool 400 to replace the failed compute node using a warm backup method. If a computing node executing the remaining subtasks fails, the management node 100 can use the computing nodes in the backup resource pool 400 to replace the failed computing node in a cold standby manner.

[0080] For ease of distinction and description, this embodiment uses the example of management node 100 replacing computing node 211 in supernode 210 with computing node 401 in backup resource pool 400.

[0081] In this embodiment, the management node 100 can add the computing node 401 in the backup resource pool 400 to the supernode 210 by adjusting the physical topology.

[0082] As an implementation example, as shown in Figure 1, the multiple switching nodes in cluster 10 include optical switching node 316 and optical switching node 317. Optical switching node 316 and optical switching node 317 can be used to connect to compute node 401 and supernode 210 in backup resource pool 400. Thus, compute node 401 can connect to each compute node in supernode 210 that has not failed through optical switching node 316 or optical switching node 317 and switching nodes (such as electrical switches) in supernode 210. Then, the management node 100 can adjust the optical path configuration of the optical switching node 316 (or optical switching node 317). For example, it can control the ports in the optical switching node 316 (or optical switching node 317) used to connect to the computing node 401 and the ports used to connect to the switching nodes in the supernode 210 to be in the open state, and control the ports in the optical switching node 316 (or optical switching node 317) used to connect to other computing nodes and other switching nodes to be in the closed state (that is, control the optical switching node to perform optical path switching). This allows the computing node 401 to communicate with the computing nodes in the supernode 210 that have not failed based on the optical switching node 316 (or optical switching node 317) whose optical path configuration has been adjusted, thereby adding the computing node 401 to the supernode 210. When a computing node is replaced in the supernode 210, it will replace the computing node in the supernode 210 that has failed. In this way, by adjusting the optical path configuration of the optical switching node, the management node 100 can not only improve the efficiency of adding the computing node 401 to the supernode 210, but also ensure that the communication speed between the computing node 401 and the computing nodes in the supernode 210 based on the optical switching node and the switching nodes in the supernode 210 is usually higher than the communication speed between the computing node 401 and the computing nodes in the supernode 210 through the switching nodes in the L1 to L3 layers. This ensures the communication performance between different computing nodes in the supernode 210, thereby reducing the impact of the communication speed between the computing node 401 and the computing nodes 212 and 214 on the overall execution efficiency of the AI ​​task.

[0083] In practical applications, management node 100 may include sub-management node 1, sub-management node 2, sub-management node 3, and a collaborative management node. Sub-management node 1 manages the switching nodes in cluster 10 that exchange data using electrical signals (e.g., configuring routing information in the switching nodes). Sub-management node 2 manages the switching nodes in cluster 10 that exchange data using optical signals (e.g., configuring ports to open or close in the switching nodes). Sub-management node 3 manages the various computing nodes in cluster 10 (e.g., controlling the computing nodes to start and run processes for executing subtasks or loading resources). The collaborative management node controls the various sub-management nodes. Therefore, when sub-management node 3 detects a failure in computing node 211 in supernode 210, it can notify the collaborative management node. The collaborative management node determines to replace the computing node 211 with the computing node 401 in the backup resource pool 400, and instructs the sub-management node 2 to perform optical path switching on the optical switching node 316 (or optical switching node 317) connected to the computing node 401, thereby adding the computing node 401 to the super node 210, and using the computing node 401 to replace the faulty computing node 211.

[0084] In practical applications, compute node 401 in backup resource pool 400 may communicate not only with compute nodes in supernode 210 but also with compute nodes in supernode 220. For example, in a model training scenario, supernodes 210 and 220 can train the AI ​​model using pipeline parallelism (PP). During the forward computation phase of model training, the data obtained from forward computation by compute node 401 in supernode 210 can be output to compute nodes in supernode 220 to continue forward computation. Subsequently, during the process of restoring the AI ​​model training, the traffic generated by compute node 401 may conflict with the traffic generated by other compute nodes during transmission (e.g., traffic congestion).

[0085] Therefore, in a further embodiment, after adding computing node 401 to supernode 210, management node 100 can also optimize the network topology of multiple switching nodes in cluster 10 for multiple computing nodes participating in the execution of AI tasks, so as to ensure the performance of traffic generated by different computing nodes in the network transmission.

[0086] Specifically, management node 100 can obtain the connection relationships between multiple switching nodes in cluster 10 and determine the network topology of multiple switching nodes based on these connection relationships. Furthermore, management node 100 can also obtain the traffic characteristics of multiple computing nodes. For example, each computing node participating in the execution of an AI task can report the communication information generated when executing sub-tasks within the AI ​​task to management node 100. This traffic characteristic can include information such as the communication domain to which the computing node belongs, the communication operator executed, and the amount of communication data. Therefore, management node 101 can determine the characteristics of the traffic generated by multiple computing nodes (such as traffic direction and traffic size) based on the communication information reported by each computing node. For example, the traffic characteristics of multiple computing nodes can be represented by a traffic bandwidth matrix. This traffic bandwidth matrix indicates the bandwidth requirements between different computing nodes participating in the execution of the AI ​​service, that is, it indicates the communication requirements of the communication operators executed by multiple computing nodes. The traffic bandwidth matrix can be an N*N two-dimensional matrix, where N is the number of participating computing nodes, N is a positive integer, each column corresponds to one computing node among the N computing nodes, and each row corresponds to one computing node among the N computing nodes. Furthermore, the elements in the two-dimensional matrix indicate the bandwidth requirements for communication between different computing nodes. Larger element values ​​indicate greater bandwidth requirements for communication between two computing nodes; smaller element values ​​indicate less bandwidth requirements.

[0087] Then, management node 100 can adjust the network topology of multiple switching nodes for supernodes 210 and 220 based on the traffic characteristics of multiple computing nodes and the aforementioned network topology, according to a load balancing strategy. For example, management node 100 can determine the logical topology 1 among multiple switching nodes in cluster 10 based on the traffic bandwidth matrix and the aforementioned topology structure. This can be achieved by using heuristic search methods based on the traffic bandwidth matrix to find logical topology 1 (wherein, the load of multiple switching nodes in cluster 10 is relatively balanced when forwarding traffic based on logical topology 1). Furthermore, management node 100 can also determine the logical topology 2 currently used by multiple switching nodes. Therefore, management node 100 can compare the differences between logical topology 1 and logical topology 2 and adjust the routing information or optical path configuration in the corresponding switching nodes in cluster 10 based on these differences. For example, management node 100 can use sub-management node 1 to adjust the routing information in switching nodes (electrical switching nodes) located at L1 and L2 layers, and use sub-management node 2 to adjust the optical path configuration in switching nodes (optical switching nodes) located at L3 layer.

[0088] In this way, the adjusted network topology can be adapted to the traffic characteristics generated when multiple computing nodes perform AI services. This makes the adjusted network topology more adaptable to the traffic distribution requirements generated by the AI ​​service, thereby effectively improving the efficiency of multiple switching nodes in cluster 10 in forwarding traffic and further improving the overall performance of cluster 10 in performing AI tasks.

[0089] After replacing the failed compute node 211 with compute node 401 from the backup resource pool 400, cluster 10 can continue executing AI tasks using supernodes 210 and 220. This can be achieved by management node 100 instructing multiple supernodes to continue executing AI tasks, or by other devices in cluster 10 instructing multiple supernodes to continue executing AI tasks. For ease of description, this embodiment uses the example of management node 100 executing multiple supernodes to continue executing AI tasks. Based on this, this embodiment may also include the following steps.

[0090] S304: Management node 100 continues to execute AI tasks using multiple supernodes that have been replaced by compute nodes.

[0091] In this embodiment, the multiple supernodes that have been replaced by computing nodes are supernode 210 with computing node 401 added and supernode 220 without faults.

[0092] After replacing compute node 211 in supernode 210 with normal compute nodes from backup resource pool 400, management node 100 can continue to execute AI tasks using compute nodes 212 to 214 in supernode 210 that have not failed, compute node 401 added to supernode 210, and compute nodes 221 to 224 in supernode 220, so that AI tasks can resume execution in cluster 10.

[0093] As an example of implementation, since compute node 211 may communicate with other compute nodes in cluster 10 before failure, after replacing compute node 211 with compute node 401, management node 100 can establish communication links between compute node 401 and other compute nodes, so that compute node 401 can communicate with other compute nodes based on the established communication links during the execution of target subtasks.

[0094] When computing node 401 communicates with computing nodes 212 to 214 in supernode 210 through optical switching node 316, during the process of establishing a communication link between computing node 401 and computing nodes in supernode 210, management node 100 can configure routing information to switching node 312 and switching node 313, so that switching node 312 and switching node 313 can forward communication data between computing node 401 and computing nodes in supernode 210 based on the routing information configured by management node 100.

[0095] Alternatively, static routing information can be pre-configured in switching nodes 312 and 313. After adjusting the optical path configuration of optical switching node 316, switching nodes 312 and 313 can use this static routing information to forward communication data between computing node 401 and the computing nodes in supernode 210, thus establishing a communication link between them. In this way, management node 100 does not need to temporarily generate routing information and configure it in switching nodes 312 and 313, thereby improving the efficiency of establishing communication links and reducing the latency of cluster 10's fault recovery. In practical applications, by configuring static routing information in switching nodes 312 and 313, the fault recovery latency of cluster 10 can be reduced to the minute level. Furthermore, when it is necessary to adjust the routing configuration in switching node 312 or switching node 313 (such as optimizing the traffic load of switching node 312 and switching node 313), management node 100 can generate dynamic routing information and send the dynamic routing information to switching node 312 or switching node 313 to update the routing information in switching node 312 and switching node 313. Thus, switching node 312 and switching node 313 forward communication data between computing node 401 and computing nodes in super node 210 based on the reconfigured dynamic routing information.

[0096] Furthermore, the management node 100 can back up the data generated by the target sub-task executed by the computing node 211 before the failure to the computing node 401, so that the computing node 401 can continue to execute the target sub-task based on the backed-up data. For example, in a model training scenario, the management node 100 can back up the model parameters trained by the computing node 211 before the failure to the computing node 401, so that the computing node 401 can continue to execute the training process for the AI ​​model based on the already trained model parameters. At the same time, the management node 100 can control the other computing nodes participating in the execution of the AI ​​task to continue running their respective processes, so that multiple computing nodes can resume executing the AI ​​task.

[0097] Furthermore, the management node 100 can also update the information of the backup resource pool 400, such as updating the number of computing nodes included in the backup resource pool 400 from 4 to 3, or updating the available computing nodes in the backup resource pool 400 from computing nodes 401 to 404 to computing nodes 402 to 404, etc.

[0098] In this embodiment, the failure of compute node 211 in supernode 210 is used as an example for illustration. When other compute nodes in supernode 210 or compute nodes in supernode 220 fail, management node 100 can also refer to the above-described similar method to replace the failed compute node with compute nodes in backup resource pool 400, and continue to execute AI tasks using the replaced compute node. This will not be elaborated further.

[0099] During the continued execution of AI tasks, the faulty compute node 211 can undergo a repair process. For example, the management node 100 can control the restart of compute node 211, or technicians can replace or repair the faulty components in compute node 211, so that compute node 211 can return to normal operation. Then, cluster 10 can reuse the restored compute node 211 to continue executing AI tasks (specifically, executing target subtasks). For example, due to the difference in communication latency between different compute nodes, the communication latency generated by using compute node 211 to execute AI tasks may be less than the communication latency generated by using compute node 401 to execute AI tasks.

[0100] Specifically, when the AI ​​task is an inference task based on an AI model, the management node 100 can replace the computing node 401 with the computing node 211 and synchronize the data on the computing node 401 to the computing node 211, so that the cluster 10 can continue to use the computing node 211 to execute the AI ​​task. The implementation process of the management node 100 replacing the computing node 401 with the computing node 211 can be found in the description of the relevant parts of the management node 100 replacing the computing node 211 with the computing node 401 above, and will not be repeated here.

[0101] When the AI ​​task is specifically a training task for an AI model, the management node 100 can replace the computing node 401 with the computing node 211 and continue training the AI ​​model using the computing node 211. The following, with reference to Figure 3, describes the implementation process of cluster 10 continuing to train the AI ​​model using the recovered computing node 211.

[0102] As shown in Figure 3, this embodiment may further include the following steps.

[0103] S305: After the compute node 211 completes fault recovery, the management node 100 waits for the AI ​​model to complete the Nth round of training, where N is a positive integer.

[0104] The Nth round of training can be the training phase that cluster 10 is executing on the AI ​​model when computing node 211 recovers from a failure.

[0105] If, upon recovery from a failure of compute node 211, the management node 100 directly replaces compute node 401 with compute node 211, then compute node 211 and the other compute nodes involved in AI model training may need to restart the Nth round of training for that AI model. This would waste the training process already performed on the AI ​​model and increase the overall latency of training the AI ​​model. For example, assuming compute node 211 recovers from a failure and the AI ​​model has completed the forward computation phase of the Nth round of training, if the management node 100 directly replaces compute node 401 with compute node 211 at this time, the AI ​​model may have to re-execute the forward computation phase, thus extending the overall training time of the AI ​​model by the computation time of one round of forward computation.

[0106] In this embodiment, after the management node 100 completes fault recovery, it can wait for the AI ​​model to complete the training process of the current round (or the specified Nth round) before switching the target subtask executed on the computing node 401 to the computing node 211 for execution.

[0107] S306: After the AI ​​model completes the Nth round of training, the management node 100 replaces the computing node 401 added to the supernode 210 with the computing node 211.

[0108] The process by which the management node 100 replaces the computing node 401 with the computing node 211 can be found in the description of the relevant parts of the above-mentioned management node 100 replacing the computing node 211 with the computing node 401, and will not be repeated here.

[0109] S307: Management node 100 continues to train the AI ​​model for the N+1th round using multiple supernodes.

[0110] As an implementation example, management node 100 can establish communication links between computing node 211 and other computing nodes, enabling computing node 211 to communicate with other computing nodes based on the established communication links during the execution of the target sub-task. Furthermore, management node 100 can synchronize the data (such as model parameters) generated by computing node 401 in executing the target sub-task to computing node 211, allowing computing node 211 to continue executing the target sub-task based on the synchronized data. This allows computing node 211 and the other computing nodes (i.e., the computing nodes in supernode 210 and supernode 220) to continue executing the N+1th round of training for the AI ​​model and subsequent training processes.

[0111] Furthermore, the management node 100 can also release compute node 401 for supernode 210 and update the information of the backup resource pool 400, such as updating the available compute nodes in the backup resource pool 400 from compute node 402 to compute node 404 to compute node 401 to compute node 404, so that the released compute node 401 can be used to replace the failed compute node in supernode 210 or supernode 220.

[0112] In this way, by dynamically adding computing nodes from the backup resource pool 400 to the faulty supernodes and reclaiming computing nodes, the management node 100 can use a small number of computing nodes from the backup resource pool 400 to perform redundant backups for a large number of supernodes in failure scenarios. This ensures the reliability of cluster 10 while effectively reducing the number of backup resources required to be deployed in cluster 10 and lowering hardware costs.

[0113] The embodiment shown in Figure 3 above illustrates fault recovery in cluster 10 by management node 100 at the granularity of compute nodes. In other embodiments, management node 100 may also perform overall replacement of faulty work frames in cluster 10 at the granularity of work frames. The following is an illustrative description with reference to Figure 4.

[0114] Referring to Figure 4, a flowchart illustrating another AI task execution method is shown. As shown in Figure 4, this method may include the following steps.

[0115] S401: Supernodes 210 and 220 in cluster 10 execute AI tasks.

[0116] S402: During the execution of AI tasks by multiple supernodes, management node 100 determines that computing node 211 located in working frame 1 of supernode 210 has failed.

[0117] S403: Management node 100 adjusts the optical path configuration of optical switching node 316 so that compute nodes 401 and 402 in working frame 5 of backup resource pool 400 can communicate with the non-faulty compute nodes in super node 210 based on the optical switching node 316 after the optical path configuration adjustment. Compute nodes 401 and 402 in working frame 5 are used to replace compute nodes 211 and 212 in working frame 1 of super node 210 to continue to participate in the execution of AI tasks.

[0118] In this embodiment, although the faulty computing node is computing node 211, the management node 100 can replace multiple computing nodes, including computing node 211, at the work frame level. Specifically, the management node 100 can select work frame 5 (or other work frames) from the backup resource pool 400 and use the selected work frame 5 to replace the faulty work frame 1 in the supernode 210.

[0119] The implementation of the management node 100 determining the working frame 5 and replacing the working frame 1 with the working frame 5 is similar to the implementation of the management node 300 determining the computing node 401 and replacing the computing node 211 with the computing node 401 in the embodiment shown in Figure 3 above. For details, please refer to the relevant descriptions above, and they will not be repeated here.

[0120] In practical application scenarios, when the working frame 1 itself fails, such as the switching node in the working frame fails, causing the computing nodes 211 and 212 in the working frame 1 to be unable to communicate with the computing nodes in other super nodes, the management node 100 can also replace the working frame 1 in the super node 210 with the working frame 5 in the backup resource pool 400 in the same way as described above.

[0121] Furthermore, in a further possible implementation, after adding the work frame 5 to the supernode 210, the management node 100 can also optimize the network topology of multiple switching nodes in the cluster 10 for multiple computing nodes participating in the execution of AI tasks, so as to ensure the performance of traffic generated by different computing nodes in the network transmission. The specific implementation method can be found in the description of the relevant aspects of network topology optimization in the embodiment shown in Figure 3 above, and will not be repeated here.

[0122] S404: Management node 100 continues to execute AI tasks using multiple supernodes that have been replaced by compute nodes.

[0123] In this embodiment, the multiple supernodes that have been replaced by computing nodes include working frame 2 in supernode 210, working frame 5 added to supernode 210, and working frames 3 and 4 in supernode 220. Thus, management node 100 can continue to execute AI tasks using the computing nodes in these working frames.

[0124] In addition, the management node 100 can also update the information of the backup resource pool 400, such as updating the number of work frames included in the backup resource pool 400 from 2 to 1, or updating the available work frames in the backup resource pool 400 from work frame 5 and work frame 6 to work frame 6, etc.

[0125] Furthermore, once workframe 1 completes fault recovery, such as the recovery of compute node 211 within workframe 1, management node 100 can reuse workframe 1 to execute AI tasks. Specifically, workframe 1 can replace workframe 5, which has been added to supernode 210. Then, management node 100 can continue executing AI tasks using workframes 1 and 2 in supernode 210, and workframes 3 and 4 in supernode 220. Additionally, management node 100 can release workframe 5 added to supernode 210 and update information in backup resource pool 400, such as updating the available workframes in backup resource pool 400 from workframe 6 to workframes 5 and 6.

[0126] In addition to performing fault recovery at the compute node or work frame level in cluster 10, management node 200 can also perform fault recovery at the supernode level in cluster 10.

[0127] In specific implementation, in cluster 20 shown in Figure 2, during the execution of AI tasks by multiple supernodes, after determining that a compute node in supernode 210 has failed, management node 100 can select multiple compute nodes from backup resource pool 400. By adjusting the optical path configuration of at least one optical switch node, the selected compute nodes in backup resource pool 400 can communicate with each other based on the adjusted optical path configuration of at least one optical switch node. This allows the selected compute nodes to replace all compute nodes in supernode 210. Then, management node 100 uses the supernodes replaced by the replaced compute nodes to continue executing AI tasks, such as continuing to train the AI ​​model. The specific implementation process is similar to the fault recovery process of management node 100 in cluster 10 at the granularity of compute nodes or work frames, as described above, and will not be repeated here.

[0128] It is worth noting that other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0129] The AI ​​task execution method provided in the embodiments of this application has been described above with reference to Figures 1 to 4. Next, the structure of the management node and computing device provided in the embodiments of this application will be described with reference to the accompanying drawings.

[0130] Referring to Figure 5, a schematic diagram of a management node structure is shown. As shown in Figure 5, the management node 500 includes:

[0131] The fault determination module 501 is used to determine that a first computing node of a target supernode among multiple supernodes has failed during the execution of artificial intelligence (AI) tasks by multiple supernodes in the cluster. The cluster also includes a management node and a backup resource pool. Each supernode among the multiple supernodes includes multiple computing nodes, the backup resource pool includes at least one computing node, and there are communication connections between different computing nodes in the cluster.

[0132] The adjustment module 502 is used to adjust the optical path configuration of the optical switching node so that the second computing node in the backup resource pool can communicate with the computing node in the target supernode that has not failed based on the optical switching node after the optical path configuration adjustment. The second computing node is used to replace the first computing node to continue to participate in the execution of AI tasks.

[0133] In one possible implementation, different computing nodes in the cluster communicate with each other through multiple switching nodes;

[0134] Adjustment module 502 is used to adjust the network topology of multiple switching nodes for multiple supernodes that have been replaced by compute nodes, based on a load balancing strategy.

[0135] In one possible implementation, the management node 500 further includes:

[0136] The acquisition module 503 is used to acquire the network topology of multiple switching nodes and the traffic characteristics generated by multiple computing nodes participating in the execution of AI tasks.

[0137] The generation module 504 is used to generate load balancing strategies based on network topology and traffic characteristics.

[0138] In one possible implementation, the AI ​​task is a training task for an AI model;

[0139] Management node 500 also includes replacement module 505, used for:

[0140] After the first computing node completes fault recovery, wait for the AI ​​model to complete the Nth round of training, where N is a positive integer;

[0141] After the AI ​​model completes the Nth round of training, the first computing node replaces the second computing node to continue training the AI ​​model.

[0142] The AI ​​model is trained for the N+1th round using multiple supernodes.

[0143] In one possible implementation, the AI ​​task includes multiple subtasks, and the first computing node runs the target subtask among the multiple subtasks before the failure.

[0144] Adjustment module 502 is used for:

[0145] Based on the target subtask running on the first computing node, determine from the backup resource pool the second computing node that has loaded the execution resources corresponding to the target subtask and is running the process used to execute the target subtask;

[0146] For the second computing node, adjust the optical path configuration of the optical switching node.

[0147] Since the management node 500 shown in Figure 5 corresponds to the management node 100 in the embodiments shown in Figures 3 and 4 above, the specific implementation method of the management node 500 shown in Figure 5 and its technical effects can be found in the relevant descriptions in the embodiments shown in Figures 3 and 4 above, and will not be repeated here.

[0148] Figure 6 is a schematic diagram of a computing device provided in this application. As shown in Figure 6, the computing device 600 includes a processor 601, a memory 602, a communication interface 603, and a bus 604. The processor 601, memory 602, and communication interface 603 communicate with each other via the bus 604. The bus 604 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 6, but this does not mean that there is only one bus or one type of bus. The communication interface 603 is used for communication with external devices, such as receiving scaling factors and data sent by other computing devices.

[0149] It should be understood that in the embodiments of this application, processor 601 may be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete device assemblies, etc. General-purpose processors may be microprocessors or any conventional processors, etc.

[0150] The memory 602 may include read-only memory and random access memory, and provides instructions and data to the processor 601. The memory 602 may also include non-volatile random access memory. For example, the memory 602 may also store device type information.

[0151] The memory 602 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0152] The memory 602 stores executable code, and the processor 601 executes the executable code to perform the method performed by the management node 100 in the embodiments shown in FIG3 and FIG4.

[0153] It should be understood that the computing device 600 according to the embodiments of this application may correspond to the management node 100 in the embodiments of this application, and may correspond to the method executed by the management node 100 in the methods shown in FIG3 and FIG4 in the embodiments of this application. The above and other operations and / or functions implemented by the computing device 600 are respectively to implement the flow of the corresponding methods in FIG3 and FIG4. For the sake of brevity, they will not be described in detail here.

[0154] It is worth noting that the computing device shown in Figure 6 is only an example. In actual applications, the computing device can be a collection of multiple devices. For example, multiple devices can be used to implement the method executed by the management node 100 in the methods shown in Figures 3 and 4 above. This is not a limitation.

[0155] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned AI task execution method.

[0156] This application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0157] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0158] The computer program product can be a software installation package. When any of the aforementioned AI task execution methods is required, the computer program product can be downloaded and executed on a computing device.

[0159] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0160] The terminology used in the above embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, “one or more” refers to one, two, or more; the character “ / ” generally indicates that the preceding and following objects are in an “or” relationship. In the embodiments of this application, “simultaneously” means within the same time period, including situations where they are at the same moment.

[0161] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0162] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cluster, characterized in that, The cluster includes a management node, multiple supernodes, and a backup resource pool. Each of the multiple supernodes includes multiple compute nodes, and the backup resource pool includes at least one compute node. The at least one compute node in the backup resource pool is connected to the multiple supernodes through an optical switching node. The multiple supernodes are used to execute artificial intelligence (AI) tasks; The management node is used to determine that the first computing node among the target supernodes has failed during the execution of the AI ​​task by the plurality of supernodes. The optical path configuration of the optical switching node is adjusted so that the second computing node in the backup resource pool can communicate with the computing node in the target supernode that has not failed based on the optical switching node whose optical path configuration has been adjusted. The second computing node is used to replace the first computing node to continue to participate in the execution of the AI ​​task.

2. The cluster according to claim 1, characterized in that, Different computing nodes in the cluster communicate with each other through multiple switching nodes; The management node is also used to adjust the network topology of the multiple switching nodes for the multiple supernodes that have been replaced by compute nodes, based on a load balancing strategy.

3. The cluster according to claim 2, characterized in that, The management node is also used for: Obtain the network topology of the multiple switching nodes; Obtain the traffic characteristics generated by multiple computing nodes participating in the execution of the AI ​​task; The load balancing strategy is generated based on the network topology and the traffic characteristics.

4. The cluster according to any one of claims 1 to 3, characterized in that, The AI ​​task is a training task for an AI model; The management node is also used for: After the first computing node completes fault recovery, it waits for the AI ​​model to complete the Nth round of training, where N is a positive integer; After the AI ​​model completes the Nth round of training, the first computing node replaces the second computing node to continue training the AI ​​model. The AI ​​model is then trained for the N+1th round using the multiple supernodes.

5. The cluster according to any one of claims 1 to 4, characterized in that, The AI ​​task includes multiple sub-tasks, and the first computing node runs the target sub-task among the multiple sub-tasks before the failure. The management node is used for: Based on the target subtask running by the first computing node, a second computing node is determined from the backup resource pool, which is loaded with the execution resources corresponding to the target subtask and runs a process for executing the target subtask. For the second computing node, adjust the optical path configuration of the optical switching node.

6. A method for executing artificial intelligence (AI) tasks, characterized in that, The method is applied to a cluster, which includes a management node, multiple supernodes, and a backup resource pool. Each of the multiple supernodes includes multiple computing nodes, and the backup resource pool includes at least one computing node. The at least one computing node in the backup resource pool is connected to the multiple supernodes through an optical switching node. The multiple supernodes are used to execute the AI ​​task. The method includes: During the execution of the AI ​​task by the plurality of supernodes, the management node determines that the first computing node among the target supernodes has failed. The management node adjusts the optical path configuration of the optical switching node so that the second computing node in the backup resource pool can communicate with the computing node in the target supernode that has not failed based on the optical switching node whose optical path configuration has been adjusted. The second computing node is used to replace the first computing node to continue to participate in the execution of the AI ​​task.

7. The method according to claim 6, characterized in that, Different computing nodes in the cluster communicate with each other through multiple switching nodes; The method further includes: The management node adjusts the network topology of the multiple switching nodes for the multiple supernodes that have been replaced by compute nodes, based on a load balancing strategy.

8. The method according to claim 7, characterized in that, The method further includes: The management node acquires the network topology of the multiple switching nodes; The management node acquires the traffic characteristics generated by multiple computing nodes participating in the execution of the AI ​​task; The management node generates the load balancing strategy based on the network topology and the traffic characteristics.

9. The method according to any one of claims 6 to 8, characterized in that, The AI ​​task is a training task for an AI model; The method further includes: After the first computing node completes fault recovery, the management node waits for the AI ​​model to complete the Nth round of training, where N is a positive integer; After the AI ​​model completes the Nth round of training, the management node replaces the second computing node with the first computing node to continue participating in the training of the AI ​​model; The management node uses the multiple supernodes to continue training the AI ​​model for the N+1th round.

10. The method according to any one of claims 6 to 9, characterized in that, The AI ​​task includes multiple sub-tasks, and the first computing node runs the target sub-task among the multiple sub-tasks before the failure. The management node adjusts the optical path configuration of the optical switching node, including: The management node determines, from the backup resource pool, a second computing node that has loaded the execution resources corresponding to the target subtask and is running a process for executing the target subtask, based on the target subtask being run by the first computing node. The management node adjusts the optical path configuration of the optical switching node for the second computing node.

11. A management node, characterized in that, The management node includes: The fault determination module is used to determine, during the execution of artificial intelligence (AI) tasks by multiple supernodes in the cluster, that a first computing node of a target supernode among the multiple supernodes has failed. The cluster also includes a backup resource pool. Each of the multiple supernodes includes multiple computing nodes, and the backup resource pool includes at least one computing node. The at least one computing node in the backup resource pool is connected to the multiple supernodes through an optical switching node. An adjustment module is used to adjust the optical path configuration of the optical switching node so that the second computing node in the backup resource pool can communicate with the computing node in the target supernode that has not experienced a failure based on the optical switching node after the optical path configuration adjustment. The second computing node is used to replace the first computing node to continue to participate in the execution of the AI ​​task.

12. A computing device, characterized in that, The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 6 to 10.

13. A computer-readable storage medium, characterized in that, Includes instructions that, when run on at least one computing device, cause the at least one computing device to perform the method as described in any one of claims 6 to 10.

14. A computer program product containing instructions, characterized in that, When it is run on at least one computing device, it causes the at least one computing device to perform the method as described in any one of claims 6 to 10.

Images