Fault handling method, computing cluster, device, and chip

Abstract

The present application relates to the technical field of machine learning, and discloses a fault handling method, a computing cluster, a device, and a chip. The method comprises: when a first node fails, acquiring a first weight corresponding to the first node, the first node being configured in a first TP domain, and computing nodes in the first TP domain being used for executing model computation of a first model on the basis of different weights allocated to the computing nodes from model weights; determining a weight allocation strategy of the first weight, the weight allocation strategy indicating a weight allocated to a second node from the first weight, and the second node being a computing node in the first TP domain different from the first node; and on the basis of the weight allocation strategy, sending, to the second node, the weight allocated to the second node from the first weight. By allocating the first weight to the second node, model computation tasks can be normally executed in the first TP domain, and the utilization rate of the computing nodes is high.

Description

Troubleshooting methods, computing clusters, devices and chips

[0001] This application claims priority to Chinese Patent Application No. 202510146899.6, filed on February 10, 2025, entitled "Fault Handling Method, Computing Cluster, Device and Chip", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of machine learning technology, and in particular to fault handling methods, computing clusters, devices and chips. Background Technology

[0003] As machine learning technology advances, model sizes continue to increase, leading to longer computation times. To reduce computation time, multiple computing nodes are used for parallel computation, and synchronization is performed based on the computation results of each node to determine the final model result.

[0004] Because different computing nodes need to remain synchronized during parallel computing, if any one or more of these nodes fail, the computing tasks of all nodes will be interrupted, resulting in a significant waste of computing power. Therefore, a fault handling method is urgently needed to continue the model's computational operations even when a computing node fails. Summary of the Invention

[0005] This application provides a fault handling method, computing cluster, device, and chip to continue the computation of the model in the event of a computing node failure. The technical solution is as follows.

[0006] Firstly, a fault handling method is provided, comprising: in the event of a first node failure, obtaining the first weight corresponding to the first node, wherein the first node is a computing node configured in a first tensor parallelism (TP) domain, and the first weight is the weight assigned to the first node from the model weights of the first model in the first TP domain, wherein each computing node in the first TP domain is used to perform model computation of the first model based on different weights in the assigned model weights; determining a weight allocation strategy for the first weight, wherein the weight allocation strategy indicates the weights assigned to a second node from the first weight, wherein the second node is a computing node in the first TP domain different from the first node; sending the weights assigned to the second node from the first weight to the second node based on the weight allocation strategy, or sending a weight allocation instruction to a third node based on the weight allocation strategy, wherein the weight allocation instruction indicates the third node to send the weights assigned to the second node from the first weight to the second node, wherein the third node is configured in the second TP domain and the weights assigned are the same as the first weights assigned to the first node.

[0007] After the first node fails, the first weight, which was originally calculated by the first node, can be allocated to the second node. This allows the second node to take over the weight calculations previously performed by the first node. Subsequently, only the model calculations of the failed first node are suspended; other computing nodes within the first TP domain can continue their model calculations normally, resulting in high utilization of the computing nodes. Allocating the first weight only within the failed first TP domain results in a small allocation range, low complexity, and high efficiency. This application does not limit the method of sending the allocated weights to the second node; they can be sent directly based on a weight allocation strategy or indirectly using a third node based on a weight allocation instruction, offering high flexibility.

[0008] In one possible implementation, before determining the weight allocation strategy for the first weight, the method further includes: if the memory overflow prediction result of the first TP domain indicates that the first TP domain does not experience a memory overflow, then determining a second node from multiple computing nodes in the first TP domain. Only when it is determined that the computing nodes in the first TP domain will not experience a memory overflow due to the allocation of the first weight will the first weight be assigned to the second node. This avoids directly assigning the first weight to the second node, which could lead to a memory overflow and runtime failure during model computation based on the additionally allocated weight.

[0009] In one possible implementation, the first weight corresponding to the first node is obtained, including: obtaining the first weight from the checkpoint file; or copying the first weight from the third node. The first weight can be obtained in multiple ways, offering high flexibility.

[0010] In one possible implementation, determining the weight allocation strategy for the first weight includes: determining the weight allocation strategy based on the number of non-faulty nodes in the first TP domain, and / or, determining the weight allocation strategy based on the node communication relationships within the first TP domain. In determining the weight allocation strategy, one or more pieces of information, such as the number of non-faulty nodes or the node communication relationships in the first TP domain, can be referenced, allowing for flexible determination using different methods based on different information.

[0011] In one possible implementation, after determining the weight allocation strategy for the first weight, the method further includes: adjusting the communication information corresponding to the second node based on the weights allocated to the second node in the first weight. The communication information indicates at least one of the following: the data being interacted with, the nodes interacting with, the interaction time, or the interaction order. Since the weights used by the second node during model calculation change—for example, by adding newly allocated weights from the first weight—the communication involved in the second node's model calculation will also change. By adjusting the communication information, communication coordination between the second node can be achieved, reducing the occurrence of communication conflicts.

[0012] In one possible implementation, model computation is also performed based on the input data of the first model. After determining the weight allocation strategy for the first weight, the process further includes: if the first TP domain is located in the first DP domain among multiple data parallel (DP) domains, determining the input data to be processed by the first DP domain from the multiple input data of the first model based on the number of nodes in each DP domain, and / or determining the input data to be processed by the first DP domain from the multiple input data of the first model based on the communication relationships between nodes in each DP domain. Due to a failure of the first node, the number of nodes performing model computation in the first DP domain where the first node is located decreases, the computation time of the first DP domain increases, and the second node will also generate additional communication when processing the weights allocated in the first weight, increasing the communication overhead and communication time within the first DP domain. If the amount of input data is the same, the processing time of the first DP domain will be longer than that of other DP domains. By reallocating the input data processed by each DP domain, the time difference in processing input data between each DP domain is reduced, thereby achieving load balancing between DP domains. Furthermore, this application does not limit the information referenced in the process of allocating input data. It can be one or more of the following: the number of normally operating nodes in each DP domain and the communication relationship between nodes in each DP domain, which provides high flexibility.

[0013] In one possible implementation, the method further includes: sending a first weight to the first node upon recovery from a failure; or, upon recovery from a failure of the first node, sending a first sending instruction to a third node, the first sending instruction instructing the third node to send the first weight to the first node. Besides assigning the first weight of the first node to the second node in the event of a failure of the first node, it is also possible to reassign the first weight processed by the second node to the first node upon recovery from a failure of the first node, supporting bidirectional adjustment of the first weight.

[0014] In one possible implementation, the first TP domain is located in the first DP domain among multiple DP domains, and / or the first TP domain is located in the first PP domain among multiple pipeline parallel (PP) domains. Computational nodes in different DP domains perform model computation based on different input data, and computational nodes in different PP domains perform model computation based on model weights located in different layers of the first model. This application can be applied to various parallel scenarios, such as DP and TP parallelism, PP and TP parallelism, or DP, TP, and PP parallelism, with wide application scenarios and high versatility.

[0015] Secondly, a computing cluster is provided, comprising a first TP domain, the first TP domain including a first node and a second node; the first node is used to perform model calculation of a first model based on a first weight, the first weight being the weight of the first model assigned to the first node in the model weights of the first model in the first TP domain; the second node is used to perform model calculation of the first model based on a second weight, the second weight being the weight of the first model assigned to the second node in the model weights of the first model in the first TP domain, the first weight and the second weight being different; the second node is also used to perform model calculation of the first model based on a third weight in the event of a failure of the first node, the third weight including the weight assigned to the second node in the first weight and the second weight.

[0016] In one possible implementation, the third weight includes a weight allocated to the second node from the first weight based on the number of non-faulty nodes in the first TP domain; and / or, the third weight includes a weight allocated to the second node from the first weight based on the node communication relationships within the first TP domain.

[0017] In one possible implementation, the second node is also used to receive the weights assigned to the second node from the first weights sent by the control device.

[0018] In one possible implementation, the second node is also used to receive weights sent by a third node in the computing cluster. The third node is configured in the second TP domain and is assigned the same weights as the first weights assigned to the first node. The third node sends the weights assigned to the second node from the first weights to the second node based on the weight allocation instructions sent by the control device.

[0019] In one possible implementation, the computing cluster further includes a fourth node and a second node, which are also used to compute the results based on the communication information and the interaction model of the fourth node. The communication information corresponding to the second node is obtained by adjusting the weight assigned to the second node in the first weight. The communication information indicates at least one of the interactive data, interactive nodes, interactive time, or interactive order.

[0020] In one possible implementation, model computation is also performed based on the input data of the first model. In the case where the computing cluster also includes multiple DP domains and the first TP domain is located in the first DP domain among the multiple DP domains, the input data processed by the first DP domain is the data determined based on the number of nodes in each DP domain from the multiple input data of the first model, and / or, the input data processed by the first DP domain is the data determined based on the node communication relationship in each DP domain from the multiple input data of the first model.

[0021] In one possible implementation, the first node is also used to receive a first weight sent by the control device in the event of a fault recovery, and to perform model calculation of the first model based on the first weight.

[0022] In one possible implementation, the first node is also used to receive a first weight sent by a third node in the computing cluster in the event of a fault recovery, and to perform model calculation of a first model based on the first weight. The third node is configured in the second TP domain and is assigned the same weight as the first node. The third node sends the first weight to the first node based on a first sending instruction sent by the control device.

[0023] In one possible implementation, the computing cluster also includes multiple DP domains, and / or multiple pipelined parallel PP domains, where computing nodes in different DP domains perform model computations based on different input data, and computing nodes in different PP domains perform model computations based on model weights located at different layers in the first model.

[0024] Thirdly, a fault handling apparatus is provided, comprising: a transceiver module for performing receiving and / or transmitting related operations in the first aspect or any possible implementation of the first aspect; and a processing module for performing other operations besides the receiving and / or transmitting related operations in the first aspect or any possible implementation of the first aspect.

[0025] In one possible implementation, the processing module is used to obtain the first weight corresponding to the first node in the event of a first node failure. The first node is a computing node configured in the first TP domain, and the first weight is the weight assigned to the first node from the model weights of the first model in the first TP domain. Each computing node in the first TP domain is used to perform model calculation of the first model based on the different weights assigned to it. The processing module is also used to determine the weight allocation strategy of the first weight, which indicates the weights assigned to the second node from the first weight. The second node is a computing node in the first TP domain that is different from the first node. The transceiver module is used to send the weights assigned to the second node from the first weight to the second node based on the weight allocation strategy, or to send a weight allocation instruction to a third node based on the weight allocation strategy. The weight allocation instruction instructs the third node to send the weights assigned to the second node from the first weight to the second node. The third node is configured in the second TP domain, and the weights assigned to it are the same as the first weights assigned to the first node.

[0026] In one possible implementation, the processing module is further configured to determine a second node from multiple computing nodes in the first TP domain if the memory overflow prediction result of the first TP domain indicates that there is no memory overflow in the first TP domain.

[0027] In one possible implementation, a processing module is used to obtain the first weight from the checkpoint file; or, to copy the first weight from the third node.

[0028] In one possible implementation, the processing module is used to determine a weight allocation strategy based on the number of non-faulty nodes in the first TP domain, and / or, based on the node communication relationships within the first TP domain.

[0029] In one possible implementation, the processing module is further configured to adjust the communication information corresponding to the second node based on the weight assigned to the second node in the first weight, wherein the communication information indicates at least one of the interactive data, the interactive node, the interactive time, or the interactive order.

[0030] In one possible implementation, the model computation is also performed based on the input data of the first model. The processing module is further configured to determine the input data to be processed in the first DP domain from the multiple input data of the first model, based on the number of nodes in each DP domain, and / or, based on the node communication relationship within each DP domain.

[0031] In one possible implementation, the transceiver module is further configured to send a first weight to the first node in the event that the first node has recovered from a failure; or, in the event that the first node has recovered from a failure, send a first transmission instruction to the third node, the first transmission instruction instructing the third node to send the first weight to the first node.

[0032] In one possible implementation, the first TP domain is located in the first DP domain among multiple DP domains, and / or the first TP domain is located in the first PP domain among multiple PP domains. Computational nodes in different DP domains perform model computation based on different input data, and computational nodes in different PP domains perform model computation based on model weights located in different layers of the first model.

[0033] Fourthly, a control device is provided, the control device being connected to a first TP domain, the device including a processor for loading and executing at least one instruction to cause the control device to perform the method of the first aspect or any possible implementation thereof.

[0034] In one possible implementation, the device includes a memory coupled to a processor, the memory storing at least one instruction.

[0035] Fifthly, a computer-readable storage medium is provided, wherein at least one instruction is stored in the computer-readable storage medium, the instruction being loaded and executed by a processor to implement the fault handling method in the first aspect or any possible implementation of the first aspect.

[0036] In a sixth aspect, a computer program (product) is provided, comprising computer program / instructions, which are executed by a processor to enable a computer to implement the fault handling method in the first aspect or any possible implementation thereof.

[0037] A seventh aspect provides a communication device comprising: a transceiver, a memory, and a processor. The transceiver, the memory, and the processor communicate with each other via an internal interconnection path. The memory stores instructions, and the processor executes the instructions stored in the memory to control the transceiver to receive and transmit signals. When the processor executes the instructions stored in the memory, it causes the processor to perform the method of the first aspect or any possible implementation thereof. Optionally, the communication device may be a chip.

[0038] Optionally, there may be one or more processors and one or more memories.

[0039] Alternatively, the memory can be integrated with the processor, or the memory can be set up separately from the processor.

[0040] In the specific implementation process, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the processor on the same chip or set on different chips. This application does not limit the type of memory or the way the memory and processor are set.

[0041] Eighthly, a chip is provided, including a processor for calling and running program instructions or code stored in a memory, causing a communication device equipped with the chip to perform the methods in the foregoing aspects.

[0042] Ninthly, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, the processor being used to execute code in the memory, and when the code is executed, the processor being used to perform the methods in the above aspects.

[0043] In a tenth aspect, a fault handling system is provided, the system including a control device and a computing cluster in the second aspect or any possible implementation of the second aspect, the control device being used to execute the fault handling method in the first aspect or any possible implementation of the first aspect.

[0044] It should be understood that the device mentioned in the third or seventh aspect above can be the chip mentioned in the eighth or ninth aspect, or the control device of the fourth aspect. The beneficial effects achieved by the technical solutions and corresponding possible implementations of the second to tenth aspects of this application can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0045] Figure 1 is a schematic diagram of a hybrid parallelism provided in an embodiment of this application;

[0046] Figure 2 is a schematic diagram of the implementation environment provided in the embodiments of this application;

[0047] Figure 3 is a schematic diagram of the structure of a computing cluster provided in an embodiment of this application;

[0048] Figure 4 is a schematic diagram of another computing cluster structure provided in an embodiment of this application;

[0049] Figure 5 is a schematic diagram of another implementation environment provided in the embodiments of this application;

[0050] Figure 6 is a flowchart of a fault handling method provided in an embodiment of this application;

[0051] Figure 7 is a schematic diagram of weight allocation provided in an embodiment of this application;

[0052] Figure 8 is a schematic diagram of the allocation of a first weight provided in an embodiment of this application;

[0053] Figure 9 is a schematic diagram of a memory overflow judgment process provided in an embodiment of this application;

[0054] Figure 10 is a schematic diagram of the distribution of model weights provided in an embodiment of this application;

[0055] Figure 11 is a schematic diagram of the distribution of model weights provided in another embodiment of this application;

[0056] Figure 12 is a schematic diagram of a communication change provided in an embodiment of this application;

[0057] Figure 13 is a schematic diagram of the interaction of a PP domain provided in an embodiment of this application;

[0058] Figure 14 is a schematic diagram of the interaction of a DP domain provided in an embodiment of this application;

[0059] Figure 15 is a schematic diagram of input data allocation provided in an embodiment of this application;

[0060] Figure 16 is a schematic diagram of the distribution of a first weight provided in an embodiment of this application;

[0061] Figure 17 is a schematic diagram of a fault handling device provided in an embodiment of this application;

[0062] Figure 18 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0063] Figure 19 is a schematic diagram of the structure of another computing device provided in an embodiment of this application. Detailed Implementation

[0064] The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. The implementation of this application will be further described in detail below with reference to the accompanying drawings.

[0065] As model size increases, the computation time for input data becomes longer. In some cases, parallel computation using dynamic programming (DP) can reduce this computation time. In DP parallelism, the input data to be computed by the model is divided into multiple subsets, each containing different input data. Multiple computing nodes then compute these subsets in parallel. These computing nodes are called DP nodes. Each DP node can be configured with the complete model, such as the model weights. Alternatively, it can be configured with a partial model, such as some of the model's weights. Regardless of the model configured, each DP node can compute the subsets of data samples in parallel using its configured model. The computation results from each DP node are then aggregated via communication for subsequent operations.

[0066] On the other hand, as the model size increases, the memory of a single computing node becomes insufficient to store the entire model. Therefore, the model can be partitioned, and each partitioned part can be deployed to different computing nodes for computation. PP parallelism and TP parallelism are two methods for achieving parallel computation by partitioning the model. In PP parallelism, the model's weights can be split into multiple layers according to their hierarchy, and the weights of different layers can be distributed across multiple computing nodes, with each computing node processing one or more layers of the model. TP parallelism vertically partitions the model's weights into multiple parts, with each part being computed by a different computing node.

[0067] Optionally, a computing node can be a computing device or a component of a computing device, such as a computing card configured on the computing device. A computing card is also called a computation card, accelerator card, or data processing card. In the case where one computing node corresponds to one computing card, the 3D hybrid parallel mode of DP, PP, and TP can be seen in Figure 1. In Figure 1, a rectangle indicates one computing card, different rows indicate different DP domains, different columns indicate different PP domains, and each stage of the PP domain in Figure 1 can be assigned to multiple computing cards to form a TP domain.

[0068] Because the computation results of multiple computing nodes need to be synchronized during parallel model calculations, a failure in any one of these nodes will interrupt the overall computation task involving all nodes. Furthermore, without backup computing power, the failed node cannot be replaced promptly after a node failure, requiring manual on-site repair or hardware replacement. This results in lengthy recovery times and significant wasted computing power due to prolonged interruptions across multiple nodes. Therefore, a fault handling method is urgently needed to allow the model calculations to continue even when a computing node fails.

[0069] In related technologies, when any computing node fails, the control device isolates the entire DP domain containing the failed computing node. All computing nodes in the DP domain stop computing, while computing nodes in other DP domains continue to perform computing tasks normally. However, this method requires suspending all normal computing nodes within the entire DP domain, resulting in significant waste of computing power.

[0070] This application provides a fault handling method for continuing the computation of a model using the computing cluster when a computing node in the computing cluster fails. Please refer to Figure 2, which shows a schematic diagram of the implementation environment of the fault handling method provided in this application embodiment. This implementation environment includes a control device 01, a first node 02, and a second node 03. The first node 02 and the second node 03 are nodes in the computing cluster. The control device 01 can establish a communication connection with the computing cluster via wired or wireless means, thereby enabling communication with the first node 02 and the second node 03.

[0071] In Figure 2, the first node 02 and the second node 03 are configured in the same TP domain in the computing cluster. In some cases, the TP domain where the first node 02 and the second node 03 are located can be called the first TP domain. In the absence of node failure, the first node 02 will perform model calculation of the first model based on the assigned first weight, and the second node 03 will perform model calculation of the first model based on the assigned second weight.

[0072] If a node failure occurs, such as the failure of the first node 02, the control device 01 can determine a weight allocation strategy based on the first weight and use this strategy to send the weights allocated to the second node 03 from the first weight, so that the second node 03 can perform model calculations based on the allocated weights. Furthermore, Figure 2 is intended to illustrate the connection relationship between the control device 01, the first node 02, and the second node 03, rather than to limit the number of second nodes 03 in the first TP domain. There can be one or more; ellipses are used in Figure 2 to indicate other possible second nodes 03 in the first TP domain.

[0073] Furthermore, a computing cluster can consist of multiple TP domains, or it can include multiple DP domains in addition to TP domains. For example, as shown in Figure 3, multiple computing nodes are divided into multiple DP domains. Each DP domain includes multiple computing nodes. Computing nodes in different DP domains perform model calculations based on different input data. Each computing node in the same DP domain can calculate the input data of the first model according to the different parts of the model weights of the first model. In the multidimensional parallel scenario shown in Figure 3, the first node 02 shown in Figure 2 can be any computing node in Figure 3 that has failed, such as the first computing node in the DP2 domain (marked in black). In this case, the remaining three computing nodes in the DP2 domain can all serve as the second node 03 shown in Figure 2.

[0074] Optionally, the computing cluster can also include multiple PP domains in addition to the TP domain. Referring to Figure 4, multiple computing nodes are divided into multiple PP domains. Each PP domain includes multiple computing nodes. Computing nodes in different PP domains perform model computation based on model weights located at different layers in the first model. Model weights allocated to any layer in any PP domain are distributed to different computing nodes, and each computing node performs model computation using its assigned weights. In the computing cluster shown in Figure 4, the first node 02 shown in Figure 2 is, for example, the first computing node in the PP2 domain (marked in black). In this case, the remaining three computing nodes in the PP2 domain can all serve as the second node 03 shown in Figure 2.

[0075] For example, a computing cluster can also include multiple PP domains and multiple DP domains in addition to a TP domain. The computing cluster can then be as shown in Figure 1, where one computing card corresponds to one computing node. The first TP domain can be the TP domain marked with a dashed box in Figure 1. When any computing card in the first TP domain fails, that computing card can act as the first node 02 in Figure 2, and the other computing cards in the first TP domain can act as the second node 03 in Figure 2. Combining Figures 1, 3, and 4, it can be seen that the first TP domain configuring the first node 02 and the second node 03 can be located in the first DP domain among multiple DP domains, and / or, the first TP domain can be located in the first PP domain among multiple PP domains.

[0076] Figure 5 is a schematic diagram of another implementation environment provided by the embodiments of this application, showing the interaction between the control device and the computing cluster when the computing cluster includes DP domain, TP domain and PP domain. The management system in Figure 5 corresponds to the control device 01 in Figure 2, and the multiple computing cards included in the hardware correspond to the multiple computing nodes included in the computing cluster in the above embodiments. In Figure 5, the management system uses the monitoring system to monitor the operation of multiple computing cards. When a computing card failure is detected, the management system performs fault diagnosis and location on the faulty card and determines the fault type. In addition, the management system also performs out-of-memory (OOM) prediction after scaling down, and determines whether an out-of-memory error will occur when the computing card in the faulty TP domain where the faulty card is located adds the weight allocated in the first weight to the original model weight, and executes the calculation task corresponding to the original model weight and the calculation task of the weight in the newly added first weight.

[0077] Based on prediction results and fault type, the management system can isolate faulty cards when the fault type is hardware failure and there is no memory overflow. It can then re-allocate the weights of the faulty TP domain containing the faulty card, allocating the first weight assigned to the faulty card to a normal computing card within that domain for computational support. Furthermore, the management system can re-allocate the communication relationships within the faulty TP domain, obtaining the communication information of the computing cards performing computations within that domain. This communication information is then sent to the collective communication library (CCL), where the CCL's computational framework is used to perform dynamic link establishment, thereby controlling the interaction between computing cards in the computing cluster.

[0078] For example, the management system will also perform load assessment on each DP domain based on the redefined weights, optimize the data sample splitting based on the load assessment results, and redetermine the input data to be processed by each DP domain from among the multiple input data included in the data sample. Afterwards, the management system can send the redefined weights and the reassigned data samples to the computing cluster, where each computing card in the computing cluster will execute the model calculation of the first model based on the newly assigned weights and the newly assigned data samples.

[0079] Optionally, the control device 01, the first node 02, and the second node 03 can be any device with processing capabilities, and can be any terminal device, including but not limited to desktop computers, laptops, tablets, or smartphones. Optionally, the control device 01, the first node 02, and the second node 03 can also be servers, such as central servers, edge servers, or local servers in a local data center. The server can be a physical server or a cloud server providing cloud computing services. The entity executing this fault handling method can be a single device as a whole, or a component of a computing device, such as a single board, line card, or functional module on the computing device, or a chip used to implement the method. Optionally, the first node 02 and the second node 03 can be different components on the same computing device, or components on different computing devices. Furthermore, the control device 01, the first node 02, and the second node 03 can also be multiple devices. These multiple devices can be the same device, such as all being terminals, or they can be different devices, such as one device being a terminal and another device being a server. In some cases, multiple devices can be referred to as a device cluster. This application embodiment does not specifically limit the execution subject of the method.

[0080] This application provides a fault handling method that can be applied to the implementation environment shown in Figure 2 or Figure 5. Taking the method as being executed by control device 01 as an example, the flowchart of the method is shown in Figure 6, including S601-S603.

[0081] S601, in the event of a failure of the first node, obtain the first weight corresponding to the first node. The first node is a computing node configured in the first TP domain. The first weight is the weight assigned to the first node by the first model in the model weights of the first model in the first TP domain. Each computing node in the first TP domain is used to perform model calculation of the first model based on the different weights in the assigned model weights.

[0082] For example, the first model can be an artificial intelligence (AI) model applicable to any field, such as natural language processing, image recognition, speech recognition, or other fields like risk prediction, autonomous driving, and personalized education. In some cases, the AI ​​model can be a large AI model, which refers to a deep learning model with a huge number of parameters, containing billions or trillions of parameters. Furthermore, the first model can be a model in training or a model that has already been trained and is in use.

[0083] In one possible implementation, the parameters of the first model include model weights and bias vectors for each layer, used to determine the connection strength between neurons in the first model and the degree of influence of signal propagation, thereby controlling the mapping and extraction of features from the input data by the first model. When using TP parallelism, multiple computing nodes in the same TP domain perform model computation of the first model based on different weights assigned to them in the model weights.

[0084] In some cases, the control device can assign the model weights of the first model located in any TP domain to each computing node in that TP domain. Optionally, the model weights in any TP domain can be the complete weights of the first model. For example, the computing cluster includes multiple computing nodes divided into different TP domains, and the computing cluster does not use DP parallelism or PP parallelism during the parallel computing of the model. Alternatively, the computing cluster includes multiple computing nodes divided into TP domains in different DP domains, but the complete weights of the first model are configured in different DP domains, and each DP domain includes only one TP domain. In this case, the model weights of the first model located in that TP domain are the complete weights of the first model.

[0085] For example, the model weight of the first model located in the TP domain can also be a partial weight of the first model. For example, the computing cluster includes multiple computing nodes, as shown in Figure 4, which are divided into multiple PP domains and TP domains. In this case, the model weight configured in a TP domain is the model weight of the first model located in one or more layers. Regardless of the model weight of the first model located in the TP domain, the control device can determine the weight assigned to each computing node in the TP domain. Different computing nodes are assigned different weights. Here, different weights refer to weights located in different positions in the model weights. For example, the two weights shown in Figure 7 (2) are located in different columns of the model weights shown in Figure 7 (1), and belong to different weights located in different positions. The values ​​of weights located in different positions can be the same or different. The difference can be completely different or partially different. Partial difference is, for example, the two columns of weights shown in Figure 7 (2). The two columns of weights include the same values ​​20 and 30, and also include different values ​​35, 22, 21 and 33.

[0086] Optionally, the control device can split the model weights by row and assign the split weights of multiple rows to multiple nodes in the same TP domain. Alternatively, the control device can split the model weights by column and assign the split weights of multiple columns to multiple nodes in the same TP domain. For example, as shown in Figure 7, Figure 7(1) shows the model weights before splitting, and Figure 7(2) shows the model weights after splitting. The control device can split the model weights by column and assign the two split weights to two computing nodes.

[0087] By assigning model weights within the TP domain, the computational load on each computing node can be reduced and its efficiency improved without affecting the calculation results. For example, after determining the weights for each computing node, the control device can send these weights to each node, enabling them to perform model calculations for the first model based on their assigned weights. The computing node calculates the input vector within its own TP domain based on the assigned weights. Therefore, computing nodes within the same TP domain will calculate the same input vector using different assigned weights.

[0088] Optionally, the input vector in the TP field can be the encoded result of a token of any type of input data. A token refers to the smallest processing unit of the input data. Taking text as an example, a token refers to a word, part of a word, or a common combination of characters. Taking an image as an example, a token can be a small piece of the image, such as a pixel or an image patch containing multiple pixels. Taking audio as another example, a token can be a short segment of audio. Each token can be encoded into a vector, which can be any number indicating the meaning of the token.

[0089] In some cases, the input vector of the TP field can also be the result of processing the encoded result. For example, in the computing cluster shown in Figure 4, the input vector of the first TP field from left to right is the encoded result of the token, while the input vector of the second TP field is the result of the computing card in the first TP field.

[0090] Regardless of the input vector, the computation nodes within the TP domain can perform model calculations based on the input vector and the assigned weights. Continuing with Figure 7 as an example, Figure 7 also shows the input vector. Figure 7(1) shows the calculation result obtained by multiplying the input vector and the model weights without splitting the model weights, and Figure 7(2) shows the calculation result obtained by multiplying the input vector and the two split weights after splitting the model weights. As can be seen from Figure 7, even if the model weights are split into multiple weights and assigned to different computation nodes for calculation, the calculation result of the model weights before splitting can be obtained based on the calculation results of multiple computation nodes. Furthermore, Figure 7 is intended to illustrate the process of splitting model weights, rather than to limit the model weights and input vector of the first model. Also, the control device can split the model weights into two weights as shown in Figure 7, or it can split them into more weights.

[0091] In one possible scenario, a computing node in the computing cluster may fail. If the control device detects a failure in the first node, it can execute the fault handling method provided in this application embodiment. For ease of understanding, the faulty computing node will be referred to as the first node, the weight assigned to the first node will be called the first weight, the TP domain where the first node is located will be called the first TP domain, the computing node in the first TP domain that calculates the first weight will be called the second node, and the weight assigned to the second node will be called the second weight.

[0092] The control device can detect whether a computing node is malfunctioning based on the usage of its central processing unit (CPU), memory, and disk space. It can also detect malfunctions based on the computing node's runtime logs. In some cases, the control device can detect malfunctions using a heartbeat mechanism. Under this mechanism, a non-malfunctioning computing node periodically sends heartbeat information to the control device. The control device determines that the computing node sending the heartbeat information is not malfunctioning based on receiving the heartbeat information within a reference time period. If the control device does not receive heartbeat information from any computing node within the reference time period, it determines that the computing node is the first malfunctioning node, and the TP domain where this computing node is located is the first TP domain. Figure 8 is a schematic diagram of a fault handling method provided in an embodiment of this application. Referring to Figure 8(a), the malfunctioning computing node is a W8 computing card marked in black, and the TP domain where this computing node is located is the TP domain located between the DP#1 and PP#1 domains.

[0093] Since multiple computing nodes need to maintain synchronization during parallel computing, the control device can pause the model computing tasks of the computing cluster after detecting a faulty first node, and then perform fault diagnosis and location on the first node to determine the type of fault. Optionally, the control device can first determine whether the fault type of the first node is a software fault or a hardware fault. Software faults may be caused by program errors or operational mistakes, and these types of faults can be repaired quickly; for example, program errors and operational mistakes can be repaired by restarting the first node. Hardware faults may be caused by physical damage or hardware component failure, such as hard drive failure, memory module failure, or power supply problems. Hardware faults require hardware replacement to repair, such as replacing the damaged components in the first node or replacing the first node with a new, unfaulty computing node. In this case, the control device can further determine whether there is a backup idle node in the first TP domain. If there is a backup idle node, the idle node can be replaced with the first node to achieve fault repair, with a short repair time. If there is no backup idle node, the fault of the first node needs to be repaired manually, which takes longer.

[0094] If the failure recovery time of the first node is long, the prolonged suspension of model calculation in the computing cluster will lead to a waste of computing power. In this case, the control device can obtain the first weight corresponding to the first node and allocate the first weight of the first node to the second node in the first TP domain. The second node can then take over the weight calculation task of the first node, thereby avoiding the suspension of the entire computing cluster due to the failure of the first node.

[0095] If the control device caches the first weight, it can access local storage and determine the first weight corresponding to the first node from the cached weights based on the first node's identifier. For example, after assigning weights to the model weights of the first model, the control device caches the weights allocated to each computing node in the first TP domain. The identifier of the first node can be its location or its identity document (ID), etc.

[0096] If the control device does not cache the first weights, this could be due to limited storage space, meaning the control device did not cache the weights corresponding to each computation node after assigning model weights to them. Alternatively, although the control device cached the weights after allocation, multiple computation nodes, after calculating the input vector based on the assigned weights and determining the output data obtained by the first model processing the input data based on the calculation results, will adjust the weights based on the output data. In this case, the first weight corresponding to the first node will differ from the weight cached by the control device. Regardless of the reason why the control device does not cache the first weights, the first weights assigned to the first node can be obtained through methods including but not limited to the following two methods.

[0097] Method 1: Obtain the first weight from the checkpoint file.

[0098] In one possible scenario, the first node is configured with a checkpointing mechanism. During the calculation of the input vector based on the assigned first weight, it periodically saves the node's state, forming a "checkpoint file." After saving the checkpoint file, the first node also reports it to a remote device for storage. The remote device can be a control device or other devices with storage capabilities. The checkpoint file contains all the state information of the first node at a given moment, such as the assigned first weight, the received input vector, or the calculation result obtained by calculating the input vector based on the first weight. Therefore, in the event of a failure of the first node, the control device can access the storage space containing the checkpoint file and retrieve the first weight from it.

[0099] Method 2: Copy the first weight from the third node.

[0100] In one possible scenario, the computing cluster also includes a third node. This third node is configured in the second TP domain and is assigned the same weight as the first node. The third node being assigned the same weight as the first weight means that the weights assigned to the third node and the first weight are in the same position within the model weights and have the same numerical value. Optionally, if the computing cluster includes multiple TP domains and multiple DP domains, the second TP domain and the first TP domain are TP domains within different DP domains, and the third node's position in the second TP domain is the same as the first node's position in the first TP domain.

[0101] The computing cluster includes multiple TP domains and multiple DP domains, as shown in Figure 3, and also as shown in Figure 8. If the weights of the first model configured in different DP domains are the same, for example, all are the full weights of the first model, then the weight assigned to the third node at the corresponding position is the same as the first weight assigned to the first node.

[0102] Continuing with Figure 8 as an example, the faulty card of the first node is located as the 8th card in the PP#1 domain within the DP#1 domain. Therefore, the 8th card in the PP#1 domain within the DP#2 domain (the calculation card marked with a diagonal line in Figure 8(a)) is located in the same position as the first node, belonging to the third node corresponding to the first node. Since the first node is faulty, the control device cannot read the first weight from it. The control device can, based on the fact that the third node is not faulty and can communicate normally, copy the assigned weight from the third node to obtain the first weight corresponding to the first node.

[0103] Optionally, the control device can randomly select one of acquisition methods (Method 1 or Method 2) to obtain the first weight, or it can select different acquisition methods depending on different situations. For example, if there is no corresponding third node for the first node, Method 1 can be selected to obtain the first weight. The absence of a third node could be due to the first model's parallel strategy not involving DP parallelism, such as the first model only using TP parallelism, or the first model using TP parallelism + PP parallelism. Alternatively, the absence of a third node could also be due to the first model's parallel strategy involving DP parallelism, but the computing cluster being a cluster that has experienced multiple failures. Previously, the first TP domain may have already failed, and the first node undertook the weight allocation task for the failed node. In this case, nodes located in the same position in other DP domains will be assigned different weights than the first weight assigned to the first node.

[0104] For example, if the first node has a corresponding third node, the control device can also choose to obtain the first weight using method two. Since the first weight obtained in method two is the weight assigned to the third node at the current moment, it is more timely than the first weight in the periodically cached checkpoint file. The control device can give priority to using method two to obtain the latest first weight.

[0105] S602, determine the weight allocation strategy for the first weight, the weight allocation strategy indicates the weight of the first weight to be allocated to the second node, the second node is a computing node in the first TP domain that is different from the first node.

[0106] In one possible scenario, the model weights of the first model are distributed to different computing nodes to perform computational tasks because the first model is large enough that a single computing node cannot handle the computation of all the model weights. For example, the memory space of a single computing node may not be sufficient to support the weight computation. In this case, distributing the weight computation task of the first node to the second node may cause the second node to experience a memory overflow and malfunction. Therefore, before distributing the first weights to the second node, the control device will also perform a memory overflow prediction for the first TP domain.

[0107] For example, the process of controlling the device to perform memory overflow prediction includes: obtaining model parameters of a first model and memory parameters of a first TP domain, wherein the model parameters are used to describe the memory usage of the first model and the memory parameters are used to describe the memory space of each computing node in the first TP domain; determining the memory overflow prediction result of the first TP domain based on the model parameters and the memory parameters, wherein the memory overflow prediction result indicates whether there is a memory overflow in the memory space of the computing nodes in the first TP domain when the computing nodes in the first TP domain perform model calculations based on the assigned weights.

[0108] In one possible scenario, model parameters include, but are not limited to, the number of model layers in the first model, the length or type of input data, the number of parameters in the first model, the quantization bit width, and the batch size. The number of model layers and the amount of memory occupied are positively correlated, as are the length of the input data. The number of parameters indicates the number of weights and biases in the first model; the number of parameters and the amount of memory occupied are positively correlated. The quantization bit width controls the parameter width of the first model; the quantization bit width and the amount of memory occupied are positively correlated. The batch size indicates the number of input data points processed by the first model each time; the batch size and the amount of memory occupied are positively correlated. Regarding the type of input data, processing image and audio data requires more memory than processing text data.

[0109] Optionally, the control device also acquires the parallel strategy of the first model, which indicates the number of TP, DP, and PP domains in the computing cluster and the number of computing nodes in the first TP domain. Based on the parallel strategy and model parameters, the control device can determine the model weights and input vectors allocated to the first TP domain. Based on the input vectors and model weights, it predicts the amount of memory occupied by each computing node during the computation process in the first TP domain at different stages. It compares the predicted memory amount with the size of the computing node's memory space to determine whether there is a memory overflow. The memory amount predicted by the control device can be the memory space occupancy rate. In this case, if the occupancy rate is greater than a reference threshold, the memory overflow prediction result indicates that a memory overflow exists; if the occupancy rate is not greater than the reference threshold, the memory overflow prediction result indicates that no memory overflow exists. The reference threshold can be any positive number set based on experience, such as 1.

[0110] For example, the control device can determine the amount of memory used by other computing nodes to collaboratively compute the weights assigned to that computing node in the event of a computing node failure, to determine whether a memory overflow exists in the first TP domain. The control device can also determine the minimum number of computing nodes that will not experience a memory overflow based on the number of computing nodes in the first TP domain and in conjunction with model parameters. Then, in the event of a first node failure, the control device compares the number of non-failed nodes in the first TP domain with the minimum number. If the number of non-failed nodes is not less than the minimum number, the memory overflow prediction result indicates that a memory overflow has not occurred.

[0111] In one possible scenario, the control device can use a model to determine whether a memory overflow occurs in the computing node. Referring to Figure 9, the model used by the control device is a computing card memory usage assessment model. The control device inputs the model parameters, hardware parameters, and parallel strategy into the computing card memory usage assessment model. The computing card memory usage assessment model can perform similar operations as described in the above embodiments to determine whether a memory overflow occurs, thereby determining the possibility of OOM occurring after scaling down and obtaining a memory overflow prediction result.

[0112] This application does not limit the timing of the control device's determination of whether a memory overflow exists. It can be performed before the first node fails, for example, before assigning weights to each computing node. Alternatively, it can be performed after the first node fails but before obtaining the first weight, or after obtaining the first weight but before determining the weight allocation strategy. Regardless of the timing or method used by the control device to determine the memory overflow situation, it can stop determining the weight allocation strategy when the memory overflow situation indicates the existence of a memory overflow.

[0113] If the memory overflow prediction result indicates that there is no memory overflow in the first TP domain, the control device begins to determine a weight allocation strategy for the first weight within the first TP domain. For example, it determines a second node from multiple computing nodes in the first TP domain, and then determines the weights allocated to each second node from the first weight to obtain the weight allocation strategy. Optionally, the control device may determine the weight allocation strategy through methods including but not limited to the following two methods.

[0114] Method 1: Determine the weight allocation strategy based on the number of non-faulty nodes in the first TP domain.

[0115] Taking an example with N-1 non-faulty nodes, where N indicates the TP parallelism and is any positive integer greater than 1, the control device divides the model weights into N-1 equal parts and then determines the second node corresponding to each part. In some cases, since the model weights may not be evenly divided into N-1 parts, the control device can first select some computing nodes from multiple non-faulty computing nodes in the first TP domain as second nodes. The number of selected second nodes is M, where M is any positive integer less than N-1. The control device then divides the model weights into M equal parts and determines the second node corresponding to each part. The selection of second nodes can be random or based on the hardware parameters of each computing node, such as the amount of memory in the computing node.

[0116] Method 2: Determine the weight allocation strategy based on the node communication relationship within the first TP domain.

[0117] In some cases, node communication relationships include, but are not limited to, the connection relationships between computing nodes within the first TP domain, the bandwidth of the connection links, and the interaction strategies. The interaction strategies indicate the interactions that each computing node needs to perform. In one possible scenario, after computing nodes calculate the input vector based on the assigned weights, they also fuse the calculation results of the input vector to obtain the calculation results of the first weight and the input vector. The fusion process can be seen in Figure 7(2). The two calculation results calculated based on the two weights will be fused into a result matrix. Therefore, there are multiple computing nodes in the first TP domain responsible for calculating the input vector, and there will be information interaction needs between these multiple computing nodes. In this case, a second node can be selected from the computing nodes within the first TP domain based on the node communication relationships to control the communication cost between the second nodes.

[0118] After selecting a second node from multiple computing nodes in the first TP domain, the control device can allocate a first weight to each second node. The control device can distribute the first weight equally, or it can allocate the first weight based on the communication complexity between the second nodes. For example, second nodes with low communication complexity are allocated more weight, and second nodes with high communication complexity are allocated less weight.

[0119] The control device can determine the model allocation strategy for the first weight based on either determination method one or determination method two. Alternatively, it can combine determination method one and determination method two to determine the model allocation strategy for the first weight. For example, the weight allocation strategy can be determined based on the node communication relationship within the first TP domain and the number of non-faulty nodes.

[0120] For example, if the first weight cannot be evenly divided into N-1 parts, when selecting a second node from the non-faulty computing nodes in the first TP domain, the control device will select M computing nodes as the second node based on the node communication relationship, thereby controlling the communication cost when the second node calculates the first weight.

[0121] S603, based on the weight allocation strategy, send the weight allocated to the second node from the first weight to the second node, or, based on the weight allocation strategy, send a weight allocation instruction to the third node, the weight allocation instruction instructing the third node to send the weight allocated to the second node from the first weight to the second node, the third node is configured in the second TP domain, and the weight allocated is the same as the first weight allocated to the first node.

[0122] Regardless of the method used by the control device to determine the weight allocation strategy, it can send the newly allocated weights from the first weight to the second node based on the weight allocation strategy. Optionally, the control device can directly send the newly allocated weights to the second node. For example, after determining the weight allocation strategy, the control device can divide the first weight based on the weight allocation strategy and send each portion of the weight to the corresponding second node. Exemplarily, the control device can send the corresponding weight to each second node individually, or it can integrate the newly allocated weights with other messages and send them together. Other messages include communication information; for a description of communication information, please refer to the relevant description below, which will not be repeated here.

[0123] In one possible scenario, the control device can also indirectly send newly assigned weights to the second node based on a weight allocation strategy. Since a third node exists that has been assigned the same weight as the first node, and the third node is configured in the same computing cluster as the second node, communication efficiency within the computing cluster is higher than communication efficiency outside the computing cluster. Therefore, the time it takes for the third node to send newly assigned weights to the second node is shorter than the time it takes for the control device to send newly assigned weights to the second node. The control device can delegate the weight sending task to the third node, sending weight allocation instructions based on the weight allocation strategy. The third node can then divide the first weight based on the weight allocation instructions to obtain the weights corresponding to each second node, and then send the corresponding weights to each second node. Referring to Figure 8(b), the W8 computing card, acting as the third node in the DP#2 domain, will send a portion of its local first weights to the seven computing cards in the DP#1 and PP#1 domains.

[0124] In one possible scenario, the second node may perform model calculations based not only on a portion of the weights in the first weights but also on the second weights. Continuing with Figure 8(b) as an example, for the first computation card located in the DP#1 and PP#1 domains, the assigned W1' weights include the W1 weights (corresponding to the second weights in the above embodiment) and a portion of the W8 weights (corresponding to the weights in the first weights in the above embodiment). When the weights calculated by the second node include both the second and first weights, the weights calculated by the second node can be referred to as the third weights.

[0125] Figures 10 and 11 are schematic diagrams of weight distribution within a TP domain provided in an embodiment of this application. Figure 10 shows the weights assigned to each computing node in the first TP domain when the first node is functioning correctly. In Figure 10, the first node that has failed is D7. The model weights of the first model in the first TP domain include weight A and weight B. The first weights assigned to the first node include A7 and B7. The weights assigned to other computing nodes are shown in Figure 10 and will not be repeated here.

[0126] Referring to Figure 11, if D7 fails, the first weight assigned to D7 will be distributed to D0-D6, which will act as the second nodes. In this case, the weight assigned to the second nodes includes both the second weight and the weight from the first weight. Taking D0 in Figure 11 as an example, the second weight includes A0 and B0, and the weight from the first weight assigned to D0 is A. 7(0) and B 7(0) As shown in Figures 10 and 11, the model weights after the fault (A' and B' in Figure 11) and the model weights before the fault (A and B in Figure 10) show that the positions of some weights have changed.

[0127] Because the weight calculated by the second node is adjusted from the second weight to the third weight, the interaction between the second node and other nodes will also change. For example, as shown in Figure 12, the point-to-point communication relationship changes to a multi-point-to-point or point-to-multi-point communication relationship. Next, using Figure 12 as an example, we will illustrate the changes in the interaction between the second node and other nodes before and after the fault. Figure 12(a) shows communication within the same TP domain, Figure 12(b) shows communication between different PP domains, and Figure 12(c) shows communication between different DP domains. The fault card in Figure 12 corresponds to the first node in the above embodiment, and the proxy card corresponds to the second node in the above embodiment.

[0128] In Figure 12(a), the first TP domain includes four computing cards. Before a faulty card malfunctions, after calculating the input vector, each of the four computing cards sends its calculation result to the first computing card. The first computing card then merges the three received calculation results with its locally calculated result to obtain the complete calculation result of the input vector based on the model weights. Afterward, the first computing card sends the complete calculation result to the other three computing cards respectively.

[0129] After a faulty card malfunctions, the three proxy cards in the first DP domain calculate the input vector based on the third weight. After obtaining the calculation result, they first send a partial result 1, obtained based on a portion of the first weight, to the first calculation card. The first proxy card then merges the three partial results 1 to obtain the calculation result based on the first weight. This calculation result reflects the partial result 2 obtained by the faulty card based on the first weight when the faulty card was not faulty. Afterward, the remaining two proxy cards send a partial result 3, obtained based on the second weight, to the first calculation card. The first proxy card then merges the two received partial results 3, the locally calculated partial result 3, and the merged partial result 2 to obtain the complete calculation result. Referring to Figure 12(a), it can be seen that for proxy cards in the same TP domain, the number of communications changes before and after the fault, and the interaction objects also change. For example, for the first calculation card, the interaction objects change from the other three calculation cards to the second and third calculation cards.

[0130] In Figure 12(b), the computing cards in different PP domains also perform forward and backward communication. In the forward phase, each computing card in PP#1 domain transmits activation values ​​to each computing card in the downstream PP#2 domain. In the backward phase, each computing card in PP#2 domain transmits gradient values ​​to each computing card in PP#1 domain. The activation values ​​transmitted by each computing card are the same, and the gradient values ​​received by each computing card are also the same. In Figure 12(b), a faulty card in PP#1 domain malfunctions and cannot interact with the fourth computing card in PP#2 domain.

[0131] Optionally, in addition to canceling communication between the faulty card and the normal card as shown in Figure 12(b), the interaction between different PP domains will also include increased interaction between the proxy card and other computing cards. Figure 13 is a schematic diagram of cross-PP domain interaction provided by an embodiment of this application. Since the faulty card cannot send the activation value to the corresponding computing card in the downstream PP domain, the normal card corresponding to the faulty card in the downstream PP domain cannot receive the activation value. The computing cards in the downstream PP domain in Figure 13 will also add one AllGather communication on the basis of maintaining the original point-to-point (P2P) communication. D0-D6 in the downstream PP domain will send the activation value to D7 in the downstream PP domain so that D7 can obtain the activation value. In the reverse communication, since the faulty card does not need to receive the gradient value, no additional communication interaction is required.

[0132] In Figure 12(c), before the faulty card malfunctions, the faulty card located in the DP#1 domain interacts with the computation card located in the DP#2 domain, and the data exchanged is, for example, the gradient corresponding to the first weight. The other three proxy computation cards do not involve interaction with the fourth computation card in the DP#2 domain. After the faulty card malfunctions, the three proxy computation cards will replace the faulty card in interacting with the fourth computation card in the DP#2 domain.

[0133] In cases where the interaction status of the second node changes, the control device, after determining the weight allocation strategy, can adjust the communication information corresponding to the second node based on the weights allocated to the second node in the first weight allocation. The communication information indicates at least one of the following: the data being interacted, the nodes interacting, the interaction time, or the interaction sequence. In some cases, the communication information may be referred to as a communication chain. Optionally, the control device determines at least one communication strategy based on the correspondence between the second node in the weight allocation strategy and a portion of the weights in the first weight allocation; determines the calculation status of each communication strategy within the at least one communication strategy, where the calculation status indicates at least one of the calculation accuracy of the first TP domain or the interaction duration; selects a communication strategy based on the calculation status; and adjusts the communication information corresponding to the second node based on the selected communication strategy. In some cases, the communication strategies determined by the control device include, but are not limited to, the following two:

[0134] Communication Strategy 1: The second node adjusts the third weight based on the concept permutation matrix; the adjusted third weight is used to perform model calculation to obtain the first calculation result; multiple second nodes interact with each other to obtain the complete calculation result of the model weight.

[0135] In one possible scenario, the concept permutation matrix P obtained by the second node is used to adjust at least one dimension in the columns or rows of the third weight. Wherein, PP T =I, where I is the identity matrix. See Formula 1 for the process of left-multiplying P to adjust the rows.

[0136] In formula 1, Indicator Concept Permutation Matrix P, Indicates the weight matrix before adjustment. Indicates the weight matrix after row adjustment.

[0137] Optionally, multiply P on the left. T For the process of adjusting the columns, please refer to Formula 2.

[0138] In formula 2, Indicates the weight matrix before adjustment. Instruction P T , Indicates the weight matrix after adjusting the columns.

[0139] The third weight of the second node is then denoted as A′. i ×B′ i For example, let's illustrate the concept permutation matrix corresponding to the second node. Where A′ i =[A i A k(i) ], B′ i =[B i B k(i) Let i be the identifier of the second node and k be the identifier of the first node. Taking the second node as D0 shown in Figure 11 as an example, then i = 0 and k = 7. For any second node, the concept permutation matrix P of the second node can be found in Formula 3, and the transpose of the concept permutation matrix can be found in Formula 4.

[0140] In Formulas 3 and 4, i indicates the second node, n indicates the number of weighted elements to be arranged forward, m indicates the number of weighted elements whose positions remain unchanged, and l indicates the number of weighted elements to be arranged backward. In Formula 3, P is used to... i n i The column is moved forward, and the remaining column is left. i The columns are arranged sequentially, and in formula 4, n is... i Line ahead, remaining l i Rows are arranged sequentially. If the third weight is adjusted using a concept substitution matrix for each second node, the weight calculations before and after the fault can be mathematically equivalent.

[0141] The model weights reassigned after the fault are A′=[A′0,A′1,…A′ k-1 ], B′=[B′0, B′1,…B′ k-1 For example, the model weights before the fault are A = [A0, A1, ... A1]. k-1 ], B = [B0, B1, ... Bk-1 Since A′=AP0P1…P k-1 , therefore,

[0142] Since the model weights before and after the fault are computationally equivalent, after each second node calculates the input vector using the adjusted third weight, the first calculation results of each second node can be fused. Based on the fusion result, the complete calculated weights of the model can be obtained. The process of exchanging the first calculation results between multiple second nodes is similar to the AllReduce communication process in Strategy 2, and can be found in the relevant descriptions, which will not be repeated here.

[0143] Communication Strategy 2: Each second node performs model calculations based on the weights in the first weights it is assigned to, and obtains a second calculation result; each second node interacts with the second calculation results to merge them and obtain a third calculation result corresponding to the first weights; each second node performs model calculations based on the second weights, and obtains a fourth calculation result; each second node interacts with the fourth calculation results to merge them and obtain a complete calculation result corresponding to the model weights.

[0144] Continuing with the example of the seven computing cards shown in Figure 11 as the second node, each computing card first calculates the weight in the first weight calculation task. After the calculation, an AllReduce communication is performed among the seven computing cards to restore the third calculation result of the faulty card. Then, the seven computing cards perform the original second weight calculation task and perform AllReduce communication between the fourth and third calculation results.

[0145] Optionally, AllReduce communication between compute cards can involve multiple compute cards sending computation results to the same compute card. For example, D1-D6 send the second computation result to D0, where D0 merges the various second computation results to obtain the third computation result. AllReduce communication can also involve two compute cards exchanging computation results and then interacting with other compute cards based on these results. For example, D6 sends the second computation result to D5, which is then merged to obtain second computation result 1. D5 then sends second computation result 1 to D4, where D4 performs the merging.

[0146] After determining each communication strategy, the control device can compare the interaction duration corresponding to each communication strategy based on the bandwidth and interaction strategy between each computing node in the node communication relationship. For example, in communication strategy one of the above embodiments, only one AllReduce communication is required between the second nodes, while in communication strategy two, two AllReduce communications are required between the second nodes. Therefore, the interaction duration of communication strategy one is shorter than that of communication strategy two.

[0147] In some cases, the interaction duration is determined not only based on the interaction between the second nodes within the first TP domain, but also based on the interaction between nodes outside the first TP domain. Figure 14 is a schematic diagram of node interaction between DP domains provided in an embodiment of this application. Figure 14(a) corresponds to communication strategy one. The DP domain where the faulty card is located in Figure 14 is referred to as the faulty DP domain below. The normal card in the normal DP domain that is at the same position as the faulty card will conduct the first AllReduce communication with the normal card in the faulty TP domain. The communication content is that the normal card calculates the weight gradient value corresponding to a portion of the weight in the first weight. Then, the other seven proxy cards in the faulty DP domain and the normal cards corresponding to the seven proxy cards in the normal TP domain will conduct the second AllReduce communication. The communication content is that each proxy card calculates the weight gradient value corresponding to the second weight.

[0148] Figure 14(b) corresponds to communication strategy two. In the normal DP domain, the normal card at the same position as the faulty card will calculate the weight gradient value corresponding to part of the weight in the first weight and transmit it to other cards in the same TP domain through scatter communication. Then, the seven proxy cards in the faulty DP domain and the normal cards corresponding to the seven proxy cards in the normal DP domain will perform AllReduce communication.

[0149] Referring to Figure 14, although both communication strategy 1 and communication strategy 2 can guarantee strict order preservation before and after scaling down, communication strategy 1 only requires two AllReduce communications between DP domains; while communication strategy 2 adds one scatter operation within the TP domain. During AllReduce communications between DP domains, the average communication load per compute card increases by 1 / 7, and the communication time is longer. Furthermore, if the load on normal DP is increased by adjusting the sample splitting method, the risk of OOM (Out of Memory) increases. Based on this, the control device can choose communication strategy 1 from communication strategy 1 and communication strategy 2.

[0150] Furthermore, since different interaction strategies may indicate different interaction sequences, and changes in the interaction sequence will also lead to changes in the calculation sequence in the first model, the calculation results obtained based on different calculation sequences will also differ. Therefore, the control device can also compare the accuracy of the calculation results corresponding to different interaction strategies and select the communication strategy with the highest accuracy. Continuing with communication strategy one and communication strategy two as examples, in communication strategy one, the change in the sequence calculation order in the model weights may affect the calculation accuracy. However, in communication strategy two, since the calculation order in the model weights does not change, the calculation accuracy is not affected. Based on this, the processing device can select communication strategy two.

[0151] In some cases, the control device can also select a communication strategy from the communication strategies based on accuracy and interaction duration. For example, multiple communication strategies that meet the system's accuracy requirements can be determined based on the accuracy of each strategy, and the communication strategy with the shortest interaction duration can be selected from these strategies. Regardless of the method used by the control device to select the communication strategy, the communication information of the second node can be adjusted based on the selected communication strategy.

[0152] In one possible scenario, if the first TP domain is located within the first DP domain of multiple DP domains, under fault-free conditions, the computational and communication overhead of each DP domain is essentially the same. Distributing the same amount of input data to each DP domain results in roughly the same processing time for each DP domain, and after each round of computation, the DP domains do not need to wait for each other, thus achieving load balancing. However, if the first node in the first TP domain fails, the number of computational nodes in the first DP domain containing the first TP domain will be less than the number of computational nodes in other DP domains, resulting in a loss of computational power for the first DP domain.

[0153] Furthermore, the second node in the first TP domain incurs additional communication overhead, resulting in longer communication time within the first TP domain. Consequently, the model computation time in the first DP domain will be longer than that in other DP domains. If the same amount of input data is distributed to each DP domain, the time required by the first DP domain will be longer than that of the normal DP domains. This leads to a long waiting time after the normal DP domains finish their computation tasks, resulting in wasted computing power and load imbalance. Therefore, the control device can also re-segment the multiple input data of the first model to determine the input data processed by each DP domain. The segmentation methods include, but are not limited to, the following two.

[0154] Segmentation Method 1: Based on the number of nodes in each DP domain, determine the input data to be processed in the first DP domain from the multiple input data of the first model.

[0155] Optionally, the number of nodes in the DP domain reflects the computation time within that domain; a DP domain with more nodes has a shorter computation time. Therefore, the control device can redetermine the input data allocated to each DP domain based on the principle that DP domains with fewer nodes require less input data for computation. The time difference between different DP domains processing different input data is less than a time threshold, which can be any duration set empirically.

[0156] The second segmentation method involves determining the input data for processing in the first DP domain from multiple input data of the first model based on the communication relationships between nodes in each DP domain.

[0157] The node communication relationships within the DP domain reflect the overall communication duration. A description of these relationships can be found in the section on node communication relationships in Method Two, and will not be repeated here. After determining the overall communication duration in each DP domain based on the node communication relationships, the control device can redetermine the input data allocated to each DP domain based on the principle that DP domains with longer overall communication durations require less input data. The time difference between different DP domains processing different input data will also be less than a time threshold.

[0158] Optionally, the control device can choose either segmentation method one or segmentation method two to redistribute the input data, or it can combine segmentation method one and segmentation method two. For example, based on the node communication relationships and number of nodes in each DP domain, and according to the computation time and overall communication time of different DP domains, the amount of data allocated to each DP domain from multiple input data can be adjusted. The adjustment principle is that the overall computation and communication time of each DP domain after adjustment is basically equal, for example, the time difference is less than a time threshold. Based on the adjusted amount of data, multiple input data are allocated to determine the input data processed by each DP domain. In one possible case, the control device can also refer to other information, such as operator fusion strategies. Operator fusion strategies reflect the operator merging situation in the DP domain. By merging multiple operators into a composite operator, executing the composite operator at once can reduce memory access, data transfer, and computational overhead, thereby achieving the effect of accelerating the computation speed.

[0159] In one possible scenario, the control device can use a model to execute the above-mentioned segmentation process. Referring to Figure 15, the control device inputs the number of nodes, node communication relationships, and operator fusion strategies in each DP domain into the load assessment model. The load assessment model is used to determine the time required for each DP domain to process different amounts of input data. Based on the correspondence between the data volume and the time output by the load assessment model, and according to the principle of minimizing the difference in the completion time of one round of computation tasks in each DP domain, the multiple input data are sampled and adaptively optimized to redetermine the input data to be processed in each DP domain.

[0160] For example, the control device can resume model computation of the computing cluster after re-determining the input data, communication information, and weights calculated by the second node. After receiving the input vector obtained by processing the input data, the second node calculates the input vector using the newly assigned third weights, obtains the calculation result, interacts with other nodes based on the communication information to determine the output result of the first TP domain through interaction, and sends the output result to the next layer.

[0161] In one possible implementation, the failure of the first node in the computing cluster can be recovered. In this case, the control device can send a first weight to the first node to utilize the first node to perform model calculations based on the first weight. This process can be referred to as the expansion of the first TP domain. The control device can directly send the first weight to the first node. The method by which the control device obtains the first weight is described in S601 and will not be repeated here. The control device can also indirectly send the first weight to the first node. For example, in the case of the first node's recovery, a first sending command is sent to the third node, instructing the third node to send the first weight to the first node. Referring to Figure 16(b), if the first node is a W8 computing card in the DP1 domain and the third node is a W8 computing card in the DP2 domain, the third node in the DP2 domain will send the first weight to the computing node in the DP1 domain.

[0162] Optionally, since the second nodes in the first TP domain are configured with the weights allocated in the first weight, when the first TP domain includes multiple second nodes, the first weight of the first node can be determined using the partial weights allocated to the multiple second nodes. For example, the partial weights obtained from the split can be concatenated together to obtain the first weight. In this case, the control device can send a second sending command to each second node, which instructs the second node to send the weights allocated in the first weight to the first node. After receiving the weights sent by each second node, the first node combines the received weights into the first weight. Referring to Figure 16(a), when the first node is the W8 computing card in the DP1 domain, and the second nodes are the seven computing cards W1'-W7' in the DP1 domain, each second node will send a partial weight (the part marked by the diagonal line in Figure 16) of the allocated first weight to the first node so that the first node can re-acquire the first weight to be calculated.

[0163] Furthermore, the first weight in the above embodiments is intended to describe the weight assigned to the first node. The first weight can be a fixed weight, for example, if the first model is a model that has been trained and is in use, and the model parameters of the first model do not change with model calculation. In some cases, the first weight can also be a dynamic weight, for example, if the first model is a model that is being trained, and since the model weights of the first model are adjusted based on the output data after one round of training. Therefore, if the model weights of the first model have been adjusted based on the output data during the time period between the first node's failure and recovery, the first weight assigned to the first node after recovery will be different from the first weight when the first node failed.

[0164] For example, since the first node has resumed calculating the first weight, the second node no longer needs to undertake the task of calculating the first weight. In this case, the control device can send a weight recovery instruction to the second node to prompt the second node to use only the second weight for subsequent model calculations. The weight recovery instruction can be the second sending instruction in the above embodiment. Based on the received second sending instruction, the second node will not only send the weights from the first weight assigned to the first node, but also delete the weights from the first weight from its local storage. When performing subsequent model calculations, only the second weight will be used for calculation.

[0165] In some cases, the second node may also be updated during the repair of the first node. For example, when the repair personnel perform hardware replacement on the first node, they replace multiple computing nodes in the first TP domain as a whole. In this case, the second node after replacement is not configured with the second weight. The control device can still send the second weight to the second node so that the second node can use the second weight for model calculation. The process of sending the second weight is similar to that of sending the first weight, and it can be sent by the fifth node corresponding to the second node. The weight assigned to the fifth node is the same as that of the second weight. The description of the fifth node is similar to that of the third node. The process of sending the second weight to the second node using the fifth node is shown in Figure 16(b). For a detailed description, please refer to the relevant description of sending the first weight in the above embodiments, which will not be repeated here.

[0166] In cases where the control device not only assigns the first weight to the second node but also adjusts communication information, the control device will update the communication information based on the recovery of the first node. Furthermore, in cases where the control device reallocates input data based on a first node failure, the control device will also update the input data allocation based on the first node's recovery. If the control device saved the initial communication information and initial allocation strategy before adjusting the communication information and input data allocation strategy during the first node failure, the control device can read the saved communication information and allocation strategy, adjusting the currently adjusted communication information to match the initial communication information, and adjusting the currently adjusted allocation strategy to match the initial communication information.

[0167] Optionally, the control device can also determine the allocation strategy online, for example, by re-establishing communication information based on the weights assigned to the first and second nodes. After the first node recovers, the allocation strategy is re-determined based on the number of nodes and their communication relationships within each DP domain. The process of determining the communication information online is similar to the process of adjusting the communication information in the above embodiments, and the process of determining the allocation strategy online is similar to the process of adjusting the allocation strategy shown in the above embodiments; please refer to the relevant descriptions, which will not be repeated here.

[0168] The above example is intended to illustrate the process by which the control device allocates the first weight to the second node in the first TP domain to perform model calculations when the first node fails, rather than to limit the allocation method of the first weight. In some cases, the control device can allocate the model weights of the first model in the first TP domain as a whole. For example, it can directly distribute the model weights equally to the second node in the first TP domain. In this case, the weights used by the second node to perform subsequent model calculations may include both the weights in the first weight and the second weight, or it may only include the weights in the first weight, or it may include other weights in the model weight that are different from the first and second weights.

[0169] This application does not limit the number of second nodes and the number of first nodes. The example here takes the first node as a computing node. However, in actual applications, there may be multiple first nodes that fail in the computing cluster. Multiple first nodes may be located in the same TP domain or in different TP domains.

[0170] In summary, the fault handling method provided in this application allows for the allocation of the first weight corresponding to the first node to a second node within the same TP domain after the first node fails. The second node then performs model calculations in place of the first node. The suspended computing node in the computing cluster is the failed first node, while other computing nodes can operate normally. This results in high utilization of computing nodes and automatic and rapid restart of the first model's calculation services. Furthermore, allocating the first weight within the same TP domain results in a small allocation range, low complexity, and high efficiency. Since the second node interacts with other nodes based on the second weight even when the first node is functioning correctly, the communication status of the second node changes little, and the adjustment time for communication information is short. After the first node fails, the input data processed by the first model is redistributed based on the weight allocation strategy to achieve load balancing across DP domains, minimizing computing power loss.

[0171] In one possible implementation, this application embodiment also provides a computing cluster, which includes a first TP domain, and the first TP domain includes a first node and a second node. The structural schematic diagram of the computing cluster can be seen in Figures 2-5, where one computing card in Figure 5 corresponds to one computing node.

[0172] For example, a first node is used to perform model calculation of a first model based on a first weight, where the first weight is the weight assigned to the first node from the model weights of the first model in the first TP domain. A second node is used to perform model calculation of the first model based on a second weight, where the second weight is the weight assigned to the second node from the model weights of the first model in the first TP domain, and the first weight and the second weight are different.

[0173] Regarding the first weight obtained by the first node and the second weight obtained by the second node, please refer to the description of the control device allocating model weights to the computing nodes in the embodiment shown in Figure 6. Regarding the description of the first node and the second node performing model calculations, please refer to the description of the computing nodes performing model calculations based on the allocated weights in the embodiment shown in Figure 6. It will not be repeated here.

[0174] In one possible scenario, the first node may fail. In this case, the second node can perform model calculations of the first model based on a third weight, where the third weight includes the weights allocated to the second node from the first weight and the second weight. Optionally, the weights allocated to the second node from the first weight are sent to the second node by the control device using the fault handling method shown in Figure 6. The weights allocated to the second node from the first weight can be obtained based on at least one of the following: the number of non-faulty nodes in the first TP domain or the node communication relationships within the first TP domain. This can be understood as the third weight including the weights allocated to the second node from the first weight based on the number of non-faulty nodes in the first TP domain, and / or the weights allocated to the second node from the first weight based on the node communication relationships within the first TP domain. Detailed procedures for allocating the first weight can be found in determination methods one and two of S602 in the embodiment shown in Figure 6, and will not be repeated here.

[0175] Regardless of how the first weight is assigned to the second node, the second node can receive the weight allocated to it from the first weight sent by other devices. In the embodiment shown in Figure 6, where the control device directly sends weights to the second node, the second node can receive the weight allocated to it from the first weight sent by the control device. In the embodiment shown in Figure 6, where the control device indirectly sends weights to the second node, for example, if the computing cluster also includes a third node configured in the second TP domain and assigned the same weight as the first node, then the third node can receive the weight allocation instruction sent by the control device and, based on the instruction, send the weight allocated to it from the first weight. The second node can then receive the first weight sent by the third node and perform model calculations for the first model based on the first weight.

[0176] In one possible scenario, the computing cluster may also include a fourth node. After the second node obtains its computation result, it may also calculate the result based on the communication information and the interaction model of the fourth node. The communication information corresponding to the second node is adjusted based on the weights assigned to the second node in the first weighting. The communication information indicates at least one of the following: the data being interacted, the nodes interacting, the interaction time, or the interaction order. The fourth node can be any computing node in the computing cluster that is different from any second node. The fourth node can be configured with any second node within the first TP domain. For example, the fourth node is another second node shown in Figure 12(a). The fourth node can also be configured in a TP domain different from the first TP domain. For example, the fourth node is a computing node configured in the downstream TP domain as shown in Figure 13, or a computing node configured in the normal DP domain as shown in Figure 14.

[0177] For example, the model computation is also performed based on the input data of the first model. In the case where the computing cluster includes multiple DP domains, and the first TP domain is located in the first DP domain among the multiple DP domains, the input data processed by the first DP domain is the data determined based on the number of nodes in each DP domain from among the multiple input data of the first model, and / or, the input data processed by the first DP domain is the data determined based on the node communication relationships within each DP domain from among the multiple input data of the first model. The process for determining the input data processed by the first DP domain can be found in the segmentation method one and segmentation method two in the embodiments shown in Figure 6 above, and will not be repeated here.

[0178] Optionally, if the first node in the computing cluster recovers from a failure, the first node can perform model calculations for the first model based on the first weight. The first node can receive the first weight sent by the control device or by a third node. For example, if the first node recovers from a failure, the control device will receive a first transmission command from the control device and send the first weight to the first node based on the first transmission command.

[0179] In some cases, the computing cluster also includes multiple DP domains, and / or multiple pipelined parallel PP domains, where computing nodes in different DP domains perform model computations based on different input data, and computing nodes in different PP domains perform model computations based on model weights located at different layers in the first model.

[0180] In summary, the computing cluster provided in this application embodiment allows a second node to perform model calculations in place of the first node after the first node fails. The suspended computing node in the cluster is the failed first node, while other computing nodes can operate normally, resulting in high computing power utilization. Since the second node interacts with other nodes based on a second weight even when the first node is functioning correctly, the communication status of the second node changes little, and the adjustment time for communication information is short. After the first node fails, the input data of each DP domain in the computing cluster is redistributed based on the weight allocation strategy, resulting in load balancing among the DP domains and minimal computing power loss.

[0181] The above describes the fault handling method of the embodiments of this application. Corresponding to the above method, the embodiments of this application also provide a fault handling device. Figure 17 is a schematic diagram of the structure of a fault handling device provided in an embodiment of this application. Based on the following multiple modules shown in Figure 17, the fault handling device shown in Figure 17 can perform all or part of the operations shown in Figure 6 above. It should be understood that the device may include more additional modules than the modules shown or omit some of the modules shown, and the embodiments of this application do not limit this. As shown in Figure 17, the device includes:

[0182] Transceiver module 1701 is used to perform the receiving and / or sending related operations in the method shown in Figure 6;

[0183] Processing module 1702 is used to perform operations other than the receiving and / or sending related operations in the method shown in FIG6.

[0184] In one possible implementation, processing module 1702 is used to obtain the first weight corresponding to the first node in the event of a first node failure. The first node is a computing node configured in the first TP domain, and the first weight is the weight assigned to the first node from the model weights of the first model in the first TP domain. Each computing node in the first TP domain is used to perform model calculation of the first model based on the different weights assigned to it. Processing module 1702 is also used to determine the weight allocation strategy of the first weight, which indicates the weight assigned to the second node from the first weight. The second node is a computing node in the first TP domain that is different from the first node. Transceiver module 1701 is used to send the weight assigned to the second node from the first weight to the second node based on the weight allocation strategy, or to send a weight allocation instruction to the third node based on the weight allocation strategy. The weight allocation instruction instructs the third node to send the weight assigned to the second node from the first weight to the second node. The third node is configured in the second TP domain, and the weight assigned to it is the same as the first weight assigned to the first node.

[0185] In one possible implementation, the processing module 1702 is further configured to determine a second node from multiple computing nodes in the first TP domain if the memory overflow prediction result of the first TP domain indicates that there is no memory overflow in the first TP domain.

[0186] In one possible implementation, processing module 1702 is used to obtain the first weight from the checkpoint file; or, to copy the first weight from the third node.

[0187] In one possible implementation, the processing module 1702 is used to determine a weight allocation strategy based on the number of non-faulty nodes in the first TP domain, and / or, based on the node communication relationships within the first TP domain.

[0188] In one possible implementation, the processing module 1702 is further configured to adjust the communication information corresponding to the second node based on the weight assigned to the second node in the first weight, wherein the communication information indicates at least one of the interactive data, the interactive node, the interactive time, or the interactive order.

[0189] In one possible implementation, the model computation is also performed based on the input data of the first model. The processing module 1702 is further configured to determine the input data to be processed in the first DP domain from the multiple input data of the first model, based on the number of nodes in each DP domain, and / or to determine the input data to be processed in the first DP domain from the multiple input data of the first model, based on the node communication relationship within each DP domain.

[0190] In one possible implementation, the transceiver module 1701 is further configured to send a first weight to the first node in the event that the first node has recovered from a failure; or, in the event that the first node has recovered from a failure, send a first transmission instruction to the third node, the first transmission instruction instructing the third node to send the first weight to the first node.

[0191] In one possible implementation, the first TP domain is located in any DP domain, and / or the first TP domain is located in any PP domain. Computational nodes in different DP domains perform model computation based on different input data, and computational nodes in different PP domains perform model computation based on model weights located in different layers of the first model.

[0192] After the first node fails, the first weight, which was originally calculated by the first node, can be assigned to the second node. This allows the second node to take over the weight calculations previously performed by the first node. Subsequently, only the model calculation task of the failed first node is paused, while the model calculation task of the first TP domain continues normally, resulting in high utilization of the computing nodes. The first weight is assigned only within the same TP domain (the first TP domain), resulting in a small allocation range and high efficiency. This application does not limit the method of sending the assigned weight to the second node; it can be sent directly based on a weight allocation strategy or indirectly using a third node based on a weight allocation instruction, offering high flexibility.

[0193] It should be understood that the device provided in Figure 17 is only illustrated by the division of the above-described functional modules when implementing its functions. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, which will not be repeated here. In addition, the fault handling device provided in Figure 17 can be a chip or a device; for example, the fault handling device can be the following computing device.

[0194] Referring to Figure 18, Figure 18 shows a schematic diagram of the structure of a computing device 1800 provided in an exemplary embodiment of this application. The computing device 1800 shown in Figure 18 can act as a control device to perform the operations involved in the fault handling method shown in Figure 6 above, or the computing device 1800 shown in Figure 18 can include at least one computing node among the first node or the second node in a first TP domain. The computing device 1800 is, for example, a switch, a router, etc., and can be implemented by a general bus architecture. The computing device 1800 includes multiple threads.

[0195] As shown in Figure 18, the computing device 1800 includes at least one processor 1801, a memory 1803, and at least one communication interface 1804.

[0196] In Figure 18, the XPU in processor 1801 indicates different types of processing units. An XPU can be, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits used to implement the solutions of this application. For example, processor 1801 includes an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A PLD can be, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof. It can implement or execute various logic blocks, modules, and circuits described in conjunction with the disclosure of embodiments of this application. The processor can also be a combination of components that implement computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. For example, when the computing device 1800 is a control device, the XPU can be a CPU; when the computing device 1800 includes computing nodes, the XPU can be a GPU, NPU, etc., as shown in the above embodiments.

[0197] Optionally, the computing device 1800 also includes a bus. The bus is used to transfer information between the components of the computing device 1800. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is used in Figure 18, but this does not indicate that there is only one bus or one type of bus.

[0198] Memory 1803 may be, for example, read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; electrically erasable programmable read-only memory (EEPROM); compact disc read-only memory (CD-ROM) or other optical disc storage; optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.); magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 1803 may exist independently and be connected to processor 1801 via a bus. Memory 1803 may also be integrated with processor 1801.

[0199] Communication interface 1804 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), or Wireless Local Area Network (WLAN). Communication interface 1804 may include wired and wireless communication interfaces. Specifically, communication interface 1804 may be an Ethernet interface, a Fast Ethernet (FE) interface, a Gigabit Ethernet (GE) interface, an Asynchronous Transfer Mode (ATM) interface, a WLAN interface, a cellular network communication interface, or a combination thereof. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. In this embodiment, communication interface 1804 can be used by computing device 1800 to communicate with other devices.

[0200] In a specific implementation, as one embodiment, processor 1801 may include one or more XPUs, such as XPU0 and XPU1 shown in FIG18. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0201] In a specific implementation, as one embodiment, computing device 1800 may include multiple processors, such as processor 1801 and processor 1805 as shown in FIG18. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0202] In a specific implementation, as one embodiment, the computing device 1800 may further include an output device and an input device. The output device communicates with the processor 1801 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 1801 and can receive user input in various ways. For example, the input device may be a mouse, a keyboard, a touchscreen device, or a sensing device, etc.

[0203] In some embodiments, memory 1803 stores program code 1810 for executing the solution of this application, and processor 1801 can execute the program code 1810 stored in memory 1803. That is, computing device 1800 can implement the fault handling method provided in the method embodiment through processor 1801 and program code 1810 in memory 1803. Program code 1810 may include one or more software modules. Optionally, processor 1801 itself may also store program code or instructions for executing the solution of this application.

[0204] In a specific embodiment, the computing device 1800 of this application embodiment may correspond to the computing device in the above-described method embodiments.

[0205] In the fault handling method shown in Figure 6, each step is completed through the integrated logic circuitry of the hardware or the instructions in the software form of the processor in the computing device 1800. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by the hardware processor, or executed by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. Since this storage medium is located in memory, the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method; to avoid repetition, these will not be described in detail here.

[0206] Referring to Figure 19, which illustrates a schematic diagram of a computing device 1900 provided in another exemplary embodiment of this application, the computing device 1900 shown in Figure 19 can act as a control device to perform all or part of the operations involved in the fault handling method shown in Figure 6 above. Alternatively, the computing device 1900 shown in Figure 19 may include at least one computing node among the first node or the second node in the first TP domain. The computing device 1900 is, for example, a switch, a router, etc., and can be implemented using a general bus architecture. The computing device 1900 includes multiple threads.

[0207] As shown in Figure 19, the computing device 1900 includes a main control board 1910 and an interface board 1930.

[0208] The main control board, also known as the main processing unit (MPU) or route processor card, 1910 is used to control and manage the various components in the computing device 1900, including routing calculation, device management, device maintenance, and protocol processing functions. The main control board 1910 includes a central processing unit 1911 and a memory 1912.

[0209] Interface board 1930, also known as line processing unit (LPU), line card, or service board, provides various service interfaces and enables packet forwarding. Service interfaces include, but are not limited to, Ethernet interfaces, POS (Packet over SONET / SDH) interfaces, etc., with Ethernet interfaces including, for example, Flexible Ethernet Clients (FlexE Clients). Interface board 1930 includes: a central processing unit 1931, a network processor 1932, a forwarding table entry memory 1934, and a physical interface card (PIC) 1933.

[0210] The central processing unit 1931 on the interface board 1930 is used to control and manage the interface board 1930 and communicate with the central processing unit 1911 on the main control board 1910.

[0211] Network processor 1932 is used to implement packet forwarding processing. Network processor 1932 can be in the form of a forwarding chip. The forwarding chip can be a network processor (NP). In some embodiments, the forwarding chip can be implemented using an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). Specifically, network processor 1932 forwards received packets based on the forwarding table stored in forwarding table entry memory 1934. If the destination address of the packet is an address of computing device 1900, the packet is sent to the CPU (such as central processing unit 1931) for processing; if the destination address of the packet is not an address of computing device 1900, the next hop and outgoing interface corresponding to the destination address are looked up in the forwarding table according to the destination address, and the packet is forwarded to the outgoing interface corresponding to the destination address. Uplink packet processing may include: packet inbound interface processing, forwarding table lookup; downlink packet processing may include: forwarding table lookup, etc. In some embodiments, the central processing unit can also perform the functions of the forwarding chip, such as implementing software forwarding based on a general-purpose CPU, thus eliminating the need for a forwarding chip on the interface board.

[0212] The physical interface card 1933 is used to implement physical layer interfacing functions. Raw traffic enters the interface board 1930 through this card, and processed packets are sent out from the physical interface card 1933. The physical interface card 1933, also known as a daughter card, can be installed on the interface board 1930. It is responsible for converting photoelectric signals into packets, performing validity checks on the packets, and forwarding them to the network processor 1932 for processing. In some embodiments, the central processing unit 1931 can also perform the functions of the network processor 1932, such as implementing software forwarding based on a general-purpose CPU, thus eliminating the need for the network processor 1932 in the physical interface card 1933.

[0213] Optionally, the computing device 1900 includes multiple interface boards. For example, the computing device 1900 also includes an interface board 1940, which includes a central processing unit 1941, a network processor 1942, a forwarding table entry memory 1944, and a physical interface card 1943. The functions and implementation methods of the components in the interface board 1940 are the same as or similar to those in the interface board 1930, and will not be described again here.

[0214] Optionally, the computing device 1900 also includes a switching fabric board 1920. The switching fabric board 1920 can also be referred to as a switch fabric unit (SFU). When the computing device 1900 has multiple interface boards, the switching fabric board 1920 is used to complete data exchange between the interface boards. For example, interface boards 1930 and 1940 can communicate via the switching fabric board 1920.

[0215] The main control board 1910 and the interface boards are coupled. For example, the main control board 1910, interface boards 1930 and 1940, and the switching network board 1920 communicate with each other via a system bus connected to the system backplane. In one possible implementation, an inter-process communication (IPC) channel is established between the main control board 1910 and interface boards 1930 and 1940, and communication between the main control board 1910 and interface boards 1930 and 1940 is achieved through the IPC channel.

[0216] Logically, the computing device 1900 includes a control plane and a forwarding plane. The control plane includes a main control board 1910 and a central processing unit 1911, while the forwarding plane includes various components that perform forwarding, such as a forwarding table entry memory 1934, a physical interface card 1933, and a network processor 1932. The control plane performs functions such as router operation, generating forwarding tables, processing signaling and protocol messages, and configuring and maintaining the status of the computing device. The control plane distributes the generated forwarding tables to the forwarding plane. In the forwarding plane, the network processor 1932 looks up and forwards messages received by the physical interface card 1933 based on the forwarding tables distributed by the control plane. The forwarding tables distributed by the control plane can be stored in the forwarding table entry memory 1934. In some embodiments, the control plane and the forwarding plane can be completely separated and not on the same computing device.

[0217] It's worth noting that a computing device may have one or more main control boards, including a primary and a backup main control board. It may also have one or more interface boards; the more powerful the computing device's data processing capabilities, the more interface boards it can provide. Each interface board may also have one or more physical interface cards. A switching network board may or may not exist; multiple boards can share the load and provide redundancy. In a centralized forwarding architecture, the computing device may not need a switching network board, as the interface boards handle the entire system's business data processing. In a distributed forwarding architecture, the computing device can have at least one switching network board, enabling data exchange between multiple interface boards and providing high-capacity data exchange and processing capabilities. Therefore, the data access and processing capabilities of a distributed architecture computing device are greater than those of a centralized architecture computing device. Alternatively, the computing device can also be a single board, without a switching network board. The functions of the interface board and the main control board are integrated on this one board. In this case, the central processing unit (CPU) on the interface board and the CPU on the main control board can be combined into a single CPU to execute the combined functions. This type of computing device has lower data exchange and processing capabilities (e.g., low-end switches or routers). The specific architecture adopted depends on the specific network deployment scenario, and no restrictions are imposed here.

[0218] In a specific embodiment, computing device 1900 corresponds to the fault handling apparatus shown in FIG17 above. In some embodiments, processing module 1702 in the fault handling apparatus shown in FIG17 is equivalent to central processing unit 1911 or network processor 1932 in computing device 1900.

[0219] This application embodiment also provides a communication device, which includes a transceiver, a memory, and a processor. The transceiver, the memory, and the processor communicate with each other via an internal connection path. The memory stores instructions, and the processor executes the instructions stored in the memory to control the transceiver to receive and transmit signals. Furthermore, when the processor executes the instructions stored in the memory, it causes the processor to perform the fault handling method shown in Figure 6.

[0220] It should be understood that the aforementioned processor can be a CPU, or other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting the Advanced Reduced Instruction Set Computing (RISC) machine (ARM) architecture.

[0221] Furthermore, in an alternative embodiment, the memory described above may include read-only memory and random access memory, and provide instructions and data to the processor. The memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0222] The memory can be volatile or non-volatile, or may include both. Non-volatile memory can be ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), EEPROM, or flash memory. Volatile memory can be RAM, used as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0223] This application embodiment also provides a control device connected to a first TP domain. The device includes a processor for loading and executing at least one instruction to enable the control device to implement the fault handling method shown in FIG6. Optionally, the device further includes a memory coupled to the processor for storing at least one instruction.

[0224] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to enable the computer to implement the fault handling method shown in FIG6.

[0225] This application also provides a computer program (product) that, when executed by a computer, causes the processor or computer to perform the corresponding steps and / or processes in the above method embodiments.

[0226] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device equipped with the chip to perform the fault handling method shown in FIG6.

[0227] This application embodiment also provides another chip, including: an input interface, an output interface, a processor, and a memory. The input interface, the output interface, the processor, and the memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the fault handling method shown in FIG6.

[0228] This application embodiment also provides a fault handling system, which includes a control device and a computing cluster. The control device is used to execute the fault handling method shown in FIG6 when the first node in the computing cluster fails.

[0229] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).

[0230] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the first weight involved in this application was obtained with full authorization.

[0231] Those skilled in the art will recognize that the method steps and modules described in conjunction with the embodiments disclosed herein can be implemented in software, hardware, firmware, or any combination thereof. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0232] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0233] When implemented using software, it can be implemented wholly or partially as a computer program product. This computer program product includes one or more computer program instructions. As an example, the methods of this application embodiment can be described in the context of machine-executable instructions, such as program modules that execute on a device on a real or virtual processor of the target. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., which perform specific tasks or implement specific abstract data structures. In various embodiments, the functionality of program modules can be combined or divided among the described program modules. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside on both local and remote storage media.

[0234] Computer program code used to implement the methods of the embodiments of this application may be written in one or more programming languages. This computer program code may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable fault-handling device, such that when executed by the computer or other programmable fault-handling device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0235] In the context of the embodiments of this application, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, etc.

[0236] Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0237] A machine-readable medium can be any tangible medium that contains or stores programs for or relating to an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More detailed examples of machine-readable storage media include electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0238] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0239] In the embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or modules, or they may be electrical, mechanical, or other forms of connection.

[0240] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0241] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0242] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0243] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image. Both the first image and the second image can be images, and in some cases, they can be separate and distinct images.

[0244] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0245] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple second messages refer to two or more second messages. The terms "system" and "network" are often used interchangeably in this document.

[0246] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0247] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0248] It should also be understood that the term “comprising” (also referred to as “includes”, “including”, “comprises” and / or “comprising”) as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0249] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0250] It should be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0251] It should also be understood that the phrases "an embodiment," "an embodiment," and "a possible implementation" used throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment or implementation is included in at least one embodiment of this application. Therefore, the phrases "in an embodiment," "an embodiment," or "a possible implementation" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

Claims

1. A fault handling method, characterized in that, The method includes: In the event of a failure of the first node, the first weight corresponding to the first node is obtained. The first node is a computing node configured in the first tensor parallel TP domain. The first weight is the weight assigned to the first node from the model weights of the first model in the first TP domain. Each computing node in the first TP domain is used to perform model calculation of the first model based on the different weights in the assigned model weights. Determine the weight allocation strategy for the first weight, wherein the weight allocation strategy indicates the weight allocated to the second node in the first weight, and the second node is a computing node in the first TP domain that is different from the first node; Based on the weight allocation strategy, the weight allocated to the second node from the first weight is sent to the second node; or, based on the weight allocation strategy, a weight allocation instruction is sent to the third node, the weight allocation instruction instructing the third node to send the weight allocated to the second node from the first weight to the second node, the third node being configured in the second TP domain, and the weight allocated is the same as the first weight allocated to the first node.

2. The method according to claim 1, characterized in that, Before determining the weight allocation strategy for the first weight, the method further includes: If the memory overflow prediction result of the first TP domain indicates that there is no memory overflow in the first TP domain, the second node is determined from multiple computing nodes in the first TP domain.

3. The method according to claim 1 or 2, characterized in that, Obtaining the first weight corresponding to the first node includes: Obtain the first weight from the checkpoint file; Alternatively, the first weight can be copied from the third node.

4. The method according to any one of claims 1-3, characterized in that, The weight allocation strategy for determining the first weight includes: The weight allocation strategy is determined based on the number of non-faulty nodes in the first TP domain, and / or based on the node communication relationships within the first TP domain.

5. The method according to any one of claims 1-4, characterized in that, After determining the weight allocation strategy for the first weight, the method further includes: The communication information corresponding to the second node is adjusted based on the weight assigned to the second node in the first weight, wherein the communication information indicates at least one of the interactive data, interactive nodes, interactive time, or interactive order.

6. The method according to any one of claims 1-5, characterized in that, The model calculation is also performed based on the input data of the first model. After determining the weight allocation strategy for the first weight, the calculation further includes: When the first TP domain is located in the first DP domain among multiple data parallel DP domains, the input data processed by the first DP domain is determined from the multiple input data of the first model according to the number of nodes in each DP domain, and / or, the input data processed by the first DP domain is determined from the multiple input data of the first model according to the node communication relationship in each DP domain.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: If the first node recovers from a failure, the first weight is sent to the first node; Alternatively, if the first node recovers from a failure, a first transmission instruction is sent to the third node, instructing the third node to send the first weight to the first node.

8. A computing cluster, characterized in that, The computing cluster includes a first tensor parallel TP domain, and the first TP domain includes a first node and a second node; The first node is used to perform model calculation of the first model based on the first weight, wherein the first weight is the weight assigned to the first node in the model weight of the first model in the first TP domain; The second node is used to perform model calculation of the first model based on the second weight, where the second weight is the weight assigned to the second node from the model weight of the first model in the first TP domain, and the first weight and the second weight are different; The second node is also configured to perform model calculation of the first model based on a third weight in the event of a failure of the first node, wherein the third weight includes the weight assigned to the second node from the first weight and the second weight.

9. The cluster according to claim 8, characterized in that, The third weight includes a weight allocated to the second node from the first weight based on the number of non-faulty nodes in the first TP domain; And / or, the third weight includes a weight assigned to the second node from the first weight based on the node communication relationship within the first TP domain.

10. The cluster according to claim 8 or 9, characterized in that, The second node is also configured to receive the weights assigned to the second node from the first weights sent by the control device.

11. The cluster according to claim 8 or 9, characterized in that, The second node is also configured to receive weights sent by a third node in the computing cluster. The third node is configured in the second TP domain and is assigned the same weights as the first weights assigned to the first node. The third node sends the weights assigned to the second node from the first weights to the second node based on the weight allocation instructions sent by the control device.

12. The cluster according to any one of claims 8-11, characterized in that, The computing cluster further includes a fourth node and the second node, which are also used to interact with the fourth node based on communication information to obtain the calculation results of the model. The communication information corresponding to the second node is obtained by adjusting the weight allocated to the second node in the first weight. The communication information indicates at least one of the data to be interacted, the nodes to be interacted, the interaction time, or the interaction order.

13. The cluster according to any one of claims 8-12, characterized in that, The model computation is also performed based on the input data of the first model. In the case that the computing cluster further includes multiple data parallel DP domains and the first TP domain is located in the first DP domain among the multiple DP domains, the input data processed by the first DP domain is the data determined based on the number of nodes in each DP domain among the multiple input data of the first model, and / or, the input data processed by the first DP domain is the data determined based on the node communication relationship in each DP domain among the multiple input data of the first model.

14. The cluster according to any one of claims 8-13, characterized in that, The first node is also configured to receive the first weight sent by the control device in the event of fault recovery, and perform model calculation of the first model based on the first weight.

15. The cluster according to any one of claims 8-13, characterized in that, The first node is also configured to receive the first weight sent by the third node in the computing cluster in the event of fault recovery, and perform model calculation of the first model based on the first weight. The third node is configured in the second TP domain and is assigned the same weight as the first weight assigned to the first node. The third node sends the first weight to the first node based on the first sending instruction sent by the control device.

16. A control device, characterized in that, The control device is connected to a first tensor parallel TP domain. The control device includes a processor and a memory. The memory is coupled to the processor. The memory stores at least one instruction. The processor is used to load and execute the at least one instruction to enable the control device to implement the fault handling method as described in any one of claims 1-7.

17. A chip, characterized in that, The chip includes a processor for calling and running program instructions or code stored in a memory, causing a device equipped with the chip to perform the fault handling method as described in any one of claims 1-7.

18. A computer program product, characterized in that, The computer program product includes a computer program / instruction that is executed by a processor to cause a computer to perform the fault handling method as described in any one of claims 1-7.

Images