Model training method and apparatus
By directly driving the replaced training node in the AI model training system, and utilizing the training process and data link, the problem of high performance overhead and low efficiency in existing technologies is solved, achieving more efficient model training.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- HUAWEI TECH CO LTD
- Filing Date
- 2026-01-27
- Publication Date
- 2026-07-30
AI Technical Summary
Existing technologies suffer from high performance overhead and low efficiency when changing AI model training nodes, resulting in high performance overhead and low efficiency in model training.
The management device receives model blocks in the training system and deploys them to multiple training nodes. The training process directly drives the replaced training nodes without the need for a data proxy to forward instructions, thus establishing a data link to ensure migration efficiency.
It reduces the performance overhead of model training and improves the efficiency of model training.
Smart Images
Figure CN2026075045_30072026_PF_FP_ABST
Abstract
Description
A model training method and apparatus
[0001] This application claims priority to Chinese Patent Application No. 202510127682.0, filed on January 27, 2025, entitled “A Model Training Method and Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model training method and apparatus. Background Technology
[0003] To improve the intelligence and application accessibility of AI models, they are trained on large amounts of data over extended periods. During training, there may be a need to change the training nodes for the AI model. For example, a better-performing training node may become available, requiring the model to be trained on that node. Another example is when multiple training nodes are located on different servers, necessitating the relocation of these nodes to a single server. All these scenarios require changing the training nodes for the AI model during training.
[0004] Typically, AI model training is performed by training nodes driven by a central processing unit (CPU). Therefore, when changing training nodes, the CPU's ability to drive the new training node needs to be considered. While current solutions can achieve CPU-driven training nodes, the driving overhead is high and the efficiency is low, resulting in high model training performance overhead and low training efficiency. Summary of the Invention
[0005] This application provides a model training method and apparatus that can reduce driving performance overhead and improve driving efficiency while driving the replaced training nodes, thereby reducing the performance overhead of model training and improving the efficiency of model training.
[0006] Firstly, a model training method is provided. This method can be executed by a management device in a training system, and the training system includes multiple training nodes, such as a first training node and a second training node. The first training node and / or the second training node can be any one of a neural network processor, a graphics processor, a tensor processor, or a field-programmable gate array (FPGA).
[0007] In this method, the management device may receive at least one model block obtained by splitting the model to be trained. Alternatively, the management device may receive the model blocks to be trained and split the model to be trained into at least one model block. The management device may deploy the at least one model block to at least one training node among a plurality of training nodes. One model block may be used to deploy to one training node; for example, a first model block among the at least one model blocks may be used to deploy to a first training node among the at least one training nodes. Furthermore, the at least one training node does not include a second training node.
[0008] The management device runs a training process for each of the at least one training node, for example, a training process for the first training node. For ease of description, the training process for the first training node can be referred to as the first training process. The management device can drive the training node to train the corresponding model block through the training process for the training node. For example, the management device drives the first training node to train the first model block through the first training process.
[0009] Due to poor performance of the first training node or other reasons, a second training node may need to take over training the first model block. In this case, the user can input a change instruction to the management device, indicating that the second training node will take over training the first model block. The management device can respond to this change instruction and start the training process for the second training node. For ease of description, the training process for the second training node can be referred to as the second training process.
[0010] The management device can migrate the first model block from the first training node to the second training node. After the migration of the first model block from the first training node to the second training node is completed, the management device can drive the second training node to train the first model block through the second training process.
[0011] The second training process is the training process for the second training node. This second training process drives the training of model blocks on the second training node, eliminating the need for other components, modules, or devices (such as data proxies) to forward related instructions. This reduces the performance overhead of model training and improves training efficiency.
[0012] In one possible implementation, the management device can establish a data link between the first training node and the second training node. For ease of description, the data link between the first training node and the second training node can be referred to as the first data link. The management device can instruct either the first training node or the second training node to establish the first data link. For example, the first data link can be established simultaneously with or before the start of the second training process.
[0013] The first model block can be migrated from the first training node to the second training node via the first data link. The first data link is the link between the first training node and the second training node, which can ensure the efficiency of the migration of the first model block from the first training node to the second training node.
[0014] In one possible implementation, the at least one model block further includes a second model block, and the at least one training node further includes a third training node. The management device deploys the second model block to the third training node; before driving the second training node to train the first model block, the method further includes: establishing a data link between the second training node and the third training node. For ease of description, the data link between the second training node and the third training node can be referred to as the second data link. The second training node can send the calculation results of the first model block to the third training node through the second data link. The third training node can send the calculation results of the second model block to the second training node through the second data link. The calculation results can be either forward calculation results or backward calculation results.
[0015] The second data link is the link between the second training node and the third training node. The second training node and the third training node exchange calculation results through the second data link, which can improve the efficiency of model training.
[0016] In one possible implementation, the management device can start the second training process before the first training node stops training the first model block. This allows for the pre-start of the second training process, so that when the second training node needs to take over training the first model block from the first training node, it can immediately be driven to train the first model block through the second training process, thus improving the efficiency of model training.
[0017] Secondly, a management device is provided, wherein the training system in which the management device is located further includes multiple training nodes; the management device includes: a deployment module, used to deploy at least one model block to at least one training node among the multiple training nodes; wherein the at least one model block is obtained by splitting the model to be trained, and a first model block among the at least one model block is used to deploy to the first training node among the at least one training node, and the management device runs a first training process for the first training node; a first driving module, used to drive the first training node to train the first model block through the first training process; a startup module, used to start a second training process for a second training node in response to a change instruction input by a user; wherein the change instruction is used to indicate that the second training node will take over the training of the first model block from the first training node, and the second training node is a node other than at least one training node among the multiple training nodes; the second driving module, used to drive the second training node to train the first model block through the second training process when the first model block in the first training node is migrated to the second training node.
[0018] In one possible implementation, the first model block is migrated from the first training node to the second training node via a first data link; wherein, the first data link is the link between the first training node and the second training node.
[0019] In one possible implementation, at least one model block further includes a second model block, which is used to deploy to a third training node among at least one training node; the startup module is used to: establish a second data link between the second training node and the third training node before the second training node trains the first model block through the second training process; wherein the second data link is used for the second training node to send the calculation results of the first model block to the third training node, and / or, the second data link is used for the third training node to send the calculation results of the second model block to the second training node.
[0020] In one possible implementation, the startup module is used to: start the second training process before the first training node stops training the first model block.
[0021] In one possible implementation, the second training node is any one of a neural network processor (NPU), a graphics processing unit (GPU), a tensor processor (TPU), or a field-programmable gate array (FPGA).
[0022] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method provided in the first aspect.
[0023] Fourthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a cluster of computing devices, perform the method as provided in the first aspect.
[0024] Fifthly, a computer program product containing instructions is provided, which, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method provided in the first aspect.
[0025] The beneficial effects of the second to fifth aspects can be referred to the introduction of the beneficial effects of the first aspect above, and will not be repeated here. Attached Figure Description
[0026] Figure 1 is a schematic diagram of a training system;
[0027] Figure 2 is a schematic diagram of a training system provided in an embodiment of this application;
[0028] Figure 3 is a schematic diagram of a training system provided in an embodiment of this application;
[0029] Figure 4 is a schematic diagram of a training system provided in an embodiment of this application;
[0030] Figure 5 is a schematic diagram of a model training method provided in an embodiment of this application;
[0031] Figure 6 is a flowchart of a model training method provided in an embodiment of this application;
[0032] Figure 7 is a flowchart of a model training method provided in an embodiment of this application;
[0033] Figure 8 is a schematic diagram of a management device provided in an embodiment of this application;
[0034] Figure 9 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;
[0035] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;
[0036] Figure 11 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation
[0037] The solutions provided in the embodiments of this application will now be described with reference to the accompanying drawings. In the embodiments of this application, "multiple" refers to two or more objects, and "various types" refers to two or more types. Terms such as "first," "second," etc., are only used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.
[0038] To facilitate understanding of the solutions provided in the embodiments of this application, the technical terms that may be involved in the embodiments of this application will be introduced first.
[0039] A model, also known as an AI model, is typically constructed from artificial neural networks, consisting of multiple layers. Generally, these layers are arranged sequentially, with the output of one layer serving as the input to the next. A neural network layer can be simply referred to as a network layer or layer. Additionally, a model can be divided into program code and model parameters.
[0040] The program code of a model refers to the code that describes or represents the structure of the model, the data interaction relationships between the model's operators, etc.
[0041] Model parameters are numerical values that describe or represent the various variables and relationships in a model, such as weights and biases. Model parameters are learned through model training.
[0042] Model blocks are obtained by breaking down a model. For example, they can be obtained by splitting the neural network that makes up the model into layers, or by splitting the model at the operator level, or by splitting the model according to stages.
[0043] Model training refers to the process of updating the parameters of a model. Typically, model parameters are preset or randomly generated. By updating these parameters, the model learns the ability to handle corresponding tasks, such as image recognition, speech recognition, natural language translation, computer game playing, and content generation. Model training requires multiple iterations until convergence or a predetermined number of training iterations are reached. Each iteration requires a certain amount of data as input. The model performs forward computation on the input dataset to obtain a loss function. Then, the model performs backward computation on the loss function to obtain the gradient. Finally, the gradient is used to update the model parameters.
[0044] Forward propagation, also known as forward pass, refers to the process of calculating and passing data layer by layer through the model, starting from the data input, to obtain the model's output. The purpose of forward propagation is to calculate the model's output.
[0045] Model computation refers to the calculations performed on a relevant dataset using the model's parameters. Model computation can be divided into forward computation and backward computation.
[0046] Calculation results refer to the results obtained by using the model's parameters on a relevant dataset. The results of the forward computation include the loss function, while the results of the backward computation include the gradient.
[0047] Forward computation refers to the computation performed on layers or blocks in the model during forward propagation, which is used to obtain the model's loss function.
[0048] Back propagation, also known as reverse propagation, refers to the process of obtaining a loss function based on the difference between the model's output and the labels. Then, using the loss function, the gradient is calculated layer by layer in reverse using the chain rule. Through back propagation, the gradient of each parameter with respect to the loss function can be calculated, and these gradients are used to update the parameters.
[0049] Backward computation refers to the computation performed on layers or blocks of the model during backward propagation. Backward computation is used to obtain gradients based on the loss function obtained from forward computation, and these gradients are then used to update the model parameters.
[0050] Gradient: The gradient refers to the direction in which the loss function value decreases the fastest. It is obtained through backward computation and is a result of that computation. The gradient is actually a vector representing the direction; each element in the gradient represents the rate of change of the loss function value relative to the weights at that position. The gradient is used to update the model's parameters. Specifically, the gradient is multiplied by the learning step size to obtain a product, which is then added to (or subtracted from) the model's parameters to obtain the updated parameters.
[0051] Activation values refer to the outputs of network layers or model blocks. The activation values of a network layer or model block are used for computational calculations in the next network layer or model block, generating its activation values. Furthermore, during backpropagation, the activation values of a network layer or model block are used for computational calculations in the next network layer or model block, generating its activation values and gradients.
[0052] Micro-batch: A fine-grained dataset. In pipelined parallelism, if the dataset input to each model block is large (e.g., mini-batch), the computation time required for that model block on that dataset is also long, resulting in low concurrency in pipelined parallelism. To improve the concurrency of pipelined parallelism, the large dataset (e.g., mini-batch) can be further split into finer-grained datasets. This finer-grained dataset is called a micro-batch.
[0053] Training node: refers to a device or apparatus used to perform training of a model or sub-model. In some embodiments, training nodes include neural network processing units (NPUs), graphics processing units (GPUs), tensor processing units (TPUs), and field-programmable gate arrays (FPGAs). The training node in this embodiment has good parallel computing capabilities, but its general computing capabilities are generally poor. In some embodiments, the training node can be a virtual instance.
[0054] Virtual instances, also known as virtual computing devices or virtual computing instances, refer to complete computer systems simulated by software, running in a completely isolated environment. The hardware system of a computing instance is physical hardware virtualized into a virtual hardware system. A complete computing instance has an independent virtual computing system (e.g., a central processing unit (CPU)) and a virtual disk. Instances that need to communicate with the outside world also have independent virtual network interface cards (NICs). Typical computing instances include virtual machines (VMs), containers, and elastic cloud servers (ECS).
[0055] Management device: refers to the module, device, equipment, cluster, or platform used to drive training nodes to execute model or model block training. The management device can be a central processing unit (CPU).
[0056] Training process: Also known as the driver process, this refers to the training management process running in the management device. This process manages the device to drive the training nodes to train the model or model block. One training process is dedicated to one training node, and the training process drives the training node it is targeting.
[0057] Central processing unit (CPU): As the core of a computer system for computation and control, it is the final execution unit for information processing and program execution.
[0058] A driving link refers to the link between the training process and training nodes in the management device, which can be used by the management device to drive the training nodes. Driving links can be implemented through application programming interfaces (APIs), etc.
[0059] High-bandwidth memory (HBM) is the memory of training nodes, used to store model weights, optimizer states, intermediate computation results, and a portion of the training data. Due to their powerful parallel computing capabilities, training nodes are typically used to efficiently execute forward propagation, backpropagation, and optimization algorithms for the model. The size of the HBM limits the size of the models that can be trained and the number of samples processed in each batch.
[0060] Memory: The primary storage area in a computer system, used to store program code, data, and runtime state. During model training, memory is mainly used to store the model's source code, load training data, perform preprocessing, and handle some intermediate computation results. Compared to HBM, memory typically has a larger capacity but is relatively slower.
[0061] Model training requires the cooperation of a CPU and training nodes (such as a GPU). The training nodes specifically train the model, for example, by performing calculations using the model's parameters. The CPU drives the training nodes in training the model, such as deploying the model to the training nodes, providing resources like memory, and updating the model's parameters using the gradients calculated by the training nodes. Therefore, the efficiency of the CPU in driving the training nodes is crucial to the overall training efficiency. During model training, training nodes may be changed.
[0062] In one approach, as shown in Figure 1, the training process corresponding to the original training node continues to drive the replaced training node. To identify the replaced training node, a data proxy is set up between the CPU and the training node. The training process in the CPU drives the replaced training node through the data proxy. This data proxy is used to identify the training node where the model resides; therefore, after the training node replacement is completed, the driving instructions can be forwarded to the replaced training node through this data proxy. In this approach, the data proxy needs to store relevant information about the training node to identify it, which increases storage performance overhead. Furthermore, regardless of whether a training node replacement has occurred, the CPU drives the training node through the data proxy. This requires the data proxy to intercept and forward numerous instructions, increasing computational overhead, driving latency, and reducing driving efficiency.
[0063] This application provides a model training method. In this method, regardless of whether the training node is before or after the change, the training node is driven by the training process of the training node itself, without the need for a data proxy to forward related instructions. This not only reduces performance overhead but also improves driving efficiency, thereby reducing the performance overhead of model training and improving the efficiency of model training.
[0064] Next, the model training method provided in the embodiments of this application will be introduced.
[0065] Figure 2 illustrates a training system 100 that can be used to implement this method. The training system 100 may include a management device 110 and multiple training nodes, such as training node 120, training node 130, training node 140, etc. In some embodiments, the training system 100 may employ a Kubernetes (K8s) architecture, wherein the management device 110 may be a master node in the K8s architecture, and the training nodes may be worker nodes in the K8s architecture. In some embodiments, the training system 100 may be a server or other form of computing device, wherein the management device 110 may be a CPU, and the training nodes may be GPUs, NPUs, or other forms of processors or computing units.
[0066] The training system 100 can be used to train the model 200. The user can input the model 200 into the management device 110. The management device 110 can divide the model 200 into at least one model block. In some embodiments, the model 200 can be trained using distributed training, in which case the at least one model block is multiple model blocks, where different model blocks are different parts of the model 200. In some embodiments, the model 200 can also be trained without distributed training, in which case the at least one model block is a single model block, which is the model 200 itself; that is, the model 200 is not divided.
[0067] The management device 110 can select at least one training node from the plurality of training nodes, and the at least one training node is used to train the model 200. Specifically, the management device 110 can select a corresponding number of training nodes based on the number of model blocks to obtain the at least one training node.
[0068] The management device 110 can deploy at least one model block to at least one selected training node. For example, each training node can deploy one model block, such as training node 120 deploying model block 210, training node 130 deploying model block 220, and so on.
[0069] The management device 110 can initiate and run training processes for training nodes, such as initiating and running training process A1 for training node 210 and training process A2 for training node 230. Through these training processes, the management device 110 drives the training node to train the model blocks or model within that node. For example, taking training node 120 as an example, the management device 110 can send a start command and training data to training node 120 through training process A1. Training node 120 can respond to the start command by inputting the training data into model block 210, enabling model block 210 to perform relevant calculations based on the training data, thus achieving forward computation of model block 210. The management device 110 can receive the gradients obtained from the backward computation of model block 210 from training node 120 through training process A1. After receiving a gradient, the management device 110 can update the parameters of model block 210 based on the gradient and send the updated parameters to training node 120 through training process A1, so that training node 120 can replace the original parameters of model block 210 with the updated parameters. The management device 110 can also send a stop command to training node 120 through training process A1. Training node 120 can respond to the stop command and stop training model block 210. In this way, the management device 110 can drive the training node without the need for data brokers or other devices, improving training efficiency while reducing performance overhead.
[0070] The training of a model block or model by a training node can refer to the calculations performed by the training node to train the model block or model, which may include forward calculations and / or backward calculations.
[0071] In some embodiments, the management device 110 can establish driving links between training processes and the training nodes targeted by those training processes, such as driving link A11 between training process A1 and training node 120, and driving link A21 between training process A2 and training node 130. The training process can drive the training node through the driving links.
[0072] During the training of model 200, management device 110 can change the training node where one or more model blocks of model 200 reside, that is, migrate one or more model blocks from the current training node to other training nodes. Next, we will take the migration of model block 210 from training node 120 to training node 140 as an example.
[0073] The management device 110 can confirm that training node 140 will take over from training node 120 to train model block 210. As shown in Figure 3, the management device 110 can receive a replacement instruction from the user, which indicates that training node 140 will take over from training node 120 to train model block 210. The replacement instruction may include the identifier of training node 120, the identifier of training node 140, and the replacement requirement. The change requirement is that the training node for training model block 210 will be changed from training node 120 to training node 140.
[0074] If it is confirmed that training node 140 will take over from training node 120 to train model block 210, management device 110 can start and run training process A3 for training node 140. In this way, management device 110 can drive the training node to train model block 210 through training process A3.
[0075] In some embodiments, before migrating model block 210 to training node 140 or before training node 120 stops training model block 210, management device 110 may start and run training process A3. Thus, after model block 210 is migrated to training node 140, management device 110 can immediately drive training node 140 to train model block 210 through training process A3, further improving training efficiency.
[0076] In some embodiments, the management device 110 may establish a drive link A31 between the training process A3 and the training node 140.
[0077] In some embodiments, model block 210 is migrated from training node 120 to training node 140 via data link B1 between training node 120 and training node 140. That is, the migration of model block can be achieved through an end-to-end link, thereby improving the migration efficiency of model block.
[0078] In one example of this embodiment, the management device 110 can establish a data link B1 between training node 120 and training node 140. Specifically, the management device 110 can send an establishment instruction to training node 120 (or training node 140) to instruct the establishment of data link B1 between training nodes 120 and 140. In response to the establishment instruction, training node 120 (or training node 140) negotiates the establishment of data link B1 with training node 120 (or training node 120).
[0079] After the data link B1 is established, the management device 110 can instruct the training node 120 to send the model block 210 to the training node 140 through the data link B1, thereby realizing the migration of the model block.
[0080] In some embodiments, as shown in FIG3, the management device 110 may also establish a data link B2 between the training node 130 and the training node 140 where the model block 220 is located. After the training node 140 starts training the model block 210, the data link B2 can be used for the training node 130 and the training node 140 to interact and calculate results. For example, the data link B2 can be used by the training node 130 to send the calculation results of the model block 220 to the training node 140, such as the calculation results of the forward calculation or the calculation results of the backward calculation of the model block 220. For example, the data link B2 can be used by the training node 140 to send the calculation results of the model block 210 to the training node 130, such as the calculation results of the forward calculation or the calculation results of the backward calculation of the model block 210.
[0081] In one example of this embodiment, before training node 140 trains model block 210 via training process A3, management device 110 establishes data link B2. This ensures that training node 130 and training node 140 can exchange computation results in a timely manner.
[0082] In some embodiments, training of each model block of model 200 can be stopped before migrating model block 210 from training node 120 to training node 140. During the stop, model block 210 is migrated from training node 120 to training node 140. After the migration of model block 210 from training node 120 to training node 140 is completed, training of each model block of model 200 is resumed. The management device 110 can send a stop command to the training node where each model block of model 200 is located. The training node can respond to the stop command received from the management device 110 and stop the training of the corresponding model block. The corresponding model block of the training node is the model block used to deploy to that training node.
[0083] In one example of this embodiment, after the training process A3 is started and the data links B1, B2, etc. are established, the training of each model block of model 200 is stopped.
[0084] In another example of this embodiment, after each model block of model 200 completes the relevant calculations for a batch of training data, it enters a stopped state. This ensures that the training progress of each model block is synchronized. Typically, the training data for a model is large in volume. The training data is divided into multiple batches according to a preset granularity (e.g., micro-batch). These multiple batches of training data are sequentially input into the foremost model block of model 200. The foremost model block refers to the model block at the very front in the forward propagation direction. The foremost model block performs forward calculations on the input batch of training data and passes the results to the next model block, and so on. The forward calculation of the last model block yields the loss function, thus completing the forward calculation. The last model block refers to the model block at the very back in the forward propagation direction. In the backward propagation, the loss function is passed sequentially from back to front, allowing each model block to perform backward calculations to obtain gradients, activation values, etc.
[0085] In this example, when the foremost model block receives a stop command, it stops receiving new batches of training data. Each model block enters a stopped state after completing the forward and backward computations corresponding to the current batch of data, thus achieving training synchronization among the model blocks. Once each model block is in a stopped state, model block 210 is migrated from training node 120 to training node 140.
[0086] To achieve the above functions, the management device 110 may include the functional modules shown in FIG4.
[0087] The migration management module 111 can receive change instructions from users and identify the model block to be migrated, the current training node of the model block, and the target training node of the model block based on the change instructions. The migration management module 111 can send a migration command to the current training node of the model block to be migrated, which triggers the current training node of the model block to send the model block to the target training node.
[0088] The start / stop module 112 can instruct training nodes to start training a model or model block, and instruct training nodes currently training a model or model block to stop training. Specifically, the start / stop module 112 can send a start command to a training node, which triggers the training node to begin training the model or model block. The start / stop module 112 can also send a stop command to a training node, which triggers the training node to stop training. The start / stop module 112 can send start and / or stop commands to training nodes based on the training process.
[0089] The communication management module 113 can manage the communication between the management node 110 and the training nodes, as well as the communication between different training nodes. The communication management module 113 can record information such as the address of the training nodes, so that the addresses of the training nodes can be used to establish drive links, data links, etc.
[0090] The resource management module 114 can manage the resources of the training nodes, such as allocating resources to the training nodes or reclaiming resources allocated to them. These resources can be CPU-side resources, such as memory resources. The training nodes can access the allocated resources through the training process.
[0091] As shown in Figure 4, a training node can be deployed with a training process. This training process can receive relevant instructions, such as start and stop instructions, sent by the management device 110 through the driver link. The training process can also access the resources allocated to the training node it resides in through the driver link.
[0092] In some embodiments, as shown in FIG4, the training system may further include a database. The database can be used to store training data. The migration management module 111 can retrieve training data from the database and send the retrieved training data to the training nodes, enabling the training nodes to train models or model blocks using the training data.
[0093] The above examples illustrate the training system 100 provided in the embodiments of this application. Next, taking the implementation in the training system 100 as an example, the model training method provided in the embodiments of this application will be described.
[0094] This method can be divided into a pre-transfer stage, a transfer preparation stage, a transfer stage, and a transfer recovery stage. Next, referring to Figure 5, we will use the training of Model 200 as an example to illustrate this method.
[0095] The pre-migration stage refers to the stage before the management platform 110 confirms that the training node 140 will take over the training model block 210 from the training node 120. In the pre-migration stage, the management device 110 drives the training node 120 to train the model block 210 through the training process A11, and the management device 110 drives the training node 130 to train the model block 220 through the training process A2.
[0096] The migration preparation phase refers to the stage where the management platform 110 confirms that training node 140 will take over training model block 210 from training node 120, but has not yet migrated model block 210 from training node 120 to training node 140. During the migration preparation phase, training process A3 can be started for training node 140, data link B1 can be established between training node 120 and training node 140, and data link B2 can be established between training node 130 and training node 140.
[0097] The migration phase refers to the phase in which model block 210 is migrated from training node 120 to training node 140. During this migration phase, training node 120 can send model block 210 to training node 140 via data link B1.
[0098] The migration and recovery phase refers to the stage where model block 210 has been migrated to training node 140, and training node 140 trains model block 210. During this phase, management device 110 drives training node 140 to train model block 210 through training process A3. This completes the replacement of training node 120 for model block 120 from training node 120 to training node 140.
[0099] The examples above illustrate the various stages of the model training method provided in this application embodiment. Next, using implementation in training system 100 as an example, the flow of the model training method provided in this application embodiment will be described. As shown in Figure 6, the method includes the following steps.
[0100] Step 602, the management device 110 confirms that the training node 140 will take over the training of the training model block 210 from the training node 120.
[0101] In some embodiments, in step 601, the management device 110 receives a change instruction from the user. In step 602, based on the change instruction, the management device 110 confirms that training node 140 will replace training node 120 in training model block 210. For example, the change instruction may include the identifier of training node 120, the identifier of training node 140, and a replacement request. The change request is that the training node training model block 210 will be changed from training node 120 to training node 140.
[0102] In step 603a, the management device 110 may initiate training process A3 for training node 140. In some embodiments, in step 603a, the management device 110 may initiate training process A3 and establish a link between training process A3 and training node 140 to obtain drive link A31.
[0103] In some embodiments, as shown in FIG6, in step 603b, the management device 110 may send a data link establishment instruction to the training node 120. The training node 120 may respond to the data link establishment instruction and execute step 604a to establish data link B1 between training node 120 and training node 140. In some embodiments, the management device 110 may send a data link establishment instruction to the training node 140. The training node 140 may respond to the data link establishment instruction and establish data link B1 between training node 120 and training node 140.
[0104] In some embodiments, as shown in FIG6, in step 603c, the management device 110 may send a data link establishment instruction to the training node 130. The training node 130 may respond to the data link establishment instruction and execute step 604b to establish data link B2 between training node 130 and training node 140. In some embodiments, the management device 110 may send a data link establishment instruction to the training node 140. The training node 140 may respond to the data link establishment instruction and establish data link B2 between training node 130 and training node 140.
[0105] In step 605a, the management device 110 may send a stop command to the training node 120. In step 605b, the management device 110 may send a stop command to the training node 130. Training nodes 120 and 130 may respectively respond to their received stop commands and stop the training of the corresponding model blocks. In some embodiments, when model blocks 210 and 220 complete the relevant calculations for the same batch of training data, they enter a stopped state to ensure that the training progress of each model block is synchronized. Related details can be found in the above description and will not be repeated here.
[0106] After each model block of model 200 is in a stopped state, management device 110 can execute step 606, sending a migration command to training node 120. Training node 120 can respond to the migration command and execute step 607, migrating model block 210 to training node 140. In some embodiments, training node 120 can migrate model block 210 to training node 140 via data link B1.
[0107] Management device 110 can execute step 608a, sending a start command to training node 130. Training node 130 can respond to the start command and begin executing step 609a, training model block 220. Management device 110 can execute step 609a, sending a start command to training node 140. Training node 140 can respond to the start command and begin executing step 609b, training model block 210.
[0108] This completes the replacement of the training nodes for model block 210.
[0109] In some embodiments, the management device 110 can issue migration instructions, data link establishment instructions, stop instructions, or start instructions through the ConfigMap. The management device 110 can record these instructions in the ConfigMap. The ConfigMap can be mounted as a file on the training node. The training process in the training node can read the relevant instructions from the ConfigMap to become aware of them.
[0110] In summary, the model training method provided in this application can initiate a training process for the new training node when the training node is changed. Then, the new training node is driven by the training process for the new training node. This reduces the performance overhead of model training and improves the training efficiency of the model.
[0111] Based on the above description, this application also provides a model training method. This method can be executed by the management device 110 in the training system 100, which also includes multiple training nodes. As shown in Figure 7, the method includes the following steps.
[0112] Step 701: The management device 110 deploys at least one model block to at least one training node among the plurality of training nodes; wherein, the at least one model block is obtained by splitting the model to be trained, and model block 210 of the at least one model block is used to deploy to training node 120 among the at least one training node, and the management device 110 runs a training process A1 for training node 120. The relevant implementation method can be referred to the description of the embodiment shown in FIG2 above, and will not be repeated here. In addition, model block 210 can also be called a first model block, and training node 120 can also be called a first training node.
[0113] In step 702, the management device 110 drives the training node 120 to train the model block 210 through the training process A1. The relevant implementation method can be found in the description of the embodiment shown in Figure 2 above, and will not be repeated here. Additionally, the training process A1 can also be referred to as the first training process.
[0114] In step 703, the management device 110 responds to the user-inputted change instruction and initiates training process A3 for training node 140; wherein, the change instruction indicates that training node 140 will take over training model block 210 from training node 120, and training node 140 is a node other than the at least one training node among the plurality of training nodes. The relevant implementation can be referred to the description of the embodiment shown in FIG6 above, and will not be repeated here. Furthermore, training process A3 can also be referred to as the second training process, and training node 140 can also be referred to as the second training node.
[0115] Step 704: When model block 210 in training node 120 is migrated to training node 140, management device 110 drives training node 140 to train model block 210 through training process A3. The relevant implementation method can be referred to the above description of the embodiment shown in Figure 6, and will not be repeated here.
[0116] In some embodiments, model block 210 is migrated from training node 120 to training node 140 via data link B1; wherein, data link B1 is the link between training node 120 and training node 140.
[0117] In some embodiments, the at least one model block further includes a model block 220, which is used to deploy to training node 130 in the at least one training node; before training node 140 is driven to train model block 210 through training process A3, management platform 110 may also establish a data link B2 between training node 140 and training node 130; wherein, data link B2 is used for training node 140 to send the calculation result of model block 310 to training node 130, and / or, data link B2 is used for training node 130 to send the calculation result of model block 220 to training node 140.
[0118] In some embodiments, the management device 110 may start the training process A3 before the training node 120 stops training the model block 210.
[0119] In some embodiments, training node 140 is a neural network processor. In some embodiments, training node 149 may be a graphics processor. In some embodiments, training node 140 may be a tensor processor. In some embodiments, training node 140 may be an FPGA.
[0120] The model training method provided in this application allows for the initiation of a training process for the new training node when the training node is changed. Then, the new training node is driven using this training process. This reduces the performance overhead of model training and improves its training efficiency.
[0121] In some embodiments, model training can also be performed using the following methods.
[0122] The scheme can be divided into three stages: warm-up, migration, and recovery. The warm-up stage initializes the new card and training process, and pre-creates the underlying ensemble communication link. The migration stage uses device-to-device (D2D) ensemble communication to transfer model data from the old card to the new card. The recovery stage controls the ensemble communication link to switch links and resume the training process.
[0123] This solution can migrate and replace the training nodes used for model training, solving problems such as resource fragmentation in training node clusters, conflicts between proactive maintenance (software and hardware repair and upgrades) and model training, and slow, stuck, and slow nodes.
[0124] This solution may involve a training task migration management component, a cluster job management component, a collective communication management component (CCL manager), and a cluster resource management component (cluster manager). The training task migration management component orchestrates the entire migration process. The cluster job management component manages the start and stop of distributed training tasks. The collective communication management component refreshes the rank table and manages the collective communication links.
[0125] In a Kubernetes (K8s) architecture, migration commands for the training node cluster are issued via ConfigMaps. These ConfigMaps are mounted as files inside the training container, and the training process reads the ConfigMaps to detect the issuance of migration commands. In non-K8s architectures, migration control commands can be issued in other ways. For example, if training is performed directly on bare metal, migration commands can be issued directly to the training or management processes through a management component.
[0126] In one example, the scheme can be divided into a warm-up phase, a migration phase, and a recovery phase.
[0127] During the warm-up phase, the training environment for the new card is initialized (e.g., package preparation, graph compilation, etc.). During the warm-up phase, the training process on the node where the new card resides is started, and the initialization of objects related to model training is completed. During the warm-up phase, 1. A pre-establishment command for ensemble communication is issued, triggering the pre-creation of the ensemble communication link.
[0128] During the migration phase, a data migration command is issued to control the transfer of training data from the old card to the new card. During the migration phase, a link switching command is issued to control the switching of the aggregate communication link from the old card to the new card.
[0129] During the recovery phase, a migration recovery command is issued, and the training process continues.
[0130] The migration aggregation communication link pre-creation technology in this solution ensures that the aggregation communication link is initialized during the migration preparation phase, without interrupting training and reducing training pause time.
[0131] This solution is application-centric and uses a collection communication method to complete the model data transfer. D2D has high bandwidth and fast data transmission, reducing training pause time.
[0132] Based on the above description, this application provides a management device 800. The training system containing the management device 800 also includes multiple training nodes. As shown in Figure 8, the management device 800 includes:
[0133] Deployment module 810 is used to deploy at least one model block to at least one training node among the plurality of training nodes; wherein, the at least one model block is obtained by splitting the model to be trained, and the first model block among the at least one model block is used to deploy to the first training node among the at least one training node; the management device runs a first training process for the first training node;
[0134] The first driving module 820 is used to drive the first training node to train the first model block through the first training process;
[0135] The startup module 830 is used to respond to a change instruction input by the user and start a second training process for the second training node; wherein, the change instruction is used to indicate that the second training node will take over the training of the first model block from the first training node, and the second training node is a node other than the at least one training node among the plurality of training nodes.
[0136] The second driving module 840 is used to drive the second training node to train the first model block through the second training process when the first model block in the first training node is migrated to the second training node.
[0137] In some embodiments, the first model block is migrated from the first training node to the second training node via a first data link; wherein, the first data link is a link between the first training node and the second training node.
[0138] In some embodiments, the at least one model block further includes a second model block, the second model block being deployed to a third training node among the at least one training nodes; the startup module 830 is configured to: establish a second data link between the second training node and the third training node before driving the second training node to train the first model block through the second training process; wherein, the second data link is used for the second training node to send the calculation result of the first model block to the third training node, and / or, the second data link is used for the third training node to send the calculation result of the second model block to the second training node.
[0139] In some embodiments, the startup module 830 is configured to: start the second training process before the first training node stops training the first model block.
[0140] The deployment module 810, the first driver module 820, the startup module 830, and the second driver module 840 can all be implemented in software or in hardware. For example, the implementation of the deployment module 810 will be described below. Similarly, the implementation of the first driver module 820, the startup module 830, and the second driver module 840 can refer to the implementation of the deployment module 810.
[0141] As an example of a software functional unit, deployment module 810 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, deployment module 810 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.
[0142] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.
[0143] As an example of a hardware functional unit, the deployment module 810 may include at least one computing device, such as a server. Alternatively, the deployment module 810 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.
[0144] The multiple computing devices included in deployment module 810 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in deployment module 810 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in deployment module 810 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0145] It should be noted that, in other embodiments, the deployment module 810 can be used to execute any step in the method shown in FIG. 7, the first driving module 820 can be used to execute any step in the method shown in FIG. 7, the startup module 830 can be used to execute any step in the method shown in FIG. 7, and the second driving module 840 can be used to execute any step in the method shown in FIG. 7. The steps implemented by the deployment module 810, the first driving module 820, the startup module 830, and the second driving module 840 can be specified as needed. By implementing different steps in the method shown in FIG. 7 through the deployment module 810, the first driving module 820, the startup module 830, and the second driving module 840, all functions of the device 800 can be realized.
[0146] This application also provides a computing device 900. As shown in FIG9, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The computing device 900 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.
[0147] Bus 902 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 9, but this does not imply that there is only one bus or one type of bus. Bus 902 can include pathways for transmitting information between various components of computing device 900 (e.g., memory 906, processor 904, communication interface 908).
[0148] Processor 904 may include a central processing unit (CPU) and a graphics processing unit (GPU).
[0149] Processing unit (GPU), microprocessor (MP), or digital signal processor (DSP) are any one or more of the following: processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).
[0150] Memory 906 may include volatile memory, such as random access memory (RAM). Memory 906 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0151] The memory 906 stores executable program code, which the processor 904 executes to implement the functions of the aforementioned deployment module 810, first drive module 820, startup module 830, and second drive module 840, thereby realizing the method shown in FIG7. That is, the memory 906 stores instructions for executing the method shown in FIG7.
[0152] The communication interface 908 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.
[0153] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
[0154] As shown in Figure 10, the computing device cluster includes at least one computing device 900. The memory 906 of one or more computing devices 900 in the computing device cluster may store the same instructions for executing the method shown in Figure 7.
[0155] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the method shown in FIG7. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the method shown in FIG7.
[0156] It should be noted that the memory 906 in different computing devices 900 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the device 900. That is, the instructions stored in the memory 906 of different computing devices 900 can implement the functions of one or more modules among the deployment module 810, the first drive module 820, the startup module 830, and the second drive module 840.
[0157] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 illustrates one possible implementation. As shown in Figure 11, two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 906 in computing device 900A stores instructions for executing the functions of the deployment module 810 and the first driver module 820. Simultaneously, the memory 906 in computing device 900B stores instructions for the functions of the startup module 830 and the second driver module 840.
[0158] It should be understood that the functions of computing device 900A shown in Figure 11 can also be performed by multiple computing devices 900. Similarly, the functions of computing device 900B can also be performed by multiple computing devices 900.
[0159] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device clusters shown in Figures 10 and 11. The difference is that the memory 906 of one or more computing devices 900 in this computing device cluster can store the same instructions for executing the method shown in Figure 7.
[0160] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the method shown in FIG7. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the method shown in FIG7.
[0161] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method shown in FIG7.
[0162] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a host migration device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the method shown in FIG7.
[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.
Claims
1. A model training method, characterized in that, The method is applied to a management device, wherein the training system in which the management device is located further includes multiple training nodes; the method includes: At least one model block is deployed to at least one of the plurality of training nodes; wherein, the at least one model block is obtained by splitting the model to be trained, the first model block of the at least one model block is used to deploy to the first training node of the at least one training node, and the management device runs a first training process for the first training node. The first training process drives the first training node to train the first model block; In response to a change instruction input by the user, a second training process is initiated for the second training node; wherein the change instruction is used to indicate that the second training node will take over training the first model block from the first training node, and the second training node is a node other than the at least one training node among the plurality of training nodes. When the first model block in the first training node is migrated to the second training node, the second training process drives the second training node to train the first model block.
2. The method according to claim 1, characterized in that, The first model block is migrated from the first training node to the second training node via a first data link; wherein, the first data link is the link between the first training node and the second training node.
3. The method according to claim 1 or 2, characterized in that, The at least one model block further includes a second model block, which is used to deploy to a third training node in the at least one training node; Before driving the second training node to train the first model block through the second training process, the method further includes: establishing a second data link between the second training node and the third training node; Wherein, the second data link is used for the second training node to send the calculation result of the first model block to the third training node, and / or, the second data link is used for the third training node to send the calculation result of the second model block to the second training node.
4. The method according to any one of claims 1-3, characterized in that, The initiation of the second training process for the second training node includes: The second training process is started before the first training node stops training the first model block.
5. The method according to any one of claims 1-4, characterized in that, The second training node is any one of a neural network processor (NPU), a graphics processing unit (GPU), a tensor processor (TPU), or a field-programmable gate array (FPGA).
6. A management device, characterized in that, The training system containing the management device also includes multiple training nodes; the management device includes: A deployment module is used to deploy at least one model block to at least one training node among the plurality of training nodes; wherein, the at least one model block is obtained by splitting the model to be trained, and the first model block among the at least one model block is used to deploy to the first training node among the at least one training node; the management device runs a first training process for the first training node; The first driving module is used to drive the first training node to train the first model block through the first training process; A startup module is used to respond to a change instruction input by the user and start a second training process for the second training node; wherein, the change instruction is used to indicate that the second training node will take over the training of the first model block from the first training node, and the second training node is a node other than the at least one training node among the plurality of training nodes. The second driving module is used to drive the second training node to train the first model block when the first model block in the first training node is migrated to the second training node through the second training process.
7. The management device according to claim 6, characterized in that, The first model block is migrated from the first training node to the second training node via a first data link; wherein, the first data link is the link between the first training node and the second training node.
8. The management device according to claim 6 or 7, characterized in that, The at least one model block further includes a second model block, which is used to deploy to a third training node in the at least one training node; The startup module is used to: establish a second data link between the second training node and the third training node before the second training node is driven to train the first model block through the second training process; Wherein, the second data link is used for the second training node to send the calculation result of the first model block to the third training node, and / or, the second data link is used for the third training node to send the calculation result of the second model block to the second training node.
9. The management device according to any one of claims 6-8, characterized in that, The startup module is used to: start the second training process before the first training node stops training the first model block.
10. The management device according to any one of claims 6-9, characterized in that, The second training node is any one of a neural network processor (NPU), a graphics processing unit (GPU), a tensor processor (TPU), or a field-programmable gate array (FPGA).
11. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 5.
12. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 5.
13. A computer program product containing instructions, characterized in that, When the instruction is executed by a cluster of computer devices, the cluster of computer devices causes the cluster of computer devices to perform the method as described in any one of claims 1 to 5.