Distributed system-based model training method, computing node, and system

Abstract

A distributed system-based model training method, a computing node, and a system. The distributed system comprises n groups of computing nodes, each group of computing nodes comprising m computing nodes, and the m computing nodes in each group respectively corresponding to m parameter sets comprised in a target model. The method comprises: the m computing nodes in each group jointly perform multiple times of training, such that each computing node in each group obtains a first value of each parameter in a parameter set corresponding to the computing node; on the basis of n first values of a first parameter, n first computing nodes corresponding to the first parameter in the n groups determine a second value corresponding to the first parameter; and the n first computing nodes respectively synchronize the values of the first parameter to the second value.

Description

Model training methods, computing nodes, and systems based on distributed systems

[0001] This application claims priority to Chinese patent application filed on December 4, 2024, with application number 202411775751.0 and entitled "Model Training Method, Computing Node and System Based on Distributed System", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments in this specification pertain to the field of machine learning, and particularly relate to a model training method, computing nodes, and system based on a distributed system. Background Technology

[0003] Distributed Data Parallel (DDP) is an important framework for implementing distributed training, allowing deep learning models to be trained in parallel across multiple processing units (such as GPUs). Specifically, training data is distributed across multiple processing units, each with a complete copy of the model's parameters. Each processing unit can use its training samples to train this copy of the model. After each backpropagation process, the gradients of the model parameters are synchronized (all-reduce), ensuring consistency across all processing units. However, this training method suffers from frequent communication issues due to the need to synchronize gradients in every training iteration. Furthermore, slow nodes are always randomly present among the multiple processing units, requiring each unit to wait for these slow nodes to complete their training before synchronizing gradients, resulting in slower training speeds.

[0004] To address the aforementioned issues, a Local Stochastic Gradient Descent (Local SGD) method is proposed where each node independently and in parallel executes multiple local update steps, and then all nodes synchronize their parameters to reduce communication frequency and mitigate the impact of random slow nodes.

[0005] However, the aforementioned Local SGD method cannot be applied to the training of Large Language Models (LLMs). This is because the Local SGD method requires storing a complete copy of the parameters, i.e., the complete model parameters, in a single node, but the storage capacity of a single node cannot store the billions of parameter copies of an LLM. Summary of the Invention

[0006] The purpose of this invention is to provide a model training method based on a distributed system, which enables more efficient model training for LLM.

[0007] The first aspect of this specification provides a model training method based on a distributed system, wherein the distributed system includes n groups of computing nodes, each group of computing nodes includes m computing nodes, and the m computing nodes in each group correspond to m parameter sets included in the target model, the method comprising:

[0008] The m computing nodes in each group jointly perform multiple training sessions, so that each computing node in each group obtains the first value of each parameter in its corresponding parameter set.

[0009] The n first computing nodes in the n groups corresponding to the first parameter determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter;

[0010] The n first computing nodes will each synchronize the value of the first parameter to the second value.

[0011] In one implementation, the m computing nodes in each group jointly perform multiple training iterations, including:

[0012] In each training session, each computing node in each group obtains the values ​​of other parameters of the target model from the other computing nodes in the group, thus obtaining the values ​​of all parameters of the target model. Based on the values ​​of all parameters of the target model, the values ​​of each parameter corresponding to the computing node are updated.

[0013] In one implementation, the target model is divided into a plurality of consecutively arranged modules, and the first parameter is a parameter in the first module among the plurality of modules.

[0014] The n first computing nodes in the n groups corresponding to the first parameter determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter, including:

[0015] Each of the n first computing nodes, during the forward propagation process of the next training iteration of its multiple training iterations, determines a second value corresponding to the first parameter based on the n first values ​​of the first parameter.

[0016] In one embodiment, the method further includes:

[0017] Each of the first computing nodes performs the calculation of the first module in the forward propagation based on the second value of the first parameter, and sends the second value to other computing nodes in the same group for use by other computing nodes in the calculation of the first module in the forward propagation.

[0018] In one implementation, during the forward propagation process of the next training iteration of the multiple training iterations, each of the n first computing nodes determines a second value corresponding to the first parameter based on n first values ​​of the first parameter, including:

[0019] The first computing node determines the second value corresponding to the first parameter in parallel with the calculation of the second module in the forward propagation process, and the second module is the preceding module of the first module.

[0020] In one implementation, the n first computing nodes in the n groups corresponding to the first parameter determine a second value corresponding to the first parameter based on n first values ​​of the first parameter, including:

[0021] The n first computing nodes calculate the difference between each first value and the current synchronization value of the first parameter, determine the outlier among the n first values ​​based on the difference, and determine the second value based on the multiple first values ​​other than the outlier among the n first values.

[0022] In one implementation, determining the second value based on a plurality of first values ​​(excluding the outlier) among the n first values ​​includes:

[0023] The weight of each difference is determined based on the multiple differences corresponding to the multiple first values. A third value is calculated based on the multiple differences and the weight of each difference. The second value is calculated based on the synchronization value and the third value.

[0024] In one implementation, calculating the third value based on the plurality of differences and the weights of each difference includes:

[0025] A fourth value is calculated based on the multiple differences and their respective weights. The fourth value is then processed based on a preset threshold and the multiple differences to obtain the third value.

[0026] In one implementation, the m computing nodes in each group jointly perform multiple training sessions, including: the number of times each group of computing nodes performs joint training is determined based on a preset first time interval.

[0027] In one implementation, the communication speed between m computing nodes in each group is faster than the communication speed between computing nodes in different groups.

[0028] In one embodiment, the method further includes a preheating training phase, the preheating training phase including backpropagation, and the method further includes: in the preheating training phase, the n first computing nodes corresponding to the first parameter in the n groups synchronize the model gradient in the backpropagation.

[0029] The second aspect of this specification provides a model training method based on a distributed system, wherein the distributed system includes n groups of computing nodes, each group of computing nodes includes m computing nodes, and the m computing nodes in each group correspond to m parameter sets included in the target model, the method being executed by a second computing node in the first group of computing nodes, the second computing node corresponding to the first parameter of the model, the method comprising:

[0030] The first parameter is obtained by performing multiple training sessions in conjunction with other computing nodes in the first group of computing nodes.

[0031] Based on the n first values ​​of the first parameter, the n-1 first computing nodes in the other groups corresponding to the first parameter determine the second value corresponding to the first parameter;

[0032] Synchronize the value of the first parameter to the second value.

[0033] A third aspect of this specification provides a second computing node in a distributed system, the distributed system comprising n groups of computing nodes, each group comprising m computing nodes, the m computing nodes in each group corresponding to m parameter sets included in the target model, the second computing node corresponding to the first parameter of the model, and the second computing node comprising:

[0034] The training unit is used to perform multiple training sessions in conjunction with other computing nodes in the first group of computing nodes to obtain the first value of the first parameter.

[0035] The determining unit is used to determine the second value corresponding to the first parameter based on n first values ​​of the first parameter by the n-1 first computing nodes in the other group corresponding to the first parameter.

[0036] A synchronization unit is used to synchronize the value of the first parameter to the second value.

[0037] The fourth aspect of this specification provides a distributed system, which includes n groups of computing nodes, each group of computing nodes including m computing nodes, and the m computing nodes in each group correspond to m parameter sets included in the target model.

[0038] The m computing nodes in each group are used to jointly perform multiple training sessions, so that each computing node in each group obtains the first value of each parameter in its corresponding parameter set.

[0039] The n first computing nodes in the n groups corresponding to the first parameter are used to determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter;

[0040] The n first computing nodes are also used to synchronize the value of the first parameter to the second value.

[0041] The fifth aspect of this specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the methods described in the first or second aspect.

[0042] A sixth aspect of this specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method described in the first or second aspect.

[0043] The seventh aspect of this specification provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0044] The model training scheme provided in the embodiments of this specification solves the problem of storage overflow of a single computing node by adopting a hierarchical distributed structure of a two-dimensional device network. This is achieved by dividing a single model into m computing nodes, each storing a portion of the model's parameters. Furthermore, during model training, the n computing nodes in a single model replication group only need to synchronize their corresponding partial parameters, rather than synchronizing all model parameters between computing nodes. This reduces the communication load of a single computing node, and the parallel synchronization of model parameters across multiple model replication groups improves the efficiency of parameter synchronization. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 is a system architecture diagram of distributed training in the embodiments of this specification;

[0047] Figure 2 is a schematic diagram of the warm-up phase of distributed model training in the embodiments of this specification;

[0048] Figure 3 is a flowchart of the model training method based on a distributed system in the embodiments of this specification;

[0049] Figure 4 is a schematic diagram of the periodic synchronous training phase in the embodiments of this specification.

[0050] Figure 5 is a schematic diagram of setting the number of joint training iterations in an embodiment of this specification;

[0051] Figure 6 is a schematic diagram of a process for reducing parameter values ​​in an embodiment of this specification.

[0052] Figure 7 is a schematic diagram of the synchronization parameter process in the forward propagation of an embodiment of this specification;

[0053] Figure 8 is an architecture diagram of a computing node provided in an embodiment of this specification. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0055] To address the aforementioned problems in the prior art, this specification proposes a distributed system-based model training method for LLM, enabling efficient distributed training of LLM.

[0056] Figure 1 is a system architecture diagram of distributed training in an embodiment of this specification. As shown in Figure 1, the system may include n*m ​​computing nodes. A computing node refers to a unit capable of independent computation, which may include a separate computing device, a virtual computing device, or a processing unit (e.g., a GPU), etc., and is not limited thereto.

[0057] For ease of understanding, these n*m ​​computing nodes can be arranged into an array of n rows and m columns, where any i-th row in the array contains m computing nodes W. i1 ~W im The m computing nodes W i1 ~W im This forms a model partitioning group (i.e., the i-th model partitioning group out of n model partitioning groups), where each computing node W in this group... ik The parameters θ corresponding to the model (e.g., LLM) k Furthermore, these m computing nodes maintain a single copy of all the parameters of the model. It can be understood that the parameter θ... k This is used to represent some parameters of the model, which may include one or more parameters. In one implementation, the model may include L modules (module 1 to module L), and each computing node W ik The corresponding partial parameters θ k It may include partial parameters θ of each of the L modules. k1 ~θ kL .

[0058] In each model training iteration, the m compute nodes within a single model partition group can locally update the model parameter values ​​through joint training. In one implementation, the m compute nodes in a single model partition group can be m GPUs within a single computing device, thereby reducing the communication time between the m compute nodes during model training. It is understood that the m compute nodes in a single model partition group are not limited to m GPUs within a single computing device; for example, they can be m computing devices within a local area network, where the communication speed between these m computing devices is faster than the communication speed between compute nodes in different model partition groups.

[0059] The j-th column of the array includes n computing nodes W 1j ~W nj These n computation nodes are the computation nodes corresponding to the same model parameters in the n model partitioning groups, forming a model replication group (i.e., the j-th model replication group out of m model replication groups). For example, as shown in Figure 1, W 11 ~W n1 Corresponding to parameter θ1, W 12 ~W n2 Corresponding to parameter θ2. Similar to the Local SGD described above, each model replication group can periodically synchronize parameters between the various computing nodes within the same group to reduce the communication overhead caused by parameter synchronization.

[0060] In other words, the system in the embodiments of this specification adopts a hierarchical distribution structure of a two-dimensional device network. By adopting this structure, the m computing nodes in a single model partition group store a portion of the model's parameters, thereby solving the problem of storage overflow of a single computing node. In addition, during model training, the n computing nodes in a single model replication group only need to synchronize their corresponding partial parameters, rather than synchronizing all the model's parameters between computing nodes, reducing the communication load of a single computing node. Furthermore, multiple model replication groups synchronize the model parameters corresponding to each group in parallel, improving the efficiency of parameter synchronization.

[0061] The model training process in the embodiments of this specification may include two stages: an initial warm-up stage and a periodic synchronous training stage. In the warm-up stage, the number of warm-up steps t can be preset. w In the first t of model training w In each training iteration, the gradients of the model parameters are synchronized across the computational nodes within each model replication group, thus making the initial training phase more stable. This is followed by a periodic synchronous training phase. The training process for these two phases will be described below with reference to the accompanying diagrams.

[0062] Figure 2 is a schematic diagram of the warm-up phase of distributed model training in an embodiment of this specification. Figure 2 only shows the computation node W in the first model partitioning group.11 and computing node W 12 and the computation node W in the second model partitioning group. 21 and computing node W 22 As shown in Figure 1, the computation node W 11 and computing node W 21 In the first model replication group.

[0063] As shown in Figure 2, each model training process can include three parts: forward propagation, backward propagation, and local parameter update.

[0064] In one training session, the computation node W is used. 11 For example, in the forward propagation part, the computation node W 11 Processing proceeds sequentially from module 1 to module L. Specifically, for module 1, the computation node W... 11 First, gather the other parameters (θ) from module 1 of the model from the other computing nodes in the model partitioning group (i.e., the first model partitioning group). i1 The value of (i≠1) is obtained to acquire all parameters (θ) of module 1. i1 The value of i = [1, m] will be used to compute node W. 11 The corresponding training samples are input into module 1. Based on all the parameters of module 1, these training samples are calculated to obtain the output of module 1. The parameters (θ) of other calculation nodes in the model group are then compared with those of the training samples. i1 Release (i ≠ 1), retain only the compute node W. 11 The corresponding parameter θ of module 1 11 To save storage space (e.g., video memory), the output of module 1 is fed into module 2. Similar to module 1, in module 2, the compute node W... 11 The other parameters of module 2 are aggregated from other computing nodes in the model group to obtain all parameters of module 2. Based on the parameters of module 2, the output of module 1 is calculated to obtain the output of module 2. The parameters corresponding to other computing nodes are released, and the output of module 2 is input to module 3. This process continues until the calculation of module L is completed, and the output of module L, which is the output of the complete model, is obtained.

[0065] Then, the backpropagation part of this training process begins. Referring to Figure 2, in the backpropagation part, node W is computed. 11 Processing proceeds sequentially from module L to module 1. Specifically, in module L, the computation node W... 11 First, the values ​​of other parameters (θ) of module L in the module partitioning group are aggregated from other computing nodes in the model. iL (i≠1), thus obtaining all parameters (θ) of module L. iLThe values ​​of i = [1, m] are used to perform backward gradient calculations based on all parameters of module L, the output of the above model, and the labels in the above training samples, to obtain the parameters θ in each module L. L The gradient of node W is then calculated. 11 Other computing nodes in the model group are clustered together with computing node W. 11 The corresponding parameter θ 1L The gradient is used to obtain the parameter θ. 1L Given m gradient values, perform a reduction calculation (e.g., average) on these m gradient values ​​to obtain the updated parameters θ. 1L The gradient value. Here, reduction is an operation used to combine multiple elements in a set into a single result through a certain operation. The reduction operation in the embodiments of this specification may include operations such as summation, product, average, maximum, and minimum, etc., and the embodiments of this specification do not limit this.

[0066] Next, compute node W 11 With other compute nodes W in its model replication group k1 ,k=[2,m]synchronization parameter θ 1L The gradient value, i.e., the parameter θ calculated for each computation node in the model replication group. 1L The gradient value is reduced to obtain the final parameter θ. 1L The gradient value of node W is then calculated. 11 Release the model parameter values ​​corresponding to other computation nodes in this model partition group, and release the parameter θ calculated by module L. 1L The gradient value is input to module L-1. In module L-1, the calculation node W... 11 Similarly, the parameters are gathered, backward computation is performed, gradients are reduced, gradients are synchronized, and parameters are released, before proceeding to module L-2. This process continues until module 1 is completed.

[0067] After completing the backpropagation section described above, compute node W 11 Update parameters locally, that is, use the parameters θ calculated for each module. 11 ~θ 1L Update parameters θ using the gradient values ​​of each gradient. 11 ~θ 1L The value of .

[0068] As mentioned above, each computing node performed t w After the initial warm-up phase of training, the periodic synchronous training phase can begin.

[0069] Figure 3 is a flowchart of the model training method based on a distributed system according to an embodiment of this specification. This process corresponds to the periodic synchronous training stage described above. For ease of illustration, only the computation node W in the first model partitioning group is shown in the figure. 11 and computing node W 12 and the computation node W in the second model partitioning group. 21 and computing node W 22 As shown in Figure 1, the computation node W 11 and computing node W 21 In the first model replication group. It can be understood that in the embodiment shown in Figure 3, the model partitioning group may include two or more computing nodes, and the model replication group may include two or more computing nodes. The method shown in Figure 3 can also be applied to the case where the model partitioning group includes multiple computing nodes and / or the model replication group includes multiple computing nodes.

[0070] As shown in Figure 3, in step S301, the m computing nodes in each model group jointly perform multiple training sessions.

[0071] For n model partitions, training is performed multiple times within each partition to update model parameters locally. For example, in the first model partition, there are m computing nodes (including computing node W). 11 and computing node W 12 The second model is divided into m computational nodes (including computational node W) and trained multiple times in a joint manner. 21 and computing node W 22 They will conduct multiple joint training sessions, and so on.

[0072] Figure 4 is a schematic diagram of the periodic synchronous training phase in the embodiments of this specification.

[0073] As shown in Figure 4, the difference between this periodic synchronization phase and the aforementioned warm-up phase includes that the computational nodes in the same model replication group no longer synchronize gradients during backpropagation, but instead periodically synchronize parameters before the forward computation of each module during forward propagation. Similarly, compared to Local SGD, in this embodiment, each computational node in the model replication group only needs to synchronize a portion of the parameters corresponding to that model replication group, rather than synchronizing all the parameters of the model. This reduces the communication load of each computational node, and the parallel synchronization of model parameters by multiple model replication groups accelerates the synchronization speed of model parameters.

[0074] Specifically, in a single model training iteration across all computational nodes within a single model partitioning group, the operations for each model module during forward propagation include parameter aggregation, forward computation, and parameter release—excluding synchronization parameter operations. Figure 4 shows the "synchronization parameters" in forward propagation as dashed lines, indicating that this operation is not executed in every training iteration but only periodically. During backpropagation, the operations for each model module include parameter aggregation, backward computation, gradient reduction, and gradient release. After backpropagation, each computational node locally updates its corresponding portion of the model parameters.

[0075] In one of the multiple joint training sessions, the computation node W is used. 11 For example, in the forward propagation part, the computation node W 11 Processing proceeds sequentially from module 1 to module L. Specifically, for module 1, the computation node W... 11 First, gather the other parameters (θ) from module 1 of the model from the other computing nodes in the model partitioning group (i.e., the first model partitioning group). i1 The value of (i≠1) is obtained to acquire all parameters (θ) of module 1. i1 The value of i = [1, m] will be used to compute node W. 11 The corresponding training samples are input into module 1. Based on all the parameters of module 1, these training samples are calculated to obtain the output of module 1. The parameters (θ) of other calculation nodes in the model group are then compared with those of the training samples. i1 Release (i ≠ 1), retain only the compute node W. 11 The corresponding parameter θ of module 1 11 To save storage space (e.g., video memory), the output of module 1 is fed into module 2. Similar to module 1, in module 2, the compute node W... 11 The other parameters of module 2 are aggregated from other computing nodes in the model group to obtain all parameters of module 2. Based on the parameters of module 2, the output of module 1 is calculated to obtain the output of module 2. The parameters corresponding to other computing nodes are released, and the output of module 2 is input to module 3. This process continues until the calculation of module L is completed, and the output of module L, which is the output of the complete model, is obtained.

[0076] Then, the backpropagation part of this training process begins. Referring to Figure 4, in the backpropagation part, node W is computed. 11 Processing proceeds sequentially from module L to module 1. Specifically, in module L, the computation node W... 11 First, the values ​​of other parameters (θ) of module L in the module partitioning group are aggregated from other computing nodes in the model. iL (i≠1), thus obtaining all parameters (θ) of module L. iLThe values ​​of i = [1, m] are used to perform backward gradient calculations based on all parameters of module L, the output of the above model, and the labels in the above training samples, to obtain the parameters θ in each module L. L The gradient of node W is then calculated. 11 Other computing nodes in the model group are clustered together with computing node W. 11 The corresponding parameter θ 1L The gradient is used to obtain the parameter θ. 1L Given m gradient values, perform a reduction calculation (e.g., average) on these m gradient values ​​to obtain the updated parameters θ. 1L The gradient value.

[0077] Next, compute node W 11 Release the model parameter values ​​corresponding to other computation nodes in this model partition group, and release the parameter θ calculated by module L. 1L The gradient value is input to module L-1. In module L-1, the calculation node W... 11 Similarly, the parameters are gathered, backward computation is performed, gradients are reduced, and parameters are released, then the process proceeds to module L-2. This process continues until module 1 is completed.

[0078] After completing the backpropagation section described above, compute node W 11 Update parameters locally, that is, use the parameters θ calculated for each module. 11 ~θ 1L Update parameters θ using the gradient values ​​of each gradient. 11 ~θ 1L The value of .

[0079] The number of times each model group is jointly trained can be set in various ways. Figure 5 is a schematic diagram of setting the number of joint training sessions in an embodiment of this specification. As shown in the upper part of Figure 5, in one implementation, the number of joint training sessions t can be preset. s For example, t s =2, where each computing node in a single model partition group trains twice after the end of the last synchronization, and then performs the next synchronization. In this case, the fast nodes in a single model partition group need to spend more time (as shown by "Idle" at the top of Figure 5) waiting for the slow nodes to complete their two training sessions, and then all nodes in the single model partition group synchronize together.

[0080] As shown in the lower part of Figure 5, in another implementation, a pre-set joint training time interval T can be used. Each computing node in a single model partition group will synchronize again after a training duration of T since the end of the previous synchronization. In this case, the fast nodes in a single model partition group require less time (as shown by the "idle" period in the lower part of Figure 5) to wait for the slow nodes to complete training, compared to the upper part of Figure 5, and then all nodes in the single model partition group synchronize together.

[0081] In step S303, after multiple training iterations, each computation node in each model partition group receives its corresponding updated parameter values. For example, computation node W in the first model partition group... 11 The value of the corresponding parameter θ1, V1, is obtained, and the computation node W in the second model partition group is obtained. 21 The value of the corresponding parameter θ1, V2, is obtained.

[0082] In step S305, each model replication group determines the synchronization value of the parameter based on n values ​​of its corresponding parameter.

[0083] As shown in Figure 1, each computation node in the model replication group corresponds to the same model parameters. For example, the n computation nodes W in the first model replication... 11 ~W n1 All correspond to parameter θ1. The n computation nodes respectively calculate the values ​​V1 to Vn of parameter θ1 in the above steps, and the n computation nodes W... 11 ~W n1 The values ​​of parameter θ1, V1 to Vn, can be reduced to obtain the synchronization value of parameter θ1.

[0084] Figure 6 is a schematic diagram of a process for reducing parameter values ​​in an embodiment of this specification.

[0085] As shown in Figure 6, assume that the first model replication group includes computing node W. 11 ~W 41 The values ​​V1 to V4 of parameter θ1 are calculated respectively. The current synchronization value V of parameter θ1 (i.e. the value of θ1 determined in the last synchronization) is subtracted from the parameter values ​​V1 to V4 respectively to obtain the virtual gradients g1 to g4.

[0086] Outliers in the virtual gradients g1 to g4 can be identified. In one implementation, the norm of the virtual gradient can be used as a metric. Specifically, the z-test method can be used for statistical analysis, letting G... i =||g i ||2 represents node W i1 If the virtual gradient norm is given, then the corresponding z-score is:

[0087] in, Representing G i The average value, σ represents G. i The standard deviation. When node W i1 Satisfy z i Points greater than δ are identified as outliers, where δ is a manually set threshold. During training, σ is updated using an exponential moving average. and σ':

[0088] Where α is the weighting factor. During this process, if all computation nodes in the model replication group are determined to be abnormal nodes, all parameters are rolled back to the parameters V of the last synchronization.

[0089] Referring back to Figure 6, after removing the abnormal nodes in the model replication group, considering that large virtual gradients might affect the overall orientation, the remaining virtual gradients can be weighted based on their norm. For each normal node, the weights can be set as follows:

[0090] Then, the virtual gradients are weighted and summed to obtain the synchronized virtual gradient g: g = ∑w i g i

[0091] Alternatively, the update step size can be limited using a gradient pruning strategy, and the pruning factor β can be set as follows:

[0092] Where G = ||g||2, represents the virtual gradient norm of synchronization. The threshold is set to ε, where ε is a small positive number. The virtual gradient can then be clipped as follows:

[0093] Then, based on virtual gradients Update the current parameter synchronization value V to obtain the updated synchronization value.

[0094] In step S307, the computing nodes in each model replication group synchronize their parameter values ​​to synchronized values.

[0095] Specifically, referring to Figure 6, the computing nodes in the first model replication group calculate the synchronized value of parameter θ1. Then, the values ​​of parameter θ1 are synchronized to the synchronized values.

[0096] In one implementation, each compute node W in the model replication group 11 ~Wn1 After completing multiple joint training sessions within the aforementioned model group, the synchronization value can be calculated and the parameters updated to that synchronization value.

[0097] In another implementation, the individual compute nodes W in the model replication group 11 ~W n1 The parameters θ of each module can be calculated during the forward propagation of the next training iteration after the completion of the aforementioned joint training. 11 ~θ 1L The synchronization value is calculated, and the parameter values ​​of each module are updated to this synchronization value.

[0098] Specifically, as shown in Figure 4, in training that includes parameter synchronization operations, parameter synchronization is first performed within each model replication group during forward propagation. Taking the first model replication group as an example, each computing node W... 11 ~W n1 In module 1, the parameter θ 11 Multiple parameter values ​​V 11 ~V n1 The parameter θ is obtained by performing reduction calculations. 11 Synchronization value Then calculate node W. 11 ~W n1 Based on this synchronization value Perform the subsequent parameter aggregation, forward computation, and parameter release operations from Module 1. Similarly, each computation node W... 11 ~W n1 In module 2, the parameter θ 12 Multiple parameter values ​​V 12 ~V n2 The parameter θ is obtained by performing reduction calculations. 12 Synchronization value Then calculate node W. 11 ~W n1 Based on this synchronization value Perform the subsequent parameter aggregation, forward computation, and parameter release operations in Module 2. Continue in this manner until the parameters for synchronization, aggregation, forward computation, and release are completed for Module L.

[0099] Figure 7 is a schematic diagram of the synchronization parameter process during forward propagation in an embodiment of this specification.

[0100] As shown in Figure 7, the computing node W in the first model replication group 11 ~W n1 For example (Figure 7 shows the computing node W) 11 and computing node W 21 In step S701, each computation node in the model replication group, during the forward propagation of module 1, is based on parameter θ.11 Multiple values ​​determine the parameter θ 11 Synchronization value

[0101] In step S703, each computation node in the model replication group will transfer the parameter θ 11 The value is synchronized with the value

[0102] In step S705, in parallel with the forward propagation in module 1 in step S707, the computation node W in the model replication group... 11 ~W n1 Based on the parameter θ corresponding to module 2 12 Multiple values ​​determine the parameter θ 12 Synchronization value

[0103] In step S709, each computation node in the model replication group will transfer the parameter θ 12 The value is synchronized with the value

[0104] As shown in Figure 7, each computing node in the model replication group can perform synchronization parameter operations in module 2 while processing the forward computation of module 1, thereby saving the time cost of synchronization parameters and further accelerating the model training process.

[0105] Figure 8 is an architecture diagram of a computing node provided in an embodiment of this specification. This computing node belongs to a distributed system, which includes n groups of computing nodes. Each group of computing nodes includes m computing nodes, and the m computing nodes in each group correspond to the m parameter sets included in the target model. The computing nodes include:

[0106] Training unit 81 is used to perform multiple training sessions in conjunction with other computing nodes in the first group of computing nodes to obtain the first value of the first parameter;

[0107] The determining unit 82 is used to determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter by the n-1 first computing nodes in the other group corresponding to the first parameter.

[0108] Synchronization unit 83 is used to synchronize the value of the first parameter to the second value.

[0109] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0110] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0111] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude the possibility that, with the future development of computer technology, the computer implementing the functions of the above embodiments can be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0112] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the elements is not excluded. For example, the use of terms such as "first," "second," etc., is to denote names and does not indicate any particular order.

[0113] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0117] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0118] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0119] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0120] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0122] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0123] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of these embodiments. Various modifications and variations can be made to these embodiments by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A model training method based on a distributed system, wherein the distributed system includes n groups of computing nodes, each group of computing nodes includes m computing nodes, and the m computing nodes in each group correspond to m parameter sets included in the target model, the method comprising: The m computing nodes in each group of computing nodes jointly perform multiple training sessions, so that each computing node in each group obtains the first value of each parameter in its corresponding parameter set; The n first computing nodes in the n groups corresponding to the first parameter determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter; The n first computing nodes will synchronize the value of the first parameter to the second value.

2. The method according to claim 1, wherein the m computing nodes in each group jointly perform multiple training iterations, comprising: In each training session, each computing node in each group obtains the values ​​of other parameters of the target model from the other computing nodes in the group, thus obtaining the values ​​of all parameters of the target model. Based on the values ​​of all parameters of the target model, the values ​​of each parameter corresponding to the computing node are updated.

3. The method according to claim 1, wherein the target model is divided into a plurality of consecutively arranged modules, and the first parameter is a parameter in the first module among the plurality of modules. The n first computing nodes in the n groups corresponding to the first parameter determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter, including: Each of the n first computing nodes, during the forward propagation process of the next training iteration of its multiple training iterations, determines a second value corresponding to the first parameter based on the n first values ​​of the first parameter.

4. The method according to claim 3, further comprising: Each of the first computing nodes performs the calculation of the first module in the forward propagation based on the second value of the first parameter, and sends the second value to other computing nodes in the same group for use by other computing nodes in the calculation of the first module in the forward propagation.

5. The method according to claim 3, wherein each of the n first computing nodes, during the forward propagation process of the next training iteration of the multiple training iterations, determines a second value corresponding to the first parameter based on n first values ​​of the first parameter, comprising: The first computing node determines the second value corresponding to the first parameter in parallel with the calculation of the second module in the forward propagation process, and the second module is the preceding module of the first module.

6. The method according to claim 1, wherein the n first computing nodes in the n groups corresponding to the first parameter determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter, comprising: The n first computing nodes calculate the difference between each first value and the current synchronization value of the first parameter, determine the outlier among the n first values ​​based on the difference, and determine the second value based on the multiple first values ​​other than the outlier among the n first values.

7. The method according to claim 6, wherein determining the second value based on a plurality of first values ​​other than the outlier among the n first values ​​comprises: The weight of each difference is determined based on the multiple differences corresponding to the multiple first values. A third value is calculated based on the multiple differences and the weight of each difference. The second value is calculated based on the synchronization value and the third value.

8. The method of claim 7, wherein calculating the third value based on the plurality of differences and the weights of the differences comprises: A fourth value is calculated based on the multiple differences and their respective weights. The fourth value is then processed based on a preset threshold and the multiple differences to obtain the third value.

9. The method of claim 1, the m computing nodes in each group jointly performing multiple training, comprising: The number of times each group of computing nodes performs joint training is determined based on a preset first time interval.

10. The method of claim 1, wherein, The communication speed between m computing nodes in each group is faster than the communication speed between computing nodes in different groups.

11. The method of claim 1, further comprising a warm-up training phase, the warm-up training phase including backpropagation, the method further comprising: During the warm-up training phase, the n first computing nodes corresponding to the first parameter in the n groups synchronize the gradients of the model parameters during the backpropagation.

12. A model training method based on a distributed system, wherein the distributed system includes n groups of computing nodes, each group of computing nodes includes m computing nodes, and the m computing nodes in each group correspond to m parameter sets included in the target model, the method being executed by a second computing node in the first group of computing nodes, the second computing node corresponding to a first parameter of the model, the method comprising: The first parameter is obtained by performing multiple training sessions in conjunction with other computing nodes in the first group of computing nodes. Based on the n first values ​​of the first parameter, the n-1 first computing nodes in the other groups corresponding to the first parameter determine the second value corresponding to the first parameter; Synchronize the value of the first parameter to the second value.

13. A second computing node in a distributed system, the distributed system comprising n groups of computing nodes, each group comprising m computing nodes, the m computing nodes in each group corresponding to m parameter sets included in a target model, the second computing node corresponding to a first parameter of the model, the second computing node comprising: The training unit is used to perform multiple training sessions in conjunction with other computing nodes in the first group of computing nodes to obtain the first value of the first parameter. The determining unit is used to determine the second value corresponding to the first parameter based on n first values ​​of the first parameter by the n-1 first computing nodes in the other group corresponding to the first parameter. A synchronization unit is used to synchronize the value of the first parameter to the second value.

14. A distributed system, the distributed system comprising n groups of computing nodes, each group comprising m computing nodes, wherein the m computing nodes in each group correspond to m parameter sets included in the target model, respectively. The m computing nodes in each group are used to jointly perform multiple training sessions, so that each computing node in each group obtains the first value of each parameter in its corresponding parameter set. The n first computing nodes in the n groups corresponding to the first parameter are used to determine the second value corresponding to the first parameter based on the n first values ​​of the first parameter; The n first computing nodes are further configured to respectively synchronize the value of the first parameter to the second value.

15. A computing device comprising a memory having executable code stored therein and a processor, wherein the processor executes the executable code to implement the method of claim 12.

Images