Model training method and apparatus, and device, medium and product

Abstract

A model training method, which is executed by a computer device. The method comprises: acquiring an integrated network model to be trained, wherein the integrated network model comprises M network sub-models, M being an integer greater than 1 (S101); performing strategy decision processing on each network sub-model, so as to obtain a parallel strategy of each network sub-model, wherein the parallel strategy of any network sub-model is used for indicating a parallel training task of the network sub-model during model training and an execution mode of the parallel training task (S102); on the basis of the parallel strategies of the M network sub-models, constructing a training system for the integrated network model, wherein the training system comprises M communication groups, one communication group corresponds to one network sub-model, and each communication group is obtained by performing construction according to the parallel strategy of the corresponding network sub-model (S103); and during the process of performing model training on the integrated network model, invoking the M communication groups in the training system to respectively execute the parallel training tasks of the corresponding network sub-models according to the execution modes indicated by the parallel strategies of the corresponding network sub-models, so as to obtain a trained network model (S104).

Description

A model training method, apparatus and equipment, medium, and product

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on November 5, 2024, application number 202411563583.9, entitled "A model training method, apparatus and equipment, medium and product", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and more particularly to the field of artificial intelligence technology, specifically to a model training method, a model training device, a computer equipment, a computer-readable storage medium, and a computer program product. Background Technology

[0004] An ensemble network model is a network model created by combining multiple different individual network models. An individual network model refers to a standalone neural network model, typically composed of one or more neural network layers, used to perform a specific task or learn specific data features. Parallel strategies are usually designed for ensemble network models. These strategies instruct the parallel training tasks and their execution methods, thereby accelerating the model training process. However, training ensemble network models is still limited by computational efficiency and scalability. Therefore, improving the computational efficiency and scalability of ensemble network training, and further accelerating the model training process, has become a key focus for engineers. Summary of the Invention

[0005] This application provides a model training method, apparatus and equipment, medium and product.

[0006] On the one hand, embodiments of this application provide a model training method, the method comprising:

[0007] Obtain the ensemble network model to be trained. The ensemble network model includes M sub-network models, where M is an integer greater than 1.

[0008] Each sub-network model is processed for policy decision-making to obtain the parallel policy of each sub-network model; the parallel policy of any sub-network model is used to indicate the parallel training task of the sub-network model during the model training process and the execution method of the parallel training task.

[0009] Based on the parallel strategy of M sub-network models, a training system for an integrated network model is constructed. The training system includes M communication groups; each communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model.

[0010] During the training of the ensemble network model, M communication groups in the training system are invoked to execute the parallel training tasks of the corresponding sub-network models according to the execution method indicated by the parallel strategy of the corresponding sub-network model, so as to obtain the trained network model.

[0011] On the other hand, embodiments of this application provide a model training apparatus, which includes:

[0012] The acquisition unit is used to acquire the ensemble network model to be trained. The ensemble network model includes M sub-network models, where M is an integer greater than 1.

[0013] The processing unit is used to perform policy decision processing on each sub-network model to obtain the parallel policy of each sub-network model; the parallel policy of any sub-network model is used to indicate the parallel training task of the sub-network model in the model training process and the execution method of the parallel training task.

[0014] The processing unit is also used to construct a training system for an integrated network model based on a parallel strategy of M sub-network models. The training system includes M communication groups; each communication group corresponds to a sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model.

[0015] The processing unit is also used to, during the model training process of the integrated network model, call M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network models according to the execution mode indicated by the parallel strategy of the corresponding sub-network models, so as to obtain the trained network model.

[0016] On the other hand, embodiments of this application provide a computer device, which includes:

[0017] A processor is used to load and execute computer programs;

[0018] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described model training method.

[0019] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described model training method.

[0020] On the other hand, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the above-described model training method.

[0021] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0023] Figure 1 is a schematic diagram of hardware resource utilization provided in an embodiment of this application;

[0024] Figure 2 is a schematic diagram of an example of an assembly line provided in an embodiment of this application;

[0025] Figure 3 is a schematic diagram of the structure of a training system for a large-scale speech model provided in an embodiment of this application;

[0026] Figure 4 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0027] Figure 5 is a flowchart illustrating a model training method provided in an embodiment of this application;

[0028] Figure 6 is a flowchart illustrating a strategy decision processing method provided in an embodiment of this application;

[0029] Figure 7 is a flowchart illustrating a training system construction method provided in an embodiment of this application;

[0030] Figure 8 is a partial structural schematic diagram of a training system provided in an embodiment of this application;

[0031] Figure 9 is a schematic diagram of a model training device provided in an embodiment of this application;

[0032] Figure 10 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] A Multimodal Large Language Model (MLLM) is an ensemble network model consisting of a modal encoder, a large language model (LLM), and a modal decoder. The modal encoder converts multimodal information (such as text, images, audio, and video) into text representations that the LLM can understand. The LLM learns language patterns, contextual information, and semantic relationships within the text representations and generates language-capable text sequences (text information that expresses linguistic information and includes grammatical structure and semantic information). The modal decoder translates the text sequences generated by the LLM back into the corresponding modal information.

[0035] Multimodal large-scale language models have shown great application potential in many fields. As an important part of the multimodal field, speech modality has received widespread attention. This application takes a large-scale speech model containing an audio encoder (for converting audio into an LLM-understandable text representation), an LLM, and an audio decoder (for translating the text sequence generated by the LLM back into the audio modality) as an example to illustrate the technical solution (referred to as the model training scheme) provided in this application.

[0036] I. This application involves parallel strategies:

[0037] Parallel strategies refer to strategies that utilize parallel computing to accelerate the model training process. Examples of parallel strategies include, but are not limited to: data parallel strategies, model parallel strategies, and hybrid parallel strategies.

[0038] ① Data Parallel Strategy: This involves dividing the training data into multiple micro-batches, each containing a micro-batch size (MBS) of training samples. These micro-batches are allocated to different training devices, each deploying a complete model instance. Each model instance on each device can be trained using its assigned micro-batches. Model parameters are updated by summing or averaging the gradients obtained from training on all devices. This approach distributes the entire model across multiple training devices, allowing each device to handle different micro-batches, thus effectively utilizing the hardware resources of multiple devices and accelerating the model training process.

[0039] Microbatch size (MBS) refers to the number of training samples processed by each training device in each training iteration. For example, MBS = 5 means that each training device can be allocated 5 training samples in each training iteration, and each training device can simultaneously input the 5 allocated training samples into its own model instance for processing. A training iteration refers to a complete parameter update process that a neural network model undergoes during model training. In a training iteration, the predicted results of the neural network model are typically compared with the actual results, the loss function value (e.g., mean squared error) is calculated, the gradient is calculated using the loss function value, and optimization algorithms such as gradient descent are used to update the model parameters.

[0040] A neural network model is a machine learning model inspired by the neural network structure of the human brain. It consists of multiple neurons (nodes) forming multiple neural network layers (input layer, hidden layer, and output layer), interconnected by weights. Neural network models learn the relationship between input data and corresponding output data, thus enabling them to make predictions or classifications without explicit rules. Both ensemble network models and the individual sub-networks that make up an ensemble network model are considered neural network models.

[0041] In data parallelism strategies, complete model instances are distributed across multiple training devices. Data parallelism (DP) reflects the number of training devices with deployed complete model instances. For example, DP = 3 indicates that complete model instances are deployed on 3 training devices. Neural network models are typically defined using classes, which contain the structure and parameters of the neural network model. A class is a user-defined reference data type used for information encapsulation. Each class contains data descriptions and a set of functions for manipulating data or passing messages. An instance of a class is an object created based on the class definition. When the class of a neural network model is initialized (or instantiated, referring to the process of creating an instance based on the class definition), a model instance of the neural network model is obtained. Obtaining a model instance means that a neural network model is ready for learning and inference, and tasks such as training, testing, and prediction can be performed using the model instance. A complete model instance can be understood as an object that encapsulates the complete structure and parameters of the neural network model.

[0042] Training equipment refers to computing devices used to train neural network models, which may include, but are not limited to, hardware devices such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), and TPUs (Tensor Processing Units). GPUs are generally considered a common training device when training neural network models; this application will use a GPU as an example for illustration.

[0043] ② Model parallelism strategy: This refers to a strategy that places different parts of a neural network model on different training devices for parallel computation during training, in order to accelerate the model training process. Model parallelism strategies include, but are not limited to: pipelined parallelism strategy and tensor parallelism strategy.

[0044] Pipeline parallelism refers to breaking down a neural network model into multiple network modules, distributing these modules across different training devices, and having each device handle the computation of a specific network module, passing the intermediate results to the next device for further processing. This allows each device to perform computations on different network modules in parallel, thus accelerating the overall training process. A network module is a component of a neural network model and may contain a set of related neural network layers. Pipeline parallelism (PP) reflects the degree of decomposition of the neural network model in a pipeline parallelism strategy. For example, PP = 3 indicates that the neural network model has been divided into 3 network modules; PP = 1 indicates that no network modules have been decomposed.

[0045] Tensor parallelism refers to splitting the model parameters (such as weight matrices and bias terms) of a neural network model into multiple tensors and distributing them to different training devices for training. Each training device is responsible for calculating the gradient of its assigned tensor. This allows each device to independently handle the calculation of a portion of the parameters, and the model parameters can be updated by aggregating the gradients calculated by each device. A tensor is a mathematical object representing a multidimensional array; it can contain scalars (0-dimensional tensors), vectors (1-dimensional tensors), matrices (2-dimensional tensors), and arrays of higher dimensions. For example, this embodiment uses the tensor parallelism strategy to split the model parameters of each neural network layer as an example for illustration. For example, a neural network model includes an input layer, a hidden layer, and an output layer. The weight matrix from the input layer to the hidden layer is W1, the bias term from the input layer to the hidden layer is b1, the weight matrix from the hidden layer to the output layer is W2, and the bias term from the hidden layer to the output layer is b2. The weight matrix W1 can be split into W1_1 and W1_2 and assigned to GPU1 and GPU2 respectively. The bias term b1 can be split into b1_1 and b1_2 and assigned to GPU1 and GPU2 respectively. The weight matrix W2 can be split into W2_1 and W2_2 and assigned to GPU1 and GPU2 respectively. The bias term b2 can be split into b2_1 and b2_2 and assigned to GPU1 and GPU2 respectively. In this way, GPU1 can be responsible for the calculation of the tensor: W1_1, b1_1, W2_1, b2_1, and GPU2 can be responsible for the calculation of the tensor: W1_2, b1_2, W2_2, b2_2. Tensor parallelism (TP) reflects the degree to which model parameters are split in a tensor parallelism strategy. For example, tensor parallelism refers to the number of tensors obtained after the model parameters are split. For instance, if the model parameters of each neural network layer are split into 2 tensors, then TP equals 2; when all model parameters remain in their original form and are not split into multiple tensors, TP equals 1.

[0046] ③ Hybrid Parallel Strategy: This refers to a strategy that simultaneously uses data parallelism and model parallelism to maximize the utilization of hardware resources and improve training speed and efficiency. For example, a neural network model can be split into network module 1 and network module 2. The model parameters of network module 1 can be split into tensor 1 and tensor 2, and the model parameters of network module 2 can be split into tensor 3 and tensor 4. Tensor 1 of network module 1 can be deployed on GPU1 and GPU2, tensor 2 of network module 1 on GPU3 and GPU4, tensor 3 of network module 2 on GPU5 and GPU6, and tensor 4 of network module 2 on GPU7 and GPU8. In this way, GPU1, GPU3, GPU5, and GPU7 can implement the functionality of a complete neural network model (i.e., a complete model instance), and GPU2, GPU4, GPU6, and GPU8 can implement the functionality of a complete neural network model (i.e., a complete model instance).

[0047] II. Technical Background of this Application

[0048] Taking the Xeus speech encoder (open source) and the 7B moe LLM (7B represents one billion model parameters and moe represents hybrid expert model) in a large speech model as examples, we analyze the shortcomings of existing model training schemes.

[0049] ① Hardware resource idleness caused by uneven computational load: From a computational load perspective, when DP=1, TP=1, PP=1, and MBS=1 (i.e., a complete model instance is deployed on a training device, and the batch size processed by the training device is 1), the computational load of the Xeus speech encoder is 3.75T FLOPs (i.e., 3.75 trillion floating-point operations), while the computational load of the 7B moe LLM is 180.55T FLOPs (i.e., 180.55 trillion floating-point operations), a difference of 48 times, indicating a very uneven computational load. FLOPs refers to the number of floating-point operations, reflecting the computational load required by the neural network model during training or inference. Under existing model training schemes, the audio encoder and audio decoder are treated as additional layers of the LLM, using a unified parallel strategy. Therefore, the MBS of the audio encoder and audio decoder are limited by the LLM; that is, the audio encoder and audio decoder need to have the same MBS as the LLM, which leads to a significant amount of idle computational resources when training the Xeus speech encoder. Referring to Figure 1, which illustrates hardware resource utilization, box 11 shows the hardware resource utilization during Xeus speech encoder training. It can be seen that the hardware resource utilization is low at this point. This is because the MBS (Mean Size Segmentation) is small for the Xeus speech encoder, resulting in low parallelism during training. This can lead to decreased computational efficiency and slow down the model training process. Hardware resources provide the infrastructure required for computational resources to operate, while computational resources refer to the actual capacity used to perform computational tasks.

[0050] ② Scalability limitations: Treating the audio encoder, audio decoder, and LLM as a single unit means they all use the same hardware resources for training. Because the audio encoder and decoder consume additional hardware resources such as video memory, the LLM may not be able to fully utilize these resources to handle larger-scale tasks or data, thus limiting its scalability. Scalability refers to the ability to effectively adapt to increasing demands or scale.

[0051] ③ Pipeline Bubbles: Treating the audio encoder and decoder as part of the LLM (Liquid Module Model) can lead to uneven partitioning of large speech models, resulting in pipeline bubbles and further reducing computational efficiency. A pipeline bubble refers to a period of idle time in the pipeline caused by certain instructions waiting for the result of the previous instruction, preventing subsequent instructions from executing. See Figure 2 for an example of a pipeline; the large computational load of the LLM results in pipeline bubbles in the audio decoder.

[0052] III. Model training scheme of this application:

[0053] This application can construct a training system for large-scale speech models and use this training system to train large-scale speech models. Referring to Figure 3, which shows the structural diagram of the training system for large-scale speech models, it can be seen that the model training scheme provided in this application can achieve independent decoupling of the parallel strategies of the audio encoder, LLM, and audio decoder. That is, the audio encoder, LLM, and audio decoder can each configure their own parallel strategies, thus cleverly eliminating the parallel dependency between the audio encoder, LLM, and audio decoder during model training. For example, in Figure 3, the parallel strategy of the audio encoder indicates data parallelism = 2, tensor parallelism = 2, pipeline parallelism = 1, and micro-batch size = 6; the parallel strategy of the LLM indicates data parallelism = 4, tensor parallelism = 4, pipeline parallelism = 2, and micro-batch size = 3; and the parallel strategy of the audio decoder indicates data parallelism = 1, tensor parallelism = 1, pipeline parallelism = 1, and micro-batch size = 12. For example, multiple reference parallel strategies can be pre-configured for the audio encoder, and the strategy that minimizes the training time of the audio encoder can be selected as the parallel strategy for the audio encoder. The same applies to the LLM and audio decoder. Computational efficiency generally refers to the ability to complete computational tasks per unit of time; therefore, shorter training time indicates higher computational efficiency. Hardware resource utilization refers to the actual degree to which hardware resources (such as CPU, GPU, TSU, memory, hard disk, network, etc.) are used. Shorter training time usually also means higher hardware resource utilization, because computational efficiency largely depends on the utilization efficiency of hardware resources. Based on the decoupling of the parallel strategies of the audio encoder, LLM, and audio decoder, separate communication groups can be built for the audio encoder, LLM, and audio decoder during the construction of a large-scale speech model training system. Each communication group includes one or more training devices. In this way, during model training, the audio encoder, LLM, and audio decoder will use independent hardware resources, which is beneficial to improving scalability. For example, in Figure 3, the DP indicated by the parallel strategy of the LLM determines that the communication group of the LLM contains 4 layers. The product between TP and PP indicated by the parallel strategy of the LLM determines that each layer contains 8 GPUs3, which is equivalent to each layer implementing the function of a complete LLM, and a total of 4 complete LLMs are included. The same applies to the audio encoder and audio decoder.

[0054] Furthermore, the model training scheme is highly flexible, supporting the use of different types of hardware resources (i.e., training devices) for the audio encoder, LLM, and audio decoder. This allows for precise matching of suitable hardware resources among the three components, further optimizing hardware resource utilization. For example, the audio encoder typically requires lower computing power for audio data compression and encoding, so selecting GPU1 with lower computing power can meet its needs. The audio decoder, on the other hand, requires higher computing power for decoding and data reconstruction, so selecting GPU2 with higher computing power provides better decoding performance. The LLM, however, needs to complete data processing with minimal latency, so selecting GPU3 with the highest computing power provides faster data processing speeds.

[0055] The model training scheme provided in this application can achieve more efficient training of large-scale speech models by using more flexible and fine-grained parallel strategy configuration and more efficient hardware resource allocation, thereby accelerating the model training process of large-scale speech models.

[0056] The following is a brief description of the application scenarios to which the model processing solution provided in this application is applicable. It should be noted that the following scenarios are for illustrative purposes only and are not intended to limit the scope of this application. In specific implementations, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs. Please refer to Figure 4, which is a schematic diagram of an application scenario provided by an embodiment of this application. This application scenario includes a terminal device 41 and a server 42, which can communicate with each other through a communication network.

[0057] The communication network can be a wired network or a wireless network. Therefore, the terminal device 41 and the server 42 can be connected directly or indirectly through wired or wireless communication. For example, the terminal device 41 can be indirectly connected to the server 42 through a wireless access point, or the terminal device 41 can be directly connected to the server 42 through the Internet. This application does not impose any restrictions on this.

[0058] Terminal devices 41 include, but are not limited to, mobile phones, tablets, laptops, desktop computers, smart home appliances, and in-vehicle terminals. Servers 42 can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0059] For example, server 42 may include: training data for training an ensemble network model (such as a large speech model) to be trained, and hardware resources (such as a GPU) for training the ensemble network model to be trained. Thus, server 42 can execute the following model training process according to the model training scheme provided in this application: determine suitable parallel strategies for each of the M sub-network models in the ensemble network model to be trained, and construct a training system for the ensemble network model to be trained based on the parallel strategies of each sub-network model. The training system includes M communication groups, each corresponding to one sub-network model. During the model training of the ensemble network model, the training data is used, and the M communication groups are invoked to execute the parallel training tasks of the corresponding sub-network models according to the execution methods indicated by the parallel strategies of the corresponding sub-network models, so as to obtain the trained network model.

[0060] In one feasible embodiment, a training framework may be deployed on server 42. A training framework is a software tool or platform for training machine learning models (such as neural network models). The training framework provides a series of tools and interfaces to facilitate developers in designing, training, optimizing, and deploying various machine learning models. The model training process of the model training scheme provided in this application embodiment can be encapsulated into a target functional module within the training framework. The ensemble network model to be trained and the training data used to train the ensemble network model can be uploaded to the training framework. The training framework then calls the target functional module to execute the model training process of the model training scheme provided in this application to obtain the trained network model.

[0061] Optionally, after training the ensemble network model to be trained, the model training scheme based on the embodiments of this application can be directly deployed on the server 42 or the terminal device 41 to provide model services. In one embodiment, the trained network model can be directly deployed on the server 42 to provide model services to the terminal device 41.

[0062] Figure 4 is just an example. In reality, the number of terminal devices 41 and servers 42 is unlimited. For example, there may be multiple servers 42, and each server 42 may include GPUs of different device types.

[0063] In this application embodiment, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations. Obtaining personal information requires the knowledge or consent of the individual (or a legal basis for information acquisition), and subsequent data use and processing should be carried out within the scope of laws, regulations, and the authorization of the personal information subject. For example, when this application embodiment is applied to specific products or technologies, such as obtaining training data and the ensemble network model to be trained, permission or consent from the creators of the training data and the ensemble network model to be trained is required. Furthermore, the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0064] The specific embodiments of the model training scheme provided in this application are described below.

[0065] Please refer to Figure 5, which is a flowchart illustrating a model training method provided in an embodiment of this application. This model training method can be executed by a model training device, such as server 42 in Figure 4. The model training method mainly includes, but is not limited to, the following steps S101 to S104:

[0066] S101. Obtain the ensemble network model to be trained. The ensemble network model includes M sub-network models, where M is an integer greater than 1.

[0067] The ensemble network model to be trained is an ensemble network model composed of M sub-network models, where each sub-network model can be a single network model. For example, the ensemble network model to be trained can be a multimodal large-scale language model, where the modal encoder, LLM, and modal decoder are three sub-network models included in the multimodal large-scale language model. This application does not limit the structure and function of the ensemble network model to be trained; for example, the ensemble network model to be trained can be any network model such as a multimodal large-scale language model or a diffusion model (an image generation model). For example, the ensemble network model to be trained can be determined according to the actual model training requirements. For instance, when the server 42 receives a model training instruction, it can determine the ensemble network model specified in the model training instruction as the ensemble network model to be trained.

[0068] S102. Perform policy decision processing on each sub-network model to obtain the parallel policy of each sub-network model; the parallel policy of any sub-network model is used to indicate the parallel training task of the sub-network model in the model training process and the execution method of the parallel training task.

[0069] In this embodiment of the application, any sub-network model is represented as sub-network model M. i , where i is an integer less than or equal to M. The following discussion uses the sub-network model M... i For example, for the sub-network model M iThe parallel strategy will be explained. Sub-network model M i The parallel strategy refers to: during the model training process (i.e., the process of training the ensemble network model to be trained), the sub-network models M are parallelized. i The strategy involves distributing computational tasks across multiple computing resources for parallel processing. Subnetwork model M i The parallel strategy can be a data parallel strategy, a model parallel strategy, or a hybrid parallel strategy. The model parallel strategy can include one or both of pipeline parallel strategies and tensor parallel strategies. Subnetwork model M i The parallel strategy can be used to instruct the sub-network model M i The parallel training task in the model training process, the execution method of the parallel training task, and the sub-network model M i Parallelism parameters. Parallelism parameters include any one or more of the following: data parallelism parameters, model parallelism parameters; data parallelism parameters include one or both of micro-batch size and data parallelism; model parallelism parameters include one or both of pipeline parallelism and tensor parallelism.

[0070] In one implementation, when the subnetwork model M i When the parallel strategy includes a data parallel strategy, the sub-network model M i The parallel strategy indicates that the parallel training tasks include data parallel tasks. Data parallel tasks refer to dividing the training data into multiple micro-batches and simultaneously running computational tasks on these micro-batches across multiple training devices. Sub-network model M i The data parallelism strategy can indicate how data parallel tasks are executed and the data parallelism parameters. For example, when the subnetwork model M... i The data parallelism strategy is: to divide the sub-network model M i When deployed on 3 training devices, with each device receiving 5 training samples per training iteration, the subnetwork model M... i The data parallel strategy indicates that the data parallel task is executed in the following way: the sub-network model M... i The training model is deployed to three training devices, and in each training iteration, five training samples are allocated to each of the three devices; sub-network model M i The data parallelism strategy indicates a micro-batch size of 5 and a data parallelism of 3.

[0071] In one implementation, when the subnetwork model M i When the parallel strategy includes pipelined parallel strategy, the subnetwork model M i The parallel strategy indicates that the parallel training tasks include pipelined parallel tasks. Pipeline parallel tasks refer to training sub-network models M... iThe network is broken down into multiple modules, which are then assigned to multiple training devices. These devices are connected in the order of execution of the network modules, forming a pipeline structure. In this pipeline structure, each training device performs the computational task of a specific network module. Subnetwork model M i The pipelined parallelism strategy can indicate the execution method and degree of pipelined parallelism for pipelined parallel tasks. For example, when the subnetwork model M... i The pipelined parallel strategy is: to divide the sub-network model M i Neural network layer 1 and neural network layer 2 are deployed on training device 1, and the sub-network model M is used. i When neural network layer 3 is deployed to training device 2, subnetwork model M i The pipelined parallel strategy indicates that the pipelined parallel task is executed as follows: training samples are input into training device 1, training device 1 collects the intermediate results generated by neural network layer 1 and neural network layer 2, and sends the intermediate results to neural network layer 3 in training device 2 for processing; sub-network model M i The pipeline parallelism strategy indicates a pipeline parallelism of 2.

[0072] In one implementation, when the subnetwork model M i When the parallel strategy includes tensor parallel strategy, the subnetwork model M i The parallel strategy indicates that the parallel training tasks include tensor parallel tasks. Tensor parallel tasks can refer to training sub-network models M... i The model parameters for each layer of the neural network are split into multiple tensors, and these tensors are assigned to multiple training devices, which are responsible for performing computational tasks on specific tensors. Sub-network model M i The tensor parallelism strategy can indicate the execution method and degree of tensor parallelism of a tensor parallelism task. For example, when the subnetwork model M... i The tensor parallel strategy is: to use the sub-network model M i The model parameters of layer 1 in the neural network are split into tensor 1 and tensor 2, and the sub-network model M is divided into tensor 1 and tensor 2. i The model parameters of layer 2 in the neural network are split into tensors 3 and 4. When tensors 1 and 3 are deployed on training device 1, and tensors 2 and 4 are deployed on training device 2, the sub-network model M... i The tensor parallel strategy indicates that the tensor parallel tasks are executed as follows: training device 1 is responsible for performing the computation tasks of tensor 1 and tensor 3, and training device 2 is responsible for performing the computation tasks of tensor 2 and tensor 4; sub-network model M i The tensor parallelism indicated by the tensor parallelism strategy is 2.

[0073] In a feasible embodiment, when the sub-network model M iWhen the parallel strategy does not include a data parallel strategy, set the sub-network model M. i The parallel strategy indicates a micro-batch size of 1 and a data parallelism of 1; when the sub-network model M i When the parallel strategy does not include a pipelined parallel strategy, set the sub-network model M. i The parallel strategy indicates a pipeline parallelism of 1; when the subnetwork model M i When the parallel strategy does not include the tensor parallel strategy, set the sub-network model M. i The parallel strategy indicates that the tensor parallelism is 1. Thus, the parallelism parameters can include data parallelism parameters and model parallelism parameters; the data parallelism parameters can include micro-batch size and data parallelism; the model parallelism parameters can include pipeline parallelism and tensor parallelism.

[0074] The following uses the subnetwork model M as an example. i For example, this illustrates the process of policy decision processing for each sub-network model. Policy decision processing refers to the process of determining a suitable parallel policy for each sub-network model. For instance, minimizing the training time of each sub-network model can be used as the execution rule for policy decision processing. Sub-network model M i The training time refers to the time taken to execute the sub-network model M in one training iteration. i The time required for parallel training tasks. Please refer to Figure 6, which is a flowchart illustrating a policy decision processing method provided in an embodiment of this application. This policy decision processing method is used to perform parallel training on sub-network model M. i The strategy decision-making process is performed to obtain the sub-network model M. i The parallel strategy process includes the following steps S201-S203:

[0075] S201, Obtain the sub-network model M i The policy search space, subnetwork model M i The strategy search space includes subnetwork model M i Multiple pre-configured reference parallel strategies.

[0076] Sub-network model M i The strategy search space includes subnetwork model M i Multiple pre-defined reference parallel strategies can be used from sub-network model M i In the strategy search space, the subnetwork model M i Decision-making for a suitable parallel strategy (i.e., sub-network model M) i Parallelism strategies). For example, a reference parallelism strategy could be an expert combining domain knowledge and practical experience with the model, tailored to the sub-network model M. i Pre-defined parallel strategy.

[0077] When setting up a reference parallel strategy for each sub-network model, the problem of data mismatch during model training needs to be considered. For example, suppose that in one training iteration, the audio encoder processes 5 training samples per batch (i.e., the number of samples processed by the audio encoder per batch is 5), and the LLM processes 4 training samples per batch (i.e., the number of samples processed by the LLM per batch is 4). After the audio encoder processes 5 training samples, it can only pass 4 of them to the LLM, and the remaining 1 training sample cannot continue to be passed, resulting in data mismatch. Data mismatch may require additional processing or adjustment to resolve, which can lead to a decrease in model training speed. In some cases, if data mismatch is not handled correctly, it may result in the loss or misprocessing of some data, affecting training accuracy. This application can first determine a suitable parallel strategy (i.e., the parallel strategy of the target sub-network model) for the target sub-network model with the largest unit computational cost (such as LLM in a large speech model). Since the product between the micro-batch size and the data parallelism indicated by the parallel strategy of the target sub-network model is equal to the number of samples processed per batch by the target sub-network model, the reference parallel strategy of other sub-network models can be limited by keeping the number of samples processed per batch of other sub-network models consistent with the number of samples processed per batch of the target sub-network model. Other sub-network models refer to network models other than the target sub-network model in the ensemble network model to be trained. The unit computational cost of a sub-network model refers to the time required for the sub-network model to process a training sample (i.e., one complete forward computation) without using parallel computing.

[0078] In one implementation, the unit computational cost of M sub-network models on the same training device can be obtained. For example, the M sub-network models can be deployed to M second training devices, with one second training device corresponding to one sub-network model. Each sub-network model can be deployed to its corresponding second training device. By obtaining the computational cost of each sub-network model on the second training device for processing one training sample, the unit computational cost of the M sub-network models on the same training device can be obtained.

[0079] Specifically, prepare M second training devices with identical performance specifications. Deploy the M sub-network models onto these M second training devices, ensuring that one second training device corresponds to one sub-network model. Provide each sub-network model with the same training sample, start the training process, and use performance monitoring tools (such as NVIDIA SMI for GPU training devices) to record the computational resources (e.g., floating-point operations, FLOPs) consumed by each second training device when processing the training sample. This is used as the unit computational cost of each sub-network model on the same training device. For example, for sub-network model M1, if the recorded FLOPs when processing the training sample on the corresponding second training device is X1, then X1 is the unit computational cost of sub-network model M1. Similarly, the unit computational cost of all M sub-network models can be obtained.

[0080] From M sub-network models, determine the target sub-network model with the largest unit computational cost. Sort the obtained M sub-network models by their unit computational cost. A sorting algorithm (such as bubble sort or quicksort) can be used to sort the data in ascending or descending order. Taking descending order as an example, after sorting, the sub-network model at the top is the target sub-network model with the largest unit computational cost. For example, the unit computational costs of the M sub-network models are X1, X2, ..., X... M After sorting in descending order, we get Sub-network model It is the target sub-network model.

[0081] Experts can pre-set multiple reference parallel strategies for the target subnetwork model and then decide on the parallel strategy for the target subnetwork model from these pre-set strategies. Based on the structural characteristics of the target subnetwork model (such as the number of network layers and neurons), computational complexity (such as computational cost per unit), and previous model training experience, experts comprehensively consider different combinations of data parallelism and model parallelism strategies (including pipelined parallelism and tensor parallelism) to pre-set multiple reference parallel strategies for the target subnetwork model. For example, for a target subnetwork model with many network layers and a large number of neurons, experts might pre-set several reference parallelism strategies that include high pipeline parallelism and tensor parallelism.

[0082] When making decisions on the parallel strategies of the target sub-network model, experts will conduct simulation training evaluations on each reference parallel strategy. Using simulation training tools, under conditions similar to the actual training environment, let the target sub-network model be trained according to each reference parallel strategy, and record various indicators during the training process, such as training time consumption, computing resource utilization, etc. Then, select the optimal reference parallel strategy as the parallel strategy of the target sub-network model according to the evaluation indicators. Usually, the reference parallel strategy with the shortest training time consumption and higher computing resource utilization will be preferentially selected. For example, after simulation training, the training time consumption of reference parallel strategy S1 is T1, and the computing resource utilization is U1; the training time consumption of reference parallel strategy S2 is T2, and the computing resource utilization is U2. If T1 < T2 and U1 > U2, then select reference parallel strategy S1 as the parallel strategy of the target sub-network model.

[0083] Experts can preset reference parallel strategies for other sub-network models according to the product of the micro-batch size and data parallelism indicated by the parallel strategy of the target sub-network model. For example, if the product of the micro-batch size and data parallelism indicated by the parallel strategy of the target sub-network model is 6, then when presetting reference parallel strategies for other sub-network models, it is necessary to ensure that the product of the micro-batch size and data parallelism indicated by the reference parallel strategies of other sub-network models is also 6.

[0084] S202. Based on the reference parallel parameters indicated by each reference parallel strategy in the strategy search space of sub-network model M i determine the training time consumption of sub-network model M i under each reference parallel strategy.

[0085] Among them, the reference parallel parameter refers to the parallel parameter indicated by the reference parallel strategy. If the problem of data mismatch is not considered and reference parallel strategies are preset for each sub-network model separately, then before determining the training time consumption of sub-network model M i under each reference parallel parameter indicated by each reference parallel strategy in the strategy search space of sub-network model M i the following steps need to be executed: If the target sub-network model is not sub-network model M i , then update each reference parallel strategy in the strategy search space of sub-network model M i according to the parallel strategy of the target sub-network model.

[0086] Sub-network model M iThe policy search space contains multiple reference parallel policies, including a first reference parallel policy. Taking the updating of the first reference parallel policy based on the parallel policy of the target sub-network model as an example, the specific implementation involves obtaining a second product between the micro-batch size indicated by the parallel policy of the target sub-network model and the data parallelism. The micro-batch size indicated by the first reference parallel policy is then updated using a second ratio between the second product and the data parallelism indicated by the first reference parallel policy. In other words, the second ratio is determined as the micro-batch size indicated by the first reference parallel policy. For example, assuming the target sub-network model is an LLM, the micro-batch size indicated by the parallel policy of the LLM is denoted as MBS. L Data parallelism is represented by DP L The micro-batch size indicated by the reference parallel strategy of the audio encoder is denoted as MBS. e Data parallelism is represented by DP e The micro-batch size indicated by the reference parallel strategy of the audio decoder is denoted as MBS. d Data parallelism is represented by DP d ; can be based on the formula: MBS e =(MBS) L ×DP L )÷DP e The micro-batch size, referencing the parallel policy indication in the policy search space for updating the audio encoder, can be determined according to equation: MBS d =(MBS) L ×DP L )÷DP d Update the micro-batch size of the reference parallel policy indication in the policy search space of the audio decoder. Alternatively, update the data parallelism of the first reference parallel policy indication using a third ratio between the second product and the micro-batch size of the first reference parallel policy indication; that is, determine the data parallelism of the first reference parallel policy indication using the third ratio. For example, suppose the target sub-network model is an LLM, and the micro-batch size of the parallel policy indication of the LLM is represented as MBS. L Data parallelism is represented by DP L The micro-batch size indicated by the reference parallel strategy of the audio encoder is denoted as MBS. e Data parallelism is represented by DP e The micro-batch size indicated by the reference parallel strategy of the audio decoder is denoted as MBS. d Data parallelism is represented by DP d ; can be based on the formula: DP e =(MBS) L ×DP L )÷MBS e The data parallelism indicated by the reference parallel policy in the policy search space of the updated audio encoder can be determined according to equation: DPd =(MBS) L ×DP L )÷MBS d Update the micro-batch size of the reference parallel policy indicated in the policy search space of the audio decoder.

[0087] For example, if the target subnetwork model is not the subnetwork model M i Obtain the micro-batch size (MBS) of the parallel strategy indication for the target sub-network model. target and data parallelism DP target Calculate their product: product = MBS target ×DP target For subnetwork model M i For each reference parallel policy in the policy search space, if the data parallelism indicated by the reference parallel policy is DP... ref Then use product and DP ref The ratio updates the micro-batch size, i.e., the new micro-batch size MBS. new =product÷DP ref If the micro-batch size indicated by the reference parallel strategy is MBS ref Then use product and MBS ref The ratio is used to update the data parallelism, i.e., the new data parallelism DP. new =product÷MBS ref For example, suppose the target subnetwork model is an LLM, and its parallel strategy indicates the micro-batch size MBS. L =5. Data Parallelism (DP) L =3, then product = 5 × 3 = 15. If the subnetwork model M i A reference parallel strategy indicates the data parallelism DP ref =5, then the updated micro-batch size MBS new =15÷5=3; if the reference parallel strategy indicates a micro-batch size MBS ref =3, then the updated data parallelism DP new =15÷3=5.

[0088] In one implementation, the parallel parameters include data parallelism parameters and model parallelism parameters. The data parallelism parameters include the micro-batch size, and the model parallelism parameters include pipeline parallelism and tensor parallelism. Based on the reference parallelism parameters indicated by the first reference parallelism strategy, the sub-network model M is determined. i The training time under the first reference parallel strategy includes: the time spent training the sub-network model M in the first training device. i Initialize as subnetwork model M iThe model instance (equivalent to deploying the subnetwork model M on the first training device) i ), call sub-network model M i The model instance processes a training sample (equivalent to calling the sub-network model M). i (By processing a training sample), a sub-network model M is obtained. i The unit training time on the first training device. The first training device can be any type of training device. The unit training time of the sub-network model refers to the time required for the sub-network model to process a training sample (i.e., one complete forward computation) without using parallel computing. The micro-batch size in the reference parallel parameters indicated by the first reference parallel strategy can be obtained, along with the sub-network model M. i The first product of the unit training time on the first training device; the first product is equivalent to the sub-network model M when only the data parallelism strategy is considered, without considering the model parallelism strategy. i The training time. The first ratio between the first product and the model parallel parameters indicated by the first reference parallel policy is determined as the training time of the sub-network model M. i The training time under the first reference parallel strategy, which considers both data parallelism and model parallelism, is the time consumption of the sub-network model M. i The training time.

[0089] For example, suppose the micro-batch size indicated by the first reference parallel policy of LLM is denoted as MBS. L Data parallelism is represented by DP L Tensor parallelism is represented as TP L The parallelism of a pipeline is expressed as PP. L The training time of LLM under the first reference parallel strategy This represents the unit training time of the LLM on the first training device. This allows us to obtain the training time of the LLM under various pre-set reference parallel strategies. By selecting the reference parallel strategy with the shortest training time, a suitable parallel strategy can be determined for the LLM. Similarly, assuming the micro-batch size indicated by the first reference parallel strategy of the audio encoder is denoted as MBS... e Data parallelism is represented by DP e Tensor parallelism is represented as TP e The parallelism of a pipeline is expressed as PP. e The training time of the audio encoder under the first reference parallel strategy This represents the unit training time of the audio encoder on the first training device. It allows us to obtain the training time of the audio encoder under various pre-set reference parallel strategies. By selecting the reference parallel strategy with the shortest training time, a suitable parallel strategy can be determined for the audio encoder. Assume the micro-batch size indicated by the first reference parallel strategy of the audio decoder is denoted as MBS. d Data parallelism is represented by DP d Tensor parallelism is represented as TP d The parallelism of a pipeline is expressed as PP. d The training time of the audio decoder under the first reference parallel strategy This represents the unit training time of the audio decoder on the first training device. It can be used to obtain the training time of the audio decoder under various reference parallel strategies preset for the audio decoder. By selecting the reference parallel strategy with the minimum training time, a suitable parallel strategy can be determined for the audio decoder.

[0090] For example, in the first training device, the sub-network model M i Initialize it as a model instance, let this model instance process a training sample, and record the time spent processing the sample. This time is the sub-network model M. i The unit training time T1 on the first training device. Obtain the micro-batch size MBS from the reference parallel parameters indicated by the first reference parallel policy. ref Calculate MBS ref The product of T1 and product1 = MBS ref ×T1, this product is equivalent to the subnetwork model M when only the data parallelism strategy is considered. i The training time. Then obtain the pipeline parallelism PP indicated by the first reference parallel policy. ref and tensor parallelism TP ref Calculate their product: product2 = PP ref ×TP ref The ratio of product1 to product2 is determined as the sub-network model M. i The training time T under the first reference parallel strategy is T = product1 ÷ product2. For example, if the micro-batch size indicated by the first reference parallel strategy is MBS... ref =4, Subnetwork Model M i The unit training time T1 on the first training device is 3 seconds, and the pipeline parallelism PP is... ref =2, Tensor parallelism TP ref =2, then product1 = 4 × 3 = 12 seconds, product2 = 2 × 2 = 4, subnetwork model M iThe training time under the first reference parallel strategy is T = 12 ÷ 4 = 3 seconds.

[0091] S203, Subnetwork Model M i The reference parallel policy corresponding to the minimum training time in the policy search space is used as the decision for the sub-network model M. i The parallel strategy. That is, when the sub-network model M is obtained. i In the subnetwork model M i After determining the training time under multiple pre-set reference parallel strategies, the reference parallel strategy corresponding to the minimum training time can be decided as the sub-network model M. i Parallel strategy.

[0092] Therefore, in this embodiment, on the one hand, the problem of data mismatch is considered to ensure that the number of samples processed by the target sub-network model in each batch is consistent with the number of samples processed by other sub-network models in each batch. On the other hand, for a certain sub-network model, the time required to perform model training using different reference parallel strategies under the same hardware resource conditions is considered, and the reference parallel strategy with the least time consumption is selected as the parallel strategy of the corresponding sub-network model, which is beneficial to improving computational efficiency.

[0093] S103. Based on the parallel strategy of M sub-network models, construct a training system for an integrated network model. The training system includes M communication groups; each communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model.

[0094] An ensemble network model training system is a system used to train ensemble network models. The system comprises M communication groups, each corresponding to a sub-network model. For example, the audio encoder, LLM, and audio decoder in Figure 3 each have their own communication group. A communication group is a collection of one or more training devices. By implementing data exchange and communication mechanisms within the communication group, different training devices can transmit data and model parameters to each other to collaboratively execute the parallel training tasks of their corresponding sub-network models, thus accelerating the computation process of those sub-network models.

[0095] Please refer to Figure 7, which is a flowchart illustrating a training system construction method provided in an embodiment of this application. This construction method is used to execute a parallel strategy based on M sub-network models to construct a training system for an integrated network model. The construction method includes the following steps S301-S302:

[0096] S301. Construct a communication group corresponding to each sub-network model according to the parallel strategy of each sub-network model in the M sub-network models.

[0097] In one implementation, the subnetwork model M i For communication group i, the parallel parameters include data parallelism parameters and model parallelism parameters. Data parallelism parameters include data parallelism degree, and model parallelism parameters include tensor parallelism degree and pipeline parallelism degree. According to the sub-network model M... i The parallel strategy constructs communication group i, including the following steps 11-13:

[0098] Step 11: Based on sub-network model M i The parallel strategy indicates the data parallelism and model parallelism parameters in the parallel parameters, determining the number of training devices in communication group i. In specific implementation, the sub-network model M... i The product between the data parallelism indicated by the parallelism strategy and the model parallelism parameters is determined as the number of training devices in communication group i. For example, the data parallelism indicated by the parallelism strategy of LLM is denoted as DP. L Tensor parallelism is represented as TP L The parallelism of a pipeline is expressed as PP. L The number N of training devices in the communication group corresponding to LLM. LLM =DP L ×TP L ×PP L The parallelism of the audio encoder, indicated by its parallel strategy, is represented as DP. e Tensor parallelism is represented as TP e The parallelism of a pipeline is expressed as PP. e The number N of training devices in the communication group corresponding to the audio encoder. e =DP e ×TP e ×PP e The parallelism strategy of the audio decoder indicates the degree of data parallelism, represented by DP. d Tensor parallelism is represented as TP d The parallelism of a pipeline is expressed as PP. d The number N of training devices in the communication group corresponding to the audio decoder d =DP d ×TP d ×PP d .

[0099] Step 12: Obtain the types of devices in the training device cluster used for model training, and classify them according to the sub-network model M. i Parallel strategy to determine sub-network model M i Training time on training devices of each device type, based on sub-network model M i The training time on the training device for each device type determines the device type of the training device in communication group i.

[0100] Device type refers to different types of training equipment, such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), etc. During model training, different sub-network models may require different types of training equipment to meet their computational needs, thereby improving training efficiency and performance.

[0101] For example, suppose the training device cluster has K (integers greater than 1) types of training devices (e.g., K types of GPUs), with the corresponding numbers 1, 2, ..., K, and the number of training devices for each of the K types being N1, N2, ..., N... K That can be done by minimizing the subnetwork model M. i The principle of training time consumption is used to determine the type of training equipment in communication group i.

[0102] In practical implementation, the types of devices in the training device cluster can be obtained, and then classified according to the sub-network model M. i Parallel strategy to determine sub-network model M i The training time on training devices for each type of device. For example, the sub-network model M can be obtained. i The unit training time on each type of training device, that is, the time spent training the sub-network model M. i Deploy the sub-network model M on each type of training device to obtain the sub-network model M on each type of training device. i The time required to process a single training sample. Based on subnetwork model M. i The parallel strategy indicates the micro-batch size, model parallel parameters, and sub-network model M. i Determine the sub-network model M based on the unit training time on the training device for each device type. i Training time on training devices for each device type. If any device type corresponds to the number k, where k is an integer less than k, then the sub-network model M... i Training time on the k-th training device = sub-network model M i The parallel strategy indicates the micro-batch size × sub-network model M i Unit training time on the k-th training device ÷ (subnetwork model M) i The parallel strategy indicates the pipeline parallelism × subnetwork model M i The parallel strategy indicates the degree of tensor parallelism.

[0103] Specifically, we obtain the types of devices in the training device cluster used for model training. Let there be K types of devices, numbered k = 1, 2, ..., K. We then define the sub-network model M... iThe training models are deployed on each of the K types of devices, each processing a training sample. The processing time is recorded to obtain the sub-network model M. i Unit training time on the k-th type of equipment Obtain the sub-network model M i The micro-batch size (MBS) indicated by the parallel strategy i PP of pipeline parallelism i and tensor parallelism TP i Then the subnetwork model M i Training time on the k-th type of equipment For example, if the subnetwork model M i The micro-batch size (MBS) indicated by the parallel strategy i =3, pipeline parallelism PP i =2, Tensor parallelism TP i =2, the unit training time on the training equipment of the second type of equipment. Seconds, then the sub-network model M i Training time on the second type of equipment Second.

[0104] Then it can be based on the sub-network model M i The training time on each type of training device determines the type of training device in communication group i. In practice, this can be based on the sub-network model M. i For each device type, determine the target device type corresponding to the minimum training time on the training devices. Obtain the number of training devices for the target device type from the training device cluster. If the number of training devices for the target device type is greater than or equal to the number of training devices in communication group i, then the target device type is determined as the device type of the training devices in communication group i. If the number of training devices for the target device type is less than the number of training devices in communication group i, then the device type with the second shortest training time is determined as the target device type, and the judgment process continues. If the number of training devices for the second shortest training time is less than the number of training devices in communication group i, then the device type with the third shortest training time is determined as the target device type, and so on.

[0105] For example, the training time of LLM on K types of training devices is expressed as: T 1 LLM T 2 LLM ... T K LLMThe training time of the audio encoder on K types of training devices is expressed as: T 1 e T 2 e ... T K e The training time of the audio decoder on K types of training devices is expressed as: T 1 d T 2 d ... T K d Minimize the objective function: T PPStage =max{T 1 LLM T 2 LLM ... T K LLM ,T 1 e T 2 e ... T K e ,T 1 d T 2 d ... T K d}, can find T 1 LLM T 2 LLM ... T K LLM The minimum value in T 1 e T 2 e ... T K e The minimum value in T 1 d T 2 d ... T K dThe minimum value in the objective function is determined by minimizing this objective function, thus allowing us to determine the types of training devices included in the communication groups corresponding to the LLM, audio encoder, and audio decoder. During this process, it's necessary to check if quantity constraints are met. For example, the number of training devices of a certain type selected for the LLM in the training device cluster must be greater than the number of training devices in the communication group corresponding to the LLM; the number of training devices of a certain type selected for the audio encoder in the training device cluster must be greater than the number of training devices in the communication group corresponding to the audio encoder; and the number of training devices of a certain type selected for the audio decoder in the training device cluster must be greater than the number of training devices in the communication group corresponding to the audio decoder. This ensures sufficient hardware resources are available to establish the communication groups.

[0106] Specifically, the sub-network model M i Training time on K types of training equipment Sort the data and find the device type number k corresponding to the shortest training time. min This device type is taken as the initial target device type. The number N of training devices of this target device type is obtained from the training device cluster. target If N target The number N of training devices in communication group i is greater than or equal to the number of training devices in communication group i. i Then the target device type is determined as the device type of the training device in communication group i; if N target <N i Then, select the device type with the second shortest training time as the new target device type, and repeat the above judgment process. If the number of training devices for the new target device type is still less than N, then... i Then, continue selecting the device type with the third shortest training time, and so on, until the target device type that meets the quantity requirement is found. For example, subnetwork model M i The training time on the three types of training equipment is as follows: Second, Second, Seconds, the device type number k corresponding to the shortest training time after sorting. min =2. If the number N of training devices for this type of equipment is... target The number N of training devices in communication group i is greater than or equal to the number of training devices in communication group i. i Then the device type is determined as the device type of the training device in communication group i; if N target <N i If the training time is the second shortest, then select the device type (number k=3) as the new target device type and continue the evaluation.

[0107] Step 13: According to the sub-network model M iThe parallel strategy is adopted, and the sub-network model M is initialized in the training device cluster based on the number and type of training devices in communication group i. i To construct communication group i. Based on the number and type of training devices in communication group i, it is possible to determine which training devices in the training device cluster can be used to construct communication group i, and based on the sub-network model M i Parallel strategies can determine the sub-network model M i Initialization methods, such as those based on sub-network model M i Data parallelism strategies can determine the need for sub-network model M i Initialize and generate several complete model instances based on sub-network model M i The pipelined parallel strategy can determine the need to modify the subnetwork model M. i The process involves initializing the network into several modules, and so on. Additionally, when constructing communication group i, communication channels are also established between training devices within communication group i to enable data interaction between the various training devices within communication group i.

[0108] Therefore, the embodiments of this application can analyze the computational performance of different hardware under the same parallel strategy by measuring the training time of the sub-network model on training devices of different types of devices, thereby allocating more suitable hardware resources to the sub-network model, which is conducive to optimizing hardware resource utilization and improving scalability.

[0109] S302. Establish the component communication topology between M communication groups and generate a training system for the integrated network model.

[0110] Component communication topology refers to the organizational structure of interconnected communication groups. Typically, if two sub-network models are cascaded, a communication channel is established between their corresponding communication groups. This channel enables data transmission and communication between different sub-network models. A cascade relationship means that the output of one sub-network model becomes the input of another. When two sub-network models are cascaded, their corresponding communication groups establish a communication channel to facilitate data transmission and communication between different sub-network models.

[0111] The subnetwork model M is used as an example below. i The integrated network model also includes sub-network model M. i Subnetwork model M with cascading relationships j Sub-network model M i Corresponding to communication group i, sub-network model M jTaking communication group j as an example, where i and j are both integers less than or equal to M, and i is not equal to j. In one embodiment, establishing a component communication topology between M communication groups includes the following steps 21-22:

[0112] Step 21: Perform group partitioning on communication group i and communication group j to obtain P communication groups, where P is a positive integer. Each communication group includes at least one sub-communication group obtained from the partitioning of communication group i and at least one sub-communication group obtained from the partitioning of communication group j. Each sub-communication group contains one or more training devices, and the number of training devices in each sub-communication group is equal to the number of sub-network models M. i The parallel strategy indicates the product between pipeline parallelism and tensor parallelism.

[0113] In practical implementation, the sub-network model M can be obtained. i The parallel strategy indicates the degree of data parallelism and the sub-network model M. j The parallel strategy indicates the greatest common divisor of the data parallelism. The greatest common divisor is determined as the number of groups P. Based on the number of groups P, communication groups i and j are partitioned into P communication groups. Each communication group includes at least one sub-communication group obtained from the partitioning of communication group i and at least one sub-communication group obtained from the partitioning of communication group j. Each sub-communication group obtained from the partitioning of communication group i can realize a complete sub-network model M. i The function is that each sub-communication group obtained by dividing communication group j can realize a complete sub-network model M. j The function of this application is as follows: The number of groups refers to the number of communication groups obtained when performing group partitioning on communication groups corresponding to two cascaded sub-network models. In this application, the number of groups P is determined by the greatest common divisor between the data parallelism degrees indicated by the parallel strategies of the two sub-network models.

[0114] Specifically, obtain the sub-network model M i Parallelism strategy indicates the degree of data parallelism (DP) i Sub-network model M j Parallelism strategy indicates the degree of data parallelism (DP) j Calculate their greatest common divisor P = gcd(DP) using the Euclidean algorithm. i ,DP j Divide the training devices in communication group i into P groups, each group forming a sub-communication group. The number of training devices in each sub-communication group is equal to the number of sub-network models M. i The parallel strategy indicates the product between pipeline parallelism and tensor parallelism; similarly, the training devices in communication group j are also divided into P groups, each forming a sub-communication group, and the number of training devices in each sub-communication group is equal to the number of sub-network models M. jThe parallel strategy indicates the product between pipeline parallelism and tensor parallelism. Each communication group contains a sub-communication group obtained by partitioning communication group i and a sub-communication group obtained by partitioning communication group j. For example, if DP i =8, DP j =6, then P = gcd(8,6) = 2. The training devices in communication group i are divided into two groups, and the training devices in communication group j are also divided into two groups, forming two communication clusters. Each communication cluster contains one sub-communication group from both communication group i and communication group j. The Euclidean algorithm is an algorithm for calculating the greatest common divisor (GCD) of two numbers. In this application, this algorithm is used to calculate the GCD between the data parallelism indicated by the parallel strategy of sub-network model Mi and the data parallelism indicated by the parallel strategy of sub-network model Mj, to determine the number of clusters P in the cluster partitioning process.

[0115] For example, refer to Figure 8, which shows a partial structural diagram of a training system. Figure 8 details the structure between the LLM and the audio encoder in the training system shown in Figure 3. Since the DP of the LLM is 4 and the DP of the audio encoder is 2, the greatest common divisor of 4 and 2 is 2. Based on the greatest common divisor (i.e., 2), the communication groups corresponding to the LLM and the audio encoder can be divided into two communication groups (i.e., communication group 1 and communication group 2). Each communication group contains one sub-communication group of the audio encoder and two sub-communication groups of the LLM. As we know from the TP=2 of the audio encoder, the model parameters of the audio encoder are split into two parts (such as tensor 1 and tensor 2). Thus, one GPU1 in communication group 1 can deploy tensor 1 of the audio encoder, and one GPU1 can deploy tensor 2 of the audio encoder. In this way, the two GPU1s in communication group 1 (i.e., the sub-communication groups of the audio encoder) can realize the function of a complete audio encoder. The same applies to the LLM.

[0116] Step 22: In the process of establishing the component communication topology between communication group i and communication group j, allocate inter-machine communication bandwidth to each communication group, and establish communication channels between sub-communication groups belonging to different communication groups in each communication group.

[0117] In one embodiment, during the establishment of the component communication topology between communication group i and communication group j, inter-machine communication bandwidth can be allocated to each communication group. This inter-machine communication bandwidth determines the maximum amount of data that can be transmitted within a given time period. In corporate or home networks, there is typically a total communication bandwidth resource. To ensure the rational allocation and use of network resources, a certain inter-machine communication bandwidth limit can be set for each device. Similarly, to ensure the communication efficiency and performance of each communication group, a total communication bandwidth resource can be divided (perhaps evenly) to obtain the inter-machine communication bandwidth allocated to each communication group. This ensures that each communication group can obtain sufficient communication bandwidth resources for communication and data transmission, thereby effectively utilizing and managing communication bandwidth resources.

[0118] Establishing a component communication topology between communication group i and communication group j involves creating a communication channel between them. This communication channel serves as the connection path for data transmission and communication interaction between communication group i and communication group j. In this embodiment, during the process of establishing the component communication topology between communication group i and communication group j, a communication channel is established between sub-communication groups belonging to different communication groups within each communication group. Specifically, this can be implemented based on the sub-network model M. i With sub-network model M j The cascading relationships between them determine the communication order of components between communication group i and communication group j; for example, a sub-network model M. i The output will become the subnetwork model M. jIf the input is from a communication group, then the communication order of components in communication group i precedes that of communication group j. Within each sub-communication group derived from the communication group with the earlier component communication order, a training device can be selected as the communication transmitting device. For example, if GPU1 in a sub-communication group in Figure 8 has an audio encoder output layer deployed on it, then GPU1 can be designated as the communication transmitting device. Similarly, within each sub-communication group derived from the communication group with the later component communication order, a training device can be selected as the communication receiving device. For example, if GPU3 in a sub-communication group in Figure 8 has an LLM input layer deployed on it, then GPU3 can be designated as the communication receiving device. A communication channel can be established between the communication transmitting device and the communication receiving device. This allows each sub-communication group in communication group i to perform data transmission and communication interaction with each sub-communication group in communication group j within the same communication group. For example, the sub-communication group of the audio encoder in communication group 1 can process 6 training samples. After processing these 6 training samples, the communication transmitting device (such as GPU 01) in the sub-communication group can aggregate the calculation results of each training device in the sub-communication group and generate two sets of training data. Each set of training data contains 3 training samples. The communication transmitting device (such as GPU 01) in the sub-communication group can send these two sets of training data to the two sub-communication groups of the LLM in communication group 1 through the communication channel with the communication receiving device (such as GPU 43 and GPU 123) in each sub-communication group of the LLM. In this way, each sub-communication group of the LLM can obtain one set of training data to perform the calculation task.

[0119] The Gather communication shown in Figure 8 is a communication operation in parallel computing. Specifically, Gather communication is a setter operation used in parallel computing to aggregate data from multiple training devices onto a single target training device (such as a communication transmitter), enabling distributed data collection and integration. Each training device in a sub-communication group can use Gather communication to send its computation results to the communication transmitter within that sub-communication group. In one implementation, the communication transmitter can use send communication within the communication channel to send data to the communication receiver. Send communication is a widely used communication operation in parallel computing for sending data from one training device to another. Send communication is a point-to-point communication method where the sender sends data to the receiver and continues performing other operations after the communication is complete.

[0120] Therefore, the embodiments of this application can abstract independent communication groups for each sub-network model and define the communication modes within and between communication groups, thus realizing a more flexible and finer-grained parallel strategy configuration.

[0121] S104. During the training of the integrated network model, the M communication groups in the training system are called to execute the parallel training tasks of the corresponding sub-network models according to the execution method indicated by the parallel strategy of the corresponding sub-network models, so as to obtain the trained network model.

[0122] In the process of training an ensemble network model, a pipeline for training the ensemble network model needs to be constructed. This pipeline decomposes the model training process into multiple stages and connects these stages in a certain order to achieve efficient model training. Specifically, the model training process of the ensemble network model can be divided into M stages in the pipeline, with each stage corresponding to the parallel training task of a sub-network model. These M stages are connected sequentially according to the cascading relationship between the M sub-network models (which can include serial or parallel connections). For example, in the pipeline, the stage corresponding to the parallel training task of the audio encoder is connected to the stage corresponding to the parallel training task of the LLM, and the stage corresponding to the parallel training task of the LLM is connected to the stage corresponding to the parallel training task of the audio decoder. The pipeline execution process sequentially completes the parallel training tasks corresponding to each stage according to the order of the stages within the pipeline. Therefore, during pipeline execution, a target communication group can be determined for the current stage, which is the stage currently being executed out of M stages. This allows the target communication group to be invoked and its parallel training tasks to be executed according to the parallel strategy of the corresponding sub-network model. For example, if the current stage corresponds to sub-network model M... i In the parallel training task, the target communication group used to perform the current stage is the sub-network model M. i The corresponding communication group can call the sub-network model M i The corresponding communication group is based on the sub-network model M. i The parallel strategy indicates the execution method, executing the sub-network model M. i Parallel training tasks.

[0123] When executing a pipeline, the first step is to initialize the M stages within it. Each stage is allocated corresponding computing resources (such as training equipment) to ensure sufficient resources for executing its corresponding parallel training task. Simultaneously, data transmission channels are established between stages to ensure smooth data flow. During pipeline operation, the parallel training tasks of each stage are started sequentially according to the cascading relationship between the M sub-network models. Once a stage completes its current parallel training task, it transmits the processing results to the next stage via the data transmission channel, and the next stage immediately begins processing the received data. For example, in the pipeline, after the stage corresponding to the audio encoder's parallel training task completes processing the training samples, it transmits the results to the stage corresponding to the LLM's parallel training task, and the LLM stage begins processing these data. To improve pipeline execution efficiency, asynchronous processing and buffering mechanisms can be employed. A buffer is set up at each stage to temporarily store data to be processed, preventing stage idleness due to data transmission delays. Simultaneously, asynchronous processing techniques allow training tasks in different stages to be executed in parallel to a certain extent, reducing overall training time. Asynchronous processing is a technique that allows training tasks at different stages to be executed in parallel to some extent. Applying this technique in a model training pipeline can reduce overall training time and improve training efficiency. By setting buffers at each stage to temporarily store data to be processed, idle time at each stage due to data transmission delays is avoided, thus enabling partial parallel execution of training tasks at different stages.

[0124] During the training of the ensemble network model, each pipeline execution involves a training iteration. This involves comparing the predictions of the ensemble network model with the actual results, calculating the loss function value, using the loss function value to calculate the gradient, and employing optimization algorithms such as gradient descent to update the model parameters. By executing the pipeline multiple times, the model parameters of the ensemble network model can be continuously adjusted to reduce the loss function value generated by the network model. In other words, it is ensured that the loss function value generated by the ensemble network model after each parameter adjustment is less than the loss function value generated by the network model before the parameter adjustment. When the ensemble network model reaches the training stopping condition after multiple parameter adjustments, the model training process can be considered complete, and the network model trained last can be identified as the trained network model. The training stopping condition includes, but is not limited to, at least one of the following: the loss function value is less than a set loss threshold, or the number of training iterations reaches a preset number.

[0125] When setting the loss threshold, you can refer to the training experience and performance metrics of similar models from the past. First, conduct multiple small-scale pre-training experiments, recording the changes in the loss function value at different training stages. Analyze this experimental data to identify the range within which the loss function value tends to stabilize in the later stages of training. Then, based on the model's accuracy requirements and training time constraints, select a suitable value from this stable range as the loss threshold. For example, if multiple pre-training experiments show that the loss function value stabilizes between 0.1 and 0.2 in the later stages of training, and if high model accuracy is required and sufficient training time is available, the loss threshold can be set to 0.1; if there are strict limitations on training time, the loss threshold can be set to 0.2. Pre-training experiments are small-scale training experiments conducted before formally training the neural network model. By conducting multiple pre-training experiments, recording the changes in the loss function value at different training stages, and analyzing the experimental data to identify the range within which the loss function value tends to stabilize in the later stages of training, a reference basis is provided for setting the loss threshold.

[0126] The preset number of training iterations can be determined based on the model's complexity and the scale of the training data. Models with complex structures and many parameters require more iterations to converge; similarly, larger training datasets require more iterations to fully learn the data features. A suitable range for the preset number of iterations can be determined through theoretical analysis and experimental verification. For example, experiments have shown that for a moderately complex model and a certain scale of training data, the model's performance reaches a good level when the number of iterations is between 100 and 200. In this case, the preset number of iterations can be set to 150. Furthermore, during training, the preset number of iterations can be dynamically adjusted based on changes in the loss function value. If the loss function value does not show significant convergence even when approaching the preset number of iterations, the preset number of iterations can be increased to continue training.

[0127] In this embodiment, by reducing the training time of each sub-network model, a suitable parallel strategy is determined for each sub-network model, and the type of training device in the corresponding communication group is determined for each sub-network model, thus achieving pipeline balancing. Pipeline balancing refers to the reasonable adjustment and optimization of workload and efficiency at each stage of the pipeline to ensure the smooth operation of the entire pipeline and improve the efficiency and quality of model training.

[0128] Therefore, this application proposes a parallel strategy decoupling method in the model training scheme. This method can create its own parallel strategy for each sub-network model in the ensemble network model to be trained, and create independent communication groups for each sub-network model in the ensemble network model to be trained. It also defines the communication modes within and between communication groups, thus achieving more flexible and fine-grained parallel strategy configuration. In addition, a hardware resource allocation method based on performance profiling is proposed. By profiling the computational performance of different sub-network models under different hardware and different reference parallel strategies, more suitable hardware resources and parallel strategies are allocated to different sub-network models. This effectively utilizes the computing power resources of non-high-end training equipment, reduces training costs, improves computational efficiency, and accelerates the model training process.

[0129] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0130] Figure 9 is a schematic diagram of a model training device provided in an embodiment of this application; the model training device can be used to execute some or all of the steps in the method embodiment shown in Figure 5. Referring to Figure 9, the model training device includes the following units: an acquisition unit 901 and a processing unit 902.

[0131] Acquisition unit 901 is used to acquire the ensemble network model to be trained. The ensemble network model includes M sub-network models, where M is an integer greater than 1.

[0132] The processing unit 902 is used to perform policy decision processing on each sub-network model to obtain the parallel policy of each sub-network model; the parallel policy of any sub-network model is used to indicate the parallel training task of the sub-network model in the model training process and the execution method of the parallel training task.

[0133] The processing unit 902 is also used to construct a training system for an integrated network model based on a parallel strategy of M sub-network models. The training system includes M communication groups; each communication group corresponds to a sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model.

[0134] The processing unit 902 is also used to, during the model training process of the integrated network model, call M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network models according to the execution mode indicated by the parallel strategy of the corresponding sub-network models, so as to obtain the trained network model.

[0135] In one embodiment, the parallel strategy of any sub-network model is further used to indicate the parallel parameters of the sub-network model, wherein any sub-network model is denoted as sub-network model M. i, where i is an integer less than or equal to M;

[0136] Processing unit 902 is used for processing sub-network model M i The strategy decision-making process is performed to obtain the sub-network model M. i When using a parallel strategy, the following steps are executed:

[0137] Call the acquisition unit 901 to obtain the sub-network model M i The policy search space, subnetwork model M i The strategy search space includes subnetwork model M i Pre-configured multiple reference parallel strategies;

[0138] Based on subnetwork model M i The reference parallel parameters indicated by each reference parallel policy in the policy search space determine the sub-network model M. i Training time under each reference parallel strategy;

[0139] Subnetwork model M i The reference parallel policy corresponding to the minimum training time in the policy search space is used as the decision for the sub-network model M. i Parallel strategy;

[0140] The parallel parameters include any one or more of the following: data parallel parameters and model parallel parameters; the data parallel parameters include one or both of micro-batch size and data parallelism; the model parallel parameters include one or both of pipeline parallelism and tensor parallelism.

[0141] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include data parallel parameters and model parallel parameters, and the data parallel parameters include micro-batch size;

[0142] Processing unit 902 is used to determine the sub-network model M based on the reference parallel parameters indicated by the first reference parallel strategy. i During the training process under the first reference parallel strategy, the following steps are specifically executed:

[0143] In the first training device, the sub-network model M i Initialize as subnetwork model M i The model instance calls the sub-network model M i The model instance processes a training sample to obtain the sub-network model M. i The unit training time on the first training device;

[0144] The micro-batch size in the reference parallel parameters indicated by the first reference parallel strategy is obtained by calling the acquisition unit 901, and compared with the sub-network model M. iThe first product between the unit training time on the first training device;

[0145] The first ratio between the first product and the model parallel parameters indicated by the first reference parallel strategy is determined as the sub-network model M. i Training time under the first reference parallel strategy.

[0146] In one embodiment, the processing unit 902 is used to process data based on the sub-network model M. i The reference parallel parameters indicated by each reference parallel policy in the policy search space determine the sub-network model M. i Before the training time under each reference parallel strategy, the following steps are also performed:

[0147] Call the acquisition unit 901 to obtain the unit computation cost of M sub-network models on the same training device;

[0148] Identify the target subnetwork model with the highest unit computational cost from among the M subnetwork models;

[0149] If the target subnetwork model is not subnetwork model M i Then, the subnetwork model M is updated according to the parallel strategy of the target subnetwork model. i Each reference parallel policy in the policy search space.

[0150] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include data parallel parameters, which include micro-batch size and data parallelism degree.

[0151] Processing unit 902 is used to update subnetwork model M according to the parallel strategy of the target subnetwork model. i When searching for the first reference parallel policy in the policy search space, the following steps are specifically performed:

[0152] The acquisition unit 901 is invoked to obtain the second product between the micro-batch size and the data parallelism of the parallel strategy indication of the target sub-network model;

[0153] The micro-batch size indicated by the first reference parallel strategy is updated using the second ratio between the second product and the data parallelism indicated by the first reference parallel strategy; or,

[0154] The data parallelism indicated by the first reference parallel strategy is updated using a third ratio between the second product and the micro-batch size indicated by the first reference parallel strategy.

[0155] In one embodiment, when processing unit 902 constructs a training system for an ensemble network model based on a parallel strategy of M sub-network models, it specifically performs the following steps:

[0156] According to the parallel strategy of each sub-network model, construct the communication group corresponding to each sub-network model;

[0157] Establish a component communication topology among M communication groups to generate a training system for an integrated network model.

[0158] In one embodiment, any subnetwork model is represented as subnetwork model M. i The integrated network model also includes sub-network model M. i Subnetwork model M with cascading relationships j Sub-network model M i Corresponding to communication group i, sub-network model M j The corresponding communication group is j; i and j are both integers less than or equal to M, and i is not equal to j;

[0159] When processing unit 902 establishes the component communication topology between M communication groups, it specifically performs the following steps:

[0160] Perform group partitioning on communication group i and communication group j to obtain P communication groups, where P is a positive integer; each communication group includes at least one sub-communication group obtained from the partitioning of communication group i and at least one sub-communication group obtained from the partitioning of communication group j.

[0161] In the process of establishing the component communication topology between communication group i and communication group j, inter-machine communication bandwidth is allocated to each communication group, and communication channels between sub-communication groups belonging to different communication groups are established in each communication group.

[0162] In one embodiment, the parallel strategy of any sub-network model is also used to indicate the parallel parameters of the sub-network model, including data parallel parameters, which include the degree of data parallelism.

[0163] Processing unit 902 is used to perform group partitioning on communication group i and communication group j to obtain P communication groups. Specifically, it executes the following steps:

[0164] Call the acquisition unit 901 to obtain the sub-network model M i The parallel strategy indicates the degree of data parallelism and the sub-network model M. j The parallel strategy indicates the greatest common divisor among the data parallelisms;

[0165] The greatest common divisor is determined as the group size P;

[0166] Based on the number of groups P, communication groups i and j are partitioned to obtain P communication groups.

[0167] In one embodiment, each sub-communication group contains one or more training devices;

[0168] When processing unit 902 establishes a communication channel between sub-communication groups belonging to different communication groups in each communication group, it specifically performs the following steps:

[0169] According to the subnetwork model M i With sub-network model M j The cascading relationships between them determine the communication order of components between communication group i and communication group j;

[0170] In each sub-communication group obtained by dividing the communication group with the component communication order first, a training device is selected as the communication transmitting device.

[0171] In each sub-communication group obtained by dividing the communication group with the component communication sequence later, a training device is selected as the communication receiving device.

[0172] Establish a communication channel between the communication transmitting equipment and the communication receiving equipment.

[0173] In one embodiment, any subnetwork model is represented as subnetwork model M. i Sub-network model M i The corresponding communication group i is an integer less than or equal to M; the parallel strategy of any sub-network model is also used to indicate the parallel parameters of the sub-network model, including data parallel parameters and model parallel parameters, and the data parallel parameters include data parallelism degree.

[0174] Processing unit 902 is used to process the sub-network model M i The parallel strategy, when constructing communication group i, specifically executes the following steps:

[0175] Based on subnetwork model M i The parallel strategy indicates the data parallelism and model parallelism parameters in the parallel parameters, and determines the number of training devices in communication group i;

[0176] Obtain the types of devices in the training device cluster used for model training, and categorize them according to the sub-network model M. i Parallel strategy to determine sub-network model M i Training time on training equipment for each type of equipment;

[0177] Based on subnetwork model M i The training time on the training device for each device type determines the device type of the training device in communication group i.

[0178] According to the subnetwork model M i The parallel strategy is adopted, and the sub-network model M is initialized in the training device cluster based on the number and type of training devices in communication group i. i To construct communication group i.

[0179] In one embodiment, the processing unit 902 is used to process the sub-network model M. i When determining the training time on training devices of each device type, the specific steps are as follows:

[0180] Based on subnetwork model M i For each type of equipment, determine the target equipment type corresponding to the minimum training time on the training equipment;

[0181] Obtain the number of training devices of the target device type from the training device cluster;

[0182] When the number of training devices of the target device type is greater than or equal to the number of training devices in communication group i, the target device type is determined to be the device type of the training devices in communication group i.

[0183] In one embodiment, the data parallelism parameter also includes the micro-batch size;

[0184] Processing unit 902 is used to process the sub-network model M i Parallel strategy to determine sub-network model M i During the training process on each type of training device, the following steps are specifically performed:

[0185] Call the acquisition unit 901 to obtain the sub-network model M i The unit training time on training equipment for each type of equipment;

[0186] Based on subnetwork model M i The parallel strategy indicates the micro-batch size, model parallel parameters, and sub-network model M. i Determine the sub-network model M based on the unit training time on the training device for each device type. i Training time on training equipment for each type of equipment.

[0187] In one embodiment, when the processing unit 902 calls M communication groups in the training system to execute the parallel training task of the corresponding sub-network model according to the execution mode indicated by the parallel strategy of the corresponding sub-network model during the model training of the ensemble network model, the specific steps are as follows:

[0188] In the process of training the ensemble network model, the training process of the network model is divided into M stages in the pipeline, and each stage corresponds to the parallel training task of a sub-network model.

[0189] The pipeline is executed, and during the execution of the pipeline, the target communication group for the current stage is determined. The target communication group is invoked to execute the parallel training task of the sub-network model corresponding to the target communication group in accordance with the execution method indicated by the parallel strategy of the sub-network model corresponding to the target communication group. The current stage is the stage currently being executed among M stages.

[0190] According to one embodiment of this application, the various units in the model training device shown in FIG9 can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This can achieve the same operation without affecting the technical effect of the embodiment of this application. The above units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the model training device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by multiple units working together. According to another embodiment of this application, the model training device shown in FIG9 and the model training method of the embodiment of this application can be constructed by running a computer program (including program code) capable of executing the steps involved in the corresponding method shown in FIG5 on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). Computer programs can be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0191] The technical solution provided in this application can obtain an ensemble network model to be trained (an ensemble network model), which includes M sub-network models (a single network model), where M is an integer greater than 1. Policy decision processing is performed on each sub-network model to obtain a parallel policy for each sub-network model. The parallel policy of any sub-network model indicates the parallel training task and its execution method during model training. Compared to existing technical solutions that treat the M sub-network models as a whole and train them using a unified parallel policy, the technical solution provided in this application can achieve parallel policy decoupling. This means that the M sub-network models can be configured with different parallel policies, and a suitable parallel policy can be determined for each sub-network model during policy decision processing, improving the computational efficiency during network model training. Based on the parallel policies of the M sub-network models, a training system for the ensemble network model is constructed. The training system includes M communication groups; each communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel policy of the corresponding sub-network model. Based on the decoupling of parallel strategies, each sub-network model can have its own communication group. This allows for precise matching of suitable hardware resources to each sub-network model, resolving the resource idleness issue caused by the uneven computational load of the M sub-network models when training them as a whole, thus improving the scalability of the network model training. During the training of the ensemble network model, the M communication groups in the training system are invoked to execute the parallel training tasks of the corresponding sub-network models according to the execution methods indicated by the parallel strategies of the respective sub-network models. This accelerates the model training process, resulting in the trained network model.

[0192] Figure 10 is a schematic diagram of a computer device provided in an embodiment of this application. Referring to Figure 10, the computer device includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, communication interface 1002, and computer-readable storage medium 1003 can be connected via a bus or other means. The communication interface 1002 is used to receive and send data. The computer-readable storage medium 1003 can be stored in the memory of the computer device. The computer-readable storage medium 1003 is used to store computer programs, including program instructions. The processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003. The processor 1001 (or CPU (Central Processing Unit)) is the computing and control core of the computer device, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions.

[0193] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 1001, which may be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0194] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor 1001 loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-described model training method embodiment; specifically, the processor 1001 loads and executes one or more instructions in the computer-readable storage medium for the following steps:

[0195] Obtain the ensemble network model to be trained. The ensemble network model includes M sub-network models, where M is an integer greater than 1.

[0196] Each sub-network model is processed for policy decision-making to obtain the parallel policy of each sub-network model; the parallel policy of any sub-network model is used to indicate the parallel training task of that sub-network model during the model training process and the execution method of the parallel training task.

[0197] Based on the parallel strategy of M sub-network models, a training system for an integrated network model is constructed. The training system includes M communication groups; each communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model.

[0198] During the training of the ensemble network model, M communication groups in the training system are invoked to execute the parallel training tasks of the corresponding sub-network models according to the execution method indicated by the parallel strategy of the corresponding sub-network model, so as to obtain the trained network model.

[0199] In one embodiment, the parallel strategy of any sub-network model is further used to indicate the parallel parameters of the sub-network model, wherein any sub-network model is denoted as sub-network model M. i , where i is an integer less than or equal to M;

[0200] Processor 1001 is used for sub-network model M i The strategy decision-making process is performed to obtain the sub-network model M. i When using a parallel strategy, the following steps are executed:

[0201] Obtain the sub-network model M i The policy search space, subnetwork model M i The strategy search space includes subnetwork model M i Pre-configured multiple reference parallel strategies;

[0202] Based on subnetwork model M i The reference parallel parameters indicated by each reference parallel policy in the policy search space determine the sub-network model M. i Training time under each reference parallel strategy;

[0203] Subnetwork model M i The reference parallel policy corresponding to the minimum training time in the policy search space is used as the decision for the sub-network model M. i Parallel strategy;

[0204] The parallel parameters include any one or more of the following: data parallel parameters and model parallel parameters; the data parallel parameters include one or both of micro-batch size and data parallelism; the model parallel parameters include one or both of pipeline parallelism and tensor parallelism.

[0205] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include data parallel parameters and model parallel parameters, and the data parallel parameters include micro-batch size;

[0206] Processor 1001 is used to determine sub-network model M based on reference parallel parameters indicated by a first reference parallel strategy. i During the training process under the first reference parallel strategy, the following steps are specifically executed:

[0207] In the first training device, the sub-network model M i Initialize as subnetwork model M i The model instance calls the sub-network model M i The model instance processes a training sample to obtain the sub-network model M. i The unit training time on the first training device;

[0208] Obtain the micro-batch size in the reference parallel parameters indicated by the first reference parallel strategy, and compare it with the sub-network model M. i The first product between the unit training time on the first training device;

[0209] The first ratio between the first product and the model parallel parameters indicated by the first reference parallel strategy is determined as the sub-network model M. i Training time under the first reference parallel strategy.

[0210] In one embodiment, processor 1001 is used to perform subnetwork model M i The reference parallel parameters indicated by each reference parallel policy in the policy search space determine the sub-network model M. i Before the training time under each reference parallel strategy, the following steps are also performed:

[0211] Obtain the unit computation cost of M sub-network models on the same training device;

[0212] Identify the target subnetwork model with the highest unit computational cost from among the M subnetwork models;

[0213] If the target subnetwork model is not subnetwork model M i Then, the subnetwork model M is updated according to the parallel strategy of the target subnetwork model. i Each reference parallel policy in the policy search space.

[0214] In one embodiment, the plurality of reference parallel strategies include a first reference parallel strategy; the parallel parameters include data parallel parameters, which include micro-batch size and data parallelism degree.

[0215] Processor 1001 is used to update subnetwork model M according to the parallel strategy of the target subnetwork model. i When searching for the first reference parallel policy in the policy search space, the following steps are specifically performed:

[0216] Obtain the second product between the micro-batch size and the data parallelism indicated by the parallel strategy of the target sub-network model;

[0217] The micro-batch size indicated by the first reference parallel strategy is updated using the second ratio between the second product and the data parallelism indicated by the first reference parallel strategy; or,

[0218] The data parallelism indicated by the first reference parallel strategy is updated using a third ratio between the second product and the micro-batch size indicated by the first reference parallel strategy.

[0219] In one embodiment, when the processor 1001 constructs a training system for an ensemble network model based on a parallel strategy of M sub-network models, it specifically performs the following steps:

[0220] According to the parallel strategy of each sub-network model, construct the communication group corresponding to each sub-network model;

[0221] Establish a component communication topology among M communication groups to generate a training system for an integrated network model.

[0222] In one embodiment, any subnetwork model is represented as subnetwork model M. i The integrated network model also includes sub-network model M. i Subnetwork model M with cascading relationships j Sub-network model M i Corresponding to communication group i, sub-network model M j The corresponding communication group is j; i and j are both integers less than or equal to M, and i is not equal to j;

[0223] When processor 1001 establishes a component communication topology among M communication groups, it performs the following steps:

[0224] Perform group partitioning on communication group i and communication group j to obtain P communication groups, where P is a positive integer; each communication group includes at least one sub-communication group obtained from the partitioning of communication group i and at least one sub-communication group obtained from the partitioning of communication group j.

[0225] In the process of establishing the component communication topology between communication group i and communication group j, inter-machine communication bandwidth is allocated to each communication group, and communication channels between sub-communication groups belonging to different communication groups are established in each communication group.

[0226] In one embodiment, the parallel strategy of any sub-network model is also used to indicate the parallel parameters of the sub-network model, including data parallel parameters, which include the degree of data parallelism.

[0227] When processor 1001 performs group partitioning on communication group i and communication group j to obtain P communication groups, it specifically executes the following steps:

[0228] Obtain the sub-network model M i The parallel strategy indicates the degree of data parallelism and the sub-network model M. j The parallel strategy indicates the greatest common divisor among the data parallelisms;

[0229] The greatest common divisor is determined as the group size P;

[0230] Based on the number of groups P, communication groups i and j are partitioned to obtain P communication groups.

[0231] In one embodiment, each sub-communication group contains one or more training devices;

[0232] When processor 1001 establishes a communication channel between sub-communication groups belonging to different communication groups in each communication group, it specifically performs the following steps:

[0233] According to the subnetwork model M i With sub-network model M j The cascading relationships between them determine the communication order of components between communication group i and communication group j;

[0234] In each sub-communication group obtained by dividing the communication group with the component communication order first, a training device is selected as the communication transmitting device.

[0235] In each sub-communication group obtained by dividing the communication group with the component communication sequence later, a training device is selected as the communication receiving device.

[0236] Establish a communication channel between the communication transmitting equipment and the communication receiving equipment.

[0237] In one embodiment, any subnetwork model is represented as subnetwork model M. i Sub-network model M i The corresponding communication group i is an integer less than or equal to M; the parallel strategy of any sub-network model is also used to indicate the parallel parameters of the sub-network model, including data parallel parameters and model parallel parameters, and the data parallel parameters include data parallelism degree.

[0238] Processor 1001 is used according to subnetwork model M i The parallel strategy, when constructing communication group i, specifically executes the following steps:

[0239] Based on subnetwork model M i The parallel strategy indicates the data parallelism and model parallelism parameters in the parallel parameters, and determines the number of training devices in communication group i;

[0240] Obtain the types of devices in the training device cluster used for model training, and categorize them according to the sub-network model M. i Parallel strategy to determine sub-network model M i Training time on training equipment for each type of equipment;

[0241] Based on subnetwork model M i The training time on the training device for each device type determines the device type of the training device in communication group i.

[0242] According to the subnetwork model M i The parallel strategy is adopted, and the sub-network model M is initialized in the training device cluster based on the number and type of training devices in communication group i. i To construct communication group i.

[0243] In one embodiment, processor 1001 is used for sub-network model M iWhen determining the training time on training devices of each device type, the specific steps are as follows:

[0244] Based on subnetwork model M i For each type of equipment, determine the target equipment type corresponding to the minimum training time on the training equipment;

[0245] Obtain the number of training devices of the target device type from the training device cluster;

[0246] When the number of training devices of the target device type is greater than or equal to the number of training devices in communication group i, the target device type is determined to be the device type of the training devices in communication group i.

[0247] In one embodiment, the data parallelism parameter also includes the micro-batch size;

[0248] Processor 1001 is used according to subnetwork model M i Parallel strategy to determine sub-network model M i During the training process on each type of training device, the following steps are specifically performed:

[0249] Obtain the sub-network model M i The unit training time on training equipment for each type of equipment;

[0250] Based on subnetwork model M i The parallel strategy indicates the micro-batch size, model parallel parameters, and sub-network model M. i Determine the sub-network model M based on the unit training time on the training device for each device type. i Training time on training equipment for each type of equipment.

[0251] In one embodiment, when the processor 1001 calls M communication groups in the training system to execute the parallel training task of the corresponding sub-network model according to the execution mode indicated by the parallel strategy of the corresponding sub-network model during the model training of the ensemble network model, the processor specifically performs the following steps:

[0252] In the process of training the ensemble network model, the training process of the network model is divided into M stages in the pipeline, and each stage corresponds to the parallel training task of a sub-network model.

[0253] The pipeline is executed, and during the execution of the pipeline, the target communication group for the current stage is determined. The target communication group is invoked to execute the parallel training task of the sub-network model corresponding to the target communication group in accordance with the execution method indicated by the parallel strategy of the sub-network model corresponding to the target communication group. The current stage is the stage currently being executed among M stages.

[0254] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the model training method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, they will not be repeated here.

[0255] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0256] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned model training method.

[0257] In summary, this application provides a model training method, apparatus, device, computer-readable storage medium, and computer program product. By acquiring an ensemble network model to be trained, which includes M sub-network models (M being an integer greater than 1), policy decision processing is performed on each sub-network model to obtain a parallel policy for each sub-network model. This parallel policy indicates the parallel training tasks and execution methods of the sub-network models during model training, achieving decoupling of parallel policies. This allows each sub-network model to select a suitable parallel policy based on its own characteristics, avoiding the problem of unreasonable allocation of computing resources caused by differences in factor network models under a unified parallel policy, improving the utilization rate of computing resources, and thus enhancing the computational efficiency when training the network model. This system constructs a training system for an ensemble network model based on a parallel strategy using M sub-network models. The system comprises M communication groups, each corresponding to a sub-network model. Each communication group is constructed according to the parallel strategy of its corresponding sub-network model, allowing each sub-network model to be trained within an independent communication group. This reduces interference between sub-network models, facilitates precise matching of appropriate hardware resources for each sub-network model, solves the resource idleness problem caused by uneven computational load in factor network models, and improves the scalability of the network model training. During the training of the ensemble network model, the M communication groups in the training system are invoked to execute the parallel training tasks of the corresponding sub-network models according to the execution method indicated by their respective parallel strategies, thereby obtaining the trained network model and accelerating the model training process.

[0258] Furthermore, the parallel strategy of any sub-network model is also used to indicate the parallel parameters of that sub-network model. When performing strategy decision processing on any sub-network model Mi (where i is an integer less than or equal to M), the strategy search space of the sub-network model Mi is first obtained. This space includes multiple reference parallel strategies pre-set for it. Then, based on the reference parallel parameters indicated by each reference parallel strategy, the training time of the sub-network model Mi under each reference parallel strategy is determined. Finally, the reference parallel strategy corresponding to the minimum training time is decided as the parallel strategy of the sub-network model Mi. This strategy decision method based on training time, through a comprehensive evaluation of multiple reference parallel strategies, can select the most suitable parallel strategy for the sub-network model Mi from many strategies, enabling the sub-network model Mi to complete the training task in the shortest time during the training process, further improving computational efficiency and reducing unnecessary waste of computational resources.

[0259] In determining the training time of the sub-network model Mi under the first reference parallel strategy based on the reference parallel parameters indicated by the first reference parallel strategy, the sub-network model Mi is first initialized as a model instance in the first training device. This model instance is then used to process a training sample, obtaining the unit training time of the sub-network model Mi on the first training device. Next, the first product of the micro-batch size and the unit training time in the reference parallel parameters indicated by the first reference parallel strategy is obtained. Finally, the first ratio between this first product and the model parallel parameters indicated by the first reference parallel strategy is taken as the training time of the sub-network model Mi under the first reference parallel strategy. This calculation method comprehensively considers the impact of data parallelism and model parallelism strategies on training time, enabling a more accurate assessment of the training time of the sub-network model Mi under a specific reference parallel strategy. This provides a more precise basis for strategy decision-making, improves the accuracy of strategy decisions, and helps to further improve computational efficiency.

[0260] Before determining the training time of subnetwork model Mi under each reference parallel policy, the unit computation of M subnetwork models on the same training device is first obtained. The target subnetwork model with the largest unit computation is then selected from these M subnetwork models. If the target subnetwork model is not Mi, the reference parallel policies in the policy search space of subnetwork model Mi are updated according to the parallel policy of the target subnetwork model. This avoids data mismatch issues and ensures smooth data flow during the training of each subnetwork model. Data mismatch may lead to additional processing or adjustments, or even data loss or incorrect processing, affecting training accuracy and computational efficiency. By updating the reference parallel policies, the number of samples processed in each batch by each subnetwork model is ensured to be consistent, improving training accuracy while reducing the additional computational overhead caused by data mismatch and improving computational efficiency.

[0261] When updating the first reference parallel policy in the policy search space of subnetwork model Mi according to the parallel policy of the target subnetwork model, a second product between the micro-batch size indicated by the parallel policy of the target subnetwork model and the data parallelism is obtained. The micro-batch size is updated using a second ratio between the second product and the data parallelism indicated by the first reference parallel policy, or the data parallelism is updated using a third ratio between the second product and the micro-batch size indicated by the first reference parallel policy. This update method matches the reference parallel policy of subnetwork model Mi with the parallel policy of the target subnetwork model, further ensuring data consistency and continuity, avoiding data mismatch problems, improving the efficiency of data transmission and processing, reducing errors and delays caused by data inconsistency, thereby improving training accuracy and computational efficiency.

[0262] When constructing a training system for an ensemble network model based on a parallel strategy using M sub-network models, the process begins by building a communication group for each sub-network model according to its parallel strategy. Then, a component communication topology is established between the M communication groups to generate the training system for the ensemble network model. During communication group construction, the number of training devices in each group is determined based on the data parallelism and model parallelism parameters specified in the parallel strategy of the sub-network models. The type of training device in the communication group is determined by evaluating the training time of the sub-network models on different types of training devices. Finally, the sub-network models are initialized in the training device cluster according to the parallel strategy and based on the number and type of training devices to construct the communication groups. This construction method allows for precise allocation of hardware resources based on the actual needs of the sub-network models. Different sub-network models have different requirements for computing power, storage capacity, etc. By evaluating the training time and selecting appropriate device types, hardware resources can be utilized more effectively, avoiding the problem of low computational efficiency caused by using unsuitable hardware resources, and improving hardware resource utilization and scalability. When establishing the component communication topology, the communication groups corresponding to the cascaded sub-network models are partitioned into P communication groups. Each communication group includes sub-communication groups derived from the partitioning of the communication group. Inter-machine communication bandwidth is allocated to each communication group, and communication channels are established between sub-communication groups belonging to different communication groups within each communication group. This partitioning process makes the division of communication groups more rational, conforms to the parallel strategy requirements of the sub-network models, and improves the efficiency of parallel training. The rational allocation of inter-machine communication bandwidth ensures the communication efficiency and performance of each communication group. The establishment of communication channels enables data transmission and communication interaction across different sub-network models, improving the efficiency and reliability of data transmission and further accelerating the model training process.

[0263] When partitioning the communication group into P communication groups, the greatest common divisor (GCD) between the data parallelism indicated by the parallel strategy of sub-network model Mi and the data parallelism indicated by the parallel strategy of sub-network model Mj is obtained and determined as the group size P. The communication groups are then partitioned based on the group size P. This method of determining the group size based on the GCD makes the partitioning of communication groups more scientific and reasonable, ensuring that the number and distribution of training devices in each communication group meet the parallel strategy requirements of the sub-network model, improving the efficiency of parallel training, and reducing the waste of computing resources caused by unreasonable communication group partitioning.

[0264] When establishing communication channels between sub-communication groups belonging to different communication groups within each communication group, the communication order of components between communication groups is determined based on the cascading relationship between sub-network models Mi and Mj. In each sub-communication group derived from the communication group with the earlier component communication order, a training device is selected as the communication transmitting device, and in each sub-communication group derived from the communication group with the later component communication order, a training device is selected as the communication receiving device, thus establishing a communication channel between the transmitting and receiving devices. This method of establishing communication channels based on cascading relationships ensures that data between different sub-network models can be transmitted accurately in the correct order, improving the efficiency and accuracy of data transmission, accelerating the model training process, and reducing the increase in training time caused by data transmission errors or delays.

[0265] In the training process of the ensemble network model, the training process is divided into M stages in a pipeline. Each stage corresponds to the parallel training task of a sub-network model. The pipeline executes, and during execution, a target communication group is determined for the current stage. This target communication group is then invoked to execute the parallel training task according to the parallel strategy indicated by its corresponding sub-network model. The pipeline approach makes the model training process more ordered, allowing each stage to be executed in parallel, reducing overall training time and improving training efficiency. During pipeline operation, buffers and asynchronous processing techniques are used to avoid stage idleness caused by data transmission delays, further improving training efficiency. Simultaneously, multiple pipeline executions continuously adjust the model parameters of the ensemble network model until the training stopping condition is met, ensuring the accuracy and stability of the model training and improving the quality of the trained network model.

[0266] Furthermore, asynchronous processing and buffering mechanisms are employed during model training. Asynchronous processing allows training tasks at different stages to be executed in parallel to some extent, reducing overall training time and improving training efficiency. Buffers are set up at each stage to temporarily store data to be processed, avoiding idle periods due to data transmission delays. This enables partial parallel execution of training tasks at different stages, further optimizing pipeline execution efficiency and improving the utilization of computing resources.

[0267] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0268] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a model training device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0269] In this application, the use of singular pronouns to denote "one or more" rather than "one and only one," unless otherwise specified. In this application, unless otherwise specified, "at least one" is intended to mean "one or more," and "more than" is intended to mean "two or more."

[0270] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0271] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A model training method, executed by a computer device, the method comprising: Obtain the ensemble network model to be trained, wherein the ensemble network model includes M sub-network models, where M is an integer greater than 1; Each of the sub-network models is processed for policy decision-making to obtain the parallel policy of each sub-network model; The parallel strategy of any of the sub-network models is used to indicate the parallel training tasks of the sub-network model during the model training process and the execution method of the parallel training tasks; Based on the parallel strategy of the M sub-network models, a training system for the integrated network model is constructed. The training system includes M communication groups; each communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model. and During the training of the integrated network model, M communication groups in the training system are invoked to execute the parallel training tasks of the corresponding sub-network models according to the execution method indicated by the parallel strategy of the corresponding sub-network model, so as to obtain the trained network model.

2. The method of claim 1, wherein the parallel strategy of any one of the sub-network models is further used to indicate the parallel parameters of the sub-network model, and any one of the sub-network models is represented as sub-network model M. i , where i is an integer less than or equal to M; For the sub-network model M i The sub-network model M is obtained by performing strategy decision processing. i Parallel strategies include: Obtain the sub-network model M i The strategy search space, the sub-network model M i The strategy search space includes the subnetwork model M. i Multiple pre-defined reference parallel strategies; Based on the sub-network model M i The reference parallel parameters indicated by each of the reference parallel policies in the policy search space are used to determine the sub-network model M. i Training time under each of the aforementioned reference parallel strategies; The subnetwork model M i The reference parallel policy corresponding to the minimum training time in the policy search space is determined by the sub-network model M. i Parallel strategy; The parallel parameters include any one or more of the following: data parallel parameters and model parallel parameters; the data parallel parameters include one or both of micro-batch size and data parallelism; the model parallel parameters include one or both of pipeline parallelism and tensor parallelism.

3. The method as described in claim 2, wherein the plurality of reference parallel strategies includes a first reference parallel strategy; the parallel parameters include data parallel parameters and model parallel parameters, wherein the data parallel parameters include micro-batch size; Based on the reference parallel parameters indicated by the first reference parallel strategy, the sub-network model M is determined. i The training time under the first reference parallel strategy includes: The sub-network model M is placed in the first training device i Initialize to the sub-network model M i The model instance calls the sub-network model M. i The model instance processes a training sample to obtain the sub-network model M. i The unit training time on the first training device; Obtain the micro-batch size from the reference parallel parameters indicated by the first reference parallel strategy, and compare it with the sub-network model M. i The first product between the unit training time on the first training device; The first ratio between the first product and the model parallel parameters indicated by the first reference parallel strategy is determined as the sub-network model M. i Training time under the first reference parallel strategy.

4. The method as described in claim 2 or 3, wherein the sub-network model M... i The reference parallel parameters indicated by each of the reference parallel policies in the policy search space are used to determine the sub-network model M. i Before the training time under each of the aforementioned reference parallel strategies, the method further includes: Obtain the unit computational cost of the M sub-network models on the same training device; From the M sub-network models, determine the target sub-network model with the largest unit computational cost; If the target subnetwork model is not the subnetwork model M i Then, the sub-network model M is updated according to the parallel strategy of the target sub-network model. i Each reference parallel policy in the policy search space.

5. The method of claim 4, wherein the plurality of reference parallel strategies includes a first reference parallel strategy; the parallel parameters include data parallel parameters, wherein the data parallel parameters include micro-batch size and data parallelism degree; Update the subnetwork model M according to the parallel strategy of the target subnetwork model. i The first reference parallel policy in the policy search space includes: Obtain the second product between the micro-batch size and the data parallelism indicated by the parallel strategy of the target sub-network model; The micro-batch size indicated by the first reference parallel policy is updated using the second ratio between the second product and the data parallelism indicated by the first reference parallel policy; or, The data parallelism indicated by the first reference parallel policy is updated using a third ratio between the second product and the micro-batch size indicated by the first reference parallel policy.

6. The method according to any one of claims 1 to 5, wherein the training system for constructing the ensemble network model based on the parallel strategy of the M sub-network models comprises: According to the parallel strategy of each sub-network model, construct the communication group corresponding to each sub-network model; Establish a component communication topology among M communication groups to generate a training system for the integrated network model.

7. The method as described in claim 6, wherein any one of the sub-network models is represented as sub-network model M. i The integrated network model also includes the sub-network model M. i Subnetwork model M with cascading relationships j The sub-network model M i Corresponding to communication group i, the sub-network model M j Corresponding communication group j; Both i and j are integers less than or equal to M, and i is not equal to j; The establishment of the component communication topology among the M communication groups includes: The communication group i and the communication group j are divided into P communication groups, where P is a positive integer; each communication group includes at least one sub-communication group obtained from the division of communication group i and at least one sub-communication group obtained from the division of communication group j. In the process of establishing the component communication topology between communication group i and communication group j, inter-machine communication bandwidth is allocated to each communication group, and communication channels between sub-communication groups belonging to different communication groups are established in each communication group.

8. The method of claim 7, wherein the parallel strategy of any one of the sub-network models is further used to indicate the parallel parameters of the sub-network model, the parallel parameters including data parallel parameters, the data parallel parameters including data parallelism degree; The process of grouping communication group i and communication group j to obtain P communication groups includes: Obtain the sub-network model M i The parallel strategy indicates the degree of data parallelism and the sub-network model M. j The parallel strategy indicates the greatest common divisor among the data parallelisms; The greatest common divisor is determined as the group number P; Based on the number of groups P, the communication group i and the communication group j are divided into P communication groups.

9. The method of claim 7 or 8, wherein each of the sub-communication groups comprises one or more training devices; The establishment of communication channels between sub-communication groups belonging to different communication groups in each of the communication groups includes: According to the cascade relationship between the sub-network model M i and the sub-network model M j , the component communication order between the communication group i and the communication group j is determined. In each of the sub-communication groups obtained by dividing the communication groups according to the component communication sequence, a training device is selected as the communication transmitting device. In each of the sub-communication groups obtained by dividing the communication groups with the component communication sequence later, a training device is selected as the communication receiving device. Establish a communication channel between the communication transmitting device and the communication receiving device.

10. The method according to any one of claims 6 to 9, wherein any one of the sub-network models is represented as sub-network model M. i The sub-network model M i The corresponding communication group i, where i is an integer less than or equal to M; The parallel strategy of any of the sub-network models is also used to indicate the parallel parameters of the sub-network model, the parallel parameters including data parallel parameters and model parallel parameters, the data parallel parameters including data parallelism degree; According to the parallel strategy of the sub-network model M i The communication group i is constructed, including: based on the sub-network model M i a data parallelism in a parallel parameter indicated by the parallel strategy and a model parallel parameter, determine the number of training devices in the communication group i; obtaining device categories in a training device cluster for model training, and determining training time consumptions on training devices of each device category for the sub-network model M i according to a parallel strategy of the sub-network model M i ​ based on the sub-network model M i a training duration on a training device of each device category, the device category of the training devices in the communication group i being determined; According to the parallel strategy of the sub-network model M i , and based on the number and types of training devices in the communication group i, the sub-network model M i is initialized in the training device cluster to build the communication group i.

11. The method of claim 10, determining the device class of the training devices in the communication group i based on the sub-network model M i training on each device class of training devices, determining the device class of the training devices in the communication group i comprises: based on the sub-network model M i a training time on the training device of each device category is consumed, and a target device category corresponding to a minimum training time is determined. Obtain the number of training devices of the target device type from the training device cluster; When the number of training devices of the target device type is greater than or equal to the number of training devices in communication group i, the target device type is determined as the device type of the training devices in communication group i.

12. The method of claim 10 or 11, wherein the data parallelism parameter further includes the micro-batch size; According to the sub-network model M i The parallel strategy determines the sub-network model M. i Training time on training devices for each device type includes: acquiring the sub-network model M i unit training time consumption on the training device of each device category; Based on the sub-network model M i The parallel strategy indicates the micro-batch size, model parallel parameters, and the sub-network model M. i The sub-network model M is determined based on the unit training time on the training device for each device type. i Training time on training equipment for each type of equipment.

13. The method according to any one of claims 1 to 12, wherein during the model training of the ensemble network model, calling M communication groups in the training system to execute the parallel training task of the corresponding sub-network model according to the execution mode indicated by the parallel strategy of the corresponding sub-network model, comprises: During the training of the integrated network model, the training process of the integrated network model is divided into M stages in the pipeline, and each stage corresponds to the parallel training task of a sub-network model. The pipeline is executed, and during the execution of the pipeline, a target communication group for executing the current stage is determined. The target communication group is invoked to execute the parallel training task of the sub-network model corresponding to the target communication group in accordance with the execution method indicated by the parallel strategy of the sub-network model corresponding to the target communication group. The current stage is the stage currently being executed among the M stages.

14. A model training apparatus, the apparatus comprising: An acquisition unit is used to acquire an ensemble network model to be trained, wherein the ensemble network model includes M sub-network models, where M is an integer greater than 1; The processing unit is used to perform policy decision processing on each of the sub-network models to obtain the parallel policy of each of the sub-network models; The parallel strategy of any of the sub-network models is used to indicate the parallel training tasks of the sub-network model during the model training process and the execution method of the parallel training tasks; The processing unit is further configured to construct a training system for the integrated network model based on the parallel strategy of the M sub-network models. The training system includes M communication groups; each communication group corresponds to one sub-network model, and each communication group is constructed according to the parallel strategy of the corresponding sub-network model. The processing unit is further configured to, during the model training process of the integrated network model, call M communication groups in the training system to execute the parallel training tasks of the corresponding sub-network models according to the execution mode indicated by the parallel strategy of the corresponding sub-network models, so as to obtain the trained network model.

15. A computer device, comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the model training method as described in any one of claims 1-13.

16. A computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed as described in any one of claims 1-13.

17. A computer program product comprising a computer program that, when executed by a processor, implements the model training method as described in any one of claims 1-13.

Images