Training method and apparatus, electronic device, computer program, and storage medium

Abstract

Embodiments of the present application disclose a training method and apparatus, an electronic device, a computer program, and a storage medium. The method in the embodiments of the present application comprises: determining a plurality of model components that together constitute a multi-modal model (110); on the basis of a structural parameter of each model component, determining a training resource requirement and a component dependency relationship of each model component (120); on the basis of idle computing device resources and the training resource requirement, determining a resource allocation solution corresponding to each model component (130); and on the basis of the component dependency relationship and the resource allocation solution, allocating the computing device resources to each model component to perform distributed model training, so as to obtain a trained multi-modal model (140). In the embodiments of the present application, by determining the training resource requirement of each model component, the resource allocation of a computing device can be optimized, thereby realizing a higher resource utilization rate; and it is ensured that each component can be trained in the most suitable environment, thereby increasing the training speed, and improving the overall training effect. Thus, the present solution can improve the speed and efficiency of model training.

Description

Training methods, devices, electronic equipment, computer programs, and storage media

[0001] This application claims priority to Chinese Patent Application No. 2024116210003, filed on November 14, 2024, entitled “Training Method, Apparatus, Electronic Device, Computer Program and Storage Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computers, and more specifically to a training method, apparatus, electronic device, computer program, and storage medium. Background Technology

[0003] Multimodal tasks refer to the processing and analysis of data in multiple different modalities, such as images, text, and audio. Significant progress has been made in the field of artificial intelligence in multimodal tasks, with research areas including image caption generation, video content understanding, text-to-image generation, and image-to-text retrieval. To handle data in different formats, current mainstream multimodal models typically employ their own encoding and decoding components to parse each data format.

[0004] In the training of multimodal models, a common approach is to perform multiple training iterations. Each iteration fixes the parameters of a subset of the model's components while updating the parameters of another subset, repeating this process multiple times until all components of the multimodal model have been trained. While effective, this method results in slow training speed and low utilization of computational resources. Summary of the Invention

[0005] This application provides a training method, apparatus, electronic device, computer program, and storage medium that can improve training speed and resource utilization.

[0006] This application provides a distributed training method for a model, including:

[0007] Identify the multiple model components that constitute the multimodal model;

[0008] Based on the structural parameters of each model component, determine the training resource requirements and component dependencies of each model component;

[0009] Based on the available computing resources and training resource requirements, determine the resource allocation scheme for each model component;

[0010] Based on component dependencies and resource allocation schemes, computing resources are allocated to each model component for distributed model training, resulting in a trained multimodal model.

[0011] This application also provides a distributed model training device, including:

[0012] Model units are used to obtain multimodal models, which include multiple model components;

[0013] Demand unit, used to determine the resource requirements of each model component;

[0014] The allocation unit is used to allocate corresponding devices to each model component according to resource requirements based on the device parameters of each device in the computing cluster.

[0015] The training unit is used to train the model components corresponding to the device using the device, so as to obtain the trained multimodal model.

[0016] In some embodiments, a resource allocation scheme for each model component is determined based on available computing resources and training resource requirements, including:

[0017] For each model component, determine the component load and storage requirements corresponding to that model component;

[0018] Based on component load, storage requirements, and available computing resources, determine the resource allocation scheme for each model component.

[0019] In some embodiments, computing device resources are allocated to each model component according to component dependencies and resource allocation schemes for distributed model training, resulting in a trained multimodal model, including:

[0020] Based on component dependencies and resource allocation schemes, determine the parallel training components and serial training components for the model training task.

[0021] According to the resource allocation scheme, the resource configuration strategy is generated when parallel training components and serial training components execute training tasks;

[0022] The parallel training components and serial training components are trained in a distributed manner according to component dependencies and resource allocation strategies to obtain the trained multimodal model.

[0023] In some embodiments, the parallel training component includes at least two encoding components and at least two decoding components; the serial training component includes a main component; the parallel training component and the serial training component are trained in a distributed manner according to component dependencies and resource allocation strategies to obtain a trained multimodal model, including:

[0024] According to the resource allocation strategy, configure encoding training equipment and main training equipment for at least two encoding components and the main component respectively;

[0025] The coding component is trained by using a coding training device and a main training device to obtain the trained coding component.

[0026] Configure decoding training equipment for at least two decoding components according to the resource allocation strategy;

[0027] The main component and the decoding component are trained collaboratively using an encoding device, a main training device, and a decoding training device to obtain the trained main component and the trained decoding component.

[0028] Based on the trained encoding component, trained subject component, and trained decoding component, a trained multimodal model is obtained.

[0029] In some embodiments, an encoding component is trained collaboratively using an encoding training device and a main training device to obtain a trained encoding component, including:

[0030] By employing the collaboration of an encoding training device and a main training device, the parameter updates of the main component are frozen. Training data is input in parallel into at least two encoding components, and the outputs of at least two encoding components are used as the inputs of the main component to train at least two encoding components, thereby obtaining the trained encoding components.

[0031] In some embodiments, an encoding device, a main training device, and a decoding training device are used to collaboratively train the main component and the decoding component to obtain the trained main component and the trained decoding component, including:

[0032] By employing the collaboration of an encoding device, a main training device, and a decoding training device, the trained encoding components are frozen, and training data is input in parallel into at least two encoding components. The output of at least one encoding component is used as the input of the main component, and the output of the main component is used as the parallel input of at least two decoding components. The main component and at least two decoding components are trained to obtain the trained main component and the trained decoding component.

[0033] In some embodiments, a collaborative approach is adopted between an encoding training device and a main training device to freeze parameter updates of the main component, input training data in parallel into at least two encoding components, use the outputs of at least two encoding components as inputs to the main component, train at least two encoding components, and obtain trained encoding components, including:

[0034] In an encoding training device, training data is input in parallel into at least two encoding components to obtain the outputs of at least two encoding components;

[0035] Freeze the parameter updates of the main component. In the main training device, use the outputs of at least two encoding components as the inputs of the main component to obtain the output of the main component.

[0036] In the coding training device, the parameters of the coding component are iteratively updated based on the output of the main component until the iteration termination condition is met, thus obtaining the trained coding component.

[0037] In some embodiments, a collaborative approach is employed involving an encoding device, a main training device, and a decoding training device. The trained encoding components are frozen, training data is input in parallel to at least two encoding components, the output of at least one encoding component is used as the input to the main component, and the output of the main component is input in parallel to at least two decoding components. The main component and at least two decoding components are then trained to obtain the trained main component and the trained decoding component, including:

[0038] Freeze the parameter updates of the encoded components after training. In the encoded training device, the training data is input in parallel into at least two encoded components to obtain the outputs of at least two encoded components.

[0039] In the main training device, the outputs of at least two encoding components are used as the inputs of the main component to obtain the output of the main component;

[0040] In the decoding training device, the output of the main component is input in parallel to at least two decoding components to obtain the output of at least two decoding components;

[0041] In the main training device and the decoding training device, the parameters of the main component and at least two decoding components are iteratively updated according to the output of the two decoding components until the iteration termination condition is met, so as to obtain the trained main component and the trained decoding component.

[0042] In some embodiments, the main component and at least two decoding components are trained to obtain trained main components and trained decoding components, including:

[0043] The parameter update of the main component is frozen, the additional parameters of the main component are introduced, and the additional parameters of the main component and the parameters of the decoding components of at least two decoding components are iteratively updated until the iteration termination condition is met, so as to obtain the trained decoding components and the additional parameters of the main component after the iteration termination.

[0044] The extra parameters and main components of the main body after the iteration terminates are used as the main components after training.

[0045] In some embodiments, before configuring the decoding training device for at least two decoding components according to a resource configuration strategy, the method further includes:

[0046] At least two decoding components are offloaded from the multimodal model, and the structural parameters of at least two decoding groups are stored on the device hard disk;

[0047] Before configuring decoding training devices for at least two decoding components according to the resource allocation strategy, the following is also included:

[0048] Read the structural parameters of at least two decoding components from the device's hard drive, and load the at least two decoding components into the multimodal model.

[0049] In some embodiments, configuring a decoding training device for at least two decoding components according to a resource configuration strategy includes:

[0050] Determine the resource update strategy for the trained encoding components;

[0051] Shared training devices are determined from the coding training devices based on resource update strategies;

[0052] Configure at least two decoding components with a shared training device and at least one of the decoding training devices, according to the resource allocation strategy.

[0053] This application also provides an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute steps in any of the distributed training methods for models provided in this application.

[0054] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the distributed training methods for models provided in this application.

[0055] The embodiments of this application can determine multiple model components constituting a multimodal model; based on the structural parameters of each model component, determine the training resource requirements and component dependencies of each model component; determine the resource allocation scheme corresponding to each model component according to the available computing device resources and training resource requirements; and allocate computing device resources to each model component for distributed model training according to the component dependencies and resource allocation scheme to obtain the trained multimodal model.

[0056] This application decouples each model component in a multimodal model, breaking down a complete model into multiple distinct components. By clearly defining the resource requirements of each model component, it optimizes the resource allocation of the computing cluster, thereby re-matching different components with devices. This addresses the problem of uneven distribution of computing resources during the overall training process, which leads to slow training speeds and achieves more efficient training. Dynamic allocation based on device parameters ensures that each component operates in the most suitable environment, improving hardware utilization efficiency. Furthermore, assigning a corresponding device to each component for model training enables parallel execution of training for all components, further accelerating the training process and improving overall training effectiveness. The embodiments of this application are particularly effective in large-scale multimodal tasks, improving both training speed and resource utilization. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1a is a schematic diagram of the allocation relationship of the distributed training method for the model provided in the embodiments of this application;

[0059] Figure 1b is a flowchart illustrating the distributed training method for the model provided in an embodiment of this application;

[0060] Figure 1c is a schematic diagram of the multimodal model structure of the distributed training method provided in the embodiment of this application;

[0061] Figure 1d is a schematic diagram of the first stage of training of the distributed training method for the model provided in the embodiment of this application;

[0062] Figure 1e is a schematic diagram of the second stage of training of the distributed training method for the model provided in the embodiment of this application;

[0063] Figure 2 is a schematic diagram of the structure of the distributed training device for the model provided in an embodiment of this application;

[0064] Figure 3 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] This application provides a training method, apparatus, electronic device, computer program, and storage medium.

[0067] Specifically, the distributed training device for this model can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0068] In some embodiments, the distributed model training device can also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the distributed model training method of this application.

[0069] In some embodiments, the terminal may also be used as a server to perform some or all of the functions of a server.

[0070] For example, referring to Figure 1a, the electronic device can be a server. The server can acquire a multimodal model, which includes multiple model components, namely model component 1, model component 2, ..., model component i+4; determine the resource requirements of each model component; based on the device parameters of each device in the computing cluster, allocate the corresponding device to each model component according to the resource requirements, for example, allocate model component i+1 and model component i+2 to device j+1; and use the device to train the model component corresponding to the device to obtain the trained multimodal model.

[0071] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0072] In this embodiment, a distributed model training method is provided, as shown in Figure 1b. The specific process of this distributed model training method is as follows:

[0073] 110. Identify the multiple model components that constitute the multimodal model.

[0074] Multimodal models are neural network models capable of processing and understanding multiple types (or modalities) of data, including text, images, audio, and video. Multimodal models can extract features from and understand information across these different modalities, enabling them to perform more complex tasks.

[0075] A modal model comprises multiple model components (sub-models), each of which may be specifically designed to process data of a particular modality or data at a specific stage of the model inference process. Therefore, their computational resource consumption during training will differ. Examples include encoders, decoders, and backbone components.

[0076] The core components are the heart of the model, responsible for processing and fusing data from different modalities. For example, core components can include neural networks with a Transformer architecture, convolutional neural networks (CNNs), and so on. They can also be models like BERT or GPT, or even an LLM (Large Language Model).

[0077] The encoding component can extract useful features from the input data, such as converting the input data into a feature vector. For example, the encoding component can include video encoding components, audio encoding components, image encoding components, text encoding components, etc. For example, the text encoding component can be a recurrent neural network (RNN), such as LSTM and GRU; for example, the image encoding component can be a convolutional neural network (CNN), etc.

[0078] The decoding component is used to generate the final output based on the feature vector, such as translated text or generated image descriptions. For example, the decoding component may include a video decoding component, an audio decoding component, an image decoding component, and a text decoding component.

[0079] For example, referring to Figure 1c, the multimodal model includes an encoding component, a main component, and a decoding component. The encoding component includes an image encoding component, an audio encoding component, and a video encoding component. The main component is an LLM. The decoding component includes an image decoding component, an audio decoding component, and a video decoding component.

[0080] 120. Based on the structural parameters of each model component, determine the training resource requirements and component dependencies of each model component.

[0081] Training resource requirements refer to the computational resources consumed in training the model component, such as component load and storage requirements. Component load refers to the computational resources consumed by the model component during training, such as the computational load (FLOPs), memory usage, and GPU memory usage required by a certain layer during forward and backward propagation. A higher load indicates a heavier computational burden on that component within the overall model. This consumption is understood to be based on predicted values ​​or historical experience. Storage requirements refer to the storage resources consumed by the model component during training, such as cache usage and memory usage.

[0082] Component dependencies refer to the input-output relationships between components. For example, suppose a model consists of an encoding component, a main component, and a decoding component. The encoding component must complete its computation first, then the main component uses its output as input, and finally, the decoding component uses the output of the main component to perform the final task computation. The dependency relationship between these three components can be described as follows:

[0083] Encoding component: Responsible for converting raw input (such as images or text) into feature representations. The encoding component is the starting point of the entire process and therefore has no prerequisites. Components that depend on the encoding component are the main components.

[0084] The main component receives the feature representations generated by the encoding component and further processes these features (such as adding contextual information or enhancing the abstract representation of features). The main component depends on the output of the encoding component as its input. Components that depend on the main component are the decoding components.

[0085] The decoding component is responsible for converting the abstract features generated by the main component into the final output (such as classification results or generated text). The encoding component is the endpoint of the entire process and therefore has no subsequent dependencies. The decoding component depends on the output of the main component.

[0086] Step 120 can determine the component load of each model component based on its structural parameters through methods such as reading, calculation, or estimation. For example, the larger the number of parameters in a model component and the deeper the model layers, the more computational resources are required. Therefore, in some embodiments, the memory requirements for training a model component can be estimated by the number of its parameters. It is understood that the aforementioned structural parameters can be hyperparameters.

[0087] For example, computational resources can be estimated by using the model components' own attributes, such as the number of layers, convolutional kernel size, number of channels, feature map size, and complexity.

[0088] For example, the computational cost of convolutional layers in a model component can be calculated using a formula:

[0089] Convolutional FLOPs = 2 × (Number of input channels × Kernel size × Number of output channels × Output feature map size)

[0090] For example, the computational cost of fully connected layers in a model component can be calculated using a formula:

[0091] Fully connected FLOPs = 2 × (number of input neurons × number of output neurons)

[0092] For example, the computational cost of an embedded Transformer layer with dimension d and complexity n² in a model component can be calculated using the following formula:

[0093] Transformer FLOPs = 4 × n² × d

[0094] Finally, the convolutional FLOPs, fully connected FLOPs, and Transformer FLOPs are summed to obtain the total computational cost of this model component:

[0095] Total computational cost = Convolutional FLOPs + Fully connected FLOPs + Transformer FLOPs

[0096] In some embodiments, the resource requirements for training different model components vary. For example, image processing components often have high computational resource requirements, particularly in terms of GPU memory and FLOPs (Flatform Counts). Text processing components may have slightly lower GPU memory requirements, but for long texts, computational resource consumption is still significant. The computational resource consumption of audio processing components depends on the length and sampling rate of the audio. When processing long audio data, especially in tasks requiring real-time processing, the computational and GPU memory resource requirements are also substantial.

[0097] 130. Based on the available computing resources and training resource requirements, determine the resource allocation scheme for each model component.

[0098] The computing resources can be distributed computing systems, cloud computing platforms, etc., and the devices can be independent computing nodes in a computing cluster, or physical servers or virtual machines. Each device has its own hardware resource parameters, such as CPU, GPU, memory, video memory, hard disk storage, network bandwidth, etc.

[0099] Resource allocation schemes refer to how to effectively allocate available computing resources (such as CPU, GPU, memory, etc.) to different components during computation or training tasks to meet their needs and optimize overall performance. A good resource allocation scheme can ensure that each task runs efficiently within the available resources, avoid resource waste, and improve training speed and model performance.

[0100] In some embodiments, these devices can be managed and scheduled through cluster scheduling systems such as Kubernetes, Slurm, Ray, etc., and can dynamically allocate computing resources to support different deep learning tasks.

[0101] Different model components have different computational and memory requirements during training. Therefore, a reasonable resource allocation strategy can improve model training efficiency.

[0102] For example, if all model components are numbered from 1 to n, the corresponding equipment can be allocated to each model component based on the resource requirements of all model components, such as the computational requirement C = {C1, C2, ..., Cn}, the storage requirement Mf = {Mf1, Mf2, ..., Mfn} in the frozen state, and the storage requirement M = {M1, M2, ..., Mn} in the normal state, and based on the total computing resources Γ and total storage resources Θ of the equipment, so as to maximize the utilization efficiency of equipment resources.

[0103] For example, define an array dp[i][j][k], representing the maximum computational load achievable using j units of computing resources and k units of storage resources when considering the first i components. The size of this array is (n+1)×(Γ+1)×(Θ+1)(n+1). Traverse all components and fill the dp table according to the state transition equation. Finally, the values ​​in dp[n][Γ][Θ]dp[n] obtained through the traversal represent the optimal allocation scheme of model components under given resources.

[0104] This embodiment transforms the allocation problem of model components and device resources into a knapsack problem and uses dynamic programming to solve it, efficiently finding the optimal allocation scheme for model components in a device cluster. This process not only improves resource utilization efficiency but also helps to rationally allocate computing and storage resources during complex model training. For example, the number of idle computing resources is fixed. Since each component has corresponding component load and storage requirements, the goal of allocation can be to select some components to be trained on a certain computing resource device, minimizing the device's usage while maximizing the number of components without exceeding the device's capacity. Of course, other conditions can also be considered, such as maximizing the utilization of computing resources or balancing the computing resource load.

[0105] Therefore, in some embodiments, training resource requirements may include component load and storage requirements. Based on the available computing resources and training resource requirements, a resource allocation scheme is determined for each model component, including:

[0106] For each model component, determine the component load and storage requirements corresponding to that model component;

[0107] Based on component load, storage requirements, and available computing device resources, a resource allocation scheme is determined for each model component. For example, referring to Figure 1a, model components 1 and 2 can be allocated to device 1, model components 3 and 4 to device 2, model component i to device j, model components i+1 and i+2 to device j+1, and model components i+3 and i+4 to device j+2.

[0108] 140. Based on the component dependencies and resource allocation scheme, allocate computing device resources to each model component for distributed model training to obtain the trained multimodal model.

[0109] Distributed model training refers to the process of training a model in parallel on multiple computing devices. Distributed training can improve training speed and efficiency, especially when dealing with large-scale data or complex models.

[0110] For example, in some embodiments, distributed model training can be divided into parallel training and serial training, that is, model components can be divided into parallel training components and serial training components. Parallel training components refer to components that can be trained simultaneously. Typically, because there are no strong dependencies between components, they can be processed in parallel on different computing devices, thereby accelerating the overall training speed. Serial training components are those that need to be trained sequentially. Due to component dependencies, such as the output of one component being the input of another, serial training may lead to longer training times, but it ensures that the dependencies between components are satisfied.

[0111] Therefore, step 140 includes:

[0112] Based on component dependencies and resource allocation schemes, determine the parallel training components and serial training components for the model training task.

[0113] According to the resource allocation scheme, the resource configuration strategy is generated when parallel training components and serial training components execute training tasks;

[0114] The parallel and serial training components are trained in a distributed manner according to component dependencies and resource allocation strategies to obtain the trained multimodal model. The resource allocation strategy is a specific configuration generated based on a resource allocation scheme, which guides the parallel and serial training components in how to use computing resources when executing training tasks, and may include memory allocation, computing power allocation, etc.

[0115] In some embodiments, as can be seen from the component dependencies, the encoding and decoding components can be trained independently without waiting for the output of other components. Therefore, the parallel training component may include at least two encoding components and at least two decoding components. The main component needs to be trained in a specific order to satisfy the dependencies, so the serial training component may include the main component.

[0116] In some embodiments, each device may correspond to at least one model component. If the device corresponds to multiple model components, the parallel execution of training tasks for multiple model components can be planned using a greedy strategy.

[0117] For example, the component and resource allocation scheme obtained in step 130 may be: Encoder 1 is allocated to device A, Encoder 2 is allocated to device B, Decoder 1 is allocated to device C, and the main model (LLM) is allocated to device D; based on the computing load and storage requirements, a parallel strategy can be determined, for example, device A processes Encoder 1 and Encoder 2 simultaneously, device B processes Decoder 1 independently, and device C processes the main model (LLM) independently.

[0118] Stage-wise training allows for the gradual optimization of different model components. Each stage reduces computational complexity by fixing (freezing) the parameters of certain model components, allowing the focus to concentrate on optimizing other components. Fixing parameters means not updating the weights of these components during training, i.e., not calculating their gradients. This saves computational resources, maintains model stability, and prevents overfitting or over-adjustment of certain model components.

[0119] For example, the encoding component is trained in the first stage, and the body component and decoding component are trained in the second stage.

[0120] Therefore, by performing distributed training on parallel and serial training components according to component dependencies and resource allocation strategies, a trained multimodal model is obtained, including:

[0121] According to the resource allocation strategy, configure encoding training equipment and main training equipment for at least two encoding components and the main component respectively;

[0122] The coding component is trained by using a coding training device and a main training device to obtain the trained coding component.

[0123] Configure decoding training equipment for at least two decoding components according to the resource allocation strategy;

[0124] The main component and the decoding component are trained collaboratively using an encoding device, a main training device, and a decoding training device to obtain the trained main component and the trained decoding component.

[0125] Based on the trained encoding component, trained subject component, and trained decoding component, a trained multimodal model is obtained.

[0126] Collaborative training refers to the process in which multiple devices work together to train certain components, which helps to make full use of resources and improve training efficiency.

[0127] Therefore, in some embodiments, the coding component is trained collaboratively using a coding training device and a main training device to obtain the trained coding component, including:

[0128] By employing the collaboration of an encoding training device and a main training device, the parameter updates of the main component are frozen. Training data is input in parallel into at least two encoding components, and the outputs of at least two encoding components are used as the inputs of the main component to train at least two encoding components, thereby obtaining the trained encoding components.

[0129] Freezing the parameters of the main component ensures that the main component remains stable during the training of the encoding component, preventing its parameters from changing during training. This helps to focus on optimizing the performance of the encoding component, reduce computational overhead, prevent overfitting, and ensure a more efficient and effective training process.

[0130] Training frozen model components still consumes some computational resources, but significantly less than training components in their normal state. For example, training frozen model components does not involve backpropagation and gradient updates, but it still consumes some computational resources during the forward propagation phase.

[0131] In some embodiments, an encoding device, a main training device, and a decoding training device are used to collaboratively train the main component and the decoding component to obtain the trained main component and the trained decoding component, including:

[0132] By employing the collaboration of an encoding device, a main training device, and a decoding training device, the trained encoding components are frozen, and training data is input in parallel into at least two encoding components. The output of at least one encoding component is used as the input of the main component, and the output of the main component is used as the parallel input of at least two decoding components. The main component and at least two decoding components are trained to obtain the trained main component and the trained decoding component.

[0133] Freezing the parameters of the encoded component after training ensures that the encoded component remains stable during the training of the main and decoder components, preventing its parameters from changing during training. This helps to focus on optimizing the performance of the main and decoder components, reducing computational overhead, preventing overfitting, and ensuring a more efficient and effective training process.

[0134] In some embodiments, a collaborative approach is adopted between an encoding training device and a main training device to freeze parameter updates of the main component, input training data in parallel into at least two encoding components, use the outputs of at least two encoding components as inputs to the main component, train at least two encoding components, and obtain trained encoding components, including:

[0135] In an encoding training device, training data is input in parallel into at least two encoding components to obtain the outputs of at least two encoding components;

[0136] Freeze the parameter updates of the main component. In the main training device, use the outputs of at least two encoding components as the inputs of the main component to obtain the output of the main component.

[0137] In the coding training device, the parameters of the coding component are iteratively updated based on the output of the main component until the iteration termination condition is met, thus obtaining the trained coding component.

[0138] Parallel input can significantly improve training efficiency because multiple encoding components can process data simultaneously and complement each other.

[0139] In some embodiments, a collaborative approach is employed involving an encoding device, a main training device, and a decoding training device. The trained encoding components are frozen, training data is input in parallel to at least two encoding components, the output of at least one encoding component is used as the input to the main component, and the output of the main component is input in parallel to at least two decoding components. The main component and at least two decoding components are then trained to obtain the trained main component and the trained decoding component, including:

[0140] Freeze the parameter updates of the encoded components after training. In the encoded training device, the training data is input in parallel into at least two encoded components to obtain the outputs of at least two encoded components.

[0141] In the main training device, the outputs of at least two encoding components are used as the inputs of the main component to obtain the output of the main component;

[0142] In the decoding training device, the output of the main component is input in parallel to at least two decoding components to obtain the output of at least two decoding components;

[0143] In the main training device and the decoding training device, the parameters of the main component and at least two decoding components are iteratively updated according to the output of the two decoding components until the iteration termination condition is met, so as to obtain the trained main component and the trained decoding component.

[0144] By inputting training data into multiple encoding components in parallel, the efficiency of feature extraction is improved, and the training time is shortened. Simultaneously, freezing the parameters of the encoded components after training ensures the stability of the main components during training, avoiding interference caused by parameter updates and promoting convergence speed and stability. The output of the main components serves as the input to the decoding components, ensuring that the decoding process is based on the latest features, thereby enhancing the model's expressiveness. Furthermore, the collaborative training mechanism iteratively updates the parameters of both the main and decoding components, ensuring that each component can dynamically adapt to each other's needs, thus improving the overall model performance.

[0145] In some embodiments, the main component and at least two decoding components are trained to obtain trained main components and trained decoding components, including:

[0146] The parameter update of the main component is frozen, the additional parameters of the main component are introduced, and the additional parameters of the main component and the parameters of the decoding components of at least two decoding components are iteratively updated until the iteration termination condition is met, so as to obtain the trained decoding components and the additional parameters of the main component after the iteration termination.

[0147] The extra parameters and main components of the main body after the iteration terminates are used as the main components after training.

[0148] In this context, "matrix additional parameters" refers to the additional trainable parameters introduced into the principal components. These parameters aim to enhance the model's expressive power or adaptability. For example, the introduced principal additional parameters could be additional fully connected layers, feature transformation matrices, low-rank matrices, regularization parameters, etc., added to the principal components.

[0149] For example, the main components can be adaptively fine-tuned by introducing a low-rank matrix. By introducing a low-rank matrix, the number of parameters that need to be trained is significantly reduced, thereby reducing memory usage and computational burden.

[0150] In some embodiments, the configuration of device resources can be dynamically adjusted when switching between the two stages. That is, devices that can be shared are identified from the encoding training devices and used for the decoding training devices, ensuring optimal utilization of device resources and making the switching between encoding and decoding tasks more efficient.

[0151] Therefore, in some embodiments, configuring decoding training devices for at least two decoding components according to a resource allocation strategy includes:

[0152] Determine the resource update strategy for the trained encoding components;

[0153] Shared training devices are determined from the coding training devices based on resource update strategies;

[0154] Configure at least two decoding components with a shared training device and at least one of the decoding training devices, according to the resource allocation strategy.

[0155] In some embodiments, adding unloading and loading steps for the decoding component can more effectively manage computing resources.

[0156] For example, to ensure that the decoding components do not consume device resources, before configuring the decoding training device for at least two decoding components according to the resource configuration strategy, the following steps are also included:

[0157] At least two decoding components are offloaded from the multimodal model, and the structural parameters of at least two decoding groups are stored on the device's hard drive to free up computational and storage resources, ensuring that the encoding components are not disturbed during training.

[0158] Before configuring decoding training devices for at least two decoding components according to the resource allocation strategy, the following is also included:

[0159] Read the structural parameters of at least two decoding components from the device's hard drive, and load at least two decoding components into the multimodal model to ensure that the decoding components can be quickly accessed and used.

[0160] For example, referring to Figure 1d, before the first stage of training, the speech decoding component, image decoding component, video decoding component, and text decoding component are unloaded from the multimodal model and stored in memory. The first stage of training can be performed using device 1 and device N+1 to train the speech coding component and image coding component in the model components, using device 2 and device N+2 to train the video coding component and alignment component in the model components, and using device N and device N-1 to train the frozen main component, thus obtaining the speech, image, video coding components, and alignment components after the first stage of training.

[0161] Then, referring to Figure 1e, before the second phase of training, the speech decoding component, image decoding component, video decoding component, and text decoding component are read from the device's memory and loaded into the multimodal model. In the second phase of training, devices N+1 and N+2 can be identified as shared devices and used for the speech, image, video, and text decoding components. Ultimately, the second phase of training can use devices 1 and 2 to train the frozen speech, image, and video encoding components and alignment components, use device N+1 to train the speech and image decoding components, and use device N+2 to train the video and text decoding components.

[0162] In some embodiments, to further improve training efficiency, step 140 further includes:

[0163] During model training, determine the load on each device in the computing cluster;

[0164] Based on the load of each device, the corresponding device is reallocated to each model component according to resource requirements.

[0165] By assessing the current load of each device in the computing cluster to effectively utilize its resources, the allocation of devices can be dynamically adjusted. This flexible resource management strategy can significantly improve the efficiency of model training, ensuring that each component can learn in an optimal environment and maximize the potential of computing resources.

[0166] As can be seen from the above, the embodiments of this application can obtain a multimodal model, which includes multiple model components; determine the resource requirements of each model component; allocate corresponding devices to each model component according to resource requirements based on the device parameters of each device in the computing cluster; and use the devices to train the model components corresponding to the devices to obtain the trained multimodal model. Therefore, this solution, through systematic model component management, accurate resource requirement assessment, and flexible device allocation strategies, can effectively improve the efficiency and effectiveness of multimodal model training. By achieving optimal resource utilization and load balancing, this method not only improves training speed but also provides a good foundation for model expansion and adaptation, thereby improving training speed and resource utilization.

[0167] The method described in the above embodiments will be further described in detail below.

[0168] In this embodiment, the method of this application embodiment will be described in detail using the stage training of a multimodal model as an example.

[0169] 1. Decoupling of multimodal models.

[0170] The multimodal model separates components at different stages. After assessing the model's requirements for computing and storage resources for each component, the sequential dependencies between the components are decoupled, thereby reorganizing the component tasks and allocating them to the corresponding computing resources for training.

[0171] The components of a multimodal model include an Encoder component, a Decoder component, and the main LLM model component, which correspond to different inputs.

[0172] 2. Multi-stage training strategy.

[0173] (1) Component requirements assessment.

[0174] Evaluate the computational cost C, storage requirements Mf in the frozen state, and storage requirements M in the normal state for each component.

[0175] All components are numbered according to their dependencies. The computational requirements of all components are denoted as C = {C1, C2, ..., Cn}, the storage requirements in the frozen state are Mf = {Mf1, Mf2, ..., Mfn}, and the storage requirements in the normal state are M = {M1, M2, ..., Mn}.

[0176] (2) Task allocation.

[0177] Based on the total computing resources Γ and total storage resources Θ of the devices, a corresponding device is allocated to each model component to maximize the utilization efficiency of device resources.

[0178] The task allocation problem can be simplified into a knapsack problem, which uses dynamic programming to find the optimal allocation scheme of model components and equipment resources.

[0179] (3) Parallel strategy decision-making.

[0180] It is possible to schedule the training of multiple components to be performed in parallel within each device.

[0181] For example, a greedy strategy can be used to search the existing parallel strategy space to obtain the parallel strategy configuration corresponding to the current task load.

[0182] (4) Model training.

[0183] Distributed training is performed based on task allocation and parallel strategy decisions.

[0184] 3. Training task tidal scheduling strategy.

[0185] For example, multi-stage training may lead to resource imbalance. The following strategy is proposed:

[0186] Phase 1: Train all Encoder components, freeze LLM parameters, and do not participate in the Decoder computation.

[0187] Phase 2: Freeze all Encoders, train LLM using LoRa, and train all Decoders.

[0188] Before the task starts, the training method for each stage is obtained, and the resource allocation strategy is determined based on the combined optimization strategy. When switching between each training stage, the resources are dynamically reallocated according to the determined resource allocation strategy. Tidal scheduling improves resource utilization, reduces IO overhead for model migration, and avoids manual intervention.

[0189] This scheme aims to improve the training efficiency of multimodal models through precise task allocation and resource scheduling, ensuring resource balance across different stages, thereby achieving more efficient model training and resource utilization. As can be seen from the above, the embodiments of this application can improve training speed and resource utilization.

[0190] It is understood that in the specific embodiments of this application, related data such as device parameters and model training data are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0191] To better implement the above methods, this application also provides a distributed model training device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0192] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the model distributed training device specifically integrated into the server as an example.

[0193] For example, as shown in Figure 2, the distributed training device for the model may include a model unit 210, a demand unit 220, an allocation unit 230, and a training unit 240, as follows:

[0194] (a) Model Unit 210.

[0195] Model unit 210 is used to acquire a multimodal model, which includes multiple model components.

[0196] (ii) Demand Unit 220.

[0197] The requirement unit 220 is used to determine the training resource requirements and component dependencies of each model component based on the structural parameters of each model component.

[0198] (III) Allocation Unit 230.

[0199] The allocation unit 230 is used to determine the resource allocation scheme for each model component based on the available computing device resources and training resource requirements.

[0200] (iv) Training Unit 240.

[0201] The training unit 240 is used to allocate computing device resources to each model component according to the component dependency relationship and resource allocation scheme to carry out distributed model training and obtain the trained multimodal model.

[0202] In some embodiments, a resource allocation scheme for each model component is determined based on available computing resources and training resource requirements, including:

[0203] For each model component, determine the component load and storage requirements corresponding to that model component;

[0204] Based on component load, storage requirements, and available computing resources, determine the resource allocation scheme for each model component.

[0205] In some embodiments, computing device resources are allocated to each model component according to component dependencies and resource allocation schemes for distributed model training, resulting in a trained multimodal model, including:

[0206] Based on component dependencies and resource allocation schemes, determine the parallel training components and serial training components for the model training task.

[0207] According to the resource allocation scheme, the resource configuration strategy is generated when parallel training components and serial training components execute training tasks;

[0208] The parallel training components and serial training components are trained in a distributed manner according to component dependencies and resource allocation strategies to obtain the trained multimodal model.

[0209] In some embodiments, the parallel training component includes at least two encoding components and at least two decoding components; the serial training component includes a main component; the parallel training component and the serial training component are trained in a distributed manner according to component dependencies and resource allocation strategies to obtain a trained multimodal model, including:

[0210] According to the resource allocation strategy, configure encoding training equipment and main training equipment for at least two encoding components and the main component respectively;

[0211] The coding component is trained by using a coding training device and a main training device to obtain the trained coding component.

[0212] Configure decoding training equipment for at least two decoding components according to the resource allocation strategy;

[0213] The main component and the decoding component are trained collaboratively using an encoding device, a main training device, and a decoding training device to obtain the trained main component and the trained decoding component.

[0214] Based on the trained encoding component, trained subject component, and trained decoding component, a trained multimodal model is obtained.

[0215] In some embodiments, an encoding component is trained collaboratively using an encoding training device and a main training device to obtain a trained encoding component, including:

[0216] By employing the collaboration of an encoding training device and a main training device, the parameter updates of the main component are frozen. Training data is input in parallel into at least two encoding components, and the outputs of at least two encoding components are used as the inputs of the main component to train at least two encoding components, thereby obtaining the trained encoding components.

[0217] In some embodiments, an encoding device, a main training device, and a decoding training device are used to collaboratively train the main component and the decoding component to obtain the trained main component and the trained decoding component, including:

[0218] By employing the collaboration of an encoding device, a main training device, and a decoding training device, the trained encoding components are frozen, and training data is input in parallel into at least two encoding components. The output of at least one encoding component is used as the input of the main component, and the output of the main component is used as the parallel input of at least two decoding components. The main component and at least two decoding components are trained to obtain the trained main component and the trained decoding component.

[0219] In some embodiments, a collaborative approach is adopted between an encoding training device and a main training device to freeze parameter updates of the main component, input training data in parallel into at least two encoding components, use the outputs of at least two encoding components as inputs to the main component, train at least two encoding components, and obtain trained encoding components, including:

[0220] In an encoding training device, training data is input in parallel into at least two encoding components to obtain the outputs of at least two encoding components;

[0221] Freeze the parameter updates of the main component. In the main training device, use the outputs of at least two encoding components as the inputs of the main component to obtain the output of the main component.

[0222] In the coding training device, the parameters of the coding component are iteratively updated based on the output of the main component until the iteration termination condition is met, thus obtaining the trained coding component.

[0223] In some embodiments, a collaborative approach is employed involving an encoding device, a main training device, and a decoding training device. The trained encoding components are frozen, training data is input in parallel to at least two encoding components, the output of at least one encoding component is used as the input to the main component, and the output of the main component is input in parallel to at least two decoding components. The main component and at least two decoding components are then trained to obtain the trained main component and the trained decoding component, including:

[0224] Freeze the parameter updates of the encoded components after training. In the encoded training device, the training data is input in parallel into at least two encoded components to obtain the outputs of at least two encoded components.

[0225] In the main training device, the outputs of at least two encoding components are used as the inputs of the main component to obtain the output of the main component;

[0226] In the decoding training device, the output of the main component is input in parallel to at least two decoding components to obtain the output of at least two decoding components;

[0227] In the main training device and the decoding training device, the parameters of the main component and at least two decoding components are iteratively updated according to the output of the two decoding components until the iteration termination condition is met, so as to obtain the trained main component and the trained decoding component.

[0228] In some embodiments, the main component and at least two decoding components are trained to obtain trained main components and trained decoding components, including:

[0229] The parameter update of the main component is frozen, the additional parameters of the main component are introduced, and the additional parameters of the main component and the parameters of the decoding components of at least two decoding components are iteratively updated until the iteration termination condition is met, so as to obtain the trained decoding components and the additional parameters of the main component after the iteration termination.

[0230] The extra parameters and main components of the main body after the iteration terminates are used as the main components after training.

[0231] In some embodiments, before configuring the decoding training device for at least two decoding components according to a resource configuration strategy, the method further includes:

[0232] At least two decoding components are offloaded from the multimodal model, and the structural parameters of at least two decoding groups are stored on the device hard disk;

[0233] Before configuring decoding training devices for at least two decoding components according to the resource allocation strategy, the following is also included:

[0234] Read the structural parameters of at least two decoding components from the device's hard drive, and load the at least two decoding components into the multimodal model.

[0235] In some embodiments, configuring a decoding training device for at least two decoding components according to a resource configuration strategy includes:

[0236] Determine the resource update strategy for the trained encoding components;

[0237] Shared training devices are determined from the coding training devices based on resource update strategies;

[0238] Configure at least two decoding components with a shared training device and at least one of the decoding training devices, according to the resource allocation strategy.

[0239] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0240] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0241] As can be seen from the above, the distributed training device of this embodiment determines multiple model components constituting the multimodal model by the model unit; the demand unit determines the training resource requirements and component dependencies of each model component based on the structural parameters of each model component; the allocation unit determines the resource allocation scheme corresponding to each model component according to the available computing resources and training resource requirements; and the training unit allocates computing resources to each model component according to the component dependencies and resource allocation scheme to perform distributed model training, thereby obtaining the trained multimodal model. Therefore, this embodiment can improve training speed and resource utilization.

[0242] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0243] In some embodiments, the distributed model training device can also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the distributed model training method of this application.

[0244] In this embodiment, a server will be used as an example for detailed description. For instance, as shown in Figure 3, which illustrates the structural diagram of the electronic device involved in this embodiment, specifically:

[0245] The electronic device may include components such as a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will understand that the electronic device structure shown in FIG3 does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0246] The processor 310 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 320, and by calling data stored in the memory 320, it performs various functions and processes data, thereby performing overall detection of the electronic device. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 310.

[0247] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one RAM device, flash memory device, or other volatile solid-state memory device. Accordingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.

[0248] The electronic device also includes a power supply 330 that supplies power to the various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 330 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0249] The electronic device may also include an input module 340, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0250] The electronic device may also include a communication module 350. In some embodiments, the communication module 350 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 350 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0251] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 310 in the electronic device loads the executable files corresponding to the processes of one or more application programs into the memory 320 according to the following instructions, and the processor 310 runs the application programs stored in the memory 320 to realize various functions, as follows:

[0252] Identify the multiple model components that constitute the multimodal model;

[0253] Based on the structural parameters of each model component, determine the training resource requirements and component dependencies of each model component;

[0254] Based on the available computing resources and training resource requirements, determine the resource allocation scheme for each model component;

[0255] Based on component dependencies and resource allocation schemes, computing resources are allocated to each model component for distributed model training, resulting in a trained multimodal model.

[0256] As can be seen from the above, the embodiments of this application can improve training speed and resource utilization.

[0257] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0258] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the distributed training methods for models provided in embodiments of this application. For example, the instructions can execute the following steps:

[0259] Identify the multiple model components that constitute the multimodal model;

[0260] Based on the structural parameters of each model component, determine the training resource requirements and component dependencies of each model component;

[0261] Based on the available computing resources and training resource requirements, determine the resource allocation scheme for each model component;

[0262] Based on component dependencies and resource allocation schemes, computing resources are allocated to each model component for distributed model training, resulting in a trained multimodal model.

[0263] The storage medium may include: read-only memory (ROM), random access memory (RAM), RAM, or optical disc, etc.

[0264] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various optional implementations of the model training or multimodal task aspects provided in the above embodiments.

[0265] Since the instructions stored in the storage medium can execute the steps in any of the distributed training methods for models provided in the embodiments of this application, the beneficial effects that any of the distributed training methods for models provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0266] The above provides a detailed description of a distributed training method, apparatus, electronic device, and computer-readable storage medium for a model provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. For those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A distributed training method for a model, performed by an electronic device, comprising: Identify the multiple model components that constitute the multimodal model; Based on the structural parameters of each model component, determine the training resource requirements and component dependencies of each model component; Based on the available computing resources and the training resource requirements, determine the resource allocation scheme for each model component; According to the component dependencies and the resource allocation scheme, computing device resources are allocated to each model component for distributed model training to obtain the trained multimodal model.

2. The model distributed training method of claim 1, wherein, The step of determining the resource allocation scheme for each model component based on the available computing device resources and the training resource requirements includes: For each model component, determine the component load and storage requirements corresponding to that model component; Based on the component load, storage requirements, and available computing resources, a resource allocation scheme is determined for each model component.

3. The model distributed training method of claim 1, wherein, The step of allocating computing device resources to each model component for distributed model training according to the component dependencies and the resource allocation scheme to obtain the trained multimodal model includes: Based on the component dependencies and the resource allocation scheme, determine the parallel training components and serial training components for the model training task; According to the resource allocation scheme, generate the resource configuration strategy for the parallel training component and the serial training component when executing training tasks; The parallel training component and the serial training component are trained in a distributed manner according to the component dependencies and the resource allocation strategy to obtain the trained multimodal model.

4. The model distributed training method of claim 3, wherein, The parallel training component includes at least two encoding components and at least two decoding components; the serial training component includes a main component; the distributed training of the parallel training component and the serial training component according to the component dependencies and the resource allocation strategy to obtain the trained multimodal model includes: According to the resource allocation strategy, configure encoding training devices and main training devices for the at least two encoding components and the main component, respectively; The coding component is trained collaboratively using the aforementioned coding training device and the main training device to obtain the trained coding component; Configure decoding training devices for the at least two decoding components according to the resource allocation strategy; The encoding device, the main training device, and the decoding training device are used to collaboratively train the main component and the decoding component to obtain the trained main component and the trained decoding component. Based on the trained encoding component, the trained subject component, and the trained decoding component, a trained multimodal model is obtained.

5. The model distributed training method of claim 4, wherein, The step of using the encoding training device and the main training device to collaboratively train the encoding component to obtain the trained encoding component includes: By employing the collaboration of the encoding training device and the main training device, the parameter updates of the main component are frozen, training data is input in parallel to at least two encoding components, and the outputs of the at least two encoding components are used as the inputs of the main component to train the at least two encoding components, thereby obtaining the trained encoding components.

6. The model distributed training method of claim 4, wherein, The step of collaboratively training the main component and the decoding component using the encoding device, the main training device, and the decoding training device to obtain the trained main component and the trained decoding component includes: By employing the collaboration of the encoding device, the main training device, and the decoding training device, the trained encoding components are frozen, training data is input in parallel to at least two encoding components, the outputs of the at least two encoding components are used as the inputs of the main component, and the outputs of the main component are used as the parallel inputs of the at least two decoding components to train the main component and the at least two decoding components, thereby obtaining the trained main component and the trained decoding component.

7. The model distributed training method of claim 5, wherein, The method involves coordinating the encoding training device and the main training device, freezing the parameter updates of the main component, inputting training data in parallel into at least two encoding components, using the outputs of the at least two encoding components as the input of the main component, and training the at least two encoding components to obtain the trained encoding components, including: In the encoding training device, training data is input in parallel to at least two encoding components to obtain the outputs of the at least two encoding components; Freeze the parameter update of the main component, and in the main training device, use the outputs of the at least two encoding components as the input of the main component to obtain the output of the main component; In the encoding training device, the parameters of the encoding component are iteratively updated based on the output of the main component until the iteration termination condition is met, thereby obtaining the trained encoding component.

8. The model distributed training method of claim 6, wherein, The method involves coordinating the encoding device, the main training device, and the decoding training device to freeze the trained encoding components, inputting training data in parallel into at least two encoding components, using the output of the at least two encoding components as the input of the main component, and using the output of the main component as the parallel input of the at least two decoding components to train the main component and the at least two decoding components, thereby obtaining the trained main component and the trained decoding component. Freeze the parameter updates of the trained encoding components, and in the encoding training device, input the training data in parallel into at least two encoding components to obtain the outputs of the at least two encoding components; In the main training device, the outputs of the at least two encoding components are used as the inputs of the main component to obtain the output of the main component; In the decoding training device, the output of the main component is input in parallel to at least two decoding components to obtain the output of the at least two decoding components; In the main training device and the decoding training device, the parameters of the main component and the at least two decoding components are iteratively updated according to the outputs of the two decoding components until the iteration termination condition is met, so as to obtain the trained main component and the trained decoding component.

9. The model distributed training method of claim 6, wherein, The step of training the main component and the at least two decoding components to obtain the trained main component and the trained decoding components includes: Freeze the parameter update of the main component, introduce additional parameters of the main component, and iteratively update the additional parameters of the main component and the parameters of the decoding components of the at least two decoding components until the iteration termination condition is met, so as to obtain the trained decoding components and the additional parameters of the main component after the iteration termination. The additional parameters of the main body after the iteration terminates and the main body components are used as the trained main body components.

10. The model distributed training method of claim 4, wherein, Before configuring the decoding training device for the at least two decoding components according to the resource configuration strategy, the method further includes: The at least two decoding components are unloaded from the multimodal model, and the structural parameters of the at least two decoding groups are stored on the device hard disk; Before configuring the decoding training device for the at least two decoding components according to the resource configuration strategy, the method further includes: The structural parameters of the at least two decoding components are read from the device's hard disk, and the at least two decoding components are loaded into the multimodal model.

11. The model distributed training method of claim 4, wherein, Configuring the decoding training device for the at least two decoding components according to the resource configuration strategy includes: Determine the resource update strategy for the trained encoding components; Based on the resource update strategy, shared training devices are determined from the coding training devices; Configure at least one of the shared training device and the decoding training device for the at least two decoding components in accordance with the resource configuration strategy.

12. A distributed training device for a model, comprising: Model units are used to define the multiple model components that make up a multimodal model; The requirement unit is used to determine the training resource requirements and component dependencies of each model component based on the structural parameters of each model component. The allocation unit is used to determine the resource allocation scheme for each model component based on the available computing device resources and the training resource requirements. The training unit is used to allocate computing device resources to each model component for distributed model training according to the component dependencies and the resource allocation scheme, so as to obtain the trained multimodal model.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the distributed training method for the model according to any one of claims 1 to 11.

14. An electronic device comprising a processor and a memory, the memory storing a plurality of instructions; the processor loading instructions from the memory to execute steps in the distributed training method for a model according to any one of claims 1 to 11.

15. A computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform the steps of the distributed training method for a model according to any one of claims 1 to 11.

Images