Training method and apparatus for multi-modal large model

Abstract

Provided in the embodiments of the present description are a training method and apparatus for a multi-modal large model. The training solution for the multi-modal large model is provided to enable decoupling of network modules in the multi-modal large model according to functions, and phased progressive training of the decoupled network modules, thereby gradually expanding the modal support capability of the model, and achieving better performance in each modality. This training method can effectively implement stable training of data in each modality.

Description

Training methods and devices for multimodal large models

[0001] This application claims priority to Chinese Patent Application No. 202510230447.6, filed on February 27, 2025, entitled “Training Method and Apparatus for Multimodal Large Models”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This specification relates to one or more embodiments in the field of computer technology, and more particularly to a training method and apparatus for a multimodal large model. Background Technology

[0003] Large models are a general term for models with large scale. They typically have complex network structures and massive parameter counts, ranging from hundreds of millions to billions or tens of billions. Large models are usually pre-trained on a large number of text samples and exhibit good performance in natural language processing. Multimodal large models are models capable of simultaneously processing one or more types of information, including audio, video, images, and text, and can simultaneously generate at least one type of information, including text, images, and audio. They extend the application capabilities of large language models. Multimodal large models capable of handling various data formats can also be called full-modal large models.

[0004] However, during the training of a full-modal model, due to the inherent differences between modalities and the imbalance of data, conflicts and difficulties in coordination may occur when training different modalities and tasks together, making it difficult to achieve stable training or reach the best performance indicators for each modality. Summary of the Invention

[0005] This specification describes one or more embodiments of a training method and apparatus for a multimodal large model, which is used to solve one or more problems mentioned in the background art.

[0006] According to the first aspect, a training method for a multimodal large model is provided, the multimodal large model comprising: a visual encoder and a visual feature bridging module connected in sequence, an audio encoder and an audio feature bridging module connected in sequence, and a large language model connected to the visual feature bridging module and the audio feature bridging module; the method includes the following pre-training steps: encoder alignment step: training the visual feature bridging module and the audio feature bridging module using image text samples and audio text samples; image-text knowledge enhancement step: training the visual encoder, the visual feature bridging module, and the large language model using image text samples and text samples; and full-modal joint training step: training the visual encoder, the visual feature bridging module, the audio encoder, the audio feature bridging module, and the large language model using image text samples, audio text samples, and text samples.

[0007] In one embodiment, after the pre-training step, the method further includes the following steps for fine-tuning the multimodal large model: image-text instruction fine-tuning step: training a visual encoder, a visual feature bridging module, and a large language model using image text samples and text samples; visual instruction fine-tuning step: training a visual encoder, a visual feature bridging module, and a large language model using image text samples and video text samples; and multimodal instruction fine-tuning step: training a visual encoder, a visual feature bridging module, an audio encoder, an audio feature bridging module, and a large language model using image text samples, audio text samples, and text samples.

[0008] In one embodiment, the multimodal joint training step includes: sampling the same batch of image text samples, audio text samples, and text samples in turn during a single parameter update cycle; determining the model loss under the corresponding modality by processing single-modal samples in a single batch; and weighting the model losses under various modalities to determine the comprehensive model loss and adjusting the corresponding model parameters.

[0009] In one embodiment, in the multimodal joint training step, the model loss determined for a single modality sample is multiplied and balanced with a pre-determined loss weight and then used to determine the backpropagation gradient of the model parameters. The loss weight corresponding to the model loss of a single modality is determined as follows: when the multimodal large model is trained using a sample set of a predetermined size under that single modality until a first convergence condition is met, a first convergence loss that meets the first convergence condition is obtained; the loss weight corresponding to the model loss under that single modality is determined according to the ratio of the reciprocal of the first model loss to the sum of the reciprocals of the convergence losses under various modalities.

[0010] In one embodiment, the full-modal instruction fine-tuning step includes: during the model parameter adjustment process, executing the following dynamic equilibrium convergence control strategy: determining the current loss weight of the corresponding mode based on the current slope of the corresponding loss curve in each mode, wherein the current loss weight of a single mode is negatively correlated with the corresponding current slope; and using the model loss corresponding to the samples in each mode for model parameter adjustment after balancing by multiplying it with the current loss weight.

[0011] In a further embodiment, the current slope of the loss curve under a single modality is determined by obtaining the validation loss of the previous H validation cycles of the current validation cycle, wherein a single validation cycle includes multiple parameter update cycles under the single modality, and the validation loss of a single validation cycle is determined by the following normalization method: the ratio of the difference between the model loss determined by the validation set for the single validation cycle and the minimum value among the losses determined by the validation set for the previous H validation cycles, to the difference between the maximum and minimum values ​​among the model losses determined by the validation set for the previous H validation cycles;

[0012] Based on the slope coefficient of the validation loss at validation period t, determined by fitting the linear regression model, the current slope of the loss curve for the corresponding mode in the current validation period is determined.

[0013] In another further embodiment, determining the current loss weight for each modality based on the current slope of the corresponding loss curve for each modality includes: determining the corresponding convergence scores based on the normalized results of the current slopes for each modality, wherein each convergence score is negatively correlated with the corresponding current slope; mapping each convergence score to an importance coefficient through an activation function, and determining the weight coefficients of the model loss for each modality based on the product of the importance coefficients and the number of modalities; and determining the current loss weights for each modality within the current validation period based on the modality weights for each modality.

[0014] In a further embodiment, determining the corresponding current loss weights based on the weight coefficients for each modality includes: for a single modality, determining the current loss weight using an exponential moving average of the weight coefficients.

[0015] According to a second aspect, a training device for a multimodal large model is provided, the multimodal large model comprising: a visual encoder and a visual feature bridging module connected in sequence; an audio encoder and an audio feature bridging module connected in sequence; and a large language model connected to the visual feature bridging module and the audio feature bridging module; the device comprises:

[0016] The encoder alignment unit is configured to train the visual feature bridging module and the audio feature bridging module using image text samples and audio text samples.

[0017] The image and text knowledge enhancement unit is configured to train the visual encoder, visual feature bridging module, and large language model using image text samples and text samples.

[0018] The multimodal joint training unit is configured to train the visual encoder, visual feature bridging module, audio encoder, audio feature bridging module, and large language model using image text samples, audio text samples, and text samples.

[0019] In one embodiment, the apparatus further includes an instruction fine-tuning unit configured to perform instruction fine-tuning on a multimodal large model through the following steps: image-text instruction fine-tuning step: training a visual encoder, a visual feature bridging module, and a large language model using image text samples and text samples; visual instruction fine-tuning step: training a visual encoder, a visual feature bridging module, and a large language model using image text samples and video text samples; and multimodal instruction fine-tuning step: training a visual encoder, a visual feature bridging module, an audio encoder, an audio feature bridging module, and a large language model using image text samples, audio text samples, and text samples.

[0020] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.

[0021] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein executable code is stored in the memory, and the processor, when executing the executable code, implements the method of the first aspect.

[0022] The apparatus and method provided in the embodiments of this specification offer a training scheme for multimodal large models. This scheme allows for functional decoupling of each network module within the multimodal large model, followed by phased, progressive training of the decoupled network modules. This gradually expands the model's modality support capabilities and achieves optimal performance across each modality. This training method effectively addresses the problem of large differences in data distribution between modalities, enabling stable training of data across all modalities.

[0023] In a further embodiment, the technical problem of uneven training sample data volume of each modality can be solved by the technical concept of balanced sampling according to the number of iteration steps. The technical concept of measuring the convergence speed of each modality by calculating the slope of the loss curve of each modality and dynamically adjusting the loss weight of each modality according to the convergence speed can solve the technical problem of inconsistent convergence speed under training samples of each modality. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 shows a schematic diagram of a specific implementation architecture of a multimodal large model;

[0026] Figure 2 shows a schematic diagram of the training process of a multimodal large model according to an embodiment of this specification;

[0027] Figure 3 shows a schematic diagram of the pre-training process in the training process of a multimodal large model according to the embodiments of this specification;

[0028] Figure 4 illustrates a schematic diagram of the instruction fine-tuning process for a multimodal large model according to an embodiment of this specification;

[0029] Figure 5 shows a schematic diagram of the model loss and weight determination process during instruction fine-tuning in the embodiment of the specification.

[0030] Figure 6 shows a structural block diagram of a training device for a multimodal large model according to one embodiment of this specification. Detailed Implementation

[0031] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0032] Figure 1 illustrates a specific implementation architecture of a multimodal large model. As shown in Figure 1, the large model LLM is a pre-trained large language model. When using a large language model to process multimodal data, it can be extended into a multimodal large model. During the extension process, based on data of various modalities, pre-processing and post-processing architectures can be added to the original large model. Data modalities may include, for example, image-text modalities, text modalities, audio-text modalities, and video-text modalities. Among them, image-text modalities may include images and one of the following: image description information, text information in images, question-and-answer information based on images, etc. Audio-text modalities may include audio and text data, and video-text modalities may include video and text data.

[0033] A specific multimodal large model architecture may include: a visual encoder, an audio encoder, an MLP (Perceptron) linear alignment network, a large language model (LLM), an image generator, and an audio decoder. It is understood that the large language model itself is pre-trained on text; therefore, the networks in Figure 1, such as BPE encoding, word embedding, language head, and BPE decoding, can be modules corresponding to the large language model itself, used for text processing. The visual encoder and audio encoder each have a corresponding visual feature bridging module (MLP) and audio feature bridging module (MLP), used to align visual features and audio features with the feature space of the large language model. Specifically, the visual encoder and the visual feature bridging module connected to it can process text-image modal data and video modal data; the audio encoder and the audio feature bridging module connected to it can process audio modal data.

[0034] As shown in Figure 1, the input data of a multimodal large-scale model can include various modalities such as text, images, video, and audio. Each modal data is encoded by its corresponding model encoder, then aligned to a unified feature space via corresponding feature alignment networks (each MLP in Figure 1), and concatenated into an encoding tensor. Here, the unified feature space can include a space defined by at least one of the following: vector dimension and value range. After the encoding tensor is input into the large language model for unified processing, a decoding tensor is obtained. Using the decoders corresponding to each modality, at least one of the following—text, image, or audio—can be decoded.

[0035] Referring to the data processing process of the multimodal large model shown in Figure 1, it is easy to understand that the technical problems that may be encountered during the training of the multimodal large model include: stable training under various modal data; imbalance in the number of training samples under various modalities; inconsistent convergence speed under various modalities, etc.

[0036] To enable stable training of large models, this specification provides a technical concept that progressively trains a multimodal large model using various modal samples, thereby achieving stable training of the multimodal large model under various modalities. Specifically, for training samples in mixed modal scenarios, some modal processing structures can be frozen first, while only the parameters in another part of the modal processing structure are adjusted, and then the various modal processing structures are mixed for training.

[0037] Understandably, large language models are pre-trained with a large number of natural language samples and possess strong text processing capabilities. The information in audio data is typically converted into text for processing. Therefore, during the training of a multimodal large model, bridging modules (visual feature bridging module and audio feature bridging module) that align audio and image data with the feature data received by the large language model can be trained first. Since the information in audio data can be converted into text, it is relatively easy to adapt to text. After training the bridging modules for the image-text and audio modal to align visual and audio features to the feature space of the large language model, the parameters of the visual encoder, visual feature bridging module, and large language model can be further adjusted using image-text modal sample data and text modal sample data, enhancing the image-text processing capabilities of the multimodal large model architecture. Furthermore, various modal sample data can be used to adjust the parameters of each module of the entire multimodal large model, enabling the large model to process various modal data.

[0038] This progressive training approach allows for stable training of large multimodal models. Furthermore, considering training efficiency and the accuracy of the large multimodal model, some embodiments can divide the training process into two stages: first, pre-training using a large number of samples according to the methods described above; then, fine-tuning training using a smaller number of samples. Instruction fine-tuning (also known as supervised fine-tuning) is one of the important methods to enhance or activate specific capabilities (such as instruction compliance) of a large language model after pre-training. In the instruction fine-tuning stage, the focus is initially on the network architecture of the text-image modality data processing part to enhance the text-image processing capabilities of the large multimodal model. Afterwards, joint training of all modal data is performed to improve the model's processing accuracy.

[0039] The technical concept of this specification is described in detail below with reference to the accompanying drawings.

[0040] Figure 2 illustrates the training flow of a multimodal large model according to an embodiment of this specification. The execution entity of this flow can be any computer, device, or server with a certain computing power. Here, the multimodal large model is based on a large language model and also includes a visual encoder, a visual feature bridging module, an audio encoder, and an audio feature bridging module, used to process text-image modal data and audio modal data, respectively.

[0041] The visual encoder can be any network structure implemented using convolutional neural networks, such as the one shown in Figure 1, which can be implemented using NaViT and can process images of any size. The audio encoder can be implemented using SAN-M, for example. Typically, in the text generation stage, multimodal large models can consider only the text portion of the audio information.

[0042] The visual encoder and audio encoder can be pre-trained modules. The visual feature bridging module and audio feature bridging module can transform the encoded representation determined by the corresponding encoder into features that can be processed by a large language model through linear transformation, such as making the value of a single element fall within a predetermined range (e.g., greater than 0 and less than 50), transforming the dimension to a predetermined dimension (e.g., 100 dimensions), and so on.

[0043] The training samples for a multimodal large-scale model can include the following modalities: text, image, video, and audio. Each modality's training samples can also correspond to its respective modality's label data. For example, when using a multimodal large-scale model to process image-text samples (which can contain both images and related text data) to generate images, the image description text can be generated first, and then the image generator (as shown in Figure 1) can be used to generate the image. Therefore, the label data for the image-text samples can be the text description information of the image to be generated. Similarly, when using a multimodal large-scale model to process audio-text samples (which can contain both audio and text data) to generate audio information, the text information in the audio can be generated first, and then the audio decoder can add sound features to synthesize the audio. Therefore, the label data for the audio-text samples can also be the text information in the audio. In other words, the label data for samples of each modality can correspond to text information. Thus, in the processing of a multimodal large-scale model, a large language model can use an autoregressive approach to generate text.

[0044] As shown in Figure 2, the training process of the multimodal large model can include the following pre-training steps: Step 201, encoder alignment step: using image text samples and audio text samples, train the visual feature bridging module and the audio feature bridging module; Step 202, image-text knowledge enhancement step: using image text samples and text samples, train the visual encoder, the visual feature bridging module and the large language model; Step 203, full-modal joint training step: using image text samples, audio text samples and text samples, adjust the visual encoder, the visual feature bridging module, the audio encoder, the audio feature bridging module and the large language model.

[0045] The training process of a multimodal large model corresponds to stages 1, 2, and 3 in Figure 3. In Figure 3, the parts with flame patterns represent networks whose model parameters need adjustment at the current stage, while the parts with snowflake patterns represent networks whose model parameters are considered fixed at the current stage. The following describes each step of the multimodal large model training process with reference to Figure 3.

[0046] First, in step 201, the visual feature bridging module and the audio feature bridging module are trained using image text samples and audio text samples.

[0047] It is understandable that image-text samples correspond to training samples in the image-text modality. Their input data can at least include an image. In some samples, the input data can also include at least one of the following: recognizable text information from the image, image description information, or question-and-answer information about the image (e.g., "Is this a puppy?" followed by "Yes"). Typically, image-text samples can correspond to text information as image-text labels, used for comparison with text generated by a large language model. Similarly, audio-text samples can correspond to initial audio as input data and text information as label data.

[0048] Referring to stage 1 in Figure 3, at this point, the model parameters in the Vision Encoder, Audio Encoder, and Large Language Model (LLM) can be frozen for training the multimodal large model. "Freezing" here means locking the current values. At this stage, the undetermined parameters in the multimodal large model can be considered to only include the model parameters in the visual feature bridging module and the audio feature bridging module.

[0049] For audio text samples, the initial audio can be used as input data for a multimodal large-scale model. The audio encoder of the multimodal large-scale model extracts audio features, the audio feature bridging module adapts and aligns the audio features with the input of the large language model, and the large language model processes and decodes the audio text to generate audio text. The generated audio text and the label data of the audio text samples are then used for autoregressive training to calculate the loss corresponding to the audio text samples, denoted as the audio loss.

[0050] The input data of the image-text samples is processed by the visual encoder of the multimodal large model to extract visual features. The visual feature bridging module adapts and aligns the visual features with the input of the large language model. After processing by the large language model, the data is decoded to generate text information describing the image to be generated. This text information describing the image to be generated can then be processed by an image generator to generate the final image. The image generator can be implemented using a conventional generator. Here, in determining the model loss, the text information output by the multimodal model can be used for autoregressive training with the label data of the image-text samples. The resulting loss is denoted as the image-text loss, for example.

[0051] It is worth noting that the input data for both image and audio text samples may include text components, such as image description information, text identified in the image, and natural language converted from audio data. This part can be embedded by the text processing module of the large language model (as shown in word embedding in Figure 1) and then processed by the large language model together with the alignment features of the image encoding representation and the audio encoding representation.

[0052] During the current training phase, the model loss for a single parameter update cycle can include at least one of audio loss and image / text loss. In a single parameter update cycle, the model loss can be determined using the training samples of the current batch, and the gradient of the undetermined parameters in the corresponding bridging module can be determined using the model loss. This allows gradient update methods, such as gradient descent, to adjust the undetermined parameters in the corresponding alignment network towards the model loss.

[0053] In one embodiment, the visual feature bridging module and the audio feature bridging module are trained separately. In this case, the single parameter update cycle and the single batch of samples can be either image / text samples or audio / text samples. Thus, the gradient of the undetermined parameters in either the visual or audio feature bridging module can be determined using the model loss.

[0054] In another embodiment, the visual feature bridging module and the audio feature bridging module are trained together. A single parameter update cycle can be achieved by processing image and audio text samples in different batches, and the model loss from each batch is accumulated. In this way, the gradients of the undetermined parameters in both the visual and audio feature bridging modules can be determined using the model loss.

[0055] Training of the visual feature bridging module and the audio feature bridging module can be stopped when predetermined conditions are met. These predetermined conditions include: the number of image text samples and audio text samples reaches a predetermined number, such as 100,000 each; the number of parameter update cycles reaches a predetermined number, such as 1,000; the model loss for multiple consecutive parameter update cycles is less than a predetermined value, such as 2; the gradient for multiple consecutive cycles is less than a predetermined value, such as 0.1; and so on.

[0056] After training with a large number of image and audio text samples, the features output by the visual feature bridging module and the audio feature bridging module can be quickly aligned with the large language model, thus laying a solid foundation for subsequent multimodal large model training.

[0057] Then, through step 202, the visual encoder, visual feature bridging module, and large language model image-text samples are trained using image-text samples and text samples.

[0058] Through training the visual feature bridging module and the audio alignment network, the encoded features of the image-text modality and the audio modality can be basically aligned with the input feature space of the large language model. Considering the complexity of image data and its significant differences from text information, at this point, referring to stage 2 in Figure 3, the large model can be further unfrozen, and the model parameters in the visual encoder, visual feature bridging module and large language model can be adjusted to enhance the image-text processing capability of the large model.

[0059] Specifically, image-text samples and text samples can be processed using a multimodal large model. The model loss is determined based on the processing results, and the model parameters in the visual encoder, visual feature bridging module, and large language model are adjusted in the direction of reducing the model loss. In the process of processing image-text samples and text samples, a single batch can process one modality of data. In a single parameter update cycle, only one modality of sample data can be used, or multiple batches can be divided to use two modalities of data to determine the current model loss.

[0060] Thus, after initial training of the visual feature bridging module, text samples are introduced and adjusted together with image text samples to improve the image-text modality related network. The model parameters of the large language model are also fine-tuned. While ensuring the text processing capability of the large language model, its ability to process image-text modal data can be further improved, laying the groundwork for training the ability to process mixed modal data.

[0061] Next, in step 203, the visual encoder, visual feature bridging module, audio encoder, audio feature bridging module, and large language model image text sample audio text sample are trained using image text sample, audio text sample, and text sample.

[0062] After the large language model has a certain ability to integrate text and image processing, it needs to be further trained on various modal data to enable the multimodal large model to have the corresponding processing ability for any modal data input at any time. Therefore, in step 203, image text samples, audio text samples, and text samples can be used to train the multimodal large model for full modality adaptation.

[0063] Referring to the illustration of stage 3 in Figure 3, in this stage, the model parameters of all networks in the multimodal large model and the large language model are unfrozen, that is, they are all updated according to the model loss.

[0064] Considering that text samples are relatively easy to collect during actual training, while image and audio text samples are fewer, especially audio text samples which are more difficult to collect, this leads to a significant difference in the number of samples between different modalities. The significant difference in data volume across modalities may limit the performance of large multimodal models on modalities with limited data. Therefore, according to a possible design, when using samples from various modalities for comprehensive training, a sample balancing strategy can be used to balance the number of samples.

[0065] For example, in one embodiment, a portion of the samples can be sampled from a larger number of samples, such that the number of samples sampled is roughly the same as the number of samples from a smaller modality. For example, there are 1,000 audio text samples, 3 million text samples, and 100,000 image text samples. Approximately 1,000 samples are randomly sampled from each of the text and image text samples and used together with the audio text samples for comprehensive training across all modalities.

[0066] In another embodiment, sample sampling can be performed based on the number of iteration steps. Specifically, during a parameter update cycle, samples of a single modality can be sampled in a single batch, maintaining a relatively consistent sampling batch size across different modalities. For example, during a single parameter update cycle, image text samples, audio text samples, and text samples can be sampled in the same batch (e.g., a single batch) in turn, ensuring a stable and consistent number of training batches for each modality during training. During a single parameter update cycle, the sampling quantity per batch for each modality can be consistent, or the sampling probability can be positively correlated with the total amount of sample data. In a specific example, during a single parameter update cycle, n text samples, n image text samples, and n audio text samples can be processed in three batches using a multimodal large model to obtain prediction results. The prediction results are then compared with the corresponding labeled data to determine the model loss and adjust the model parameters. For example, in a single parameter update cycle, samples of one modality are used to determine the model loss and adjust the model parameters. The samples used in each parameter update cycle are rotated according to the modality. For example, n text samples are used in the first parameter update cycle, n image text samples are used in the second parameter update cycle, n audio text samples are used in the third parameter update cycle, n text samples are used in the fourth parameter update cycle, and so on.

[0067] In other embodiments, the quantity balance of various modal samples can also be achieved through other reasonable methods, which will not be elaborated here.

[0068] In optional embodiments, considering the differences between various modal samples, corresponding loss weights can be set for the loss determined based on various modal samples to balance the differences in processing capabilities of the large language model for various modal samples, enabling stable updates of the multimodal large model. When using multimodal samples in a single parameter update cycle, the loss weights can also be used to weight the model losses under various modalities to determine the comprehensive model loss, which is then used to adjust the corresponding model parameters. The loss weights can be preset.

[0069] In one example, the loss weight for text samples can be set to be the minimum, and the loss weight for image-text samples to be the maximum.

[0070] In another example, the loss weights for each modality can be determined using the range of loss values ​​for various modalities. The specific steps are as follows: Train the multimodal large model using a predetermined size sample set for each modality until the corresponding convergence condition is met, obtaining the corresponding model loss as the convergence loss for that modality, and recording the convergence loss value for each modality; for each modality, use the normalized value of its convergence loss value relative to each other's convergence loss values ​​as the loss weight for that modality. This loss weight determination process can be performed before the start of the multimodal large model training process, or in the first few cycles.

[0071] For example, for a single modality i, a smaller subset of samples is used. Train the multimodal model until it meets the first convergence condition (e.g., convergence of a large multimodal model). Record the model loss value at this point as the first convergence loss. Then, the corresponding normalized weights are calculated as the loss weights for mode i, as shown by the following formula: Where M is the number of modes and α is the balance coefficient, such as 10.

[0072] During parameter updates, the model loss determined for each modality sample can be multiplied by the loss weight and applied to the parameter update. For example, the update formula for the model parameter θ at step t using gradient descent can be adjusted as follows:

[0073] It is worth noting that after comprehensive training with samples from various modalities, the multimodal large model can approach convergence. Through the phased and progressive training scheme in steps 201 to 203, the modality support capability of the multimodal large model can be gradually expanded, and stable training of data under various modalities can be achieved.

[0074] In actual business applications, there may be higher processing accuracy requirements. Therefore, in some possible designs, the training process described above in steps 201 to 203 can be used as a pre-training process for the multimodal large model, and additional fine-tuning of the multimodal large model's instructions can be performed. In an optional embodiment, when the training process in steps 201 to 203 is used as a pre-training process, the resolution of the image and text data can be set to a lower resolution (e.g., 256×512) to accelerate the training speed.

[0075] Referring again to Figure 2, step 204 of the instruction fine-tuning can include, for example, the following steps: Step 2041, image-text instruction fine-tuning step: using image text samples and text samples, training a visual encoder, a visual feature bridging module, and a large language model; Step 2042, visual instruction fine-tuning step: using image text samples and video text samples, training a visual encoder, a visual feature bridging module, and a large language model; Step 2043, full-modal instruction fine-tuning step: using image text samples, audio text samples, and text samples, training a visual encoder, a visual feature bridging module, an audio encoder, an audio feature bridging module, and a large language model.

[0076] Steps 2041, 2042, and 2043 correspond to stages 1, 2, and 3 in Figure 4, respectively. Similar to Figure 3, the parts with flame patterns in Figure 4 represent the network whose model parameters are to be adjusted in the current stage, while the parts with snowflake patterns represent the network whose model parameters are considered to be fixed in the current stage.

[0077] In step 2041, the model parameters in the visual encoder, visual feature bridging module, and large language model are adjusted using image text samples and text samples.

[0078] Here, text samples can be samples where the input data is in the pure text modality. After training the multimodal large model with samples from various modalities, the multimodal large model has the corresponding processing capabilities for data of various modalities. At this point, further fine-tuning the model parameters related to image and text data processing, such as the visual encoder and visual feature bridging module, based on image and text samples, and performing adaptive fine-tuning on the large language model (as shown in stage 1 of Figure 4), can enhance the multimodal large model's ability to process image and text data.

[0079] It is worth noting that the pre-trained multimodal large model has its own image and text processing capabilities. In step 2041, higher resolution image data (such as 7680×4320) can be used to further fine-tune the model parameters and enhance the image and text processing capabilities.

[0080] In step 2042, the model parameters in the visual encoder, visual feature bridging module, and large language model are adjusted using image text samples and video text samples.

[0081] As shown in Stage 2 of Figure 4, this step allows for fine-tuning of the instructions for the image-text processing network using image and video text samples. Since videos can contain multiple images, they can also be processed by the image-text processing network. The adjusted network still includes the visual encoder, visual feature bridging module, and large language model. This instruction fine-tuning can further enhance the image-text understanding capabilities of the multimodal large model.

[0082] In step 2043, the model parameters in the visual encoder, visual feature bridging module, audio encoder, audio feature bridging module, and large language model are adjusted using image text samples, audio text samples, and text samples.

[0083] Following the enhancements to the multimodal large model's text and image understanding capabilities in steps 2041 and 2042, further fine-tuning using instructions from all network structures across various modalities can improve the multimodal large model's overall data processing capabilities. Similar to step 203, a sample balancing strategy can be used to balance the number of samples.

[0084] Furthermore, during model convergence, the model parameters of each network approach stable values. In full-modal training, the parameter updates of the processing networks for each modality typically depend on samples from that modality. Fine-tuning of instructions in a large language model also depends on samples from various modalities. Therefore, there may be situations where the convergence speed of model parameters in the processing networks corresponding to different modalities is inconsistent. To address this, this specification also provides a dynamic equilibrium convergence control strategy to achieve consistent convergence of model parameters during full-modal training.

[0085] This dynamic equilibrium convergence control strategy can influence the magnitude of the backpropagation gradient by adjusting the weights of the model loss corresponding to samples under various modalities, thereby controlling the consistency of model parameter convergence. Specifically, training under each modality can be regarded as a training task, and the training progress under each modality (task) can be balanced using the principle of Multi-Task Learning (MTL). That is, for modalities exhibiting a slower convergence slope (smaller slope value), lower training weights are given to prevent overfitting; for modalities exhibiting a steeper convergence slope (larger slope value), higher training weights are assigned to promote their learning.

[0086] Referring to Figure 5, a single batch of samples can be a single modality, such as text, audio, or video modality. A model loss can be determined using samples from a single modality. During a single parameter update cycle, the current model loss can be determined using samples from a single modality, or by processing samples from multiple modalities in multiple batches.

[0087] In Figure 5, the audio input can be converted into text data for processing, for example, denoted as "Audio-Text" in Figure 5. The input data for the text input is denoted as "Text" in Figure 5. The input data for the image-text input is denoted as "Image-Text" in Figure 5. When recognizing text from an image as input data, it is denoted as "OCR" in Figure 5. The input data for the video input is denoted as "Video-Text" in Figure 5, and so on.

[0088] To utilize the current slope to regulate the training progress of a given modality, this specification proposes a technique for determining the current slope for each modality using a validation set. Specifically, during training, periodic validation operations are inserted at fixed parameter adjustment intervals. The interval between two validation operations is defined as a validation period, and a single validation period can include multiple parameter adjustment periods. In each validation period, a small subset of validation data is used to calculate the validation loss for each modality, thereby enabling the tracking of training progress for each modality using the validation loss and convergence slope within a historical window. The validation subset can be a randomly segmented portion of the training data for the corresponding modality, including... Sample. Here, S i B represents the number of verification batches for a single verification segment of the i-th modality. i This indicates the validation batch size for this modality, where M represents the number of modalities. Data in the validation subset is excluded from the model training set.

[0089] Different modalities exhibit varying degrees of training difficulty, resulting in different ranges of validation loss values. To ensure balanced initialization and mitigate potential inaccuracies in the initial model loss curve, specific loss weights, such as w, can be assigned to all modalities at the beginning of training. i,0 =1, and maintains a fixed value in the first H verification segments.

[0090] At the beginning of each verification cycle (H+1), the weights of each mode can be determined by the slope. This can be achieved using method a. i,t x+b i,t A linear regression model of the form is used to fit the change of validation loss within the historical window as a function of validation period t, resulting in the slope coefficient a. i,t This indicates the current convergence rate of the mode. Based on this slope coefficient 'a'... i,t This allows us to determine the current slope of the loss curve for the corresponding mode in the current validation period. Figure 5 shows the fitted curve (a i,t x+b i,t In the case of ), where a i,tThe slope of the fitted line is represented by i = 1, 2, 3...6, which correspond to the image-text modality, image-text recognition OCR, audio-text modality, text modality, interleaved image, and video modality shown in Figure 5, respectively.

[0091] To ensure fair weight allocation across modalities, in one embodiment, the convergence slope of each modality can be calculated based on the normalized verification loss, where the normalized verification loss for modality i is as follows:

[0092] in, H represents the validation loss for the i-th modality in the t-th validation period, H represents the history window size, i.e., the number of historical validation periods referenced in the process of calculating the validation loss of the current validation period, and ∈ is a very small positive number, such as 10. -6 To prevent division by zero, the normalized validation loss can be fitted using a linear regression model to account for the variation of the validation loss within the historical window as a function of the validation period t. The resulting slope coefficient is denoted as a. i,t .

[0093] To determine the current loss weight for each modality based on the current slope of the corresponding loss curve, the normalized results of the current slopes for each modality can be used to determine the corresponding convergence scores. Each convergence score is negatively correlated with the corresponding current slope. Then, each convergence score is mapped to an importance coefficient through an activation function. The weight coefficients of the model loss for each modality are determined based on the product of the importance coefficients and the number of modalities. Finally, the current loss weights for each modality within the current validation period are determined based on the modality weights for each modality.

[0094] As a concrete example, for the t-th verification period (where t>H), the normalized slope can be calculated. The convergence score s of the i-th mode i,t For example:

[0095] The softmax operation is performed along the modality dimension. Next, the modality weight allocation for the current validation cycle is calculated: w i,t =M*softmax(f*s i,t )

[0096] Where f is a scaling factor that adjusts the weight probability distribution, multiplying by M ensures that the sum of the weights of all modes equals M.

[0097] The validation cycle approach can dynamically adjust modal weights with minimal computational overhead, thereby improving the performance of all modalities in the context of full-modal learning.

[0098] In some optional embodiments, to mitigate sudden fluctuations caused by single-step weight updates and improve the stability of model training, an exponential moving average (EMA) mechanism can be used to smoothly adjust the training weights of a single modality, as shown below:

[0099] The smoothing factor α can be preset, such as to 0.9. Adjusted modal loss weights. Each training step for the next validation cycle.

[0100] In other embodiments, the slope corresponding to the loss curve can be determined by other reasonable methods, which will not be listed here.

[0101] By using a multimodal large model (M2-omini) to process samples from various modalities forward, the model loss can be determined. The product of the model loss and the corresponding current loss weight is used as the basis for determining the parameter update gradient during the backward propagation process. This ensures that the parameters in each neural network module of the multimodal large model remain consistent during the regional convergence process, avoiding training bias caused by some networks converging faster than others.

[0102] In summary, the multimodal large-scale model training scheme provided under the technical concept of this specification can decouple the various network modules in the multimodal large-scale model according to their functions, and perform phased progressive training on the decoupled network modules to gradually expand the model's modality support capabilities and achieve optimal performance in each modality. This training method can effectively address the problem of large differences in data distribution between different modalities, and achieve stable training of data for each modality.

[0103] In a further embodiment, the technical problem of uneven training sample data volume of each modality can be solved by the technical concept of balanced sampling according to the number of iteration steps. The technical concept of measuring the convergence speed of each modality by calculating the slope of the loss curve of each modality and dynamically adjusting the loss weight of each modality according to the convergence speed can solve the technical problem of inconsistent convergence speed under training samples of each modality.

[0104] According to another embodiment, a training device for a multimodal large model is also provided. This device can be located in a computer, terminal, or server with a certain computing power. The multimodal large model may include: a visual encoder and a visual feature bridging module connected in sequence; an audio encoder and an audio feature bridging module connected in sequence; and a large language model connected to the visual feature bridging module and the audio feature bridging module. Figure 6 illustrates a training device 600 for a multimodal large model according to one embodiment.

[0105] As shown in Figure 6, the device 600 may include:

[0106] The encoder alignment unit 601 is configured to train the visual feature bridging module and the audio feature bridging module using image text samples and audio text samples.

[0107] The image and text knowledge enhancement unit 602 is configured to train the visual encoder, visual feature bridging module and large language model using image text samples and text samples.

[0108] The full-modal joint training unit 603 is configured to train a visual encoder, a visual feature bridging module, an audio encoder, an audio feature bridging module, and a large language model using image text samples, audio text samples, and text samples.

[0109] According to one possible design, device 600 may also include instruction fine-tuning unit 604, configured to perform instruction fine-tuning on the multimodal large model through the following steps:

[0110] Image-text instruction fine-tuning steps: Use image-text samples and text samples to train the visual encoder, visual feature bridging module, and large language model;

[0111] Visual instruction fine-tuning steps: Train the visual encoder, visual feature bridging module, and large language model using image text samples and video text samples;

[0112] Full-modal instruction fine-tuning steps: Using image text samples, audio text samples, and text samples, train the visual encoder, visual feature bridging module, audio encoder, audio feature bridging module, and large language model.

[0113] It is worth noting that the device 600 shown in FIG6 corresponds to the method described in FIG2, and the corresponding description in the method embodiment shown in FIG2 also applies to the device 600, and will not be repeated here.

[0114] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform the method described in conjunction with FIG2, etc.

[0115] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method described in conjunction with FIG2, etc., is implemented. Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0116] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the technical concept in this specification. It should be understood that the above description is only a specific embodiment of the technical concept in this specification and is not intended to limit the scope of protection of the technical concept in this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments in this specification should be included within the scope of protection of the technical concept in this specification.

Claims

1. A training method for a multimodal large model, wherein the multimodal large model comprises: A visual encoder and a visual feature bridging module are connected in sequence, followed by an audio encoder and an audio feature bridging module, and then a large language model connected to both the visual and audio feature bridging modules. The method includes the following pre-training steps: Encoder alignment steps: Train the visual feature bridging module and the audio feature bridging module using image text samples and audio text samples; Image-text knowledge enhancement steps: Use image-text samples and text samples to train the visual encoder, visual feature bridging module and large language model; Full-modal joint training steps: Using image text samples, audio text samples, and text samples, train the visual encoder, visual feature bridging module, audio encoder, audio feature bridging module, and large language model.

2. The method as described in claim 1, wherein, Following the pre-training step, the method further includes the following step of fine-tuning the multimodal large model: Image-text instruction fine-tuning steps: Use image-text samples and text samples to train the visual encoder, visual feature bridging module, and large language model; Visual instruction fine-tuning steps: Train the visual encoder, visual feature bridging module, and large language model using image text samples and video text samples; Full-modal instruction fine-tuning steps: Using image text samples, audio text samples, and text samples, train the visual encoder, visual feature bridging module, audio encoder, audio feature bridging module, and large language model.

3. The method as described in claim 1, wherein, The multimodal joint training steps include: During a single parameter update cycle, the same batch is sampled in turn for image text samples, audio text samples, and text samples; The model loss for a given modality is determined by processing a single batch of samples for a single modality. The overall model loss is determined by weighting the model losses under various modalities, and then used to adjust the corresponding model parameters.

4. The method of claim 1, wherein, In the full-modal joint training step, the model loss determined for a single modality sample is multiplied and balanced with a pre-determined loss weight and then used to determine the backpropagation gradient of the model parameters; the loss weight corresponding to the model loss of a single modality is determined in the following way: If the multimodal large model is trained using a sample set of a predetermined size under the single modality until the first convergence condition is met, the first convergence loss that meets the first convergence condition is obtained. The loss weight corresponding to the model loss in a single modality is determined based on the ratio of the reciprocal of the first model loss to the sum of the reciprocals of the convergence losses in various modalities.

5. The method of claim 2, wherein, The full-modal command fine-tuning steps include: During model parameter tuning, the following dynamic equilibrium convergence control strategy is implemented: The current loss weight for each mode is determined based on the current slope of the corresponding loss curve for each mode. The current loss weight for a single mode is negatively correlated with the corresponding current slope. The model loss corresponding to the samples in each modality is balanced by multiplying it with the current loss weight and then used to adjust the model parameters.

6. The method of claim 5, wherein, The current slope of the loss curve for a single mode is determined as follows: Obtain the validation loss of the previous H validation periods of the current validation period. A single validation period includes multiple parameter update periods under a single modality. The validation loss of a single validation period is determined by the following normalization method: the difference between the model loss of the single validation period determined by the validation set and the minimum value of the loss determined by the validation set of the previous H validation periods, and the difference between the maximum and minimum values ​​of the model loss determined by the validation set of the previous H validation periods. Based on the slope coefficient of the validation loss at validation period t, determined by fitting the linear regression model, the current slope of the loss curve for the corresponding mode in the current validation period is determined.

7. The method of claim 5 or 6, wherein, The step of determining the current loss weight for each mode based on the current slope of the corresponding loss curve for each mode includes: For the normalized results of each current slope under each mode, determine the corresponding convergence score. Each convergence score is negatively correlated with the corresponding current slope. Each convergence score is mapped to an importance coefficient through an activation function, and the weight coefficients of the model loss under each mode are determined based on the product of the importance coefficient and the number of modes. The current loss weights for each mode within the current validation period are determined based on the mode weights under each mode.

8. The method of claim 7, wherein, The process of determining the corresponding current loss weights based on the weighting coefficients under various modalities includes: For a single modality, the current loss weight is determined using an exponential moving average of the weighting coefficients.

9. A training apparatus for a multimodal large model, the multimodal large model comprising: A visual encoder and a visual feature bridging module are connected in sequence; an audio encoder and an audio feature bridging module are connected in sequence; and a large language model is connected to the visual feature bridging module and the audio feature bridging module. The device includes: The encoder alignment unit is configured to train the visual feature bridging module and the audio feature bridging module using image text samples and audio text samples. The image and text knowledge enhancement unit is configured to train the visual encoder, visual feature bridging module, and large language model using image text samples and text samples. The multimodal joint training unit is configured to train the visual encoder, visual feature bridging module, audio encoder, audio feature bridging module, and large language model using image text samples, audio text samples, and text samples.

10. The apparatus of claim 9, wherein, The device further includes a command fine-tuning unit configured to perform command fine-tuning on a multimodal large model through the following steps: Image-text instruction fine-tuning steps: Use image-text samples and text samples to train the visual encoder, visual feature bridging module, and large language model; Visual instruction fine-tuning steps: Train the visual encoder, visual feature bridging module, and large language model using image text samples and video text samples; Full-modal instruction fine-tuning steps: Using image text samples, audio text samples, and text samples, train the visual encoder, visual feature bridging module, audio encoder, audio feature bridging module, and large language model.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-8.

12. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-8.

Images