Method and apparatus for training mixture-of-experts model, computer device, readable storage medium, and program product

Abstract

The present application relates to a method and apparatus for training a mixture-of-experts model, a computer device, a computer-readable storage medium, and a computer program product. The method comprises: acquiring a training sample set of a mixture-of-experts model associated with input to a large model, the training sample set comprising input data of the large model and input parsing label information corresponding to the input data; inputting a target sample in the training sample set into the mixture-of-experts model, and obtaining activation probability information of each expert model in the mixture-of-experts model, a target output result of the mixture-of-experts model, and a sub-output result of a loaded expert model; obtaining expert differentiation reward information on the basis of model parameters of each expert model, the target sample, the target output result, and the sub-output result; and performing iterative training on the mixture-of-experts model on the basis of the activation probability information, the expert differentiation reward information, the target sample, and the target output result, and obtaining a trained mixture-of-experts model.

Description

Training methods, apparatus, computer equipment, readable storage media, and program products for hybrid expert models

[0001] Related applications

[0002] This application claims priority to Chinese patent application No. 2024119697317, filed on December 30, 2024, entitled “Training Method, Apparatus, Computer Equipment, Readable Storage Medium and Program Product for Hybrid Expert Models”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of machine learning technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for training a hybrid expert model. Background Technology

[0004] In recent years, with the rapid development of deep learning technology, large-scale neural networks have made significant progress in fields such as natural language processing and computer vision. However, as the model size continues to expand, the computational overhead of training and inference increases dramatically, placing higher demands on hardware resources and training efficiency. Mixture of Experts (MoE) models, as a sparse activation mechanism, effectively reduce computational costs while maintaining high model performance by dynamically activating a subset of expert models to process input data, thus attracting widespread attention. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, computer-readable storage medium, and computer program product for training hybrid expert models.

[0006] Firstly, this application provides a method for training a hybrid expert model, including:

[0007] Obtain a training sample set of a hybrid expert model associated with the input of a large model; the training sample set includes the input data of the large model and the input parsing label information corresponding to the input data;

[0008] The target samples in the training sample set are input into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model, and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0009] Based on the model parameters of each expert model, the target sample, the target output result, and the sub-output result, expert differentiation reward information is obtained;

[0010] Based on the activation probability information, the expert differentiation reward information, the target sample, and the target output result, the hybrid expert model is iteratively trained to obtain the trained hybrid expert model.

[0011] In one embodiment, obtaining expert differentiation reward information based on the model parameters of each expert model, the target sample, the target output result, and the sub-output result includes:

[0012] Based on the model parameters of each expert model, determine the model diversity reward information;

[0013] Based on the sub-output results, determine the output diversity reward information;

[0014] Based on the target sample and the target output result, determine the input-output difference reward information;

[0015] The expert differentiation reward information is determined based on the model diversity reward information, the output diversity reward information, and the input-output difference reward information.

[0016] In one embodiment, determining the model diversity reward information based on the model parameters of each expert model includes:

[0017] Based on the model parameters of each expert model, determine the parameter difference information between each pair of expert models;

[0018] Based on the parameter difference information, the model diversity reward information is determined.

[0019] In one embodiment, determining the output diversity reward information based on the sub-output results includes:

[0020] Based on the sub-output results, determine the output difference information between each pair of sub-output results;

[0021] Based on the output difference information, the output diversity reward information is determined.

[0022] In one embodiment, determining the input-output difference reward information based on the target sample and the target output result includes:

[0023] Based on the target samples and the target output results, determine the sample difference information and output result difference information between each pair of target samples;

[0024] Based on the sample difference information and the output result difference information, determine the pairwise discriminant information between the target samples;

[0025] Based on the discrimination information, the input-output difference reward information is determined.

[0026] In one embodiment, the hybrid expert model includes a gated network model and multiple expert models;

[0027] The iterative training of the hybrid expert model based on the activation probability information, the expert differentiation reward information, the target sample, and the target output includes:

[0028] In each iteration of training, a gradient product term of the parameter update gradient is determined based on the preset baseline function and the expert-discriminated reward information;

[0029] The parameters of the gated network model are updated based on the activation probability information and the gradient product term, and the parameters of the target expert model are updated based on the target sample, the target output result, and the gradient product term; the target expert model is an expert model determined from the plurality of expert models based on the activation probability information trained in this iteration.

[0030] In one embodiment, determining a gradient product term of the parameter update gradient based on a preset baseline function and the expert-differentiated reward information includes:

[0031] Obtain the historical expert differentiation reward information corresponding to the previous iteration training;

[0032] Based on the historical expert-differentiated reward information and the expert-differentiated reward information, determine the moving average of the reward information;

[0033] The difference between the expert-distinguished reward information and the moving average of the reward information is used as the gradient product term.

[0034] In one embodiment, the expert differentiation reward information is a reward value determined by evaluating the differences between expert models in parameter configuration and output characteristics.

[0035] In one embodiment, before determining the model diversity reward information based on the model parameters of each expert model, the method further includes:

[0036] The model parameters of each expert model are extracted from the hybrid expert model. The model parameters include the weight matrix, bias value and network structure characteristics of each expert model.

[0037] In one embodiment, determining the model diversity reward information based on the model parameters of each expert model includes:

[0038] The degree of difference between the model parameters of each expert model is calculated to quantify the distributional differences of the model parameters, and the distributional differences are integrated into the model diversity reward information.

[0039] In one embodiment, determining the output diversity reward information based on the sub-output results includes:

[0040] For the sub-output results of the loaded expert model, the output diversity reward information is calculated by comparing the numerical distribution or the directionality of the feature vectors of these output results.

[0041] In one embodiment, the expert differentiation reward information is determined based on the model diversity reward information, the output diversity reward information, and the input-output difference reward information, including:

[0042] The model diversity reward information, the output diversity reward information, and the input-output difference reward information are weighted and integrated to obtain the expert differentiation reward information;

[0043] The weights for weighted integration are set based on the priority of the task or the training objective of the expert model.

[0044] In one embodiment, the target sample trained in each iteration consists of multiple samples, and each sample activates a different target expert model.

[0045] In one embodiment, the sub-output of the loaded expert model is a numerical vector, a classification probability distribution, or other form of intermediate model result.

[0046] In one embodiment, a trained hybrid expert model is used to analyze and process at least one of text data, image data, and audio data.

[0047] Secondly, this application also provides a training device for a hybrid expert model, comprising:

[0048] The sample acquisition module is used to acquire a training sample set of the hybrid expert model associated with the input of the large model; the training sample set includes the input data of the large model and the input parsing label information corresponding to the input data;

[0049] The model input module is used to input the target samples from the training sample set into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model, and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0050] The reward acquisition module is used to obtain expert differentiation reward information based on the model parameters of each expert model, the target sample, the target output result, and the sub-output result;

[0051] The model training module is used to iteratively train the hybrid expert model based on the activation probability information, the expert differentiation reward information, the target sample, and the target output result to obtain the trained hybrid expert model.

[0052] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0053] Obtain a training sample set of a hybrid expert model associated with the input of a large model; the training sample set includes the input data of the large model and the input parsing label information corresponding to the input data;

[0054] The target samples in the training sample set are input into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model, and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0055] Based on the model parameters of each expert model, the target sample, the target output result, and the sub-output result, expert differentiation reward information is obtained;

[0056] Based on the activation probability information, the expert differentiation reward information, the target sample, and the target output result, the hybrid expert model is iteratively trained to obtain the trained hybrid expert model.

[0057] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0058] Obtain a training sample set of a hybrid expert model associated with the input of a large model; the training sample set includes the input data of the large model and the input parsing label information corresponding to the input data;

[0059] The target samples in the training sample set are input into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model, and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0060] Based on the model parameters of each expert model, the target sample, the target output result, and the sub-output result, expert differentiation reward information is obtained;

[0061] Based on the activation probability information, the expert differentiation reward information, the target sample, and the target output result, the hybrid expert model is iteratively trained to obtain the trained hybrid expert model.

[0062] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0063] Obtain a training sample set of a hybrid expert model associated with the input of a large model; the training sample set includes the input data of the large model and the input parsing label information corresponding to the input data;

[0064] The target samples in the training sample set are input into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model, and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0065] Based on the model parameters of each expert model, the target sample, the target output result, and the sub-output result, expert differentiation reward information is obtained;

[0066] Based on the activation probability information, the expert differentiation reward information, the target sample, and the target output result, the hybrid expert model is iteratively trained to obtain the trained hybrid expert model. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 is a flowchart illustrating the training method of a hybrid expert model in one embodiment;

[0069] Figure 2 is a flowchart illustrating the steps for obtaining expert-differentiated reward information in one embodiment;

[0070] Figure 3 is a flowchart illustrating the training method of the hybrid expert model in another embodiment;

[0071] Figure 4 is a structural block diagram of a training device for a hybrid expert model in one embodiment;

[0072] Figure 5 is an internal structure diagram of a computer device in one embodiment. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0074] In existing training processes for hybrid expert models, the control mechanisms of gating networks are relatively simple, leading to multiple experts potentially learning similar feature processing methods and parameter configurations during training. This homogenization phenomenon causes the trained expert models to produce the same or similar output responses to similar inputs, thereby reducing the actual effective capacity of the model and resulting in low robustness of the hybrid expert model.

[0075] Based on this, this application provides a training method, apparatus, computer device, computer-readable storage medium, and computer program product for hybrid expert models that can improve the robustness of hybrid expert models.

[0076] In one embodiment, as shown in Figure 1, a training method for a hybrid expert model is provided. This embodiment illustrates the method by applying it to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, and tablets. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. In this embodiment, the method includes the following steps:

[0077] Step S101: Obtain the training sample set of the hybrid expert model associated with the input of the large model.

[0078] The training sample set includes the input data of the large model and the corresponding input parsing label information.

[0079] The training sample set consists of input data and input parsing label information. Input data is the raw data that the large model needs to process, covering various types of task inputs, including but not limited to text, image, audio, and video data. Each type of input data typically requires corresponding input parsing label information. This data provides the correspondence or reference answer between the input and the target output, guiding the model's learning process and helping it optimize its parameters during training. Input parsing label information is labeled data closely related to the input data; in supervised learning tasks, label information is a crucial training basis.

[0080] For example, the terminal collects and organizes input data from a data source. This data typically includes text, images, and audio, and depending on the task, it may be raw user input, labeled image data, or text data. The terminal then prepares corresponding parsing labels for this input data. These labels are usually the correct answers or outputs provided by experts or artificial intelligence systems. This label information is used to train the model, enabling it to optimize its parameters by learning the mapping relationship between inputs and labels. Finally, the terminal feeds the organized dataset into a hybrid expert model as a training sample set for subsequent training to support the model's accurate predictions in real-world tasks.

[0081] Specifically, ① Input data: Natural language text, such as news articles, product reviews, question descriptions, etc.; corresponding input parsing label information: emotion labels for sentiment analysis tasks, translated text labeled as the target language for translation tasks, category labels for text classification tasks (such as science, sports, entertainment), and answer text labels for question-and-answer tasks. ② Input data: Image data, including still images, video images, or continuous frame image sequences; corresponding input parsing label information: category labels for image / video classification tasks (such as cat, car), object location (such as coordinate information) / object category labels for object detection tasks, and action labels for video action recognition tasks (such as running, jumping). ③ Input data: Speech signals or audio files; corresponding input parsing label information: text labels for speech-to-text conversion in speech recognition tasks, emotion labels for speech sentiment analysis, etc.

[0082] Step S102: Input the target samples in the training sample set into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model, and the sub-output result of the loaded expert model.

[0083] Here, "loaded expert models" refers to a predetermined number of expert models in the hybrid expert model. Specifically, it refers to a subset of expert models selected from all expert models according to a predetermined strategy within the hybrid expert model; these models are used to participate in reward calculation. The selection method for loaded expert models can include a Top-K strategy or a probabilistic selection strategy to ensure full utilization of resources while improving the comprehensiveness of reward calculation and model performance. Furthermore, it may not be entirely identical to the activated expert model of the hybrid expert model (the activated expert model is the expert model used to obtain the target output result of the hybrid expert model).

[0084] The activation probability information refers to the activation probability distribution of the expert model selected by the gating network for the current target sample.

[0085] For example, during each training iteration, the terminal randomly selects a target sample from the training sample set and inputs it into the hybrid expert model. The hybrid expert model generates activation probability information for each expert model through a gating network, determining which experts are activated to participate in the forward computation of the current input, and generating the target output result of the hybrid expert model. Simultaneously, the terminal further selects loading expert models from all expert models based on a preset maximum number of loading experts, obtaining sub-output results of the loading expert models for subsequent reward value calculation; the maximum number of loading experts can be determined based on hardware resources. For example, a Top-K strategy can be used to select loading expert models, in which case the K expert models with the highest probabilities are selected as loading expert models based on the activation probability information; alternatively, a subset of models can be probabilistically selected from the activated expert models of the hybrid expert model as loading expert models.

[0086] Step S103: Based on the model parameters, target samples, target output results, and sub-output results of each expert model, obtain the expert differentiation reward information.

[0087] The expert-differentiating reward information is a reward value determined by evaluating the differences in parameter configuration and output characteristics among expert models. The goal of calculating the expert-differentiating reward information is to encourage each expert model to learn independent feature representations and task specializations, thereby enhancing model diversity and avoiding the problem of expert homogenization.

[0088] For example, the terminal calculates the degree of difference between experts by analyzing the parameter information of each expert model (such as weight distribution and gradient changes) and its sub-output results on the current target sample. For instance, the terminal compares the sub-output results of the loaded expert models and evaluates the output differences between them. At the same time, it analyzes the ability of the expert models to distinguish the target sample by combining the target output results. In addition, the independence of the expert models is further quantified by evaluating the parameter differences of the expert models (such as the distance or distribution differences of parameter values).

[0089] Step S104: Based on the activation probability information, expert differentiation reward information, target sample and target output result, the hybrid expert model is iteratively trained to obtain the trained hybrid expert model.

[0090] For example, the terminal selects target samples from the training sample set and inputs them into the hybrid expert model to obtain the target output, activation probability information, and expert discrimination reward information. Combining this information, the terminal iteratively optimizes the hybrid expert model. First, the terminal uses the expert discrimination reward information and activation probability information to update the parameters of the gating network through a policy gradient optimization algorithm, enabling it to more accurately select suitable expert models and improve the rationality of the activation probability distribution. Second, the terminal combines the expert discrimination reward information to optimize the parameters of the activated expert models, adjusting the model gradient direction to encourage expert models to develop differentiated characteristics and task focus capabilities during the learning process. During training, inactive expert models indirectly participate in reward calculation and parameter updates through a loading mechanism, further improving resource utilization and the adaptability of expert models. After multiple iterations of training, the terminal obtains an optimized hybrid expert model with higher task accuracy, better generalization ability, and stronger robustness.

[0091] In the training method of the aforementioned hybrid expert model, firstly, a training sample set of hybrid expert models associated with the input of the large model is obtained. This training sample set includes the input data of the large model and the corresponding input parsing label information. The input data and corresponding parsing label information help the model accurately understand the relationship between the input and the expected output, providing sufficient supervision signals for subsequent training and improving training effectiveness and the model's adaptability to specific tasks. Next, the target samples from the training sample set are input into the hybrid expert model to obtain the activation probability information of each expert model, the target output result of the hybrid expert model, and the sub-output results of the loaded expert models. The loaded expert models represent a predetermined number of expert models in the hybrid expert model. By introducing the sub-output results of the loaded expert models, the calculation results of potentially inactive expert models are incorporated into the reward and parameter update process, thereby fully utilizing computational resources. This allows the outputs of more expert models to participate in the reward calculation, effectively avoiding idle computational resources and enhancing the expert model through comprehensive evaluation of the outputs of multiple expert models. The diversity and task adaptability of the hybrid expert model enable it to better utilize the computational power of all expert models during training, improving training efficiency and enhancing robustness and overall performance. Then, based on the model parameters, target samples, target outputs, and sub-outputs of each expert model, expert-discriminative reward information is obtained. By providing expert-discriminative rewards for iterative training, different experts are encouraged to learn different features or tasks, improving model diversity and robustness. This effectively prevents multiple experts from learning similar feature processing methods, thereby enhancing the model's adaptability to different inputs. Finally, the hybrid expert model is iteratively trained based on activation probability information, expert-discriminative reward information, target samples, and target outputs to obtain a trained hybrid expert model. Iterative training of the hybrid expert model using activation probability information, expert-discriminative reward information, target samples, and target outputs allows the gating network to more effectively learn how to allocate tasks to different experts and update model parameters based on reward signals. This gradually optimizes the expert selection strategy, improving overall model performance and expert adaptability. The above method effectively solves the problem of expert homogenization that may occur in hybrid expert models by introducing expert differentiation reward information. By calculating the expert differentiation reward for each training iteration, it encourages the diversity of different experts in feature learning and task processing, avoids similar parameter configurations and outputs among experts, improves the robustness and diversity of the model, enables the model to better adapt to complex tasks and input changes, and significantly improves the overall performance and training stability of the model.

[0092] In an exemplary embodiment, as shown in FIG2, the above step S103, which obtains expert differentiation reward information based on the model parameters, target samples, target output results, and sub-output results of each expert model, can also be implemented through the following steps:

[0093] Step S201: Determine the model diversity reward information based on the model parameters of each expert model;

[0094] Step S202: Determine the output diversity reward information based on the sub-output results;

[0095] Step S203: Determine the input-output difference reward information based on the target sample and the target output result;

[0096] Step S204: Determine the expert differentiation reward information based on the model diversity reward information, output diversity reward information, and input-output difference reward information.

[0097] For example, the terminal first extracts parameter information for each expert model from the hybrid expert models, including its weight matrix, bias values, and network structure characteristics. The terminal quantifies the distributional differences of the model parameters by calculating the dissimilarity between the expert models' parameters, such as Euclidean distance, KL divergence, or cosine similarity, and integrates these distributional differences into model diversity reward information to measure the independence and diversity of the expert models at the parameter level. Subsequently, for the sub-outputs loaded from the expert models, the terminal calculates output diversity reward information by comparing the numerical distribution or directionality of the feature vectors of these outputs. For example, the terminal can determine the difference in each expert's response to the same input by comparing the cosine similarity or distance between the output vectors loaded from the expert models, thereby deriving the degree of diversity of output characteristics. Next, the terminal analyzes the matching relationship between the input features of the target sample and the target output result. By calculating the correlation between the input sample features (such as image features, text embedding vectors, etc.) and the corresponding target output features, the terminal quantifies the input-output mapping relationship, thereby generating input-output difference reward information to evaluate the discriminative ability of the expert model in processing a specific task. Finally, the terminal performs weighted integration of model diversity reward information, output diversity reward information, and input-output difference reward information. The weights can be set based on task priority or the training objectives of the expert models. The integrated expert differentiation reward information can comprehensively reflect the differences between expert models and provide multi-dimensional guidance for the optimization of expert models.

[0098] In this embodiment, by introducing a multi-dimensional computation mechanism that includes model diversity reward information, output diversity reward information, and input-output difference reward information, the performance of expert models in parameters, output, and input-output matching can be evaluated. This provides more dimensions of reference for expert model optimization, thereby improving the adaptability and robustness of hybrid expert models, significantly enhancing the independence and diversity among expert models, reducing homogenization, and ensuring their efficient performance in complex tasks.

[0099] In an exemplary embodiment, step S201, which determines model diversity reward information based on the model parameters of each expert model, further includes: determining parameter difference information between pairs of expert models based on the model parameters of each expert model; and determining model diversity reward information based on the parameter difference information.

[0100] For example, the terminal extracts parameter information for each expert model from the hybrid expert models, including its weight matrix and bias values. The terminal calculates the differences between expert models by comparing their parameters pairwise. For instance, for each pair of expert models, the terminal can calculate the Euclidean distance, cosine similarity, or measure the difference using the Kullback-Leibler divergence of the parameter distribution. Furthermore, for expert models with different hierarchical structures, the terminal can also calculate their differences based on the statistical properties of the parameters between levels (such as mean and variance). By summarizing and generalizing the parameter differences between all expert models pairwise, the terminal generates a comprehensive parameter difference distribution matrix. Subsequently, the terminal further calculates model diversity reward information based on this difference distribution matrix. This process can quantify the distributional independence of different expert models in the overall parameter space by normalizing the difference information.

[0101] In a specific example, for any two expert models E i and E j Calculate the differences in their parameters:

[0102] Where, θ i and θ j They are expert E i and E j The parameter vector, |θ i | represents the dimension of the parameter, and |||| represents the Euclidean norm.

[0103] Calculate the average of the parameter differences among all N expert models to determine the model diversity reward information:

[0104] The reward item for model diversity reward information in the expert differentiation reward information is: λ θ These are the weighting coefficients.

[0105] In this embodiment, by introducing parameter difference information between each pair of expert models and calculating model diversity reward information based on this difference information, the independence between expert models can be effectively measured and enhanced, reducing homogenization. Furthermore, this process can dynamically adjust the allocation of reward values, promoting more diverse parameter configurations in expert models during training, thereby improving the overall adaptability and robustness of the hybrid expert model.

[0106] In an exemplary embodiment, step S202, which determines output diversity reward information based on the sub-output results, further includes: determining output difference information between each pair of sub-output results based on the sub-output results; and determining output diversity reward information based on the output difference information.

[0107] For example, the terminal obtains sub-outputs for each expert from the loaded expert models. These sub-outputs can be numerical vectors, classification probability distributions, or other forms of intermediate model results. The terminal calculates output difference information by comparing these sub-outputs pairwise. For example, the terminal can use Euclidean distance or cosine similarity to measure output differences. Furthermore, for probability distribution-type sub-outputs, the terminal can calculate KL divergence or Jensen-Shannon divergence to assess the differences in distribution between the two sub-outputs. After completing pairwise comparisons and calculating output difference information between all loaded expert models, the terminal summarizes and integrates this difference information, for example, by calculating the average output difference value to quantify the overall output diversity of the loaded expert models. Subsequently, the terminal generates output diversity reward information based on this difference information; a higher reward value indicates that the loaded expert model exhibits greater diversity in output characteristics. Ultimately, the output diversity reward information is used to guide the training process, prompting the expert models to develop more independent characteristics during task processing, thereby enhancing the robustness and task adaptability of the hybrid expert model.

[0108] In a specific example, for the input sample x t Any two loaded expert models E i and E j Sub-output results and Calculate the difference in their outputs:

[0109] For the input sample x t Calculate the average of the output differences among all K loaded expert models:

[0110] For T target samples, Further averaging is performed to determine the output diversity reward information:

[0111] The reward items in the expert differentiation reward information are: λ y These are the weighting coefficients.

[0112] In this embodiment, by calculating the diverse reward information, the differences in the sub-output results of the loaded expert model can be fully explored, avoiding the convergence of output characteristics among multiple experts. This further optimizes the training direction of the expert model, enabling the hybrid expert model to demonstrate stronger division of labor and collaboration capabilities in complex tasks, while improving the overall performance and generalization ability of the model.

[0113] In an exemplary embodiment, step S203, which determines the input-output difference reward information based on the target samples and the target output results, further includes: determining the sample difference information and output result difference information between each pair of target samples based on the target samples and the target output results; determining the discrimination information between each pair of target samples based on the sample difference information and the output result difference information; and determining the input-output difference reward information based on the discrimination information.

[0114] For example, the terminal extracts the input features and target output results for each target sample. First, the terminal performs pairwise comparisons of the input features of the target samples, such as calculating the Euclidean distance or cosine similarity of the feature vectors, to generate sample difference information between each pair of target samples, which is used to quantify the discriminative power of the input samples. Next, the terminal performs pairwise comparisons of the target output results, such as calculating the output difference information through numerical differences or distribution differences (e.g., KL divergence) of the output vectors, which is used to quantify the discriminative power of the model's output characteristics. Then, the terminal compares the input difference information and output difference information of each pair of samples, and evaluates the discriminative power information between each pair of target samples by calculating their absolute difference. For example, when the input difference information is large but the output difference information is small, the discriminative power information of this pair of samples will be reflected as a negative reward signal; while when the input difference information and output difference information are close, the discriminative power information of this pair of samples will be reflected as a positive reward signal, thereby encouraging the gating network's selection strategy to better conform to the input-output relationship. Finally, the terminal integrates all the discrimination information of the target samples, for example, by generating input-output difference reward information through weighted averaging or normalization. The reward objective is to ensure that the gating network's output discrimination matches the input discrimination when processing different inputs. If the input data has high discrimination, the gating network should output a high discrimination (which can be reflected by the output of the hybrid expert model), and vice versa. This reward information guides the hybrid expert model to more accurately match the relationship between input characteristics and output results by directly optimizing the gating network selection strategy.

[0115] In a specific example, for the input sample x m and x n The discrimination index is D input (x m ,x n ): D input (x m ,x n )=||x m -x n ||

[0116] The corresponding target output result y m and y n The sample label is y′ m and y′ n Its output discrimination D output (y m ,y n ) is: D output (y m ,y n )=||y m -y n ||·||y′ m -y′n ||

[0117] Input sample x m and x n The discrimination information is: D g (x m ,x n )=|D input (x m ,x n )-D output (y m ,y n )|

[0118] For T target samples, the discrimination information of all sample pairs is further averaged to determine the input-output difference reward information:

[0119] The reward item for input-output difference reward information in the expert differentiation reward information is: λ g These are the weighting coefficients.

[0120] In this embodiment, by comparing the input and output difference information of the target samples pairwise, the gating network ensures that it maintains consistency in distinguishing between inputs and outputs when processing target samples. Samples with large input differences should also have significant output differences, while samples with small input differences should maintain moderate consistency in output differences. This effectively optimizes the expert selection strategy of the gating network, improves the model's task adaptability and discrimination ability, and further enhances the model's robustness and generalization ability in complex scenarios.

[0121] In an exemplary embodiment, the hybrid expert model includes a gated network model and multiple expert models. Step S104 above, which iteratively trains the hybrid expert model based on activation probability information, expert discrimination reward information, target samples, and target output results, further includes: in each iteration of training, determining a gradient product term of the parameter update gradient based on a preset baseline function and expert discrimination reward information; updating the parameters of the gated network model based on the activation probability information and the gradient product term; and updating the parameters of the target expert model based on the target samples, target output results, and the gradient product term. The target expert model is an expert model determined from multiple expert models based on the activation probability information of this iteration of training.

[0122] The gradient product term, calculated by combining a preset baseline function and expert-discriminative reward information, serves as a core factor in the parameter update process. It stabilizes gradient changes, reduces variance during training, and improves the convergence efficiency of model training. The target expert model is dynamically selected in each iteration using activation probability information and is trained and optimized for the target samples.

[0123] For example, in each training iteration, the terminal first calculates the activation probability information of the target sample using a gated network model and dynamically selects the target expert model. Then, the terminal combines the expert discrimination reward information R corresponding to the current target sample with historical reward values ​​to calculate the gradient product term, which reflects the relationship between the reward signal and the baseline and its impact on parameter updates. Next, the terminal uses the activation probability information and the gradient product term to update the parameters of the gated network, enabling it to more accurately select a suitable target expert model. For the target expert model, the terminal uses the input and output of the target sample to calculate the gradient using the backpropagation algorithm and updates the parameters of the target expert model using the gradient product term.

[0124] In a specific example, the update gradient of the gated network is:

[0125] in, For the gating network based on state s t Select action a t Probability distribution of state s; t For input data sample x t It can be the original input or a feature vector processed by a pre-processing network; action a t G is the target expert model selected by the gating network for the current input; G is the gradient product term.

[0126] Update the gated network parameters θ g : α is the learning rate.

[0127] Update the target expert network parameters θ e : Where β is the learning rate; The update gradient is the loss function corresponding to the model output and label information.

[0128] In addition, the target samples for each iteration of training consist of multiple samples, and each sample activates a different target expert model. Therefore, each target expert model is updated only based on the target samples mapped to itself.

[0129] In this embodiment, by introducing a gradient product term, the guiding role of the reward signal in gradient updates is effectively enhanced, while the baseline function is used to smooth reward fluctuations during training. The gating network can gradually improve the accuracy in selecting the target expert model, while the target expert model also gradually optimizes its adaptability to specific tasks during iterative training, thereby significantly improving the robustness and overall performance of the hybrid expert model in complex task scenarios.

[0130] In an exemplary embodiment, the above-described method of determining a gradient product term of the parameter update gradient based on a preset baseline function and expert-differentiated reward information further includes: obtaining historical expert-differentiated reward information corresponding to previous iterations of training; determining a moving average of the reward information based on the historical expert-differentiated reward information and the expert-differentiated reward information; and using the difference between the expert-differentiated reward information and the moving average of the reward information as the gradient product term.

[0131] Historical expert-differentiated reward information refers to the expert-differentiated reward information calculated in several iterations prior to the current training iteration, used to provide a long-term baseline for the reward signal. The moving average of the reward information is calculated by combining current and historical reward information, used to smooth the reward signal and reduce the impact of fluctuations on gradient updates.

[0132] For example, in each training iteration, the terminal first obtains the expert-discriminative reward information r for the current iteration. t Then, if historical reward information exists prior to this iteration, the terminal averages the most recent historical reward values ​​with the current reward value to obtain a moving average value *b*. Next, the terminal calculates the difference between the current reward value and the moving average value, generating the gradient product term: G = r t -b. During the first iteration of training, due to the lack of historical reward information, the moving average cannot be calculated directly, so the terminal can initialize the moving average to 0.

[0133] In addition, r t =r θ +r y +r g .

[0134] In this embodiment, by introducing a moving average of historical reward information, the impact of reward signal fluctuations on gradient updates is significantly reduced, making training more stable. Particularly in the first iteration, a reasonable initialization strategy addresses the lack of historical reward information, ensuring the correctness of the gradient product term and the consistency of the training process. Simultaneously, the moving average mechanism balances the relationship between the current reward value and historical reward trends, thereby improving the model's convergence efficiency and robustness in complex tasks.

[0135] In one exemplary embodiment, a trained hybrid expert model is used to analyze and process at least one of text data, image data, and audio data. For example, the trained hybrid expert model can be used in natural language processing, computer vision applications, big data analysis, etc.

[0136] In another exemplary embodiment, as shown in FIG3, this application provides a method for training a hybrid expert model, the method comprising:

[0137] Step S301: Initialize network parameters.

[0138] Step S302: Forward propagation of the gated network.

[0139] Step S303: Forward propagation of the maximum loaded expert network.

[0140] Step S304: Obtain the target expert network calculation results.

[0141] Step S305 yields the calculation results of the maximum loading expert network.

[0142] Step S306 yields the output of the hybrid expert model.

[0143] Step S307: Experts differentiate and calculate reward information.

[0144] Step S308: Update gradient calculation.

[0145] Step S309, model update.

[0146] For example, the entire hybrid expert model is used as the decision-making environment for the agent, including the expert network, the gating network, and the input dataset. The gating network acts as the agent, responsible for selecting the appropriate expert network based on the input state. The feature representation of the input sample is used as the state, which can be the original input or a feature vector processed by the preceding network. The expert network selected by the gating network for the current input is used as the action. The probability distribution of the action selected by the gating network based on the state is used as the policy. The expert-discriminated reward information is used as the reward value. This enables the training and optimization of the hybrid expert model.

[0147] In this embodiment, a hybrid expert model is modeled using reinforcement learning, and a specialized reward mechanism is designed to encourage diversity among experts. By penalizing the similarity of parameters among experts, experts are encouraged to learn different features and functions, thereby improving the robustness and expressive power of the model. Meanwhile, in strategies such as sparse activation or probabilistic activation, using only activated expert models for computation can leave some loaded expert model resources idle, failing to fully utilize resources. This embodiment, based on a conventional hybrid expert model, also incorporates the computation results of inactive experts (i.e., the use of the maximum loaded expert network) into the reward, ensuring that the parameter update process fully utilizes computational resources and improves computational resource utilization.

[0148] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0149] Based on the same inventive concept, this application also provides a hybrid expert model training apparatus for implementing the hybrid expert model training method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations of one or more hybrid expert model training apparatus embodiments provided below can be found in the limitations of the hybrid expert model training method described above, and will not be repeated here.

[0150] In an exemplary embodiment, as shown in FIG4, a training apparatus for a hybrid expert model is provided, comprising: a sample acquisition module 401, a model input module 402, a reward acquisition module 403, and a model training module 404, wherein:

[0151] The sample acquisition module 401 is used to acquire the training sample set of the hybrid expert model associated with the input of the large model; the training sample set includes the input data of the large model and the input parsing label information corresponding to the input data;

[0152] The model input module 402 is used to input the target samples in the training sample set into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model, and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model;

[0153] The reward acquisition module 403 is used to obtain expert-differentiated reward information based on the model parameters, target samples, target output results, and sub-output results of each expert model.

[0154] The model training module 404 is used to iteratively train the hybrid expert model based on activation probability information, expert discrimination reward information, target samples and target output results to obtain the trained hybrid expert model.

[0155] In one embodiment, the reward acquisition module 403 is further configured to determine model diversity reward information based on the model parameters of each expert model; determine output diversity reward information based on the sub-output results; determine input-output difference reward information based on the target sample and the target output results; and determine expert differentiation reward information based on the model diversity reward information, output diversity reward information, and input-output difference reward information.

[0156] In one embodiment, the reward acquisition module 403 is further configured to determine the parameter difference information between each pair of expert models based on the model parameters of each expert model; and to determine the model diversity reward information based on the parameter difference information.

[0157] In one embodiment, the reward acquisition module 403 is further configured to determine output difference information between each pair of sub-output results based on the sub-output results; and to determine output diversity reward information based on the output difference information.

[0158] In one embodiment, the reward acquisition module 403 is further configured to determine the sample difference information and output result difference information between each pair of target samples based on the target samples and the target output results; determine the discrimination information between each pair of target samples based on the sample difference information and the output result difference information; and determine the input-output difference reward information based on the discrimination information.

[0159] In one embodiment, the hybrid expert model includes a gated network model and multiple expert models; the model training module 404 is further configured to, in each iteration of training, determine a gradient product term of the parameter update gradient based on a preset baseline function and expert differentiation reward information; update the parameters of the gated network model based on activation probability information and the gradient product term; and update the parameters of the target expert model based on the target sample, the target output result, and the gradient product term; the target expert model is an expert model determined from multiple expert models based on the activation probability information of this iteration of training.

[0160] In one embodiment, the model training module 404 is further configured to obtain historical expert-discriminative reward information corresponding to the previous iteration training; determine the moving average of the reward information based on the historical expert-discriminative reward information and the expert-discriminative reward information; and use the difference between the expert-discriminative reward information and the moving average of the reward information as the gradient product term.

[0161] The modules in the training device for the aforementioned hybrid expert model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0162] The features described in the embodiments of the above-described training method for hybrid expert models are applicable to embodiments of the training apparatus for hybrid expert models. Various implementations of the embodiments of the training apparatus for hybrid expert models can be found in the relevant descriptions in the embodiments of the aforementioned training method for hybrid expert models, and will not be repeated here.

[0163] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 5. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a hybrid expert model training method. The display unit of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0164] Those skilled in the art will understand that the structure shown in Figure 5 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0165] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above embodiments of the training methods for hybrid expert models.

[0166] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above embodiments of the training methods for hybrid expert models.

[0167] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the training method embodiments of the various hybrid expert models described above.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0169] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0170] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0171] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for a hybrid expert model, comprising: Obtain the training sample set of the hybrid expert model associated with the input of the large model; The training sample set includes the input data of the large model and the input parsing label information corresponding to the input data; The target samples in the training sample set are input into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model, and the sub-output result of the loaded expert model. The loaded expert model refers to a preset number of expert models in the hybrid expert model; Based on the model parameters of each expert model, the target sample, the target output result, and the sub-output result, expert differentiation reward information is obtained; Based on the activation probability information, the expert differentiation reward information, the target sample, and the target output result, the hybrid expert model is iteratively trained to obtain the trained hybrid expert model.

2. The method according to claim 1, wherein, The step of obtaining expert differentiation reward information based on the model parameters of each expert model, the target sample, the target output result, and the sub-output result includes: Based on the model parameters of each expert model, determine the model diversity reward information; Based on the sub-output results, determine the output diversity reward information; Based on the target sample and the target output result, determine the input-output difference reward information; The expert differentiation reward information is determined based on the model diversity reward information, the output diversity reward information, and the input-output difference reward information.

3. The method according to claim 2, wherein, The step of determining model diversity reward information based on the model parameters of each expert model includes: Based on the model parameters of each expert model, determine the parameter difference information between each pair of expert models; Based on the parameter difference information, the model diversity reward information is determined.

4. The method according to claim 2, wherein, The step of determining the output diversity reward information based on the sub-output results includes: Based on the sub-output results, determine the output difference information between each pair of sub-output results; Based on the output difference information, the output diversity reward information is determined.

5. The method according to claim 2, wherein, The step of determining the input-output difference reward information based on the target sample and the target output result includes: Based on the target samples and the target output results, determine the sample difference information and output result difference information between each pair of target samples; Based on the sample difference information and the output result difference information, determine the pairwise discriminant information between the target samples; Based on the discrimination information, the input-output difference reward information is determined.

6. The method according to claim 1, wherein, The hybrid expert model includes a gated network model and multiple expert models; The iterative training of the hybrid expert model based on the activation probability information, the expert differentiation reward information, the target sample, and the target output includes: In each iteration of training, a gradient product term of the parameter update gradient is determined based on the preset baseline function and the expert-discriminated reward information; The parameters of the gated network model are updated based on the activation probability information and the gradient product term, and the parameters of the target expert model are updated based on the target sample, the target output result, and the gradient product term; the target expert model is an expert model determined from the plurality of expert models based on the activation probability information trained in this iteration.

7. The method according to claim 6, wherein, The step of determining a gradient product term for the parameter update gradient based on the preset baseline function and the expert-differentiated reward information includes: Obtain the historical expert differentiation reward information corresponding to the previous iteration training; Based on the historical expert-differentiated reward information and the expert-differentiated reward information, determine the moving average of the reward information; The difference between the expert-distinguished reward information and the moving average of the reward information is used as the gradient product term.

8. The method according to claim 1, wherein, The expert differentiation reward information is a reward value determined by evaluating the differences between expert models in parameter configuration and output characteristics.

9. The method according to claim 2, wherein, Before determining the model diversity reward information based on the model parameters of each expert model, the method further includes: The model parameters of each expert model are extracted from the hybrid expert model. The model parameters include the weight matrix, bias value and network structure characteristics of each expert model.

10. The method according to claim 9, wherein, Based on the model parameters of each expert model, the model diversity reward information is determined as follows: The degree of difference between the model parameters of each expert model is calculated to quantify the distributional differences of the model parameters, and the distributional differences are integrated into the model diversity reward information.

11. The method according to claim 2, wherein, Based on the sub-output results, the output diversity reward information is determined as follows: For the sub-output results of the loaded expert model, the output diversity reward information is calculated by comparing the numerical distribution or the directionality of the feature vectors of these output results.

12. The method according to claim 2, wherein, Based on the model diversity reward information, the output diversity reward information, and the input-output difference reward information, the expert differentiation reward information is determined to include: The model diversity reward information, the output diversity reward information, and the input-output difference reward information are weighted and integrated to obtain the expert differentiation reward information; The weights for weighted integration are set based on the priority of the task or the training objective of the expert model.

13. The method according to claim 6, wherein, The target sample in each training iteration consists of multiple samples, and each sample activates a different target expert model.

14. The method according to claim 4, wherein, The sub-output of the loaded expert model is a numerical vector, a classification probability distribution, or other form of intermediate model result.

15. The method according to claim 1, wherein, Use the trained hybrid expert model to analyze and process at least one of the text data, image data, and audio data.

16. A training device for a hybrid expert model, comprising: The sample acquisition module is used to acquire the training sample set of the hybrid expert model associated with the input of the large model; The training sample set includes the input data of the large model and the input parsing label information corresponding to the input data; The model input module is used to input the target samples from the training sample set into the hybrid expert model to obtain the activation probability information of each expert model in the hybrid expert model, the target output result of the hybrid expert model, and the sub-output result of the loaded expert model; the loaded expert model represents a preset number of expert models in the hybrid expert model; The reward acquisition module is used to obtain expert differentiation reward information based on the model parameters of each expert model, the target sample, the target output result, and the sub-output result; The model training module is used to iteratively train the hybrid expert model based on the activation probability information, the expert differentiation reward information, the target sample, and the target output result to obtain the trained hybrid expert model.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein... When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 15.

18. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 15.

Images