The application provides a model initialization method based on cross-task shared expert inheritance, constructs a multi-task mixed expert ancestor model and trains; each task data is input into the ancestor model, and the selection probability of each expert is counted according to
layers; the importance
score is constructed based on the cross-task average selection probability and the entropy of the expert, and the shared expert is selected according to
layers to form a learning
gene; each layer of the shared expert is used as a basic expert, a derived expert set is generated through linear expansion to initialize the mixed expert module of the corresponding layer of the
offspring model; in the training or fine-tuning of the
offspring model, Top-K sparse routing is used to activate the expert and output; when the multi-task training is performed, the task-expert
mutual information loss and the task loss are introduced to form a total loss and update the
model parameters, and the convergence speed and the generalization performance of the application in unknown downstream tasks can be improved.