The invention discloses a multi-
modal large model training optimization method and device, and relates to the technical field of
large model training, and the method comprises the following steps: initializing a training environment, preparing a multi-
modal data set of texts, images and audios, and configuring hardware resources and a
software framework required by training; according to the method, through a progressive training strategy, each
modal sub-model independently learns the modal characteristics, mutual interference during early-stage multi-modal mixed training is avoided, the effect of reducing the complexity at the initial stage of training is achieved, the model training process is easier to control, and in the subsequent multi-modal interactive fusion training stage, the model training efficiency is improved. By means of a modal interaction
algorithm and an elaborately-designed
fusion mechanism, fine and complex association among different
modal data is deeply mined, the purpose of improving the comprehensive understanding ability of the model for multi-modal information is achieved, the accuracy and efficiency of parameter updating are improved for an optimization
algorithm adopted for training in each stage, and the method is suitable for being applied to multi-modal information training. And finally, the overall effect of improving the model stability and performance is achieved.