The application discloses a kind of multi-task method and
system based on multi-
modal continuous learning, the method includes: obtaining multi-
modal data and constructing multi-
modal dataset;Multi-modal dataset is mapped to unified
language model representation space, and unified
semantic feature representation is obtained;Unified
semantic feature representation is input into sparse mixed expert layer, and the previous expert is based on token level sparse activation by
router, and forward calculation is carried out by self-attention and expert
feedforward neural network;In continuous learning, expert activation frequency and average gradient norm are regularly counted, and low active cold experts or new experts are dynamically pruned, and sparse mixed expert layer is optimized;Decoupling low-rank
adaptation mechanism is used, historical task direction is frozen, and only new
task learning direction and amplitude are updated;Finally, the prediction result of classification,
question answering or generation task is outputed.The application uses the above method, and breaks through the
bottleneck of poor modality adaptability and
single task of traditional multi-modal model.