The invention discloses a multi-
modal data feature representation optimization method based on comparative learning and two-stage masks. The method comprises the following steps of: firstly, aiming at an input
single image-text pair, generating two groups of heterogeneous image-text data with mode missing as two inputs of a model by applying two random
mask strategies with different
mask areas and proportions; wherein the first group of data applies a high-proportion
mask to the image and applies a low-proportion mask to the text; the second group applies a low-scale mask to the image and a high-scale mask to the text. Then, the two groups of data respectively pass through an image
encoder and a text
encoder which share parameters, and two different multi-
modal fusion feature vectors are generated through a cross-
modal fusion
encoder; according to the method, masked modal information is recovered through a decoder, and the reconstruction loss of the difference between a
recovery result and
original data is calculated. Meanwhile, two multi-modal fusion feature vectors generated twice are subjected to comparative learning, and the comparative loss of feature representation distances under different enhanced views for approaching the same image-text pair is calculated. And finally, performing weighted summation on the reconstruction loss and the comparison loss to form an overall
loss function, and optimizing
model parameters through back propagation. When the trained encoder is used for a downstream multi-modal classification task, the classification precision and generalization ability of the model can be effectively improved.