The invention discloses a cross-
modal image
text retrieval method based on
deep learning, and provides a novel cross-
modal feature extraction and alignment framework, efficient cross-
modal retrieval is performed by learning
semantic representation of an image and a text in a common feature space, the framework comprises a plurality of modules, and in a
feature extraction stage, the cross-modal
feature extraction and alignment framework is used for extracting and aligning the images and the texts. BERT Tokenizer and BERT Embedding are adopted for text coding, and the image feature extraction is combined with a Faster R-CNN (
Convolutional Neural Network) model and a ResNet-101 model, so that the dimensional consistency of image features and text features is ensured. In the feature alignment stage, fine-grained
semantic alignment of image and text features is realized through an I2T Attention module and a T2I Attention module, and the cross-modal matching precision is remarkably improved. In the correlation scoring stage, a correlation scoring matrix is generated by calculating the similarity
score between the image features and the text features, so that the accuracy of cross-modal matching is improved. And further optimizing the feature alignment effect through normalization and attention redistribution. In an optimization stage, the
system innovatively adopts a shared semantic and sorting loss strategy, integrates an OpenCLIP model framework, efficiently mines a
semantic relationship between an image and a text from unlabeled large-scale data through comparative learning, and shows strong transfer learning ability and
inference precision.