The invention is suitable for the technical field of mixed
modal retrieval, provides a financial
risk identification method based on multi-
granularity mixed
modal retrieval, and is suitable for the field of mixed
modal retrieval. The method comprises the following steps: constructing a multi-modal retrieval
library, respectively encoding image and text modal information by using CLIP and RoBERTa models, and aligning features through a learnable projection matrix to form a searchable financial
knowledge base; a multi-
granularity retrieval scheme combining Dual-
Encoder and Cross-
Encoder is adopted, a Top10 candidate answer is rapidly positioned through coarse
granularity, and a high-relevance candidate set is accurately screened through fine granularity; and finally, through a multi-input weighted integration strategy, integrating multiple rounds of prediction results of the
large model and carrying out weighted summation based on association degrees to generate a final risk prediction result. According to the method, limitation of a single
data source is broken through, comprehensive utilization of cross-modal information is realized,
information quality and retrieval efficiency are ensured, model uncertainty is remarkably reduced, a
risk identification system with robustness and accuracy is formed, and scientific decision support is provided for financial institutions.