The invention relates to an explainable
rare disease auxiliary diagnosis
system and method based on multi-
modal fusion, and the
system comprises an input module which is used for receiving the multi-
modal data of a patient user, the input module is connected with a visual
encoder and a text
encoder, the visual
encoder and the text encoder are connected to a
modal fusion classifier, and the modal fusion classifier is connected with the input module. The input module, the visual encoder, the text encoder and the modal fusion classifier are respectively connected with the
interpretability analysis module, and the visual encoder is used for extracting visual features from multi-
modal data; the text encoder is used for extracting text features from the multi-
modal data; the modal fusion classifier is used for splicing the visual features and the text features, performing classification diagnosis and outputting a classification recognition result; and the
interpretability analysis module is used for generating a thermodynamic diagram containing importance distribution of the multi-
modal data to the classification recognition result. Compared with the prior art, the method can provide an accurate
rare disease auxiliary diagnosis result, and improves the transparency and credibility of the diagnosis process.