The invention discloses a real-time
disease deterioration
prediction system based on multi-
modal data, and belongs to the technical field of auxiliary
medical treatment based on
deep learning. Based on a bidirectional
encoder characterization model, a Transform model and a
text generation model, the ICU model can be accessed to a monitor, an
electronic medical record system and the like to obtain multi-
modal data, including real-time high-frequency physiological signals, clinical text records, image reports and the like, and the condition deterioration probability of
ICU patients in multiple different time periods in the future is predicted; and an interpretable report is generated to prompt a doctor of a basis for model evaluation. According to the invention, complex multi-
modal medical data can be processed,
disease change of the ICU patient can be detected in real time, early discovery, early intervention and early treatment can be realized, and the method has great potential in optimizing medical
resource distribution, improving working efficiency of
medical staff and reducing death rate of the ICU patient. In addition, the method is high in generalization ability, and a special
system can be developed for specific departments and specific diseases, so that the prediction accuracy and the result credibility are improved.