The invention provides a
disease course
record generation method and
system based on a mobile terminal and multi-
modal data, and the method comprises the following steps: firstly constructing a directional collection channel and a
ward round session associated with a specific patient through the
signal interaction between the mobile terminal and a sickbed; in the session, multi-
modal data streams with timestamps, such as voice, images and operation instructions, are obtained in real time, and audio semantic anchor points and visual semantic anchor points are extracted from the multi-
modal data streams. And constructing an image-text semantic unit by calculating and associating the audio and video anchor points of which the
semantic similarity exceeds a threshold. And taking the unit as a current observation state, and verifying the legality of the state transition through a finite-state
machine model. And based on the image-text semantic units passing the
verification, template slot filling and term
standardization are carried out, and a structured
disease course
record data packet is generated and written back to the HIS
system. The method has the effect of improving the working efficiency and quality of clinical documents.