The invention discloses an automatic
SOAP formatted
medical record generation method based on multi-
modal dialogues, and relates to the technical field of
artificial intelligence and
medical information processing. The method aims at solving the problems that in the prior art, only text information is relied on, so that understanding is one-sided, and factual errors exist in large
language model generation content. The method comprises the following steps: acquiring
voice data of a doctor-patient dialogue and original text data converted from the
voice data; respectively extracting acoustic
rhythm features in the speech and semantic features in the text, and fusing the acoustic
rhythm features and the semantic features into a unified multi-
modal context representation; inputting the multi-
modal context representation into a medical large
language model, and dynamically weighting the generation probability of the large
language model and the entity correlation
score of the
knowledge graph through a dynamic knowledge fusion gating mechanism by innovatively combining with the
medical knowledge graph to obtain a multi-modal context representation; therefore, a formatted
medical record text following an
SOAP (subjective information S,
objective information O, evaluation analysis A and a diagnosis and
treatment plan P) structure is generated lexical element by lexical element. According to the method, the comprehensiveness of dialogue understanding is improved by fusing the multi-modal information, the professionality and the fact accuracy of the generated content are enhanced by utilizing the
knowledge graph, and the automatic generation of the high-quality
medical record is realized.