The invention belongs to the technical field of man-
machine interaction, and discloses a diagnosis and treatment
auxiliary system and method based on a voice gesture
interaction technology, and the
system comprises a voice recognition module which is used for collecting voice information; converting the voice information into text information; inputting the text information into the large
language model to obtain a voice
system instruction, and sending the voice
system instruction to the
data processing and interaction module; the
gesture recognition module is used for collecting diagnosis and treatment scene information; identifying a preset operation gesture in the diagnosis and treatment scene information; mapping a preset operation gesture into a gesture system instruction and sending the gesture system instruction to the
data processing and interaction module; the
data processing and interaction module is used for receiving and executing a voice system instruction or a gesture system instruction; generating a calling instruction according to the voice system instruction or the gesture system instruction and sending the calling instruction to the
network communication module; converting the received
medical record data into diagnosis and treatment data in a
display format; the
network communication module is connected with a
hospital information system; the calling module is used for calling
medical record data after receiving a calling instruction of the data
processing and interaction module and sending the
medical record data to the data
processing and interaction module; the display module is used for displaying the diagnosis and treatment data. The problem that a doctor cannot conveniently operate a computer to check
patient data due to the
sterility principle in the diagnosis and treatment process is solved, the diagnosis and treatment efficiency is improved,
infection control is facilitated, and the
sterility principle is met.