Information processing device, information processing system, information processing method, and program

Abstract

This information processing device comprises: an acquisition unit that acquires first finding information based on a medical image; a generation unit that, on the basis of information pertaining to a subject of an image interpretation report, generates an image interpretation report including second finding information obtained by editing the first finding information; and a presentation unit that presents the image interpretation report to the subject.

Description

Information Processing Apparatus, Information Processing System, Information Processing Method and Program

[0001] The present disclosure relates to an information processing apparatus, an information processing system, an information processing method and a program. In particular, the present disclosure relates to a technique for editing an interpretation report generated based on a medical image in accordance with the knowledge level and needs of the presentation destination (such as specialist physicians, general practitioners, paramedical staff, patients and their families).

[0002] Techniques for creating an interpretation report based on a medical image are known. For example, Patent Document 1 discloses an apparatus that stores input values of a plurality of input items included in an interpretation report creation screen and detects inconsistencies and input omissions.

[0003] Japanese Unexamined Patent Application Publication No. 2021-168852

[0004] However, Patent Document 1 does not disclose a technique for automatically rewriting the content of an interpretation report in accordance with the knowledge level, specialized field or other attributes of the subject to whom the report is presented. Specifically, the conventional art does not consider automatically editing and sorting an interpretation report in accordance with the level of specialized knowledge and the required level of detail of information of the subject to whom the interpretation report is presented (e.g., specialist physicians, general practitioners, paramedical staff, patients and their families, etc.). For example, while specialist physicians require detailed information including advanced technical terms and abbreviations, it is desirable to convey medical conditions and findings in plain language to patients and their families. Conventionally, this separate phrasing requires a great deal of work, which increases the burden on medical staff and also causes misunderstandings and insufficient explanation in information communication.

[0005] An object of an aspect of the present disclosure is to provide an information processing technique capable of automatically editing finding information based on a medical image and presenting the information with appropriate expressions and level of detail in accordance with the knowledge level and needs of the subject to whom an interpretation report is presented. Another object is to, when the subject has questions about the interpretation report, interactively provide additional information and explanations to support efficient and accurate communication.

[0006] An information processing device according to one aspect of the present disclosure includes: an acquisition unit that acquires first finding information based on a medical image; a generation unit that generates an image interpretation report including second finding information obtained by editing the first finding information based on information about the subject of the image interpretation report; and a presentation unit that presents the image interpretation report to the subject.

[0007] According to one aspect of this disclosure, it is possible to generate image interpretation reports tailored to the target audience. Traditionally, medical image interpretation reports have been created as primary information for specialists such as radiologists. As a result, for diverse audiences with varying levels of knowledge, including general practitioners, paramedical staff, patients, and their families, each attending physician had no choice but to provide supplementary explanations verbally, leading to an explanation burden, information disparities, and misunderstandings. The information processing device according to one aspect allows patients, general practitioners, specialists, etc., to access the same primary information (first findings information), while controlling only the expression, level of detail, and scope of disclosure. This enables the information processing device according to one aspect to significantly reduce explanation time, improve understanding or patient satisfaction, and quantitatively ensure safety.

[0008] Figure 1 is a block diagram showing an example of the overall configuration of a medical support system. Figure 2 is a block diagram showing an example of a computer. Figure 3 is a block diagram showing an example of the functional configuration of a reporting device. Figure 4 is a diagram showing an example of a display screen. Figure 5 is a flowchart showing an example of a medical support method.

[0009] Hereinafter, embodiments of this disclosure will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] [Embodiment] One embodiment of the present disclosure is an example of an information processing system that supports medical procedures. Hereinafter, the information processing system according to this embodiment will be referred to as the "medical support system". In this embodiment, the medical support system has a function to output an image interpretation report based on medical images in order to support image diagnosis, which is an example of a medical procedure.

[0011] In recent years, artificial intelligence (AI) technology has been used in the analysis of medical images. Among these technologies, systems that detect findings from medical images and generate interpretation reports are well-known.

[0012] A radiology report is a document in which a radiologist diagnoses the presence or absence of lesions based on medical images and reports the findings. Typically, radiology reports are prepared based on examination requests from clinical departments to the radiology department. While radiology reports are often prepared for physicians with extensive medical knowledge, they may also be presented to other healthcare professionals (also known as paramedical staff), patients, or their families.

[0013] Ideally, radiology reports should be created according to the expertise level and needs of the subject. Conventional systems only output findings detected by radiologists or auxiliary AI as text, and lacked sufficient means to automatically rewrite the information according to the level of expertise of the recipient (e.g., doctor or patient). As a result, there was a risk of insufficient explanation or misunderstanding for patients and paramedical staff.

[0014] This embodiment enables the generation of flexible and accurate image interpretation reports by automatically changing the wording and terminology of findings obtained from medical images based on information about the subject of the image interpretation report (e.g., specialists, general practitioners, paramedical staff, patients / families, etc.).

[0015] In one respect, this embodiment allows for the creation of personalized image interpretation reports because the findings information is edited based on information about the subject of the image interpretation report. In another respect, this embodiment allows for the presentation of personalized image interpretation reports, thereby promoting the subject's understanding of the report.

[0016] <Overall Configuration> The overall configuration of the medical support system according to this embodiment will be explained with reference to Figure 1. Figure 1 is a schematic block diagram showing an example of the medical support system 1000, illustrating a configuration in which a reporting device 10 and terminal devices 20a and 20b (hereinafter collectively referred to as "terminal devices 20" unless otherwise specified) are connected to each other via a communication network N. In this example, two terminal devices 20a and 20b are shown, but one to more terminal devices 20 may be connected.

[0017] As shown in Figure 1, the medical support system 1000 includes a reporting device 10 and a terminal device 20. These can communicate data bidirectionally via a communication network N such as a LAN (Local Area Network), VPN (Virtual Private Network), or the Internet (illustrated as a cloud in the diagram).

[0018] The reporting device 10 is an example of an information processing device that generates an image interpretation report. The reporting device 10 may be a computer such as a personal computer, workstation, or server.

[0019] The reporting device 10 generates an image interpretation report in response to a generation request from the terminal device 20. The reporting device 10 transmits the generated image interpretation report to the terminal device 20. The reporting device 10 also receives questions from the subject regarding the image interpretation report from the terminal device 20. The reporting device 10 generates answers to the questions and transmits them to the terminal device 20.

[0020] Terminal device 20 is an example of an information processing device operated by a user of the medical support system 1000. Terminal device 20 may be a personal computer, tablet terminal, or smartphone, for example.

[0021] The terminal device 20 transmits a request to the reporting device 10 to generate an image interpretation report in response to user operations. The terminal device 20 presents the image interpretation report received from the reporting device 10 to the subject. The terminal device 20 also transmits questions from the subject to the reporting device 10. The terminal device 20 receives the answers to the questions from the reporting device 10 and presents them to the subject.

[0022] The user of terminal device 20 and the subject of the image interpretation report may be the same person or different people. For example, the subject of the image interpretation report may be a patient, and the user of terminal device 20 may be the doctor examining the patient.

[0023] The overall configuration of the medical support system 1000 shown in Figure 1 is just one example, and various system configurations are possible depending on the application and purpose. As shown by the dotted line in Figure 1, it is also possible to have a configuration in which multiple terminal devices 20 are connected to the reporting device 10 via a communication network N. Alternatively, multiple reporting devices 10 may be provided, and the system may be implemented as a cluster configuration or a cloud service. For example, the reporting device 10 may be implemented by multiple computers, or as a cloud computing service. For example, the medical support system 1000 may be implemented by a standalone computer that integrates the reporting device 10 and the terminal devices 20. The classification of devices such as the reporting device 10 and terminal devices 20 shown in Figure 1 is just one example.

[0024] <Hardware Configuration> The reporting device 10 and terminal device 20 included in the medical support system 1000 can both be implemented using a computer. Figure 2 is a block diagram showing an example of such a computer hardware configuration. In this embodiment, the computer 500 shown in Figure 2 will be used as an example, but this is merely one example, and it can be implemented in various forms such as a personal computer, server, workstation, or virtual machine on the cloud.

[0025] As shown in Figure 2, the computer 500 includes a processor 501, memory 502, auxiliary storage device 503, input device 504, output device 505, communication device 506, drive device 507, etc., and is interconnected via a bus 508. Furthermore, an external storage medium 509 (for example, a CD-ROM or DVD-ROM) can be connected to the drive device 507. The storage medium 509 may be any medium, such as a USB memory or flash memory.

[0026] The processor 501 is an example of a computing device that implements the control and functions of the entire computer 500. The processor 501 may include, for example, at least one of a CPU (Central Processing Unit) or a GPU (Graphic Processing Unit). The processor 501 reads various programs installed in the auxiliary storage device 503 into the memory 502 and executes them.

[0027] Memory 502 functions as a working area for temporarily storing programs and data, and RAM (Random Access Memory) is preferred, but ROM (Read Only Memory), cache memory, or other semiconductor memory may also be used in combination.

[0028] The processor 501 and the memory 502 form what is known as a computer (hereinafter also referred to as the "control unit"). The computer realizes various functions by having the processor 501 execute various programs read into the memory 502.

[0029] The auxiliary storage device 503 is an example of a non-volatile storage device that stores various programs and various data used by those programs. The auxiliary storage device 503 may include, for example, at least one of the following: an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The programs stored in the auxiliary storage device 503 may include, for example, basic software (Operating System) or various applications.

[0030] The input device 504 is a device used by the user of the computer 500 to input various signals. The input device 504 may include, for example, an operating device such as a mouse, keyboard, or touch panel, or a sound collection device such as a microphone.

[0031] The output device 505 is a device that outputs the processing results of various processes performed by the computer 500. The output device 505 may include, for example, a display device such as a display or touch panel, or an audio device such as a speaker.

[0032] The communication device 506 is a communication device for communicating with external devices. For example, the communication device 506 may be a network interface that performs wired communication or wireless communication via the communication network N.

[0033] The drive device 507 is a device for setting the storage medium 509. The storage medium 509 may include, for example, a medium that stores electronic data optically or magnetically, such as a CD-ROM, DVD-ROM, flexible disk, or magneto-optical disk. The storage medium 509 may also include a medium such as a semiconductor memory that stores electronic data electrically, such as a flash memory.

[0034] The various programs to be installed on the auxiliary storage device 503 are installed, for example, when the distributed storage medium 509 is set in the drive device 507, and the various programs stored on the storage medium 509 are read by the drive device 507. Alternatively, the various programs to be installed on the auxiliary storage device 503 may be downloaded from a cloud service via the communication device 506. Furthermore, they may be introduced as containers or virtual machine images using virtualization technology.

[0035] The computer 500 can perform various processes described later by having the hardware configuration shown in Figure 2. Note that the hardware configuration shown in Figure 2 is just one example, and the computer 500 may have other hardware configurations. For example, the computer 500 may have multiple processors 501 or multiple memory 502.

[0036] <Functional Configuration> The functional configuration of the reporting device 10 will be described in detail below with reference to Figure 3. This device is characterized by its ability to acquire findings information based on medical images, generate an image interpretation report, and consistently provide additional information in response to questions from the subject.

[0037] As shown in Figure 3, the reporting device 10 comprises a model storage unit 101, a report storage unit 102, an input unit 110, a detection unit 120, an acquisition unit 130, an assignment unit 140, a generation unit 150, a presentation unit 160, a suggestion unit 170, a reception unit 180, and a response unit 190. When a pre-installed reporting program is executed, the reporting device 10 works in conjunction as the model storage unit 101, report storage unit 102, input unit 110, detection unit 120, acquisition unit 130, assignment unit 140, generation unit 150, presentation unit 160, suggestion unit 170, reception unit 180, and response unit 190, thereby realizing "automatic generation and presentation of image interpretation reports tailored to the subject's level of expertise and needs" and "interactive question-and-answer function."

[0038] For example, the model storage unit 101 and the report storage unit 102 are implemented by the memory 502 or auxiliary storage device 503 shown in Figure 2. For example, the input unit 110, detection unit 120, acquisition unit 130, assignment unit 140, generation unit 150, presentation unit 160, proposal unit 170, reception unit 180, and response unit 190 are implemented by a process in which a program deployed from the auxiliary storage device 503 shown in Figure 2 onto the memory 502 is executed by the processor 501.

[0039] The model memory unit 101 stores trained models. A trained model is a machine learning model that has been trained to perform a predetermined task. Machine learning models may include, for example, neural networks, deep neural networks, convolutional neural networks, and neural networks with attention mechanisms. A neural network with an attention mechanism may be, for example, a neural network called a Transformer. The recognition model is referenced by the detection unit 120 when extracting findings from medical images. On the other hand, the language model is referenced by the generation unit 150 and the response unit 190 when editing and generating natural language text. Thus, this embodiment employs a configuration that calls up multiple types of trained models as needed.

[0040] A trained model may include a trained language model. A language model is an example of a machine learning model that has been trained to perform a given language processing task. A language model may be a large language model (LLM) trained to perform a variety of language processing tasks. A language model may also be a small language model (SLM) trained to perform a specific language processing task.

[0041] A trained model may include a trained recognition model. A recognition model is an example of a machine learning model trained to recognize findings contained in medical images. A recognition model may be trained to output the detection results of findings captured in a medical image when a medical image is input. A recognition model may be trained to comprehensively recognize findings when a medical image capturing a wide area of ​​the body is input.

[0042] Furthermore, the present invention includes a mechanism for reducing the risk of "hallucination" (false or inaccurate output) that may occur when a general large language model (LLM) is applied to the medical field. Specifically, this system is characterized by including a process of re-evaluating a draft report generated by a large language model against a knowledge graph and automatically correcting the draft report. The system has a separate process that determines whether an output result is correct by comparing it with the text of the citing source, based on a medical knowledge base (knowledge graph) such as regulations like cancer treatment guidelines and cutting-edge paper knowledge.

[0043] Image interpretation reports are stored in the report storage unit 102. The image interpretation reports stored in the report storage unit 102 may be generated by the generation unit 150.

[0044] The input unit 110 accepts input of a generation request. The generation request may include, for example, medical images, examination information, examinee information and subject information. The generation request may include finding information instead of medical images.

[0045] A medical image is an image obtained by imaging the structure or function of the inside of a human body, mainly for the purpose of disease diagnosis and treatment. In the present embodiment, a person who is the subject of a medical image is referred to as an examinee. Examples of the medical image include X-ray images, Computed Tomography (CT) images, Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET) images, and the like.

[0046] Examination information is information related to an examination. The examination may be an image examination for capturing medical images. The examination information may include information indicating the purpose of the examination. As an example, the examination information may include information indicating the content of a request from a clinical department to a radiology department.

[0047] Examinee information is information relating to an examinee. An examinee is a person who has undergone an examination that is the subject of creating an interpretation report, and is the person who is the subject of a medical image. The examinee information may include information relating to the examinee's health condition. As one example, the examinee information may include medical history, medical records, health check-up results, and the like.

[0048] Subject information is information relating to a subject. A subject is a person to whom an interpretation report is presented. The subject may also be described as a person who refers to the interpretation report. As one example, the subject information may include the subject's type, level of medical knowledge, language used, cultural background, and the like. As one example, the subject's type may include specialized physicians, non-specialized physicians, medical personnel other than physicians, examinees, the examinee's family members, and the like.

[0049] Finding information is information indicating findings described in an interpretation report. The finding information may be information indicating a lesion that can be diagnosed by an interpreting physician, a suspected lesion, or the like. The finding information may include text data in which lesions and the like are described in natural language. As one example, the finding information may include information indicating findings described in an image diagnosis guideline issued by the Japan Radiological Society. The finding information included in the generation request may be finding information input by an interpreting physician.

[0050] The detection unit 120 inputs the medical image included in the generation request input from the input unit 110 into a trained recognition model, and detects lesion sites and abnormal findings in the image. The findings detected here are generated as finding information (first finding information) and passed to the acquisition unit 130. When there is finding information already included in the generation request, the acquisition unit 130 also receives such finding information together, and provides it to the subsequent assigning unit 140 and generation unit 150.

[0051] The detection unit 120 may detect findings based on a trained recognition model. The detection unit 120 may acquire finding detection results by inputting a medical image into a trained recognition model. The detection unit 120 may generate finding information indicating detected findings.

[0052] The acquisition unit 130 acquires findings information. The acquisition unit 130 may acquire findings information based on medical images (first findings information). The acquisition unit 130 may acquire findings information detected from medical images included in the generation request. The acquisition unit 130 may acquire findings information generated by the detection unit 120. The acquisition unit 130 may acquire findings information included in the generation request. The acquisition unit 130 may acquire findings information input to the reporting device 10. The acquisition unit 130 may acquire findings information written by a radiologist.

[0053] The assignment unit 140 assigns priority to the findings information. The assignment unit 140 may assign priority to the findings information acquired by the acquisition unit 130. The assignment unit 140 may assign priority to the findings information detected by the detection unit 120. The assignment unit 140 may assign priority to the findings information included in the generation request.

[0054] Priority may be an indicator of the priority of treatment. Priority may also be information indicating priority. Priority may also be an indicator of the importance of a finding or the urgency of treatment. Priority may be defined in multiple levels. Priority may include, for example, a first priority requiring urgent treatment (e.g., treatment required), a second priority requiring further examination (e.g., re-examination, detailed examination, etc.), and a third priority not requiring urgent treatment or examination (e.g., observation, etc.).

[0055] The assignment unit 140 may assign priorities to the findings information in accordance with predetermined rules. The assignment unit 140 may assign priorities to the findings information based on patient information (e.g., the patient's medical history). The assignment unit 140 may assign priorities to the findings information based on examination information (e.g., the purpose of the examination). The assignment unit 140 may assign priorities to the findings information based on the comparison result between the examination information and the findings information. In this case, the assignment unit 140 refers to a priority determination table stored in the reporting device 10 and assigns priorities by comparing the "purpose of the examination" and the "type of lesion detected in the findings". For example, it is also possible to use a predefined urgency code or guideline as a key.

[0056] For example, if the findings information includes wording that requires urgent treatment, the assigning unit 140 may assign a first priority to the findings information. For example, if the findings information includes an abnormality unrelated to the patient's medical history, the assigning unit 140 may assign a second priority to the findings information. For example, if the findings information includes an abnormality unrelated to the purpose of the examination, the assigning unit 140 may assign a second priority to the findings information. For example, if the findings information includes an abnormality related to the patient's medical history that does not require urgent treatment, the assigning unit 140 may assign a third priority to the findings information. Note that the wording that requires urgent treatment may be predetermined. For example, the wording that requires urgent treatment may be determined based on imaging diagnostic guidelines.

[0057] The generation unit 150 generates an image interpretation report. The generation unit 150 may generate an image interpretation report that includes findings information (second findings information) edited from findings information (first findings information) acquired by the acquisition unit 130. The generation unit 150 may generate an image interpretation report based on subject information included in the generation request. The generation unit 150 may generate an image interpretation report based on the priority assigned by the assignment unit 140.

[0058] The generation unit 150 may edit the findings information based on a trained language model. The generation unit 150 may also obtain the result of generating a radiographic report by inputting instructions for generating a radiographic report into a trained language model. The instructions for generating a radiographic report may also be text data written in natural language, also known as a prompt.

[0059] The generation unit 150 may generate an image interpretation report with varying amounts and / or content of information according to the subject information categorized into multiple medical knowledge levels, based on the findings information acquired by the acquisition unit 130. For example, if the subject is a physician, an image interpretation report is generated that describes the detailed analysis results using technical terms. Important findings are placed at the beginning of the image interpretation report. On the other hand, if the subject is a patient, an image interpretation report is generated that avoids technical terms and is written in simple language. Diagrams or illustrations showing the location or condition of the findings may also be included as needed.

[0060] Specifically, the prompt for generating a radiographic report may include patient information, findings information, and priority. The language model edits the findings information based on the patient information, sorts the findings information based on priority, and generates a radiographic report including a detailed pathological explanation (first radiographic report) and / or a radiographic report including a simplified pathological explanation (second radiographic report). Here, a detailed pathological explanation is a detailed explanation that uses a relatively high proportion of abbreviations and medical jargon, and is intended to accurately grasp the pathological condition. A simplified pathological explanation, on the other hand, uses a lower proportion of abbreviations and medical jargon than a detailed explanation, and is intended to explain the pathological condition in a simple and easy-to-understand manner. In the case of a simplified pathological explanation, the system may automatically edit the text, such as inserting illustrations or simplified expressions. For example, if the prompt specifies "Patient: Patient, Medical Knowledge Level: Low to Medium," the language model will replace difficult technical terms with simpler terms and generate a report text that focuses on essential information. On the other hand, when "Target audience: Specialists" is specified, it is possible to control the presentation of analysis results that include detailed descriptions of the clinical course and prognosis, as well as advanced technical terms and abbreviations.

[0061] The generation unit 150 may determine the subject's attributes or level of medical knowledge based on the subject's questions, input text, or dialogue history. The generation unit 150 may change the format or level of detail of the image interpretation report according to the determination result of the subject's attributes or level of medical knowledge. The generation unit 150 may determine the subject's attributes or level of medical knowledge based on the frequency of use of medical terms, the specialization of the questions, or at least one of the past dialogue history.

[0062] The generation unit 150 may send an instruction to generate a radiographic report to another information processing device equipped with a trained language model. The other information processing device may input the instruction to generate a radiographic report to the trained language model and send the radiographic report generation result output from the trained language model to the reporting device 10. The generation unit 150 may acquire the radiographic report received from the other information processing device.

[0063] When the generation unit 150 transmits the instruction to generate a reading report to another information processing device, it may conceal certain information included in the generation instruction. For example, the generation unit 150 may anonymize personal information included in the subject information or findings information included in the generation instruction. Personal information may include, as an example, information specified in the Guidelines on the Secure Management of Medical Information Systems issued by the Ministry of Health, Labour and Welfare of Japan, the General Data Protection Regulation (GDPR) of the European Union, or the Health Insurance Portability and Accountability Act (HIPAA) of the United States.

[0064] The generation unit 150 may edit or delete information included in the image interpretation report generation instruction based on the subject's country of residence or applicable laws and regulations concerning the protection of personal information or medical information, and then transmit it to another information processing device. Laws and regulations concerning the protection of personal information or medical information may include the General Data Protection Regulation in the European Union or the Health Insurance Portability and Accountability Act in the United States. The generation unit 150 may edit or delete personally identifiable information included in the image interpretation report generation instruction. Personally identifiable information may include name, date of birth, address, location information, medical record number, facial image, or biometric identification information.

[0065] The generation unit 150 may anonymize or abstract any information that could identify the patient before transmitting the patient information to another information processing device. If the patient information includes the name of a disease, and the disease is a rare disease or a genetic disease, and the patient can be identified from the name of the disease, the generation unit 150 may replace the name of the disease with a higher-level disease category.

[0066] The presentation unit 160 outputs the image interpretation report. The presentation unit 160 may also control the presentation of the image interpretation report to the subject. The presentation unit 160 may transmit the image interpretation report to the terminal device 20. The terminal device 20 may display the image interpretation report received from the reporting device 10 on the output device 505. The presentation unit 160 may also display the image interpretation report on the output device 505 of the reporting device 10.

[0067] The presentation unit 160 may output a screen for presenting the image interpretation report (hereinafter referred to as the "presentation screen"). The presentation screen includes a display area for displaying the image interpretation report (hereinafter referred to as the "display area") and a dialogue area where the user can interactively input questions. For example, in the dialogue area, the system can accept additional inputs such as "I want to know more" or "Please tell me about related literature" in response to the findings explanation presented by the report device 10, and the response unit 190 is configured to present new information accordingly.

[0068] If the subject (for example, a physician) is operating the reporting device 10 or terminal device 20, the subject may refer to the image interpretation report displayed on the output device 505. If the subject (for example, a patient) is not operating the reporting device 10 or terminal device 20, the user of the reporting device 10 or terminal device 20 (for example, a physician) may present the image interpretation report displayed on the output device 505 to the subject.

[0069] The suggestion unit 170 refers to the contents of the image interpretation report and the subject information (specialist, general practitioner, patient, etc.) and automatically generates candidate questions such as "a list of questions frequently asked by users in similar positions" and "supplementary explanations related to the image interpretation content," and displays them on the screen. This allows the subject to submit questions with a single click, preventing insufficient explanations or oversights.

[0070] The proposal unit 170 may generate candidate questions based on a trained language model. The proposal unit 170 may obtain the result of generating candidate questions by inputting a command to generate candidate questions into the trained language model. The command to generate candidate questions may include subject information and a medical imaging report. The language model generates candidate questions appropriate for the subject based on the subject information. Examples of questions appropriate for the subject may include questions that ask for information useful to the subject, or questions that similar subjects frequently ask.

[0071] The reception unit 180 accepts questions. The reception unit 180 may accept questions from the subject. The reception unit 180 may receive questions entered into the terminal device 20 from the terminal device 20. The reception unit 180 may accept question input via the input device 504 of the reporting device 10. Question input may be by text input or voice input. The reception unit 180 may accept questions entered on the dialogue screen.

[0072] For example, if the subject is a physician, their questions may include requests for additional information, statistical data, or relevant literature regarding specific findings included in the radiology report. On the other hand, if the subject is a patient, their questions may include questions about the diagnosis or future treatment.

[0073] The response unit 190 outputs an answer. The response unit 190 may output an answer to a question from the subject. The response unit 190 may output an answer to a question received by the reception unit 180. The response unit 190 may display the answer on the dialogue screen.

[0074] The response unit 190 may generate an answer based on a trained language model. The response unit 190 may obtain the result of generating an answer by inputting an answer generation instruction to the trained language model. The answer generation instruction may include questions from the subject and an image interpretation report. The language model generates an answer to the questions from the subject based on the image interpretation report.

[0075] The response unit 190 may generate a response based on Retrieval Augmented Generation (RAG). Retrieval Augmented Generation is a technique that includes reference information obtained by searching predetermined data sources as input to a language model in order to obtain good output results from the language model. By considering reference information not included in the training data, the language model can generate more appropriate data. For example, if the response unit 190 receives a "question about prognosis," it searches the literature database and clinical guidelines within the reporting device 10 using Retrieval Augmented Generation (RAG) and adds the obtained statistical information and reference materials to the prompt. Based on this reference information and the content of the image interpretation report, the language model generates a more accurate and well-supported response.

[0076] It should be noted that the functional configuration of the reporting device 10 shown in Figure 3 is just one example, and it goes without saying that there are various functional configurations depending on the application and purpose, and module division and external implementation are also possible. For example, the model storage unit 101 may be provided by another information processing device, including an external cloud that can communicate with the reporting device 10 via a communication network N. The division of storage units or processing units such as the model storage unit 101, report storage unit 102, input unit 110, detection unit 120, acquisition unit 130, assignment unit 140, generation unit 150, presentation unit 160, proposal unit 170, reception unit 180, and response unit 190 shown in Figure 3 is just one example. Also, the detection unit 120 and the acquisition unit 130 may be implemented as a single unit.

[0077] <User Interface> The user interface of the medical support system 1000 will be described with reference to Figure 4. The user interface of the medical support system 1000 may include a display screen. The display screen may be displayed on the output device 505 of the report device 10 or terminal device 20. Figure 4 is a diagram showing an example of a display screen.

[0078] As shown in Figure 4, the presentation screen 600 has a display area 610 and an interaction area 620. The image interpretation report is displayed in the display area 610. The interaction area 620 has a question input field 621 and a send button 622. The question input field 621 is an example of a screen component that accepts the input of a question. The send button 622 is an example of a screen component that sends the question entered in the question input field 621 to the report device 10. When the user operates the send button 622 (for example, by clicking or tapping), the question entered in the question input field 621 is displayed in the interaction area 620.

[0079] The dialogue area 620 displays one or more questions q (q1, q2) and one or more answers r (r1, r2). The question q area displays the question received by the receiving unit 180 of the reporting device 10. The answer r area displays the answer output by the answering unit 190 of the reporting device 10. Answer r1 is the answer to question q1. Answer r2 is the answer to question q2. Questions q1, q2 and answers r1, r2 are displayed in chronological order from top to bottom.

[0080] Figure 4 shows an example where two questions q and two answers r are displayed, but there is no limit to the number of questions q or answers r displayed in the dialogue area 620. If it is not possible to display all of the questions q and answers r in the dialogue area 620, the dialogue area 620 may be controlled to be scrollable.

[0081] The dialogue area 620 may display one or more candidate questions c (c1, c2). Candidate c is an area that displays candidate questions generated by the suggestion unit 170 of the reporting device 10. When a candidate c is manipulated by the user (for example, by clicking or dragging), the manipulated candidate c may be entered into the question input field 621.

[0082] For example, the display screen 600 may be configured as web content. The display screen 600 may be displayed by a web browser pre-installed on the reporting device 10 or terminal device 20. Alternatively, for example, the display screen 600 may be configured as the screen of a dedicated application. The display screen 600 may be displayed by launching an application pre-installed on the reporting device 10 or terminal device 20.

[0083] It should be noted that the screen configuration of the presentation screen 600 shown in Figure 4 is just one example, and there are various other screen configurations depending on the application and purpose. For example, the presentation screen 600 may be composed of different screens for the display area 610 and the dialogue area 620. Also, for example, the presentation screen 600 may have an area for displaying other information in addition to the image interpretation report. Other information may include, for example, patient information, subject information, or examination information.

[0084] <Processing Procedure> The medical support method performed by the medical support system 1000 will be explained with reference to Figure 5. Figure 5 is a flowchart showing an example of a medical support method.

[0085] In step S1, the user of the terminal device 20 inputs medical images, examination information, patient information, and subject information into the terminal device 20. The user may input findings information instead of medical images. The terminal device 20 sends a generation request to the report device 10. The generation request includes the medical images (or findings information), examination information, patient information, and subject information input into the terminal device 20.

[0086] The reporting device 10 receives a generation request from the terminal device 20. The input unit 110 of the reporting device 10 receives the input of the generation request received by the reporting device 10. The input unit 110 sends the medical image included in the generation request to the detection unit 120. The input unit 110 also sends the examination information and patient information included in the generation request to the assignment unit 140, and the subject information to the generation unit 150.

[0087] In step S2, the detection unit 120 of the reporting device 10 receives a medical image from the input unit 110. The detection unit 120 reads a recognition model from the model storage unit 101. The detection unit 120 inputs the medical image into the recognition model. The recognition model detects findings from the input medical image and outputs the findings detection results. The detection unit 120 sends finding information indicating the findings detected by the recognition model to the acquisition unit 130.

[0088] In step S3, the acquisition unit 130 of the reporting device 10 receives the findings information from the detection unit 120. If the generation request includes findings information, the acquisition unit 130 may acquire the findings information included in the generation request. In this case, step S2 does not need to be executed. The acquisition unit 130 sends the findings information to the assignment unit 140.

[0089] The assignment unit 140 receives findings information from the acquisition unit 130. The assignment unit 140 also receives examination information and patient information from the input unit 110. Based on the examination information and patient information, the assignment unit 140 assigns priority to the findings information. The assignment unit 140 sends the prioritized findings information to the generation unit 150.

[0090] In step S4, the generation unit 150 of the report device 10 receives priority-assigned findings information from the assignment unit 140. The generation unit 150 also receives subject information from the input unit 110. The generation unit 150 reads the language model from the model storage unit 101. The generation unit 150 generates an instruction to generate a reading report. The generation unit 150 inputs the instruction to generate a reading report into the language model. The language model generates a reading report based on the generation instruction and outputs the result of the reading report generation. The generation unit 150 retrieves the reading report generated by the language model. The generation unit 150 stores the reading report in the report storage unit 102. The generation unit 150 sends the reading report to the presentation unit 160.

[0091] In step S5, the presentation unit 160 of the reporting device 10 receives the image interpretation report from the generation unit 150. The presentation unit 160 controls the presentation of the image interpretation report to the subject. Specifically, the presentation unit 160 transmits a presentation screen 600 having a display area 610 for the image interpretation report to the terminal device 20. The terminal device 20 displays the presentation screen 600 received from the reporting device 10 on the output device 505. The subject can then refer to the image interpretation report displayed in the display area 610.

[0092] In step S6, the proposal unit 170 of the report device 10 reads the image interpretation report from the report storage unit 102. The proposal unit 170 reads the language model from the model storage unit 101. The proposal unit 170 generates an instruction to generate candidate questions. The proposal unit 170 inputs the instruction to generate candidate questions to the language model. The language model generates candidate questions based on the generation instruction and outputs the result of the candidate question generation. The proposal unit 170 obtains the candidate questions generated by the language model. The proposal unit 170 sends the candidate questions to the terminal device 20. The terminal device 20 displays the candidate questions received from the report device 10 in the dialogue area 620 of the presentation screen 600.

[0093] In step S7, the user of the terminal device 20 enters a question in the question input field 621 on the display screen 600 and operates the send button 622. The terminal device 20 sends the question entered in the question input field 621 to the report device 10. The report device 10 receives the question from the terminal device 20. The receiving unit 180 of the report device 10 receives the question received by the report device 10. The receiving unit 180 sends the question to the answering unit 190.

[0094] In step S8, the receiving unit 180 of the reporting device 10 receives a question from the answering unit 190. The answering unit 190 reads the language model from the model storage unit 101. The answering unit 190 generates an instruction to generate an answer. The answering unit 190 inputs the instruction to generate an answer to the language model. The language model generates an answer to the question based on the instruction and outputs the result of the answer generation. The answering unit 190 retrieves the answer generated by the language model. The answering unit 190 sends the answer to the terminal device 20. The terminal device 20 displays the answer received from the reporting device 10 in the dialogue area 620 of the display screen 600.

[0095] In step S9, the reporting device 10 determines whether the dialogue has ended. For example, the reporting device 10 may determine that the dialogue has ended if it detects that the display screen 600 has closed. If it determines that the dialogue has ended (YES), the reporting device 10 terminates processing. On the other hand, if it determines that the dialogue has not ended (NO), the reporting device 10 returns processing to step S6.

[0096] Returning to step S6, the suggestion unit 170 of the reporting device 10 generates new question candidates. At this time, the suggestion unit 170 may input a generation instruction to the language model that includes the question received in step S7 and the answer output in step S8. The suggestion unit 170 sends the new question candidates to the terminal device 20. After that, the reception unit 180 of the reporting device 10 waits for the input of new questions. The reporting device 10 repeatedly executes the processes from step S6 to step S8 until it is determined in step S9 that the dialogue has ended. The user can continue the dialogue until there are no more questions regarding the image interpretation report.

[0097] <Effects of the Embodiment> The reporting device 10 according to one embodiment of the present disclosure acquires first finding information based on medical images, generates a reading report that includes second finding information which is an edited version of the first finding information based on information about the subject of the reading report, and presents the reading report to the subject.

[0098] In one respect, this embodiment allows for the creation of personalized image interpretation reports because the findings information is edited based on information about the subject of the image interpretation report. In another respect, this embodiment allows for the presentation of personalized image interpretation reports, thereby promoting the subject's understanding of the findings information.

[0099] By configuring the system as described above, the medical support system 1000 can improve diagnostic efficiency. For example, the reporting device 10 comprehensively detects findings from medical images, prioritizes them, and generates an image interpretation report, allowing for quick confirmation of important findings and streamlining the diagnostic process.

[0100] Furthermore, the medical support system 1000 can optimize information transmission. For example, the reporting device 10 can customize the content of the image interpretation report according to the subject, so that information is transmitted appropriately and effectively, preventing misunderstandings and insufficient information.

[0101] Furthermore, the medical support system 1000 can improve the user experience. For example, since the reporting device 10 has an interactive interface, users can resolve any questions while referring to the image interpretation report, thereby deepening their understanding of the diagnosis and treatment plan.

[0102] Furthermore, the medical support system 1000 can improve the quality of medical services. For example, the medical support system 1000 facilitates communication between healthcare professionals and patients, which in turn improves patient satisfaction and contributes to optimizing treatment effectiveness.

[0103] Traditionally, medical image interpretation reports were created as primary information for specialists such as radiologists. Therefore, for diverse audiences with varying levels of knowledge, including general practitioners, paramedical staff, patients, and their families, each attending physician had no choice but to provide supplementary verbal explanations, leading to explanation burdens, information disparities, and misunderstandings. The medical support system 1000 allows patients, general practitioners, and specialists to access the same primary information (first findings information), while controlling only the expression, level of detail, and scope of disclosure. This enables the medical support system 1000 to significantly reduce explanation time, improve understanding or patient satisfaction, and quantitatively ensure safety.

[0104] The medical support system 1000 can be used not only by medical institutions but also by medical device manufacturers, healthcare-related companies, and educational institutions, and is easily expandable internationally. For example, medical institutions can contribute to improving the efficiency and accuracy of image diagnostic work in hospitals or clinics. It can also be expected to reduce the burden on doctors and improve the quality of patient services. For example, medical device manufacturers can develop new medical devices incorporating the various functions of the reporting device 10, or improve existing medical devices. For example, healthcare-related companies can provide medical solutions linked with electronic medical record systems and medical information systems. For example, educational institutions can use it as an educational tool for medical students or residents to learn about image diagnostics. Furthermore, for example, the reporting device 10 can generate image interpretation reports that support multiple languages ​​and cultural backgrounds, making it easy to expand into the international medical market. In addition, it can provide appropriate information to patients and their families who do not have specialized knowledge, leading to faster informed consent and improved patient satisfaction. Moreover, it reduces the time and effort required for explanations by doctors and paramedical staff, contributing to a reduction in the burden on medical professionals.

[0105] [Supplement] Each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" in this specification includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each function described above.

[0106] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications or changes are possible within the scope of the gist of the present invention as described in the claims.

[0107] Furthermore, the following forms are possible for disclosure technology.

[0108] (Note 1) An information processing device comprising: an acquisition unit configured to acquire first finding information based on a medical image; a generation unit configured to generate the image interpretation report including second finding information obtained by editing the first finding information based on information about the subject of the image interpretation report; and a presentation unit configured to present the image interpretation report to the subject.

[0109] (Note 2) The information processing apparatus according to Note 1, further comprising a detection unit configured to detect the first finding information based on the medical image.

[0110] (Note 3) The information processing device according to Note 1 or 2, further comprising a prioritization unit for assigning priority to the first findings information, wherein the generation unit is configured to generate the image interpretation report including the second findings information in an order corresponding to the priority.

[0111] (Note 4) The information processing apparatus according to Note 3, wherein the assigning unit is configured to assign the priority based on the words included in the first findings information.

[0112] (Note 5) The information processing apparatus according to Note 3 or 4, wherein the acquisition unit is configured to acquire further inspection information indicating the purpose of the inspection, and the assignment unit is configured to assign the priority based on the comparison result between the inspection information and the first finding information.

[0113] (Note 6) The information processing device according to any one of Notes 1 to 5, wherein the generation unit is configured to generate a first image interpretation report including a detailed explanation of the pathological condition or a second image interpretation report including a simplified explanation of the pathological condition, depending on the level of medical knowledge indicated by the information about the subject.

[0114] (Note 7) The information processing device according to Note 6, wherein the information processing device is capable of communicating via a network with a second information processing device equipped with a trained language model, and the generation unit is configured to obtain the second observation information output from the trained language model by transmitting the first observation information to the second information processing device.

[0115] (Note 8) The information processing apparatus according to Note 7, wherein the generation unit is configured to transmit predetermined information included in the first findings information to the second information processing apparatus while concealing it.

[0116] (Note 9) The information processing apparatus according to any one of Notes 1 to 8, further comprising: a reception unit configured to receive questions from the subject; and a response unit configured to output answers to the questions.

[0117] (Note 10) The information processing device according to Note 9, wherein the answer unit is configured to output the answer generated based on a trained language model.

[0118] (Appendix 11) The information processing apparatus according to Appendix 9 or 10, further comprising a suggestion unit configured to suggest the questions based on the information relating to the subject.

[0119] (Note 12) An information processing system in which a terminal device and an information processing device can communicate via a network, wherein the information processing device comprises: an acquisition unit configured to acquire first finding information based on a medical image; a generation unit configured to generate the image interpretation report including second finding information obtained by editing the first finding information based on information about the subject of the image interpretation report; and a presentation unit configured to transmit the image interpretation report to the terminal device.

[0120] (Note 13) An information processing method in which a computer performs the following steps: a step of acquiring first findings information based on a medical image; a step of generating a reading report that includes second findings information, which is an edited version of the first findings information, based on information about the subject of the reading report; and a step of presenting the reading report to the subject.

[0121] (Note 14) A program for causing a computer to perform the following steps: a procedure for acquiring first findings information based on medical images; a procedure for generating a reading report that includes second findings information, which is an edited version of the first findings information, based on information indicating the subject of the reading report; and a procedure for presenting the reading report to the subject.

[0122] This application claims priority to Japanese Patent Application No. 2025-050123, filed with the Japan Patent Office on March 25, 2025, which is incorporated herein by reference to its entire contents.

[0123] 10: Reporting device 20: Terminal device 101: Model storage unit 102: Report storage unit 110: Input unit 120: Detection unit 130: Acquisition unit 140: Assignment unit 150: Generation unit 160: Presentation unit 170: Proposal unit 180: Reception unit 190: Response unit 1000: Medical support system

Claims

1. An information processing device comprising: an acquisition unit configured to acquire first finding information based on a medical image; a generation unit configured to generate a reading report that includes second finding information obtained by editing the first finding information based on information about the subject of the reading report; and a presentation unit configured to present the reading report to the subject.

2. The information processing apparatus according to claim 1, further comprising a detection unit configured to detect the first finding information based on the medical image.

3. The information processing apparatus according to claim 1 or 2, further comprising a assigning unit for assigning priority to the first findings information, wherein the generation unit is configured to generate the image interpretation report including the second findings information in an order corresponding to the priority.

4. The information processing apparatus according to claim 3, wherein the assigning unit is configured to assign the priority based on the words included in the first findings information.

5. The information processing apparatus according to claim 3 or 4, wherein the acquisition unit is configured to further acquire inspection information indicating the purpose of the inspection, and the assignment unit is configured to assign the priority based on the comparison result between the inspection information and the first finding information.

6. The information processing apparatus according to any one of claims 1 to 5, wherein the generation unit is configured to generate a first image interpretation report including a detailed explanation of the pathological condition or a second image interpretation report including a simplified explanation of the pathological condition, depending on the level of medical knowledge indicated by the information about the subject.

7. The information processing device according to claim 6, wherein the information processing device is capable of communicating via a network with a second information processing device equipped with a trained language model, and the generation unit is configured to obtain the second observation information output from the trained language model by transmitting the first observation information to the second information processing device.

8. The information processing apparatus according to claim 7, wherein the generation unit is configured to transmit predetermined information included in the first findings information to the second information processing apparatus while concealing it.

9. The information processing apparatus according to claim 8, wherein the generation unit is configured to edit or delete information included in the first findings information or the generation instructions to be input to the language model, based on the country where the subject is located or the laws and regulations concerning the protection of personal information or medical information applicable to the subject, and then transmit it to the second information processing apparatus.

10. The information processing apparatus according to claim 9, wherein the laws and regulations include the General Data Protection Regulation in the European Union or the Health Insurance Portability and Accountability Act in the United States, and the generating unit is configured to edit or delete personal identification information including name, date of birth, address, location information, medical record number, facial image, or biometric identification information.

11. The information processing apparatus according to any one of claims 6 to 10, wherein the generation unit is configured to determine the attributes or level of medical knowledge of the subject based on the content of the questions asked by the subject, the input text, or the dialogue history, and to change the expression format or level of detail of the image interpretation report according to the result of the determination.

12. The information processing apparatus according to claim 11, wherein the generation unit is configured to determine the attributes or medical knowledge level of the subject based on at least one of the frequency of use of medical terms, the specialization of the question, or past dialogue history.

13. The information processing apparatus according to any one of claims 8 to 10, wherein the generation unit is configured to anonymize or abstract information that can identify the patient before transmitting the patient, who is the subject of the medical image, to the second information processing apparatus.

14. The information processing apparatus according to claim 13, wherein the generation unit is configured to replace the name of a disease with a higher-level disease category if the patient information includes the name of a disease, and the disease is a rare disease or a genetic disease, and the patient can be identified from the name of the disease.

15. The information processing apparatus according to any one of claims 1 to 14, further comprising: a reception unit configured to receive questions from the subject; and a response unit configured to output answers to the questions.

16. The information processing apparatus according to claim 15, wherein the response unit is configured to output the response generated based on a trained language model.

17. The information processing apparatus according to claim 15 or 16, further comprising a suggestion unit configured to suggest the question based on information relating to the subject.

18. An information processing system in which a terminal device and an information processing device can communicate via a network, wherein the information processing device comprises: an acquisition unit configured to acquire first finding information based on a medical image; a generation unit configured to generate the image interpretation report including second finding information obtained by editing the first finding information based on information about the subject of the image interpretation report; and a presentation unit configured to transmit the image interpretation report to the terminal device.

19. An information processing method comprising: a procedure for a computer to acquire first findings information based on a medical image; a procedure for generating a medical reading report that includes second findings information, which is an edited version of the first findings information, based on information about the subject of the medical reading report; and a procedure for presenting the medical reading report to the subject.

20. A program for causing a computer to perform the following steps: a procedure for acquiring first findings information based on medical images; a procedure for generating a reading report that includes second findings information, which is an edited version of the first findings information, based on information indicating the subject of the reading report; and a procedure for presenting the reading report to the subject.

Images