Content question-answering method driven by clinical reasoning step, device and system

Abstract

Provided in the present application are a content question-answering method driven by clinical reasoning steps, a device and a system. In the method, an enquiry intent of an ophthalmic problem is analyzed to generate a step sequence consistent with a real ophthalmic clinical analysis process, and, on the basis of function description information of each analysis step in the step sequence, corresponding model tools are matched and called from a preset tool library, so as to perform staged processing on multi-modal ophthalmic images. During a reasoning process, on the basis of assessment results of reasoning results of the analysis steps in terms of confidence, integrity and consistency, the step sequence is dynamically adjusted under clinical semantic constraints, so as to realize layered analysis, verification and interpretation of ophthalmic images. By explicitly modeling the multimodal medical analysis process as a constrainable and reconfigurable clinical step path, the present application reduces the accumulation of uncertainty in end-to-end reasoning, and improves the performance of content question-answering results in terms of accuracy, interpretability and clinical application reliability.

Description

Content-based question-answering methods, devices, and systems driven by clinical reasoning steps

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese patent application filed on January 2, 2025, application number 2025100053045, entitled "Content Question Answering Method and Graphic Question Answering Device Based on Large Language Model Intelligent Agent", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the fields of artificial intelligence and smart healthcare technology, and in particular to a content question-answering method driven by clinical reasoning steps, a multimodal ophthalmology content question-answering device, a multimodal ophthalmology content question-answering system, a computer-readable storage medium, and a computer program product. Background Technology

[0004] Currently, ophthalmic image interpretation solutions based on large language models mostly adopt an end-to-end "black box" architecture. This approach, when processing complex multimodal ophthalmic data (such as combining CFP, OCT, and FFA images), often suffers from a lack of explicit medical logic constraints, easily leading to "inference illusions" that do not conform to clinical diagnostic norms. Furthermore, existing models struggle to autonomously adjust their paths during the reasoning process based on the quality or conflicts of intermediate results, resulting in interpretability and accuracy of question-and-answer results that fail to meet the safety requirements of clinical practice. Summary of the Invention

[0005] This invention provides a content-based question-answering method, a multimodal ophthalmology content-based question-answering device, a multimodal ophthalmology content-based question-answering system, a computer-readable storage medium, and a computer program product, all driven by clinical reasoning steps. These improvements enhance the accuracy, interpretability, and clinical safety of multimodal ophthalmology data analysis. By introducing a dynamic path control mechanism constrained by clinical semantics, the problem of uncontrollable medical AI reasoning is addressed.

[0006] In a first aspect, embodiments of this application provide a content-based question-answering method driven by clinical reasoning steps, the method comprising:

[0007] Acquire ophthalmic problems for the target group and an ophthalmic image set containing at least one imaging modality;

[0008] The system analyzes the inquiry intent of ophthalmological questions and generates a target step sequence based on the inquiry intent. The target step sequence is explicitly modeled as a reasoning path structure constrained by clinical semantics, consisting of multiple analysis steps. Each analysis step in the target step sequence is associated with a clinical semantic tag, which is used to limit the medical dependencies and execution order between analysis steps.

[0009] Based on the functional description information corresponding to multiple model tools in the preset tool library, the target model tool corresponding to each analysis step in the target step sequence is determined. The model tool includes at least one or more of image analysis models, structured data analysis models, and text analysis models. The model tool is bound to the analysis steps in the target step sequence through a functional semantic mapping relationship, and is used to process ophthalmic images and related data of different modalities.

[0010] The corresponding target model tool is called to process the ophthalmic image set according to the execution order of the target step sequence, and the inference results of each analysis step are obtained.

[0011] Based on the inquiry intent and the reasoning results of each analysis step, the system generates a response to the ophthalmological question and outputs the response on the interactive interface. The response includes images of lesions in the target object's eye, the condition of the lesions and their explanations, as well as the path explanation information of the target step sequence and its dynamic adjustment process corresponding to the final analysis conclusion.

[0012] In some embodiments, during the execution of each analysis step in the target step sequence, a step-level state assessment is performed on the reasoning results of each analysis step. The step-level state assessment includes at least a confidence assessment, a consistency assessment, and a completeness assessment.

[0013] If the inference results of each analysis step meet the preset clinical semantic constraints, and the step-level state evaluation result of the inference result of any analysis step meets the preset path adjustment trigger condition, a dynamic adjustment operation is performed on the target step sequence. The clinical semantic constraints are used to restrict the adjustment methods and order between analysis steps with different clinical semantic labels, ensuring from a mechanism level that the inference path conforms to the logic of real clinical decision-making.

[0014] When a conflict is detected between intermediate inference results of different analysis steps at the medical semantic level, or when the confidence or completeness of the inference results of key analysis steps is lower than a preset threshold, the system triggers the inference path control mechanism and performs at least one adjustment operation on the target step sequence according to the step adjustment strategy corresponding to the triggering condition. This includes: inserting analysis steps to supplement or verify the current inference results, replacing analysis steps that do not match the current clinical analysis goal, or reordering the execution order of analysis steps in a restricted manner under the premise of meeting clinical semantic constraints, thereby ensuring that the adjusted target step sequence meets the preset requirements in terms of clinical semantic consistency and inference completeness.

[0015] In some embodiments, clinical semantic labels include at least one of screening labels, preliminary judgment labels, confirming diagnosis labels, quantitative analysis labels, verification labels, and interpretation labels; medical dependencies include the requirement that subsequent diagnostic steps are contingent upon the completion of the corresponding screening or preliminary judgment steps, and interpretation steps are only allowed to be generated after the confirming diagnosis or quantitative analysis steps are completed.

[0016] In some embodiments, the method supports multiple model tool categories, including but not limited to classification and detection, segmentation and quantization, regression, generation, and retrieval enhancement. Each category of model tool is used for tasks such as modality recognition, ocular lateralization, image quality grading, basic lesion diagnosis, secondary lesion diagnosis, lesion annotation, lesion segmentation, anatomical structure segmentation, biomarker extraction, physiological parameter prediction, cross-site health risk prediction, report generation, text generation, cross-modal image generation, 3D reconstruction, video generation, ophthalmological knowledge retrieval, reasoning evidence retrieval, or consistency checking.

[0017] Secondly, embodiments of this application provide a multimodal ophthalmic content question-and-answer device, including: a processor, a memory, and an interactive interface. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the method as described in the first aspect and outputs response content through the interactive interface.

[0018] Thirdly, embodiments of this application provide a multimodal ophthalmology content question-answering system, including a step sequence generation module for generating a target step sequence constrained by clinical semantics; a model tool invocation module for matching and executing model tools in a preset tool library according to the target step sequence; a path adjustment control module for evaluating the reasoning result and dynamically adjusting the target step sequence when triggering conditions are met; and a content generation module for outputting response content containing explanation information of the reasoning path.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium, comprising: a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the content question-answering method based on a large language model intelligent agent as described in any embodiment of the first aspect.

[0020] Fifthly, embodiments of this application provide a computer program product that, when run on a multimodal ophthalmology content question-answering device, causes the multimodal ophthalmology content question-answering device to execute the content question-answering method based on a large language model intelligent agent as described in any of the embodiments of the first aspect above.

[0021] The advantages of the embodiments in this application compared with related technologies are:

[0022] To avoid clinical risks arising from model inference uncertainty or inappropriate path selection during multimodal question answering, a reasoning path control mechanism under clinical semantic constraints is employed to generate a sequence of target steps corresponding to the inquiry intent of ophthalmological questions. This ensures that each analysis step in the sequence is constrained by real clinical decision-making logic, realistically simulating the clinical analysis process of ophthalmological problems. Furthermore, each analysis step in the sequence is assigned a clinical semantic label, representing the medical dependencies and execution order between different analysis steps, thus limiting the logical relationships between each analysis step and improving the accuracy, interpretability, and clinical safety of intelligent analysis of multimodal ophthalmological images. Based on each analysis step in the target step sequence that accurately simulates the ophthalmological clinical process, the system automatically and accurately determines the appropriate target model tool from a pre-defined tool library for each analysis step. During the process of calling each target model tool according to the execution order of each analysis step, the inference results of multiple analysis steps are evaluated to ensure that the entire inference process consistently adheres to professional diagnostic paradigms, ultimately generating reliable, interpretable, and clinically compliant responses. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 is a flowchart illustrating a content question-and-answer method provided in an embodiment of this application.

[0025] Figure 2 is a flowchart illustrating another content question-and-answer method provided in an embodiment of this application.

[0026] Figure 3 is a flowchart illustrating a content question-and-answer method in an application scenario according to an embodiment of this application.

[0027] Figure 4 is a flowchart illustrating the content question-answering method in another application scenario of this application embodiment.

[0028] Figure 5 is a flowchart illustrating the content question-answering method in another application scenario of this application embodiment.

[0029] Figure 6 is a schematic diagram of the structure of a multimodal ophthalmology content question-and-answer system according to an embodiment of this application.

[0030] Figure 7 is a structural schematic diagram of another multimodal ophthalmic content question-and-answer device according to an embodiment of this application. Detailed Implementation

[0031] The technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0032] With the development of artificial intelligence (AI) technology, AI models, especially large language models and vision-language models, have been widely used in ophthalmology to assist in examining eye health, segmenting eye structures, generating reports, and providing interactive question-and-answer sessions. However, accurate assessment of eye health typically relies on the comprehensive analysis of ophthalmic images from multiple imaging modalities. For example, color fundus photography (CFP) primarily reflects the intuitive morphological characteristics of the retina, optical coherence tomography (OCT) is used to characterize the layered structure of the retina and choroid, and fundus fluorescein angiography (FFA) and indocyanine green angiography (ICGA) provide information on retinal and choroidal blood flow and function. The complementarity of different modalities at the structural and functional levels makes their joint analysis a crucial foundation for the diagnosis of complex eye diseases. However, the integration of multimodal ophthalmic images inherently relies on complex clinical reasoning processes, typically involving medical hypothesis testing, cross-modal evidence consistency assessment, and longitudinal follow-up information integration. Currently, ophthalmic image interpretation solutions based on large language models often employ an end-to-end "black box" architecture, lacking explicit medical logical constraints during the reasoning process. When dealing with complex, multimodal, and potentially conflicting ophthalmic data, these solutions are prone to deviating from clinical diagnostic and treatment procedures, generating "reasoning illusions" that contradict medical common sense or clinical norms. Furthermore, existing models struggle to autonomously adjust their reasoning paths based on the quality or conflict of intermediate results, lacking controllability and adaptability in the reasoning process. This results in results that fail to provide a transparent, traceable, and evidence-based argumentation process. In applications with high clinical safety requirements, these issues limit the accuracy and interpretability of the models, affecting physicians' understanding and trust in the model's output, and hindering their widespread application in real-world clinical settings.

[0033] To address the aforementioned issues, this application provides a content-based question-answering method, a multimodal ophthalmology content-based question-answering device, a multimodal ophthalmology content-based question-answering system, a computer-readable storage medium, and a computer program product driven by clinical reasoning steps. By parsing the intent of ophthalmological questions, a sequence of steps consistent with real ophthalmological clinical analysis processes is generated, accurately simulating the clinical workflow of ophthalmic images. This improves the accuracy and automation of multimodal ophthalmic image processing. Based on the functional description information of each analysis step in the step sequence, corresponding model tools are matched and invoked from a preset tool library to process multimodal ophthalmic images in stages. During the reasoning process, the step sequence is dynamically adjusted under clinical semantic constraints based on the evaluation results of the reasoning results of each analysis step in terms of confidence, completeness, and consistency. This includes step insertion, step replacement, or step reordering to achieve hierarchical analysis, verification, and interpretation of ophthalmic images. By explicitly modeling the multimodal medical analysis process as a constrained and reconfigurable clinical step path, this application reduces the accumulation of uncertainty in end-to-end reasoning and improves the performance of content question-answering results in terms of accuracy, interpretability, and clinical application reliability. Through innovative designs such as explicit clinical analysis path structure, clinical semantic tags, dynamic and controlled reasoning path adjustment mechanism, functional semantic mapping of multimodal model tools, step-level state assessment, and path interpretation information output, the intelligence level, clinical applicability, interpretability, and user trust of the multimodal ophthalmology content question-answering system are significantly improved.

[0034] Before introducing the specific embodiments of this application, it should be noted that the image acquisition process and feature extraction process involved in this application are performed with the user's knowledge and permission. That is, the image acquisition process and feature extraction comply with the requirements of laws and regulations and do not constitute acts that harm the public interest.

[0035] Figure 1 is a flowchart illustrating a content question-answering method driven by clinical reasoning steps according to an embodiment of this application. This method is applied to a multimodal ophthalmology content question-answering system (hereinafter referred to as the system). The system can be integrated into a multimodal ophthalmology content question-answering device. The system includes an inference engine and a preset tool library. The inference engine can be a large language model, and the preset tool library includes multiple model tools. The inference engine and the preset tool library can form an ophthalmology intelligent agent. The steps executed by the multimodal ophthalmology content question-answering system can also be executed by the multimodal ophthalmology content question-answering device, or steps executed by the multimodal ophthalmology content question-answering system or the multimodal ophthalmology content question-answering device through the inference engine in the ophthalmology intelligent agent. The multimodal ophthalmology content question-answering device can be a mobile phone, laptop, desktop computer, or tablet, etc. This embodiment of the application does not limit the specific type of multimodal ophthalmology content question-answering device. As shown in Figure 1, the method includes the following steps.

[0036] S101, acquire an ophthalmic image set containing at least one imaging modality and an ophthalmic problem for the target object.

[0037] The target subjects were any subject undergoing an ophthalmological examination.

[0038] At least one imaging modality may include at least one of CFP, OCT, FFA, ICGA, anterior segment optical coherence tomography (ASOCT), scanning laser ophthalmoscopy (SLO), computed tomography (CT), ultra-widefield imaging (UWF), optical coherence tomography angiography (OCTA), magnetic resonance imaging (MRI), or fundus autofluorescence (FAF). It is understood that the above are merely examples, and the embodiments of this application do not limit the specific imaging modality used in ophthalmology. Any image targeting the eye can be included in the ophthalmological image set. Furthermore, ophthalmological images can be two-dimensional or three-dimensional images, and can be single images, keyframe images extracted from medical videos, or image sets composed of multiple images.

[0039] When ophthalmological questions and images are input into the interactive interface of the multimodal ophthalmology content question-and-answer device, the text-image question-and-answer system retrieves the questions and images. The ophthalmological questions can be in the form of text, voice, images, or video. For example, a user could voice input "What disease does this fundus image show?", text input "Please assess the severity of diabetic retinopathy in the patient's left eye and generate a report", or image input "Is there macular edema based on the OCT image?" The multimodal ophthalmology content question-and-answer device can obtain the text content of the ophthalmological questions through natural language understanding, speech recognition, and / or image recognition.

[0040] The ophthalmic image set can be obtained through the interface connecting the multimodal ophthalmic content question-and-answer device to medical imaging equipment (such as fundus cameras or OCT instruments), imported from a medical image archiving and communication system, or directly uploaded by the user. Furthermore, the ophthalmic images in the image set can be images extracted from ophthalmic medical videos by the multimodal ophthalmic content question-and-answer device, single images directly input to the multimodal ophthalmic content question-and-answer device, images generated by the multimodal ophthalmic content question-and-answer device based on text content, or images synthesized from multiple images. This application embodiment does not limit the method of acquiring ophthalmic images.

[0041] S102, analyze the inquiry intent of ophthalmological questions, and generate a sequence of target steps based on the inquiry intent.

[0042] In this context, the query intent is used to characterize the core analytical goals or types of information that a user hopes to achieve when raising the ophthalmological question, such as disease category determination, lesion localization and labeling, biomarker extraction, risk assessment, or result interpretation.

[0043] The system can use its inference engine to perform natural language processing operations such as semantic understanding and contextual analysis on ophthalmology questions to understand their semantic content and thus obtain the question intent. For example, if the ophthalmology question is "What is the diagnosis?", the analyzed question intent is "Determine the disease category." Another example is "Count the drusen in this image and measure their size," which reveals the question intent is "Extract specific biomarkers." Yet another example is "If this is diabetic retinopathy, generate the corresponding FFA image and label it," which indicates the question intent is "Cross-modal diagnosis and visualization." The system can map the query intent to a preset clinical analysis scenario, then decompose it into a series of ordered analysis steps to obtain the target step sequence.

[0044] The target step sequence is an explicitly modeled clinical analysis path structure, consisting of multiple analysis steps. Each analysis step indicates an operational instruction and is associated with a corresponding clinical semantic tag. The clinical semantic tag is used to characterize the role of the analysis step in the real ophthalmological clinical workflow and to define the medical dependencies and execution order between different analysis steps.

[0045] In one implementation, clinical semantic labels may include at least one of screening labels, preliminary judgment labels, confirmatory diagnosis labels, quantitative analysis labels, verification labels, and interpretation labels. Medical dependencies include the requirement that subsequent diagnostic steps are contingent upon the completion of the corresponding screening or preliminary judgment step, and that interpretation steps are only allowed to be generated after the confirmatory diagnosis or quantitative analysis step is completed. Thus, each analytical step in the target step sequence can be assigned a clear clinical semantic label, and strict medical dependencies are stipulated, including the requirement that subsequent diagnoses must be based on screening completion, and that interpretations can only be generated after a confirmed diagnosis. This enforces the rigorous logic of the clinical pathway throughout the entire reasoning process corresponding to the target step sequence generated based on the query intent. This constraint ensures that each step of reasoning is based on correct medical premises, thus limiting dynamic path adjustments to semantically permissible limits and preventing non-compliant jumps. Therefore, the system not only flexibly adapts to changing clinical scenarios but also ensures that the final generated interpretations and suggestions originate from sufficient and orderly diagnostic evidence, significantly improving the credibility, interpretability, and clinical applicability of the question-and-answer results.

[0046] For example, the target step sequence includes multiple analysis steps, which may include image modality recognition, image quality grading, diabetic retinopathy classification, lesion segmentation, and diagnostic report generation. Image modality recognition and image quality grading correspond to preliminary judgment labels, diabetic retinopathy classification corresponds to screening labels, lesion segmentation corresponds to quantitative analysis labels, and diagnostic report generation corresponds to interpretation labels. Therefore, the execution order of the analysis steps in the target step sequence, based on the aforementioned medical dependencies, can be determined as follows: Image modality recognition -> Image quality grading -> Diabetic retinopathy classification -> Lesion segmentation -> Diagnostic report generation.

[0047] In one implementation, as shown in Figure 2, the sequence of target steps for determining the ophthalmology image set based on the inquiry intent of the ophthalmology question may include the following steps S201 to S202.

[0048] S201, in the step rule base associated with the ophthalmology knowledge graph, search for the target step template sequence corresponding to the query intent.

[0049] The step rule base includes at least one step template for multiple clinical analysis scenarios, and the step module includes the corresponding intent, multiple analysis step templates, and clinical semantic tags corresponding to each analysis step template.

[0050] The target scenarios include at least one of the following: basic diagnostic scenarios, specialist diagnostic scenarios, medical education scenarios, associated risk assessment scenarios, or decision conflict resolution scenarios.

[0051] It is understood that the basic diagnostic scenario, specialty diagnostic scenario, medical education scenario, associated risk assessment scenario, and decision conflict resolution scenario are pre-defined clinical analysis scenarios based on the diagnostic objectives or needs in the actual ophthalmological examination process. Clinical diagnostic objectives or needs often represent different inquiry intentions, thus each type of clinical analysis scenario can correspond to at least one inquiry intention. Of course, other clinical analysis scenarios can be defined based on the diagnostic objectives or needs in the actual clinical process, as long as different inquiry intentions can be distinguished. For example, the actual clinical process scenarios can also be divided into hierarchical decision support scenarios, quantitative image analysis scenarios, medical education scenarios, decision conflict resolution scenarios, and cross-specialty longitudinal analysis scenarios. This application embodiment does not impose specific limitations. It should be noted that different clinical diagnostic objectives or needs often have a framework-like general analysis logic. Therefore, a step decomposition strategy corresponding to each clinical analysis scenario can be pre-set according to this general analysis logic to decompose a type of clinical analysis scenario into analysis steps corresponding to different processing operations.

[0052] Basic diagnostic scenarios (which can also be hierarchical decision support scenarios) are mainly used to represent the initial identification of ophthalmic images and the initial determination of ophthalmic disease categories, such as modality recognition of ophthalmic images and primary classification of ophthalmic disease categories.

[0053] Specialty diagnostic scenarios (which can also be quantitative image analysis scenarios) are mainly used to represent the quantification of ocular biomarkers from ophthalmic images and the further determination of ophthalmic disease categories, such as biomarker extraction and secondary classification of ophthalmic disease categories.

[0054] Medical education scenarios are primarily used to represent education and interaction related to ophthalmic disease categories or biomarkers, such as issuing reports or case images.

[0055] Associated risk assessment scenarios (which can also be cross-specialty longitudinal analysis scenarios) are mainly used to assess the classification of disease categories in other parts of the subject's body through ophthalmology disease categories. For example, combining ophthalmology disease categories to identify internal medicine disease categories or inferring lifestyle habits, etc.

[0056] The decision conflict resolution scenario is mainly used to represent the reclassification of ophthalmic disease categories when there are conflicts in the confirmation of ophthalmic disease categories. For example, conflict ablation or cross-validation of ophthalmic disease categories by combining multimodal ophthalmic images.

[0057] When analyzing the inquiry intent of ophthalmological questions through its inference engine, the system can categorize the inquiry intent into corresponding clinical analysis scenarios, thereby obtaining at least one clinical analysis scenario to which the inquiry intent belongs. All clinical analysis scenarios to which the inquiry intent belongs are considered as target scenarios. Then, within the target scenarios, a step template matching the inquiry intent is searched to obtain a target step template sequence.

[0058] For example, the inquiry intent of "determining the disease category" needs to go from basic screening to specialist confirmation. Its corresponding target scenarios are basic diagnosis scenario and specialist diagnosis scenario. The matching step template can include: [Step 1: Image modality recognition (preliminary judgment label)], [Step 2: Image quality grading (preliminary judgment label)], [Step 3: Disease category classification (screening label)], [Step 4: Lesion segmentation (quantitative analysis label)], [Step 5: Secondary classification of disease category (verification label)].

[0059] The inquiry intent of "extracting specific biomarkers" is generally only required for specialist diagnosis. Its corresponding target scenario is a specialist diagnosis scenario. The matching step template may include: [Step 1: Image modality recognition (preliminary judgment label)], [Step 2: Image quality grading (preliminary judgment label)], [Step 3: Disease category classification (screening label)], [Step 4: Specific biomarker segmentation (quantitative analysis label)], [Step 5: Specific biomarker labeling (quantitative analysis label)].

[0060] The query intent of "cross-modal diagnosis and visualization" generally requires modality recognition, specialist diagnosis and report generation. Its corresponding target scenarios are basic diagnosis scenario, specialist diagnosis scenario and medical education scenario. The matching step template can include: [Step 1: Image modality recognition (preliminary judgment label)], [Step 2: Image quality grading (preliminary judgment label)], [Step 3: Disease category classification (screening label)], [Step 4: Age-related macular degeneration grading (MetaPM grading) (quantitative analysis label)], [Step 5: Atrophy segmentation (quantitative analysis label)], [Step 6: 3D eye shape generation (interpretation label)].

[0061] S202, based on the target step decomposition strategy and clinical semantic constraints, the inquiry intent is decomposed into multiple analysis steps to obtain the target step sequence.

[0062] The system can instantiate entities or semantic content in the ophthalmological question of the identified query intent according to the step sequence template in the target step decomposition strategy, and determine whether the instantiated analysis steps meet the clinical semantic constraints. If they do, the target step sequence is obtained; if not, the instantiated analysis steps are adjusted to meet the clinical semantic constraints to obtain the target step sequence.

[0063] Clinical semantic constraints refer to the rules and limitations derived from medical expertise, clinical practice norms, and disease diagnosis and treatment logic that must be followed when decomposing the inquiry intent into specific analysis steps. Specifically, these constraints can be determined based on the inquiry intent. For example, when the inquiry intent is glaucoma assessment, the clinical semantic constraints could be to first perform "optic disc structure assessment" (such as cup-to-disc ratio measurement), followed by "visual function assessment" (such as visual field examination), which aligns with glaucoma assessment. When the inquiry intent does not explicitly specify a disease category, the clinical semantic constraints could be to prioritize the default analysis steps for the basic diagnostic scenario before performing the default analysis steps for the specialist diagnostic scenario. The default analysis steps for each clinical analysis scenario can be set independently; this application's embodiments do not impose specific limitations.

[0064] In another implementation, the system can directly generate the target step sequence based on the query intent using a trained inference engine.

[0065] S103, based on the functional description information of multiple model tools in the preset tool library, determine the target model tool corresponding to each analysis step in the target step sequence.

[0066] The pre-defined toolkit can specifically include a collection of pre-trained and validated specialized ophthalmic AI models. At least one of these models constitutes a model tool, each focusing on processing different modalities of ophthalmic images or text content. Thus, the model tools in the pre-defined toolkit can be categorized by input object type (image or text) or by model type, such as image analysis models, structured data analysis models, or text analysis models. Therefore, the target model tool can include at least one or more of these three types. The target model tool is bound to the analysis steps in the target step sequence through a functional semantic mapping relationship, and is used to process ophthalmic images and related data of different modalities.

[0067] It is understandable that the training dataset in the model tools of the training preset tool library may include multiple sample ophthalmic images in the modality corresponding to the model, as well as the annotation results on each sample ophthalmic image. The annotation results are the capabilities that the model needs to learn, that is, the processing operations (or processing tasks) corresponding to the model tools.

[0068] Each model tool in the preset tool library has associated functional description information, which indicates the applicable modalities, functions, and input / output formats of the model tool. For example, the functional description information of a model tool A may include "Input type - Image (an example of an input format), Applicable modality - ICGA (an example of an applicable modality), Function - Choroidal vessel segmentation (an example of a function), Type - Segmentation, Output - Choroidal vessel (an example of an output format)".

[0069] For example, Table 1 below shows some of the model tools in the preset tool library and their corresponding function descriptions. It should be understood that the model tools in Table 1 are only illustrative examples. Besides those listed in Table 1, other model tools with different numbering indicators may also exist in the preset tool library. For example, model tool 8: Input type - Image, Applicable modality - Multimodal, Function - Modality recognition, Type - Classification, Output - "AS-OCT, CFP, CT, Corneal topography, External eye photograph, FAF, FFA, ICGA, IOLMaster, Infrared, MRI, OCT, OCTA, Pentacam, RetCam, SLO, Slit lamp, Cornea..." Model tool 9: “Input type - image, applicable modality - multimodal, function - left and right eye side recognition, type - classification, output - left / right eye”; Model tool 10: “Input type - image, applicable modality - CFP, function - quality grading, type - classification, output - “gradable, small pupil artifact, defocus, incomplete macular area, poor image quality, incomplete optic disc, possible severe cataract”; Model tool 11: “Input type - image, applicable modality - CFP, function - video generation, type - 3D generation, output - FFA video”, etc., are not specifically limited in the embodiments of this application.

[0070] Table 1. Information on some model tools and step categories in the preset tool library.

[0071] In one implementation, the target model tool corresponding to each analysis step in the target step sequence is determined based on the functional description information of multiple model tools in a preset tool library. This includes: extracting processing instruction information for the analysis steps, which includes the processing object modality, processing operation, and input / output format; determining the target tool category based on the processing operation, where the tool category is selected from classification and detection, segmentation and quantification, regression, generation, and retrieval enhancement; and selecting model tools whose functional description information matches the processing instruction information from the target tool category as the target model tool.

[0072] The functional description information includes at least one of the applicable modalities of the model tool, the processing operation, or the output data modalities.

[0073] The model tools under the classification and detection category are used for at least one of the following: modality recognition, ocular lateral recognition, image quality grading, basic lesion diagnosis, or secondary lesion diagnosis.

[0074] The model tools under the Segmentation and Quantization category are used for at least one of the following: lesion annotation, lesion segmentation, anatomical structure segmentation, or biomarker extraction.

[0075] Regression-based model tools are used for at least one of the following: physiological parameter prediction or cross-site health risk prediction.

[0076] Generative modeling tools are used for at least one of report generation, text generation, cross-modal image generation, 3D reconstruction, or video generation.

[0077] The model tools under the retrieval enhancement category are used for at least one of ophthalmological knowledge retrieval, reasoning evidence retrieval, or consistency checking.

[0078] For example, referring to Table 1, model tools numbered 1 and 7 belong to the classification and detection category, model tools numbered 3 and 6 belong to the segmentation and quantization category, model tools numbered 4 and 5 belong to the generation category, model tool numbered 2 belongs to the regression category, and model tool numbered 5 also belongs to the retrieval enhancement category.

[0079] The system can extract processing instruction information for each analysis step in the target step sequence. This processing instruction information indicates the modality being processed, the processing operation, and the input / output format of the analysis step. The system can search a preset tool library for the tool category to which the processing operation in each analysis step belongs, obtain the target tool category, and quickly identify the target model tool that matches the processing instruction information in that analysis step from among the various model tools under the found target tool category. For example, if an analysis step is "identifying the modality of ophthalmic images," its indicated processing operation is modality recognition, which belongs to the model tool under the classification and detection category. The system can then identify a model tool specifically for modality recognition from this classification and detection category.

[0080] In one implementation, the system can determine the target model tool that matches the processing instruction information in each analysis step through the following matching strategies: 1. The semantic similarity between the processing operation of the analysis step and the processing operation of the model tool is greater than a preset threshold, and the one with the highest semantic similarity is selected. For example, the processing operation of "image quality scoring" matches the processing operation of "image quality grading"; 2. The processing object modality of the analysis step is included in the applicable modality of the model tool; 3. The input and output formats of the analysis step are compatible with the input and output formats of the model tool. The matching priority of these three strategies decreases in that order. For example, if an analysis step is "diabetic retinopathy (DR) grading", and the system matches a model tool B with "input type - image, applicable modality - CFP, function - DR grading, type - classification, output - R0, R1, R2, R3 or R4", then the system determines that model tool B is the target model tool for the "DR grading" analysis step.

[0081] In the above implementation, processing instruction information, including the processing object modality, processing operation, and input / output format, is extracted from the analysis steps. Based on the processing operations, the target tool category (e.g., classification / detection or segmentation / quantification) of the analysis step is determined, narrowing the search range between the processing information of the analysis step and the step type information of the model tool. This allows for the rapid retrieval of functional description information matching the processing instruction information in the analysis step within the reduced search range. The resulting target model tool ensures that each analysis step is matched with the most professional and suitable model tool, thereby improving the accuracy and efficiency of individual analysis steps. It also provides crucial assurance for the reliable execution of the entire target step sequence and the final generation of high-quality and professional medical question-and-answer content.

[0082] S104. The corresponding target model tool is called to process the ophthalmic image set according to the execution order of the target step sequence, and the inference results of each analysis step are obtained.

[0083] The system executes each analysis step sequentially according to the execution order of the analysis steps in the target step sequence. For each analysis step, the system sends a call request to the target model tool corresponding to that analysis step through a predefined API or function interface. The target model tool can then perform inference based on the inference result of the previous analysis step or at least one ophthalmic image from the ophthalmic image set to obtain the inference result of that analysis step. For example, in an analysis step of "CFP to FFA generation," the target model tool is model tool C. If the previous analysis step is image modality recognition, then model tool C can extract ophthalmic images with the imaging modality of CFP from the ophthalmic image set based on the various imaging modalities of the ophthalmic images corresponding to the image modality recognition analysis step. Based on these imaging modalities, model tool C can then generate an FFA image from the CFP image set. Therefore, the inference result of the "CFP to FFA generation" analysis step is the FFA image.

[0084] It is understandable that when there is a medical dependency between analytical steps, the reasoning results of the preceding analytical steps can serve as inputs or conditions for subsequent analytical steps. For example, the mask image generated by the "lesion segmentation" analytical step can be used by the model tool that generates the report as a visualization of "evidence" in the text description.

[0085] In one implementation, the system can further adjust the analysis steps in the already generated target step sequence during the execution of the analysis steps. Referring to Figure 1, before generating the response content for the ophthalmology question based on the inquiry intent and the reasoning results of each analysis step, the method also includes S105 to S106.

[0086] S105, During the execution of each analysis step in the target step sequence, perform a step-level state evaluation on the reasoning results of each analysis step.

[0087] Among them, the step-level status assessment includes at least confidence assessment, consistency assessment and integrity assessment.

[0088] During the execution of the target step sequence, for each analysis step, the system will detect the confidence, consistency and completeness of the inference results corresponding to that analysis step, thereby obtaining the step-level state evaluation results of each executed analysis step.

[0089] S106, if the reasoning results of each analysis step meet the preset clinical semantic constraints, and the step-level state evaluation result of the reasoning result of any analysis step meets the preset path adjustment triggering condition, perform dynamic adjustment operation on the target step sequence.

[0090] Clinical semantic constraints are used to limit the medical dependencies between analysis steps.

[0091] When the step-level state evaluation result of the inference result in any analysis step meets the preset path adjustment trigger condition, the system will perform dynamic adjustment operations on the target step sequence, provided that the inference results of each analysis step meet the preset clinical semantic constraints. Specifically, different situations that meet the path adjustment trigger condition correspond to different step adjustment strategies. For example, if the confidence of the inference result of the current analysis step is lower than a preset threshold, the system can call another alternative model tool to perform a secondary judgment on the same analysis step, or call another model tool to find visual evidence that supports or opposes the low-confidence diagnosis. Furthermore, if there is a medical semantic conflict between the inference results of multiple analysis steps, the system can add a conflict verification step to the target step sequence to arbitrate the conflicting inference results or redetermine the inference results. For example, the system can call a "generative" tool to synthesize an FFA image based on a CFP image to verify whether there is leakage in the blood vessels of the target object's eye; or call the RAG knowledge base to retrieve the identification points of NPDR with other similar disease categories to arbitrate the conflict. If the inference result fails the preset integrity verification rules, the system will add a secondary verification step to the target step sequence. For example, if a segmentation model tool only finds a microaneurysm lesion but cannot classify it into a specific disease category, the system can call a segmentation tool that can detect bleeding or exudation for completion. Or, if a classification model tool gives a basic diagnosis of "myopia," the system can call a more refined grading or quantitative tool for verification and refinement, upgrading the diagnosis from "presence / absence" to "degree." If the step-level state evaluation results of all executed analysis steps do not meet the path adjustment trigger condition, the system will continue to execute the analysis steps in the target step sequence until the inference result of the last analysis step in the target step sequence is obtained, proceeding to S107. If the condition is met, proceeding to S106.

[0092] In one implementation, the path adjustment triggering condition includes at least one of the following: the confidence level of the inference result of the current analysis step is lower than a preset threshold; there is a medical semantic conflict between the inference results of multiple analysis steps; the inference result fails to pass the preset integrity verification rule.

[0093] For example, for the inference result of each analysis step, the target model tool corresponding to that analysis step outputs the confidence level of the inference result and checks the completeness of the inference result through a preset completeness verification rule. The system can detect whether the confidence level is greater than or equal to a preset threshold (e.g., 0.88 or 0.95, etc., which are not specifically limited in this embodiment) and whether the inference result passes the preset completeness verification rule. The system will detect whether there is a conflict between the obtained inference results. For example, if the inference result of analysis step A1 is no lesion, and the inference result of analysis step A4 is the segmentation of a lesion in an ophthalmic image, then it is determined that there is a medical semantic conflict between the inference results of analysis step A1 and analysis step A4. As another example, a classification model tool gives a basic diagnostic result of "myopia" but does not indicate whether the myopia is caused by any reason, and it fails the preset completeness verification rule.

[0094] In one implementation, dynamic adjustment operations are performed on the target step sequence, including: inserting a new analysis step after the current analysis step to supplement or verify evidence; replacing analysis steps that do not match the current clinical analysis objective; and rearranging the execution order of the analysis steps in the target step sequence while satisfying clinical semantic constraints.

[0095] The system can pre-store a step adjustment strategy library, which contains step adjustment strategies, clinical semantic constraints, and a list of step templates corresponding to various path adjustment trigger conditions.

[0096] The system can search the step adjustment strategy library for a specific path adjustment trigger condition and identify, based on the step template list of that strategy, any missing, replacement, or rearranged analysis steps in the current target step sequence. It then inserts new analysis steps to supplement or validate evidence after the current analysis step, replaces analysis steps that don't match the current clinical analysis goal, or rearranges the execution order of unexecuted analysis steps in the target step sequence. The system also verifies whether the analysis steps in the target step sequence after the dynamic adjustment meet the clinical semantic constraints corresponding to the path adjustment trigger condition. If they do, it continues executing the steps that call the target model tool corresponding to the analysis step to process the ophthalmic image set until the inference results of each analysis step are obtained. If not, it analyzes the specific type of the violated clinical semantic constraints (e.g., missing dependencies, sequence conflicts, or unmet conditions) and attempts to apply predefined repair rules: for sequence conflicts, it automatically adjusts the order of related analysis steps within the limits allowed by the dependency graph. If a prerequisite is missing, a supplementary step to satisfy the prerequisite is automatically inserted before the current step (e.g., if an examination requires ruling out infection, a step to 'ask for infection symptoms' is inserted). If there is a mutual exclusion constraint conflict, one of the conflicting steps is automatically removed or disabled. If the clinical semantic constraints are still not met after repair, an alarm message is output to prompt the user to set the analysis steps themselves, thereby optimizing the target step sequence.

[0097] The above technical solution, by specifying the dynamic adjustment operation into three steps of insertion, replacement and rearrangement, and relying on real-time verification of clinical semantic constraints, realizes real-time, safe and personalized adjustment of the ophthalmic auxiliary diagnosis process. It can flexibly respond to clinical dynamics, optimize the diagnostic path, and strictly follow medical standards, significantly improving the adaptability, accuracy and safety of the auxiliary diagnosis system, and providing clinicians with efficient and reliable intelligent decision support.

[0098] S107. Based on the inquiry intent and the reasoning results of each analysis step, generate the response content for the ophthalmology question and output the response content on the interactive interface.

[0099] The response includes images of lesions in the target subject's eye, their condition and explanation, as well as the path explanation information for the target step sequence and its dynamic adjustment process corresponding to the final analysis conclusion. The final analysis conclusion indicates the health status of the target subject's eye as determined by the reasoning results based on the analysis steps. It may include specific disease categories and treatment recommendations. It can be understood that when the target subject's eye has no disease, the final analysis conclusion may indicate "eye health".

[0100] The system can receive the inference results of each target model tool's corresponding analysis step through the inference engine in the ophthalmic agent. These inference results can include diagnostic labels for disease categories, confidence levels, segmented labeled images, quantitative indicators (e.g., number of lesions or hemorrhage area), generated synthetic images, or report text fragments from multiple target model tools. Based on the query intent, the system can comprehensively process each received inference result to generate a response that includes images of lesions in the target object's eye, lesion conditions and explanations, and a path explanation of the target step sequence and its dynamic adjustment process corresponding to the final analysis conclusion.

[0101] In one implementation, a response to an ophthalmological question is generated based on the inquiry intent and the reasoning results of each analysis step. This includes: extracting the reasoning path trajectory of the target step sequence during execution based on the inquiry intent and the reasoning results of each analysis step; associating the reasoning results of each analysis step with the corresponding reasoning path trajectory to generate explanatory information including lesion identification conclusions, quantitative indicators, and reasoning basis paths.

[0102] During the execution of the target step sequence, the system can automatically capture inference path trajectory information based on the query intent and the inference results of each analysis step. This includes step execution logs, input / output snapshots, evidence evolution records, decision point records, and medical dependencies. The step execution logs represent the identifier, start and end times, and execution status (success / failure) of each analysis step. Input / output snapshots represent the input data and inference results of each analysis step. Evidence evolution records represent the evidence involved in analyzing ophthalmic images, such as evidence items and their confidence levels. Decision point records represent situations that trigger dynamic adjustments. The system organizes the inference path trajectory information into a time-series trajectory graph, where nodes are instances of analysis steps and edges represent data flow or logical order. The system can find the source step node that directly generated each inference result in the trajectory graph, as well as all step nodes supporting the inference result, forming one or more chains of evidence. Based on the trajectory graph and associated inference results, the inference engine generates response content including lesion identification conclusions, quantitative indicators, and explanatory information about the inference path.

[0103] In one implementation, to ensure the domain specificity of the ophthalmology agent, a pre-built knowledge base can be constructed for the ophthalmology agent. This pre-built knowledge base may include multiple ophthalmology textbooks, and the ophthalmology agent can search for ophthalmology textbooks in the pre-built knowledge base through retrieval-enhanced generation technology.

[0104] In one application scenario, as shown in Figure 3, a user inputs a CFP image and the ophthalmology question "What is the diagnosis?" into the ophthalmology agent of a computer (an example of a system). The computer, through the inference engine in the ophthalmology agent, determines that the query intent of the question is "disease category diagnosis". The ophthalmology agent, combined with the query intent of this clinical analysis scenario set, determines the target step template sequence as follows: 1. Identify the image modality (preliminary label judgment); 2. Based on the identification result of step 1, perform image quality grading operation for the corresponding modality (preliminary label judgment); 3. Identify the eye side (preliminary label judgment); 4. Preliminarily determine the disease category (screening label); 5. Secondarily confirm the disease category (confirm diagnosis label) (each step in this template sequence is the analysis step template). In step 1, the ophthalmology AI agent uses a modality recognition tool (an example of a target pattern tool) under the general practitioner class (an example of a target tool category) to identify the modality of the user-input ophthalmology image as CFP. In step 2, it uses an image quality grading tool to grade the CFP image. In step 3, it identifies the right eye (Oculus Dexter, OD) in the graded CFP image. In step 4, it uses a multi-disease category diagnosis model tool, i.e., tool number 7 in Table 1, to determine the disease category in the graded CFP image as diabetic retinopathy. In step 5, based on the diabetic retinopathy identified in step 4, the computer uses the ophthalmology AI agent to determine and use a diabetic retinopathy category classification tool (an example of a model tool under the classification and detection category) under the retinal expert class to determine the disease category in the graded CFP image as moderate non-proliferative diabetic retinopathy (NPDR), and uses a diabetic retinopathy lesion segmentation tool (an example of a model tool under the segmentation and quantification category) to segment the lesion region in the graded CFP image. Based on the reasoning results obtained in steps 1 to 5, the computer, through an ophthalmic intelligent agent, can generate and output a fixed-format response: Modality: Color fundus photography (CFP); Quality: Scored; Side: Right eye (OD); Diagnosis: Moderate non-proliferative diabetic retinopathy (NPDR) (one example of lesion status); Evidence (one example of interpretation): The general practitioner used diabetic retinopathy (DR) as the primary diagnosis (probability: 93.7%), a specific DR classification tool confirmed it as moderate NPDR (probability: 63.1%), lesion segmentation detected 70 microaneurysms (MA) and 23 hemorrhages (HE), and output a visualized graded CFP image (one example of lesion image).

[0105] For example, as shown in Figure 4, when a user inputs an ophthalmological question, "Statistically identify and label all lesions in this image," along with a CFP image, the computer analyzes the query intent as "lesion labeling," which falls under the clinical analysis scenario of quantitative image analysis. The sequence of steps determined by the computer is consistent with that in Figure 3, except that the response generated by the computer through the ophthalmological agent displays the CFP image and the lesion image segmented from the CFP image. Specifically, as shown in Figure 3, "Modal: Color Fundus Photography (CFP); Quality: Scored; Lateral: Left Eye (OS); Segmentation Result: Microaneurysm (MA) (n=57, Area=1.0-125.2 pixels), Hemorrhage (HE) (n=46, Area=5.0-405.2 pixels)... Visualized Image."

[0106] For example, as shown in Figure 5, when a user inputs "What are the examination results?" along with a CFP image, the computer analyzes the inquiry intent as "examination result determination." This falls under the clinical analysis scenario of decision conflict resolution. The computer-determined sequence of steps, in addition to the five steps shown in Figure 3, includes generating FFA images (interpreting labels), multimodal DR, and classification (quantitative analysis labels). After sequentially calling relevant model tools under each analysis step, the computer generates the following response: Examination method: Color fundus photography (CFP); Quality: Scored; Side: Left eye (OS); Diagnosis: Normal retina; Evidence: General practitioners have identified diabetic retinopathy (DR) as the primary diagnosis; Specific diabetic retinopathy classification tools show mild non-proliferative diabetic retinopathy (NPDR); No abnormalities were found in glaucoma lesion segmentation (microaneurysms, etc.); Cross-modal synthesis did not label diabetic retinopathy, and the conflict has been resolved. Recommendation: 1. Routine follow-up and subject education, and displaying CFP images and retinal images extracted from them.

[0107] In this embodiment, to avoid clinical risks caused by model inference uncertainty or inappropriate path selection during multimodal content question answering, a reasoning path control mechanism under clinical semantic constraints is used to generate a sequence of target steps corresponding to the inquiry intent of ophthalmological questions. This ensures that each analysis step in the sequence is constrained by real clinical decision-making logic, realistically simulating the clinical analysis process of ophthalmological questions. Furthermore, each analysis step in the sequence is assigned a clinical semantic label, representing the medical dependencies and execution order between different analysis steps, which limits the logical relationships between each analysis step, thereby improving the accuracy, interpretability, and clinical safety of intelligent analysis of multimodal ophthalmological images. Furthermore, it automatically determines the appropriate target model tool from a pre-set tool library for each analysis step in the target step sequence that accurately simulates the ophthalmic clinical process. During the process of calling each target model tool according to the execution order of the analysis steps, the inference results of multiple analysis steps are evaluated. When the inference results meet the path adjustment trigger conditions, it does not simply repeat the original analysis steps, but dynamically adjusts the analysis steps in the target step sequence within the limits allowed by clinical constraints. This ensures that the analysis steps can be dynamically adjusted according to the real-time processing results of multimodal ophthalmic images, better aligning with the clinical analysis process of ophthalmic problems. This dynamic adjustment operation for the target step sequence improves the flexibility of processing multimodal ophthalmic images, ensuring that even when facing complex and ever-changing ophthalmic problems, the entire inference process always adheres to professional diagnostic paradigms, ultimately generating reliable, interpretable, and clinically compliant responses. Providing users with visual responses for explanation, i.e., the basis of reasoning, enhances user trust.

[0108] Overall, through innovative designs such as explicit clinical analysis path structure, clinical semantic tags, dynamic and controlled reasoning path adjustment mechanism, functional semantic mapping of multimodal model tools, step-level state assessment and path explanation information output, the intelligence level, clinical applicability, interpretability and user trust of the multimodal ophthalmology content question answering system have been significantly improved.

[0109] The content-based question-answering method in this application has undergone rigorous and multi-level technical verification. These verification results fully demonstrate the effectiveness and superiority of the proposed method. Specifically, the first verification is tool-level verification, where each model tool in the pre-defined tool library is independently evaluated on large and diverse datasets. For example, the classification model tool achieved near- or above 99% accuracy or Area Under the Receiver Operating Characteristic Curve (AUC) values ​​on tasks such as image modality recognition, CFP quality grading, and eye-specific recognition. The segmentation tool also demonstrated Descein similarity on various lesions and structures. Coefficients (Dice) and Intersection over Union (IoU) also demonstrate reliable performance, ensuring that each invoked model tool outputs highly reliable inference results.

[0110] The second validation method was module-level ablation analysis. System performance was evaluated on a test set containing 13 CFP diseases and 8 OCT diseases by progressively increasing the number of tool categories (e.g., from 5 model tools to 53 model tools). Results showed that the disease category diagnostic accuracy of the content-based question answering method in this embodiment steadily improved from 69.71% in the basic configuration (5 model tools) to 80.79% in the full configuration (53 model tools). The largest performance jump (+9.11%) came from the addition of disease category-specific classification tools, demonstrating the value of specialized tools over general-purpose tools. The addition of segmentation tools brought further refinement. This indicates that the design of different categories of model tools in the content-based question answering method of this embodiment is correct and efficient.

[0111] The third type of verification is a system-level comparison. Compared with currently used large models, the ophthalmic agent in this application embodiment shows significant advantages in multiple tasks. In scenarios such as hierarchical diagnosis, quantitative analysis, medical education, and adversarial cases, the ophthalmic agent, benefiting from the planning and invocation of different model tools, can provide more accurate, specific, and visually supported answers, while effectively avoiding the hallucinations, vague descriptions, or incorrect responses to misleading questions that may occur with general large models.

[0112] The fourth validation method involved expert scoring, where experts blindly scored the end-to-end output of the ophthalmic agent across 200 cases. Scoring dimensions included accuracy, completeness, security, inference quality, and interpretability. Results showed that over 85% of the cases were rated "acceptable" or "excellent" across all metrics. The accuracy rate of tool invocation reached 93.7%. This quantitatively demonstrates the clinical applicability and reliability of the ophthalmic agent output in the embodiments of this application from a domain expert perspective.

[0113] The fourth type of validation is a human-machine collaborative reader study. A multi-center study involving multiple experts from multiple centers demonstrates that the ophthalmology agent in this application achieves an independent diagnostic accuracy of 93.3%, exceeding the diagnostic level without AI assistance (75.6%), and even comparable to the diagnostic accuracy of senior experienced professionals. When used as an auxiliary tool, the diagnostic accuracy of disease categories significantly improves to 94.1%, and the clinical report completeness also increases dramatically from 0.57 to 0.76. After using AI assistance, the average time for relevant personnel to complete case reading and report writing is reduced by more than 50%, and diagnostic confidence is significantly improved. The vast majority of users are satisfied with the system, finding its output interpretable, and are willing to use it in future clinical practice or teaching. These comprehensive evaluations, from algorithm performance and clinical utility to user acceptance, validate the strong vitality and huge application potential of the technical approach proposed in this application, which involves calling a deterministic set model tool based on ophthalmology agents for analytical steps.

[0114] The content-based question-answering method driven by clinical reasoning steps in this application provides an innovative ophthalmic content-based question-answering technology. This method achieves a highly automated, intelligent, and clinically workflow-compliant decision support platform by understanding user intent, dynamically planning analysis steps, precisely scheduling a vast toolkit, executing and optimizing the analysis process, and ultimately generating well-supported and interpretable text-and-image answers. It not only improves the efficiency and accuracy of ophthalmic image interpretation but, more importantly, establishes effective communication and trust between humans and machines through a transparent reasoning process and rich multimodal outputs. The technical framework of this application is highly scalable, and its core principles can be transferred to other medical imaging disciplines, providing a blueprint for the design of next-generation clinical artificial intelligence systems.

[0115] It should be noted that, indeed, those skilled in the art can modify this application through non-inventive effort, thereby making it applicable to other medical disciplines (such as radiology or dermatology). For example, sample medical images and medical images in a medical image set can be modified to images specific to a particular medical discipline.

[0116] Figure 6 is a schematic diagram of the structure of a multimodal ophthalmology content question-answering system provided in an embodiment of this application. The multimodal ophthalmology content question-answering system 500 includes:

[0117] The step sequence generation module 510 is used to generate a target step sequence constrained by clinical semantics.

[0118] The model tool invocation module 520 is used to match and execute model tools in a preset tool library according to the target step sequence;

[0119] The path adjustment control module 530 is used to evaluate the reasoning results and dynamically adjust the target step sequence when the triggering conditions are met.

[0120] The content generation module 540 is used to output response content that includes explanations of the reasoning path.

[0121] Figure 7 is a schematic diagram of the structure of another multimodal ophthalmology content question-answering device provided in an embodiment of this application. The multimodal ophthalmology content question-answering device 6 includes: at least one processor 60 (only one processor is shown in Figure 7), a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60. When the processor 60 executes the computer program 62, it implements the steps in any of the above-described image question-answering method embodiments, an interactive interface (not shown in Figure 7), generates the response content of the ophthalmology question based on the updated step sequence structure, and outputs it to the interactive interface.

[0122] The multimodal ophthalmic content question-and-answer device 6 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This multimodal ophthalmic content question-and-answer device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that Figure 7 is merely an example of the multimodal ophthalmic content question-and-answer device 6 and does not constitute a limitation on it. It may include more or fewer components than illustrated, or combine certain components, or use different components, such as input / output devices, network access devices, etc.

[0123] The processor 60 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0124] In some embodiments, the memory 61 may be an internal storage unit of the multimodal ophthalmic content question-and-answer device 6, such as a hard disk or memory of the multimodal ophthalmic content question-and-answer device 6. In other embodiments, the memory 61 may be an external storage device of the multimodal ophthalmic content question-and-answer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the multimodal ophthalmic content question-and-answer device 6. Further, the memory 61 may include both internal storage units and external storage devices of the multimodal ophthalmic content question-and-answer device 6. The memory 61 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0125] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0126] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0128] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0129] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0130] Furthermore, in the description of this application and the appended claims, the terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0131] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0133] In the embodiments provided in this application, it should be understood that the disclosed apparatus, computer equipment, and methods can be implemented in other ways. For example, the apparatus and computer equipment embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0134] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A content-based question-answering method driven by clinical reasoning steps, characterized in that, include: Acquire ophthalmic problems for the target group and an ophthalmic image set containing at least one imaging modality; The query intent of the ophthalmological question is analyzed, and a target step sequence is generated based on the query intent. The target step sequence is explicitly modeled as a reasoning path structure constrained by clinical semantics, consisting of multiple analysis steps. Each analysis step in the target step sequence is associated with a clinical semantic tag, which is used to limit the medical dependencies and execution order between analysis steps. Based on the functional description information corresponding to multiple model tools in the preset tool library, the target model tool corresponding to each analysis step in the target step sequence is determined. The target model tool includes at least one or more of image analysis models, structured data analysis models, and text analysis models. The target model tool is bound to the analysis steps in the target step sequence through a functional semantic mapping relationship and is used to process ophthalmic images and related data of different modalities. The corresponding target model tool is called to process the ophthalmic image set according to the execution order of the target step sequence, and the inference results of each analysis step are obtained. Based on the inquiry intent and the reasoning results of each analysis step, the response content for the ophthalmological question is generated and output on the interactive interface. The response content includes the lesion image of the target object's eye, the condition of the lesion and its explanation, as well as the path explanation information of the target step sequence and its dynamic adjustment process corresponding to the final analysis conclusion.

2. The method as described in claim 1, characterized in that, The method further includes: During the execution of each analysis step in the target step sequence, a step-level state evaluation is performed on the reasoning results of each analysis step. The step-level state evaluation includes at least a confidence evaluation, a consistency evaluation, and a completeness evaluation. If the reasoning results of each analysis step meet the preset clinical semantic constraints, and the step-level state evaluation result of the reasoning result of any analysis step meets the preset path adjustment triggering condition, then a dynamic adjustment operation is performed on the target step sequence.

3. The method according to claim 1 or 2, characterized in that, The clinical semantic tags include at least one of the following: screening tags, preliminary judgment tags, confirmed diagnosis tags, quantitative analysis tags, verification tags, and interpretation tags; the medical dependency relationship includes: the execution of subsequent diagnostic steps is premised on the completion of the corresponding screening or preliminary judgment steps, and interpretation steps are only allowed to be generated after the completion of the confirmed diagnosis or quantitative analysis steps.

4. The method according to claim 2, characterized in that, The path adjustment triggering condition includes at least one of the following: The confidence level of the inference result in the current analysis step is lower than the preset threshold; There are medical semantic conflicts in the inference results between multiple analysis steps; The reasoning result failed the preset integrity verification rules.

5. The method according to claim 2 or 4, characterized in that, The dynamic adjustment operation on the target step sequence includes: Insert a new analytical step after the current analytical step to supplement or verify evidence; Replace analytical steps that are not aligned with the current clinical analysis objectives; Under the premise of satisfying clinical semantic constraints, the execution order of the analysis steps of the target step sequence is rearranged.

6. The method according to claim 1, characterized in that, The step of determining the target model tool corresponding to each analysis step in the target step sequence based on the functional description information of multiple model tools in the preset tool library includes: Extract the processing instruction information of the analysis step, which includes the processing object modality, processing operation, and input / output format; The target tool category is determined based on the processing operation, and the tool category is selected from classification and detection, segmentation and quantification, regression, generation, and retrieval enhancement. The model tools whose functional description information matches the processing instruction information are selected from the target tool categories as target model tools.

7. The method according to claim 1, characterized in that, Based on the inquiry intent and the reasoning results of each analysis step, the system generates a response to the ophthalmological question, including: Based on the inquiry intent and the reasoning results of each analysis step, the reasoning path trajectory of the target step sequence during the execution process is extracted; The reasoning results of each analysis step are associated with the corresponding reasoning path trajectory to generate a response that includes lesion identification conclusions, quantitative indicators, and explanatory information based on the reasoning path.

8. A multimodal ophthalmic content question-and-answer device, characterized in that, include: processor; A memory having a computer program stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 7; The interactive interface generates the response content for the ophthalmological problem based on the updated step sequence structure and outputs it to the interactive interface.

9. A multimodal ophthalmology content question-and-answer system, characterized in that, include: The step sequence generation module is used to generate target step sequences that are constrained by clinical semantics. The model tool invocation module is used to match and execute model tools in a preset tool library according to the target step sequence; The path adjustment control module is used to evaluate the reasoning results and dynamically adjust the target step sequence when the triggering conditions are met; The content generation module is used to output response content that includes explanations of the reasoning path.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images