Image analysis method and apparatus, and computer device, storage medium and program product

Abstract

An image analysis method and apparatus, and a computer device, a storage medium and a program product. The method comprises: acquiring eye image samples in multiple modalities, processing the eye image samples in multiple modalities to obtain visible image regions, and on the basis of the visible image regions, training an initial image encoder in an initial eye image analysis model, so as to obtain a trained image encoder; on the basis of eye image samples in at least one of the multiple modalities and the trained image encoder, determining first feature image samples; on the basis of the first feature image samples, training an initial image analysis sub-model in the initial eye image analysis model, so as to obtain a trained image analysis sub-model; on the basis of the trained image encoder, performing feature extraction on an eye image to be subjected to identification, so as to obtain a first feature image; and inputting the first feature image into the trained image analysis sub-model, so as to obtain an analysis result of the eye image to be subjected to identification.

Description

Image analysis methods, apparatus, computer equipment, storage media and software products

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on February 5, 2025, with application number 202510131499.8, entitled "Image Analysis Method, Apparatus, Computer Equipment, Storage Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer vision technology, and in particular to an image analysis method, apparatus, computer equipment, storage medium, and program product. Background Technology

[0004] With the aging population and changing lifestyles, the incidence of vision diseases is on the rise globally. These diseases not only severely affect patients' visual function but can also trigger a series of complications, thereby reducing quality of life and even threatening life. With the advancement of artificial intelligence technology, using eye image analysis models to analyze eye images to assist doctors in diagnosing eye diseases has become an important research direction in this field.

[0005] Currently, eye image analysis models are typically used to analyze fixed, single-modality eye images to obtain analysis results. However, this eye image analysis model has limitations in its application. Summary of the Invention

[0006] Therefore, it is necessary to provide an image analysis method, apparatus, computer equipment, storage medium, and program product to address the aforementioned technical problems.

[0007] Firstly, this application provides an image analysis method. The method includes:

[0008] Multiple modal eye image samples are acquired, and the multiple modal eye image samples are processed to obtain visible image regions. Based on the visible image regions, the initial image encoder in the initial eye image analysis model is trained to obtain the trained image encoder.

[0009] The first feature image sample is determined based on at least one modality of eye image samples from the multiple modalities and the trained image encoder;

[0010] The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the trained image analysis sub-model.

[0011] Based on the trained image encoder, feature extraction is performed on the eye image to be identified to obtain a first feature image;

[0012] The first feature image is input into the trained image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0013] In one embodiment, the acquisition of eye image samples of the multiple modalities, processing of the eye image samples of the multiple modalities to obtain the visible image region, and training of the initial image encoder in the initial eye image analysis model based on the visible image region to obtain the trained image encoder includes:

[0014] Obtain eye image samples of the multiple modalities, process the eye image samples of the multiple modalities to obtain the visible image region, and map the visible image region of the eye image samples of each modality to the corresponding feature vector;

[0015] The feature vectors of eye image samples corresponding to different modalities are input into the initial image encoder to obtain the corresponding second feature image samples;

[0016] Each of the second feature image samples is input into the corresponding initial image decoder to obtain the reconstructed image sample corresponding to each of the second feature image samples;

[0017] Based on eye image samples of each modality and the corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the trained image encoder.

[0018] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0019] In one embodiment, the initial image analysis sub-model includes the initial large language model. The step of training the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain the trained image analysis sub-model includes: training the initial large language model based on the first feature image sample and the corresponding diagnostic report sample to obtain the target large language model, and using the target large language model as the trained image analysis sub-model.

[0020] In one embodiment, the initial image analysis sub-model includes the initial classifier, and the step of training the initial image analysis sub-model in the initial eye image analysis model based on the first feature image samples to obtain the trained image analysis sub-model includes: training the initial classifier based on the first feature image samples to obtain a target classifier, and using the target classifier as the trained image analysis sub-model.

[0021] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0022] In one embodiment, acquiring eye image samples of multiple modalities includes: acquiring eye images of multiple modalities using multimodal imaging techniques; performing quality control on the eye images of multiple modalities to exclude images with poor quality; and adjusting the remaining image size to 256×256 pixels using cubic spline interpolation to obtain eye image samples of multiple modalities.

[0023] In one embodiment, processing eye image samples of multiple modalities to obtain the visible image region includes: for each image of the eye image samples of multiple modalities, segmenting it into an image region of the same size of 16×16 pixels; randomly selecting an image region proportionally, and covering the selected image region to obtain an uncovered visible image region.

[0024] In one embodiment, each initial image decoder includes two Transformer blocks; the occluded image regions of the second feature image sample of the corresponding modality output by the initial image encoder and the eye image sample of the corresponding modality are respectively input into the two Transformer blocks of the corresponding initial image decoder.

[0025] In one embodiment, an initial image encoder in the initial eye image analysis model is trained based on eye image samples of each modality and the corresponding reconstructed image samples to obtain a trained image encoder. This includes: normalizing the pixel values ​​of each image region of the eye image samples of each modality according to the eye image samples of each modality and the corresponding reconstructed image samples; calculating the loss between the normalized pixel values ​​of the eye image samples of each modality and the pixel values ​​of the corresponding reconstructed image samples; and training the initial image encoder based on the calculated loss to obtain the trained image encoder.

[0026] In one embodiment, the initial classifier is a multilayer perceptron classifier.

[0027] Secondly, this application also provides an image analysis apparatus. The apparatus includes: a first training module, a first determining module, a second training module, and a feature extraction module.

[0028] The first training module is used to acquire eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and train the initial image encoder in the initial eye image analysis model based on the visible image regions to obtain the trained image encoder.

[0029] The first determining module is used to determine a first feature image sample based on at least one modality of eye image samples from the multiple modalities of eye image samples and the trained image encoder;

[0030] The second training module is used to train the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample, so as to obtain the trained image analysis sub-model.

[0031] The feature extraction module is used to extract features from the eye image to be identified based on the trained image encoder to obtain a first feature image;

[0032] The second determining module is used to input the first feature image into the trained image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0033] Thirdly, this application also provides a computer device, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of any of the above methods.

[0034] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0035] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0036] The aforementioned image analysis methods, apparatus, computer equipment, storage media, and program products acquire multimodal eye image samples, process these samples to obtain visible image regions, train an initial image encoder in an initial eye image analysis model based on these visible image regions, resulting in a trained image encoder, determine a first feature image sample based on at least one modality of eye image samples from the multimodal eye image samples and the trained image encoder, train an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample, resulting in a trained image analysis sub-model, extract features from the eye image to be identified based on the trained image encoder, obtain a first feature image, and input the first feature image into the trained image analysis sub-model to obtain the analysis result of the eye image to be identified. The image analysis method in this embodiment uses unlabeled multimodal eye image samples for pre-training, allowing the initial eye image analysis model to learn effective image feature representations. This improves the disease classification and generalization capabilities of the trained eye image analysis model on limited labeled data, reduces reliance on manual annotation, saves time and costs, and provides strong technical support for the automated diagnosis of ophthalmic diseases. Furthermore, pre-training on multimodal ocular image samples allows the initial ocular image analysis model to learn shared feature representations across modalities, thereby better capturing the correlations and specificities between different modalities. In disease classification tasks based on multimodal ophthalmic images, pre-training the image analysis model on multimodal ocular image samples can better extract and fuse disease features from different modalities, thus improving classification accuracy. The ocular image analysis model can support multiple modalities of ocular images as input, improving the accuracy of ocular image analysis results and overcoming the limitations of traditional ocular image analysis models in application. Attached Figure Description

[0037] Figure 1 is an internal structural diagram of a computer device provided in an embodiment of this application;

[0038] Figure 2 is a flowchart illustrating an image analysis method provided in an embodiment of this application;

[0039] Figure 3 is a flowchart illustrating an image encoder training method provided in an embodiment of this application;

[0040] Figure 4 is a schematic diagram of the architecture of an image encoder training method provided in an embodiment of this application;

[0041] Figure 5 is a schematic diagram of the architecture of an image analysis sub-model training method provided in an embodiment of this application;

[0042] Figure 6 is a schematic diagram of the architecture of another image analysis sub-model training method provided in the embodiments of this application;

[0043] Figure 7 is a schematic diagram of the architecture of another image analysis method provided in an embodiment of this application;

[0044] Figure 8 is a flowchart illustrating another image analysis method provided in an embodiment of this application;

[0045] Figure 9 is a structural block diagram of an image analysis device provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] With the aging population and changing lifestyles, the incidence of vision diseases is on the rise globally. These diseases not only severely affect patients' visual function but can also trigger a series of complications, thereby reducing quality of life and even threatening life. With the advancement of artificial intelligence technology, using eye image analysis models to analyze eye images to assist doctors in diagnosing eye diseases has become an important research direction in this field.

[0048] Currently, eye image analysis models are used to analyze fixed single-modality eye images to obtain analysis results. However, these eye image analysis models have limitations in their application.

[0049] The image analysis method provided in this application embodiment can be applied to the application environment shown in Figure 1. Figure 1 is an internal structure diagram of a computer device provided in this application embodiment. The computer device can be a server, and its internal structure diagram is as shown in Figure 1. The computer device includes a processor, a memory, and a communication interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image analysis method.

[0050] Those skilled in the art will understand that the structure shown in Figure 1 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0051] In one embodiment, as shown in FIG2, FIG2 is a flowchart of an image analysis method provided by an embodiment of the present application. The method can be applied to the computer device in FIG1 and includes the following steps S201 to S205.

[0052] S201: Obtain eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and train the initial image encoder in the initial eye image analysis model based on the visible image regions to obtain the trained image encoder.

[0053] Ophthalmic imaging modalities may include fundus imaging (CFP), optical coherence tomography (OCT), fundus angiography (FFA), ultra-wide-angle fundus imaging (UWF), and external eye imaging (EEP). Optionally, the ocular image samples of the various modalities in this application may include at least two of the following five different ophthalmic imaging modalities: fundus imaging samples, optical coherence tomography samples, fundus angiography samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0054] In one embodiment of this application, firstly, multimodal imaging technology can be used to acquire eye images of various modalities. Then, quality control is performed on the eye images of each modality to exclude images with poor quality, such as those containing artifacts, dark colors, or missing pixels. Finally, cubic spline interpolation is used to adjust the image size to 256×256 pixels to obtain eye image samples of various modalities. The eye image samples of different modalities can be stored in different folders.

[0055] In embodiments of this application, processing multiple modalities of eye image samples to obtain visible image regions includes: dividing each image of the multiple modalities of eye image samples into image regions of equal size (16×16 pixels); randomly selecting image regions proportionally and covering the selected image regions to obtain uncovered visible image regions. Specifically, for each image of the multiple modalities of eye image samples, it can be divided into image regions (image blocks) of equal size (16×16 pixels), then a certain proportion of image regions are randomly selected and covered, and the uncovered image regions of each modality of eye image sample are the visible image regions. For example, randomly selecting image regions proportionally can cover 85% of each image of an optical coherence tomography (OCT) imaging sample, and 75% of each image of a fundus imaging sample, fundus angiography imaging sample, ultra-wide-angle fundus imaging sample, and external eye image sample.

[0056] In one embodiment, the visible image regions of different modal image samples are mapped to different feature vectors through different embedding mapping layers. That is, the visible image regions obtained from eye image samples of each modality can be mapped to corresponding feature vectors. Then, each feature vector is input into the initial image encoder to obtain the second feature image sample corresponding to each feature vector. That is, the initial image encoder uses the visible region of the image to extract global features of the image. Then, each second feature image sample is input into the corresponding decoder. Each decoder reconstructs the corresponding occluded image region based on the information of each second feature image sample and the occluded image region of the corresponding eye image sample of each modality, and obtains the reconstructed image sample corresponding to the eye image sample of each modality. Based on the eye image samples of each modality and the corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is adjusted to obtain the trained image encoder.

[0057] S202, determine the first feature image sample based on at least one modality of eye image samples from multiple modalities and a trained image encoder.

[0058] For example, one or more modal eye image samples from multiple modalities can be input into a trained image encoder for feature extraction to obtain the first feature image sample output by the trained image encoder.

[0059] S203, train the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample, and obtain the trained image analysis sub-model.

[0060] In one embodiment of this application, the initial eye image analysis model includes an initial image analysis sub-model, which may include an initial classifier and / or an initial large language model.

[0061] In one embodiment, the initial image analysis sub-model includes an initial classifier, which can be trained based on a first feature image sample to obtain a target classifier, and the target classifier is used as the trained image analysis sub-model.

[0062] In one embodiment, the initial image analysis sub-model includes an initial large language model, which can be trained based on a first feature image sample and a diagnostic report sample corresponding to the first feature image sample to obtain a target large language model, and the target large language model is used as the trained image analysis sub-model.

[0063] S204, Based on the trained image encoder, feature extraction is performed on the eye image to be identified to obtain the first feature image.

[0064] In this embodiment, the eye image to be identified can be input into a trained image encoder for feature extraction to obtain the first feature image output by the trained image encoder.

[0065] S205, input the first feature image into the trained image analysis sub-model to obtain the analysis results of the eye image to be identified.

[0066] Figure 7 is a schematic diagram of the architecture of another image analysis method provided in this application embodiment. As shown in Figure 7, specifically, the image analysis method of this application firstly inputs the eye image to be identified into the embedding mapping layer. After being mapped into a feature vector, it is input into a trained image encoder for feature extraction, obtaining the first feature image output by the trained image encoder. Then, the first feature image is input into a trained image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0067] In one embodiment, the initial image analysis sub-model includes an initial classifier, and the trained image analysis sub-model includes a target classifier. The first feature image can be input to the target classifier to obtain the analysis result of the eye image to be identified, output by the target classifier. The analysis result of the eye image to be identified output by the target classifier includes the probability that the patient corresponding to the eye image has various diseases.

[0068] In one embodiment, the initial image analysis sub-model includes an initial large language model, and the trained image analysis sub-model includes a target large language model. The first feature image can be input into the target large language model to obtain the analysis result of the eye image to be identified output by the target large language model. The analysis result of the eye image to be identified output by the target large language model includes a diagnostic report corresponding to the eye image to be identified.

[0069] In the image analysis method embodiments of this application, multiple modalities of eye image samples are acquired, and these samples are processed to obtain visible image regions. Based on these visible image regions, an initial image encoder in an initial eye image analysis model is trained, resulting in a trained image encoder. A first feature image sample is determined based on at least one modality of eye image samples from the multiple modalities and the trained image encoder. An initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample, resulting in a trained image analysis sub-model. Features are extracted from the eye image to be identified based on the trained image encoder to obtain a first feature image. The first feature image is then input into the trained image analysis sub-model to obtain the analysis result of the eye image to be identified. The image analysis method in this application embodiment, through pre-training with unlabeled multimodal eye image samples, allows the initial eye image analysis model to learn effective image feature representations. This improves the disease classification and generalization capabilities of the trained eye image analysis model on limited labeled data, reduces reliance on manual annotation, saves time and costs, and provides strong technical support for the automated diagnosis of ophthalmic diseases. Furthermore, pre-training on multimodal ocular image samples allows the initial ocular image analysis model to learn shared feature representations across modalities, thereby better capturing the correlations and specificities between different modalities. In disease classification tasks based on multimodal ophthalmic images, pre-training the image analysis model on multimodal ocular image samples can better extract and fuse disease features from different modalities, thus improving classification accuracy. The ocular image analysis model can support multiple modalities of ocular images as input, improving the accuracy of ocular image analysis results and overcoming the limitations of traditional ocular image analysis models in application.

[0070] Figure 3 is a flowchart illustrating an image encoder training method according to an embodiment of this application. As shown in Figure 3, in one embodiment of this application, step S201 obtains the visible image region based on the acquired multimodal eye image samples, and trains the initial image encoder in the initial eye image analysis model based on the visible image region to obtain the trained image encoder. This may include the following steps S301 to S304.

[0071] S301, acquire eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and map the visible image regions of eye image samples of each modality to corresponding feature vectors.

[0072] In this embodiment, each image in the eye image samples of each modality can be divided into image blocks, i.e., image regions, of the same size (16×16 pixels). Then, a certain proportion of the image regions are randomly selected for masking. For example, 85% of the image region of each image in the optical coherence tomography (OCT) imaging sample can be masked, and 75% of the image blocks of each image in the fundus imaging sample, fundus angiography imaging sample, ultra-wide-angle fundus imaging sample, and external eye image sample can be masked. The unmasked image regions are the visible image regions.

[0073] For example, referring to Figure 4, which is a schematic diagram of the architecture of an image encoder training method provided in an embodiment of this application. As shown in Figure 4, after segmentation and occlusion processing of eye image samples of various modalities, visible image regions corresponding to different modalities of eye image samples are obtained. Then, the visible image regions corresponding to different modalities of eye image samples are mapped to feature vectors corresponding to different modalities of eye image samples through different embedding mapping layers.

[0074] S302, input the feature vectors of eye image samples corresponding to different modalities into the initial image encoder to obtain the corresponding second feature image samples.

[0075] In one embodiment, as shown in Figure 4, the feature vectors of eye image samples corresponding to different modalities can be arranged together and used as input to the initial image encoder to obtain the high-dimensional feature representations corresponding to the feature vectors of each modality output by the initial image encoder, i.e., the second feature image samples. The second feature image samples represent the high-dimensional features of the visible image regions of eye image samples of different modalities.

[0076] In one embodiment of this application, the initial eye image analysis model includes an initial image encoder. Optionally, the initial image encoder can be a Vision Transformer Large version, consisting of 24 Transformer blocks.

[0077] S303, each second feature image sample is input into the corresponding initial image decoder to obtain the reconstructed image sample corresponding to each second feature image sample.

[0078] In one embodiment of this application, the initial eye image analysis model includes at least a plurality of initial image decoders corresponding one-to-one with multiple modalities. In one embodiment, each initial image decoder can receive the second feature image sample of the corresponding modality output by the initial image encoder and the occluded image region of the eye image sample of the corresponding modality, thereby reconstructing the pixel values ​​of the occluded image region of the corresponding modality and obtaining the reconstructed image sample corresponding to each second feature image sample.

[0079] In one embodiment of this application, each initial image decoder may contain two Transformer blocks. The second feature image sample of the corresponding modality output by the initial image encoder and the occluded image region of the eye image sample of the corresponding modality are respectively input into the two Transformer blocks. Each initial image decoder is configured to reconstruct the occluded image region of the eye image sample of different modalities.

[0080] S304. Based on eye image samples of each modality and the corresponding reconstructed image samples, train the initial image encoder in the initial eye image analysis model to obtain the trained image encoder.

[0081] Optionally, in one embodiment of this application, the pixel values ​​of each image region of the eye image samples of each modality can be normalized according to the eye image samples of each modality and the corresponding reconstructed image samples. Then, the loss between the pixel values ​​of the normalized eye image samples and the pixel values ​​of the corresponding reconstructed image samples can be calculated. The initial image encoder can be trained according to the calculated loss to obtain the trained image encoder.

[0082] In this embodiment, eye image samples of multiple modalities are acquired, processed to obtain visible image regions, and the visible image regions of each modal eye image sample are mapped to corresponding feature vectors. The feature vectors corresponding to different modal eye image samples are input into an initial image encoder to obtain corresponding second feature image samples. Each second feature image sample is input into a corresponding initial image decoder to obtain reconstructed image samples corresponding to each second feature image sample. Based on the eye image samples of each modality and the corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain a trained image encoder. This enables the initial image encoder to be trained based on unlabeled multimodal eye image samples, allowing the initial image encoder to learn shared feature representations across modalities, thereby better capturing the correlation and specificity between different modalities and improving the accuracy of eye image analysis results.

[0083] In one embodiment of this application, the initial image analysis sub-model in the initial eye image analysis model includes an initial large language model. Step S203 trains the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain the trained image analysis sub-model. Specifically, the initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample to obtain a target large language model, and the target large language model is used as the trained image analysis sub-model.

[0084] This application does not impose specific limitations on the initial large language model. Those skilled in the art can determine it according to the actual situation. For example, the initial large language model can be an LLaMA large language model.

[0085] Figure 5 is a schematic diagram of the architecture of an image analysis sub-model training method provided in an embodiment of this application. As shown in Figure 5, in one embodiment of this application, eye image samples of at least one modality from multiple modalities can be segmented into image regions of the same size, and then input into an embedding mapping layer and a trained image encoder to obtain an output first feature image sample. The first feature image sample represents the high-dimensional features of the image region of the corresponding modality of the eye image sample. The first feature image sample is used as the input of the initial large language model. In addition, diagnostic report samples corresponding to each eye image sample (i.e., diagnostic report samples corresponding to the first feature image sample) are obtained. After feature mapping, the diagnostic report samples are input into the initial large language model. Based on the first feature image sample and the corresponding diagnostic report sample, the initial large language model is jointly pre-trained for vision and language to obtain a target large language model, which can be used as a trained image analysis sub-model. In the embodiment of this application, the eye image samples of at least one modality from multiple modalities are segmented into 16×16 pixel image regions. In embodiments of this application, the diagnostic report used to train the initial large language model may include one or more of the following from routine ophthalmological diagnosis, ophthalmological advice and medical orders, and image reports including image descriptions, image impressions, medical records and health records.

[0086] In one embodiment, during the pre-training phase of the initial large language model, the training period can be set to 5 epochs, the batch size to 16, and the initial learning rate to 2×10⁻⁶. -3 The learning rate is adjusted during training using a cosine decay adjustment strategy. In the instruction fine-tuning phase, training also lasts for 5 epochs with a batch size of 8, and the initial learning rate for this phase is 2×10⁻⁶. -5 Similarly, the learning rate is adjusted using a cosine decay strategy.

[0087] In the embodiments of this application, an initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample to obtain a target large language model. This target large language model is then used as a trained image analysis sub-model, which can learn the complex relationship between eye images of multiple modalities and diagnostic report text. This allows the trained image analysis sub-model to generate corresponding diagnostic reports or visual question-and-answer results (answering diagnosis-related questions based on the input image) based on the eye image to be identified, thereby providing doctors with more reliable auxiliary diagnostic suggestions.

[0088] In one embodiment of this application, the initial image analysis sub-model includes an initial classifier. Step S203 trains the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain the trained image analysis sub-model. Specifically, the initial classifier is trained based on the first feature image sample to obtain a target classifier, and the target classifier is used as the trained image analysis sub-model.

[0089] Alternatively, the initial classifier can be, for example, a multilayer perceptron classifier.

[0090] Figure 6 is a schematic diagram of another image analysis sub-model training method provided in an embodiment of this application. As shown in Figure 6, in one embodiment of this application, eye image samples of at least one modality from multiple modalities can be segmented into image regions of the same size and then input into an embedding mapping layer and a trained image encoder to obtain an output first feature image sample. The first feature image sample represents the high-dimensional features of the image region of the corresponding modality of the eye image sample. The first feature image sample is input into an initial classifier to train the initial classifier to obtain a target classifier, and the target classifier is used as the trained image analysis sub-model.

[0091] In one embodiment, the eye image samples corresponding to the modality used for training the input image analysis model are adjusted to a size of 224×224 pixels. The initial classifier undergoes a total training cycle of 50 epochs, with a batch size of 16. In the first 10 epochs, the learning rate is gradually increased from 0 to 5×10⁻⁶. -4 Subsequently, a cosine decay strategy was used to reduce the learning rate from 5×10⁻⁶ over the next 40 epochs. -4 Gradually reduce to 1×10 -6 After each epoch of training, the classifier is evaluated on the validation set, and the area under the receiver operating characteristic (AUC) is obtained. At the end of training, the model weights from the epoch with the highest AUC are saved as the final training result, yielding the target classifier. The target classifier is then evaluated on the test set.

[0092] In this embodiment, an initial classifier is trained based on the first feature image sample to obtain a target classifier. The target classifier is used as a trained image analysis sub-model, which can better extract and fuse disease features of different modal eye images based on the trained image analysis sub-model, thereby improving the accuracy of the analysis results.

[0093] In the embodiments of this application, step S201 acquires eye image samples of multiple modalities, including acquiring eye image samples of at least two modalities among fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0094] For example, the fundus imaging sample is an eye image acquired using fundus imaging (CFP) technology. Fundus imaging is a non-contact imaging technique using various clinically common fundus cameras, featuring autofocus and auto-exposure, non-mydriatic 45-degree angle color fundus photographs. The fundus imaging sample is obtained by cropping the black background from the color fundus photograph, using cubic spline interpolation to resize the image to 256×256 pixels, and discarding images of poor quality.

[0095] Optical coherence tomography (OCT) images are ocular images acquired using OCT technology. OCT is performed using a clinically common OCT scanner for non-mydriatic OCT examination, extracting 128 b-scan sequence images. The OCT sample is obtained by resizing the 128 b-scan sequence images to 256×256 pixels using cubic spline interpolation.

[0096] Fundus angiography (UWF) images are ocular images acquired using fundus angiography technology. UWF is a non-mydriatic, 200-degree high-resolution ultra-wide-angle fundus color image captured by clinical ophthalmologists and technicians during routine ophthalmological procedures using a clinically common ultra-wide-angle fundus camera. Fundus angiography images are obtained by adjusting the size of the ultra-wide-angle fundus color image to 256×256 pixels using cubic spline interpolation.

[0097] Ultra-wide-angle fundus imaging (FFA) samples are ocular images acquired using ultra-wide-angle fundus imaging technology. FFA uses a clinically common fluorescein angiography fundus camera, and is used by clinicians and specialists during routine ophthalmological examinations to capture fundus fluorescence images of patients. The FFA sample is obtained by resizing the acquired fundus fluorescence image to 256×256 pixels using cubic spline interpolation.

[0098] External eye image samples are ocular images acquired using external eye photography (EEP) imaging technology. External eye photography uses a common clinical camera, and professional technicians collect images of the external eye from patients during routine ophthalmological examinations. External eye image samples are obtained by resizing the acquired external eye images to 256×256 pixels using cubic spline interpolation, and by discarding images of poor quality.

[0099] Referring to Figure 7, which is a flowchart of another image analysis method provided in an embodiment of this application, the image analysis method includes the following steps: S701 to S708.

[0100] S701, acquire eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and map the visible image regions of eye image samples of each modality to corresponding feature vectors.

[0101] S702, input the feature vectors of eye image samples corresponding to different modalities into the initial image encoder to obtain the corresponding second feature image samples.

[0102] S703, each second feature image sample is input into the corresponding initial image decoder to obtain the reconstructed image sample corresponding to each second feature image sample.

[0103] S704, based on eye image samples of each modality and the corresponding reconstructed image samples, train the initial image encoder in the initial eye image analysis model to obtain the trained image encoder.

[0104] S705, determine the first feature image sample based on at least one modality of eye image samples from multiple modalities and a trained image encoder.

[0105] S706, train an initial large language model based on the first feature image sample and the corresponding diagnostic report sample to obtain a target large language model, and use the target large language model as a trained image analysis sub-model, and / or, train an initial classifier based on the first feature image sample to obtain a target classifier, and use the target classifier as a trained image analysis sub-model.

[0106] S707 extracts features from the eye image to be identified based on the trained image encoder to obtain the first feature image.

[0107] S708, input the first feature image into the trained image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0108] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0109] Based on the same inventive concept, this application also provides an image analysis apparatus for implementing the image analysis method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image analysis apparatus embodiments provided below can be found in the limitations of the image analysis method described above, and will not be repeated here.

[0110] In one embodiment, as shown in FIG8, FIG8 is a structural block diagram of an image analysis device provided in an embodiment of the present application. The device 800 includes: a first training module 801, a first determination module 802, a second training module 803, a feature extraction module 804, and a second determination module 805.

[0111] The first training module 801 is used to acquire eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and train the initial image encoder in the initial eye image analysis model based on the visible image regions to obtain the trained image encoder.

[0112] The first determining module 802 is used to determine a first feature image sample based on at least one modality of eye image samples from multiple modalities and a trained image encoder.

[0113] The second training module 803 is used to train the initial image analysis sub-model in the initial eye image analysis model based on the first feature image samples, so as to obtain the trained image analysis sub-model.

[0114] The feature extraction module 804 is used to extract features from the eye image to be identified based on the trained image encoder to obtain the first feature image.

[0115] The second determining module 805 is used to input the first feature image into the trained image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0116] In one embodiment, the first training module 801 includes: a mapping unit, a first determining unit, a second determining unit, and a first training unit.

[0117] The mapping unit is used to acquire eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and map the visible image regions of eye image samples of each modality into corresponding feature vectors.

[0118] The first determining unit is used to input the feature vectors of eye image samples corresponding to different modalities into the initial image encoder to obtain the corresponding second feature image samples.

[0119] The second determining unit is used to input each second feature image sample into the corresponding initial image decoder to obtain the reconstructed image sample corresponding to each second feature image sample.

[0120] The first training unit is used to train the initial image encoder in the initial eye image analysis model based on eye image samples of each modality and the corresponding reconstructed image samples, so as to obtain the trained image encoder.

[0121] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0122] In one embodiment, the initial image analysis sub-model includes an initial large language model, and the second training module 803 includes a second training unit for training the initial large language model based on the first feature image sample and the corresponding diagnostic report sample to obtain a target large language model, and using the target large language model as the trained image analysis sub-model.

[0123] In one embodiment, the initial image analysis sub-model includes an initial classifier, and the second training module 803 includes a third training unit for training the initial classifier based on the first feature image samples to obtain a target classifier, and using the target classifier as the trained image analysis sub-model.

[0124] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0125] Each module in the aforementioned image analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0126] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0127] Obtain eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and train the initial image encoder in the initial eye image analysis model based on the visible image regions to obtain the trained image encoder;

[0128] The first feature image sample is determined based on at least one modality of eye image samples from multiple modalities and a trained image encoder;

[0129] The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the trained image analysis sub-model;

[0130] Based on the trained image encoder, feature extraction is performed on the eye image to be identified to obtain the first feature image;

[0131] The first feature image is input into the trained image analysis sub-model to obtain the analysis results of the eye image to be identified.

[0132] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0133] Obtain eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and map the visible image regions of eye image samples of each modality to corresponding feature vectors;

[0134] The feature vectors of eye image samples corresponding to different modalities are input into the initial image encoder to obtain the corresponding second feature image samples;

[0135] Each second feature image sample is input into the corresponding initial image decoder to obtain the reconstructed image sample corresponding to each second feature image sample.

[0136] Based on eye image samples of each modality and corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the trained image encoder.

[0137] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0138] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0139] An initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample to obtain the target large language model, and this target large language model is used as the trained image analysis sub-model.

[0140] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0141] An initial classifier is trained based on the first feature image sample to obtain the target classifier, and this target classifier is used as the trained image analysis sub-model.

[0142] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0144] Obtain eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and train the initial image encoder in the initial eye image analysis model based on the visible image regions to obtain the trained image encoder;

[0145] The first feature image sample is determined based on at least one modality of eye image samples from multiple modalities and a trained image encoder;

[0146] The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the trained image analysis sub-model;

[0147] Based on the trained image encoder, feature extraction is performed on the eye image to be identified to obtain the first feature image;

[0148] The first feature image is input into the trained image analysis sub-model to obtain the analysis results of the eye image to be identified.

[0149] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0150] Obtain eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and map the visible image regions of eye image samples of each modality to corresponding feature vectors;

[0151] The feature vectors of eye image samples corresponding to different modalities are input into the initial image encoder to obtain the corresponding second feature image samples;

[0152] Each second feature image sample is input into the corresponding initial image decoder to obtain the reconstructed image sample corresponding to each second feature image sample.

[0153] Based on eye image samples of each modality and corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the trained image encoder.

[0154] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0155] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0156] An initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample to obtain the target large language model, and this target large language model is used as the trained image analysis sub-model.

[0157] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0158] An initial classifier is trained based on the first feature image sample to obtain the target classifier, and this target classifier is used as the trained image analysis sub-model.

[0159] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0160] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0161] Obtain eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and train the initial image encoder in the initial eye image analysis model based on the visible image regions to obtain the trained image encoder;

[0162] The first feature image sample is determined based on at least one modality of eye image samples from multiple modalities and a trained image encoder;

[0163] The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the trained image analysis sub-model;

[0164] Based on the trained image encoder, feature extraction is performed on the eye image to be identified to obtain the first feature image;

[0165] The first feature image is input into the trained image analysis sub-model to obtain the analysis results of the eye image to be identified.

[0166] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0167] Obtain eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and map the visible image regions of eye image samples of each modality to corresponding feature vectors;

[0168] The feature vectors of eye image samples corresponding to different modalities are input into the initial image encoder to obtain the corresponding second feature image samples;

[0169] Each second feature image sample is input into the corresponding initial image decoder to obtain the reconstructed image sample corresponding to each second feature image sample.

[0170] Based on eye image samples of each modality and corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the trained image encoder.

[0171] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0172] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0173] An initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample to obtain the target large language model, and this target large language model is used as the trained image analysis sub-model.

[0174] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0175] An initial classifier is trained based on the first feature image sample to obtain the target classifier, and this target classifier is used as the trained image analysis sub-model.

[0176] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0177] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0178] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image analysis method, characterized in that, The method includes: Multiple modal eye image samples are acquired, and the multiple modal eye image samples are processed to obtain visible image regions. Based on the visible image regions, the initial image encoder in the initial eye image analysis model is trained to obtain the trained image encoder. The first feature image sample is determined based on at least one modality of eye image samples from the multiple modalities and the trained image encoder; The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the trained image analysis sub-model. Based on the trained image encoder, feature extraction is performed on the eye image to be identified to obtain a first feature image; The first feature image is input into the trained image analysis sub-model to obtain the analysis result of the eye image to be identified.

2. The method according to claim 1, characterized in that, The process of acquiring eye image samples of the multiple modalities, processing the eye image samples of the multiple modalities to obtain the visible image region, and training the initial image encoder in the initial eye image analysis model based on the visible image region to obtain the trained image encoder includes: Obtain eye image samples of the multiple modalities, process the eye image samples of the multiple modalities to obtain the visible image region, and map the visible image region of the eye image samples of each modality to the corresponding feature vector; The feature vectors of eye image samples corresponding to different modalities are input into the initial image encoder to obtain the corresponding second feature image samples; Each of the second feature image samples is input into the corresponding initial image decoder to obtain the reconstructed image sample corresponding to each of the second feature image samples; Based on eye image samples of each modality and the corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the trained image encoder.

3. The method according to claim 1 or 2, characterized in that, The initial image analysis sub-model includes an initial classifier and / or an initial large language model.

4. The method according to claim 3, characterized in that, The initial image analysis sub-model includes the initial large language model. The initial image analysis sub-model in the initial eye image analysis model trained based on the first feature image samples, to obtain the trained image analysis sub-model, includes: The initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample to obtain the target large language model, and the target large language model is used as the trained image analysis sub-model.

5. The method according to claim 3, characterized in that, The initial image analysis sub-model includes the initial classifier. The process of training the initial image analysis sub-model within the initial eye image analysis model based on the first feature image samples to obtain the trained image analysis sub-model includes: The initial classifier is trained based on the first feature image sample to obtain the target classifier, and the target classifier is used as the trained image analysis sub-model.

6. The method according to any one of claims 1-5, characterized in that, The multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

7. The method according to any one of claims 1-6, characterized in that, Obtain eye image samples in multiple modalities, including: Multiple modal imaging techniques are used to acquire eye images of the various modalities. Quality control is performed on the eye images of the various modalities to exclude images with poor quality; The remaining image size was adjusted to 256×256 pixels using cubic spline interpolation to obtain eye image samples of the various modalities.

8. The method according to any one of claims 1-7, characterized in that, The visible image region is obtained by processing eye image samples of multiple modalities, including: For each image of the multimodal eye image samples, it is divided into an image region of the same size, 16×16 pixels; Randomly select an image region proportionally, cover the selected image region, and obtain the uncovered visible image region.

9. The method according to claim 2, characterized in that, Each initial image decoder includes two Transformer blocks; the occluded image regions of the second feature image sample of the corresponding modality output by the initial image encoder and the eye image sample of the corresponding modality are respectively input into the two Transformer blocks of the corresponding initial image decoder.

10. The method according to claim 2, characterized in that, Based on eye image samples of each modality and the corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain a trained image encoder, including: Based on the eye image samples of each modality and the corresponding reconstructed image samples, the pixel values ​​of each image region of the eye image samples of each modality are normalized. Calculate the loss between the pixel values ​​of the normalized eye image samples of each modality and the pixel values ​​of the corresponding reconstructed image samples; The initial image encoder is trained based on the calculated loss to obtain the trained image encoder.

11. The method according to claim 3 or 5, characterized in that, The initial classifier is a multilayer perceptron classifier.

12. An image analysis device, characterized in that, The device includes: The first training module is used to acquire eye image samples of multiple modalities, process the eye image samples of multiple modalities to obtain visible image regions, and train the initial image encoder in the initial eye image analysis model based on the visible image regions to obtain the trained image encoder. The first determining module is used to determine a first feature image sample based on at least one modality of eye image samples from the multiple modalities of eye image samples and the trained image encoder; The second training module is used to train the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample, so as to obtain the trained image analysis sub-model. The feature extraction module is used to extract features from the eye image to be identified based on the trained image encoder to obtain a first feature image; The second determining module is used to input the first feature image into the trained image analysis sub-model to obtain the analysis result of the eye image to be identified.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images