Cell image recognition method and system based on model fine-tuning

Abstract

The present invention relates to the technical field of data processing. Disclosed are a cell image recognition method and system based on model fine-tuning. The method comprises: acquiring an image to be recognized comprising a cell image; inputting the image to be recognized into a trained cell clarity recognition model to obtain corresponding cell clarity; inputting the image to be recognized into a trained cell presence region recognition model to obtain a corresponding cell presence region image; and inputting the cell presence region image and the cell clarity into a trained cell classification model to obtain a plurality of corresponding cell regions and corresponding cell types. The cell clarity recognition model, the cell presence region recognition model, and the cell classification model are obtained by means of joint training and fine-tuning. Hence, in the present invention, clarity factors and prior region recognition can be fully utilized to accurately identify cell regions and types in images, thereby improving the accuracy and efficiency of cell recognition.

Description

A Cell Image Recognition Method and System Based on Model Fine-Tuning Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a cell image recognition method and system based on model fine-tuning. Background Technology

[0002] With the development of image processing technology and the popularization of medical imaging technology, an increasing number of biological cell images are being used for disease research or treatment. How to effectively identify and segment cellular regions in biological tissue images has become a core technical problem. However, existing cell image recognition technologies generally only rely on training common image recognition algorithm models based on cell-annotated images for identification, without further developing unique algorithm models tailored to different stages of the cell recognition process and factors affecting image clarity. Therefore, their efficiency and accuracy in cell recognition are somewhat lacking. Clearly, existing technologies have shortcomings that urgently need to be addressed. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a cell image recognition method and system based on model fine-tuning, which can make full use of sharpness factors and prior region recognition to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0004] To address the aforementioned technical problems, the first aspect of this invention discloses a cell image recognition method based on model fine-tuning, the method comprising:

[0005] Acquire the image to be identified, which includes cell images;

[0006] The image to be identified is input into the trained cell sharpness recognition model to obtain the corresponding cell sharpness.

[0007] The image to be identified is input into a trained cell presence region recognition model to obtain the corresponding cell presence region image;

[0008] The image of the cell-containing region and the cell clarity are input into the trained cell classification model to obtain multiple corresponding cell regions and corresponding cell types; the cell clarity recognition model, the cell-containing region recognition model, and the cell classification model are obtained based on joint training and fine-tuning.

[0009] As an optional implementation, in the first aspect of the present invention, the cell clarity recognition model, the cell presence region recognition model, and the cell classification model are jointly trained based on the same training dataset; the training dataset includes multiple training images and corresponding cell region annotations, cell type annotations, and cell clarity annotations.

[0010] As an optional implementation, in the first aspect of the present invention, the cell clarity recognition model, the cell presence region recognition model, and the cell classification model are trained through the following steps:

[0011] The training dataset is input into the cell clarity recognition model and the cell presence region recognition model for training until the first prediction accuracy of both reaches a preset first accuracy threshold.

[0012] The model weights of the cell clarity recognition model and the cell presence region recognition model are frozen, and the training dataset is input into the cell classification model for training until its second prediction accuracy reaches a preset second accuracy threshold; the second accuracy threshold is greater than the first accuracy threshold;

[0013] The model weights of the cell sharpness recognition model and the cell presence region recognition model are unfrozen, and the training dataset is simultaneously input into the cell sharpness recognition model, the cell presence region recognition model, and the cell classification model for joint training until the third prediction accuracy of the three models reaches a preset third accuracy threshold, thus obtaining the trained cell sharpness recognition model, cell presence region recognition model, and cell classification model; the third accuracy threshold is greater than the second accuracy threshold.

[0014] As an optional implementation, in the first aspect of the present invention, the cell clarity recognition model is a convolutional neural network, which uses L1 loss as the loss function during training; the cell presence region recognition model and the cell classification model are both fully convolutional neural networks, which use cross-entropy loss function as the loss function during training.

[0015] As an optional implementation, in the first aspect of the present invention, when the training dataset is simultaneously input into the cell clarity recognition model, the cell presence region recognition model, and the cell classification model for joint training, the loss function used in the joint training is a weighted sum of the values ​​of a first loss function, a second loss function, and a third loss function; wherein:

[0016] The first loss function value is the difference between the predicted sharpness output by the cell sharpness recognition model and the corresponding cell sharpness label, and its weight is proportional to the variance of the cell sharpness labels corresponding to all training images in the current input data.

[0017] The second loss function value is the cross-entropy between the predicted cell presence region output by the cell presence region recognition model and all the corresponding cell region labels, and its corresponding weight is proportional to the total area of ​​the union of all cell region labels corresponding to all training images in the current input data.

[0018] The third loss function is the cross-entropy between the predicted cell region and predicted type output by the cell classification model and the corresponding cell region label and cell type label. Its weight is proportional to the total number of cell type labels corresponding to all training images in the current input data.

[0019] As an optional implementation, in the first aspect of the invention, the cell type is PanCK+ cells, CD3+ cells, CD8- cells, or CD8+ cells.

[0020] As an optional implementation, in the first aspect of the invention, the training dataset is obtained through the following steps:

[0021] Acquire multiple training images;

[0022] Based on the operator's annotations, the cell nucleus location annotations and cell type annotations corresponding to each training image are determined;

[0023] For each nucleus location label, obtain the cell type outline corresponding to the cell type label corresponding to the nucleus location label;

[0024] Based on the cell type contour, a contour is generated with the cell nucleus location marked as the center to obtain the corresponding cell region marking;

[0025] The cell sharpness label corresponding to each training image is determined based on the time taken by the operator to annotate each training image.

[0026] As an optional implementation, in the first aspect of the present invention, the step of obtaining the training dataset, determining the cell sharpness annotation corresponding to each training image based on the time taken by the operator to annotate each training image, includes:

[0027] For each training image, obtain the movement trajectory of the operator's cursor on the training image and the corresponding movement speed;

[0028] From multiple image regions of the training image, select multiple densely populated cell regions whose corresponding cell region annotations are greater than a preset threshold.

[0029] Calculate the product of the length of the trajectory portion of the cursor movement path in each of the dense cell regions and the corresponding movement speed to obtain the region annotation movement time for each of the dense cell regions;

[0030] Calculate the weighted average of the time taken to move the region labels corresponding to all the dense cell regions to obtain the cell sharpness label corresponding to the training image; wherein, the weight corresponding to the time taken to move the region labels corresponding to each dense cell region is inversely proportional to the number of labels corresponding to that dense cell region.

[0031] A second aspect of this invention discloses a cell image recognition system based on model fine-tuning, the system comprising:

[0032] The acquisition module is used to acquire an image to be identified, including cell images;

[0033] The first recognition module is used to input the image to be recognized into the trained cell sharpness recognition model in order to obtain the corresponding cell sharpness.

[0034] The second recognition module is used to input the image to be recognized into the trained cell presence region recognition model to obtain the corresponding cell presence region image;

[0035] The third recognition module is used to input the image of the cell-containing region and the cell clarity into the trained cell classification model to obtain multiple corresponding cell regions and corresponding cell types; the cell clarity recognition model, the cell-containing region recognition model and the cell classification model are obtained based on joint training and fine-tuning.

[0036] As an optional implementation, in the second aspect of the present invention, the cell clarity recognition model, the cell presence region recognition model, and the cell classification model are jointly trained based on the same training dataset; the training dataset includes multiple training images and corresponding cell region annotations, cell type annotations, and cell clarity annotations.

[0037] As an optional implementation, in a second aspect of the invention, the cell clarity recognition model, the cell presence region recognition model, and the cell classification model are trained through the following steps:

[0038] The training dataset is input into the cell clarity recognition model and the cell presence region recognition model for training until the first prediction accuracy of both reaches a preset first accuracy threshold.

[0039] The model weights of the cell clarity recognition model and the cell presence region recognition model are frozen, and the training dataset is input into the cell classification model for training until its second prediction accuracy reaches a preset second accuracy threshold; the second accuracy threshold is greater than the first accuracy threshold;

[0040] The model weights of the cell sharpness recognition model and the cell presence region recognition model are unfrozen, and the training dataset is simultaneously input into the cell sharpness recognition model, the cell presence region recognition model, and the cell classification model for joint training until the third prediction accuracy of the three models reaches a preset third accuracy threshold, thus obtaining the trained cell sharpness recognition model, cell presence region recognition model, and cell classification model; the third accuracy threshold is greater than the second accuracy threshold.

[0041] As an optional implementation, in the second aspect of the present invention, the cell clarity recognition model is a convolutional neural network, which uses L1 loss as the loss function during training; the cell presence region recognition model and the cell classification model are both fully convolutional neural networks, which use cross-entropy loss function as the loss function during training.

[0042] As an optional implementation, in a second aspect of the invention, when the training dataset is simultaneously input into the cell clarity recognition model, the cell presence region recognition model, and the cell classification model for joint training, the loss function used in the joint training is a weighted sum of the values ​​of a first loss function, a second loss function, and a third loss function; wherein:

[0043] The first loss function value is the difference between the predicted sharpness output by the cell sharpness recognition model and the corresponding cell sharpness label, and its weight is proportional to the variance of the cell sharpness labels corresponding to all training images in the current input data.

[0044] The second loss function value is the cross-entropy between the predicted cell presence region output by the cell presence region recognition model and all the corresponding cell region labels, and its corresponding weight is proportional to the total area of ​​the union of all cell region labels corresponding to all training images in the current input data.

[0045] The third loss function is the cross-entropy between the predicted cell region and predicted type output by the cell classification model and the corresponding cell region label and cell type label. Its weight is proportional to the total number of cell type labels corresponding to all training images in the current input data.

[0046] As an optional implementation, in a second aspect of the invention, the cell type is PanCK+ cells, CD3+ cells, CD8- cells, or CD8+ cells.

[0047] As an optional implementation, in a second aspect of the invention, the training dataset is obtained through the following steps:

[0048] Acquire multiple training images;

[0049] Based on the operator's annotations, the cell nucleus location annotations and cell type annotations corresponding to each training image are determined;

[0050] For each nucleus location label, obtain the cell type outline corresponding to the cell type label corresponding to the nucleus location label;

[0051] Based on the cell type contour, a contour is generated with the cell nucleus location marked as the center to obtain the corresponding cell region marking;

[0052] The cell sharpness label corresponding to each training image is determined based on the time taken by the operator to annotate each training image.

[0053] As an optional implementation, in a second aspect of the invention, the step of acquiring the training dataset, determining the cell sharpness annotation corresponding to each training image based on the time taken by the operator to annotate each training image, includes:

[0054] For each training image, obtain the movement trajectory of the operator's cursor on the training image and the corresponding movement speed;

[0055] From multiple image regions of the training image, select multiple densely populated cell regions whose corresponding cell region annotations are greater than a preset threshold.

[0056] Calculate the product of the length of the trajectory portion of the cursor movement path in each of the dense cell regions and the corresponding movement speed to obtain the region annotation movement time for each of the dense cell regions;

[0057] Calculate the weighted average of the time taken to move the region labels corresponding to all the dense cell regions to obtain the cell sharpness label corresponding to the training image; wherein, the weight corresponding to the time taken to move the region labels corresponding to each dense cell region is inversely proportional to the number of labels corresponding to that dense cell region.

[0058] A third aspect of the present invention discloses another cell image recognition system based on model fine-tuning, the system comprising:

[0059] Memory containing executable program code;

[0060] A processor coupled to the memory;

[0061] The processor calls the executable program code stored in the memory to execute some or all of the steps in the cell image recognition method based on model fine-tuning disclosed in the first aspect of the present invention.

[0062] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the cell image recognition method based on model fine-tuning disclosed in the first aspect of the present invention.

[0063] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0064] This invention first identifies the cell clarity of an image based on a trained cell clarity recognition model, then identifies the corresponding prior high-probability cell-containing regions using a trained cell presence region recognition model, and finally obtains the corresponding cell regions and cell types based on the cell-containing region images and cell clarity recognition using a trained cell classification model. This fully utilizes clarity factors and prior region recognition to accurately identify cell regions and types in images, improving the accuracy and efficiency of cell identification and providing an accurate data foundation for disease treatment or research. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 is a schematic flowchart of a cell image recognition method based on model fine-tuning disclosed in an embodiment of the present invention.

[0067] Figure 2 is a schematic diagram of the structure of a cell image recognition system based on model fine-tuning disclosed in an embodiment of the present invention.

[0068] Figure 3 is a schematic diagram of another cell image recognition system based on model fine-tuning disclosed in an embodiment of the present invention.

[0069] Figure 4 is a schematic diagram of the recognition effect of the recognition model disclosed in the embodiment of the present invention.

[0070] Figure 5 is a schematic diagram of the statistical results of the correlation coefficient of the prediction results of the algorithm recognition scheme disclosed in the embodiment of the present invention. Embodiments of the present invention

[0071] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] The terms "first," "second," etc., used in the specification and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0073] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0074] This invention discloses a cell image recognition method and system based on model fine-tuning. It first identifies the cell sharpness of an image using a trained cell sharpness recognition model, then identifies the corresponding prior high-probability cell-containing regions using a trained cell presence region recognition model. Finally, based on a trained cell classification model, it identifies the corresponding cell regions and cell types according to the cell-containing region images and cell sharpness recognition. This fully utilizes sharpness factors and prior region recognition to accurately identify cell regions and types in images, improving the accuracy and efficiency of cell recognition and providing an accurate data foundation for disease treatment or research. Detailed explanations follow.

[0075] Example 1

[0076] Please refer to Figure 1, which is a flowchart illustrating a model-based cell image recognition method disclosed in an embodiment of the present invention. The model-based cell image recognition method described in Figure 1 can be applied to a data processing system / data processing device / data processing server (wherein the server includes a local processing server or a cloud processing server). As shown in Figure 1, the model-based cell image recognition method may include the following operations:

[0077] 101. Obtain the image to be identified, which includes cell images.

[0078] 102. Input the image to be recognized into the trained cell sharpness recognition model to obtain the corresponding cell sharpness.

[0079] 103. Input the image to be identified into the trained cell presence region recognition model to obtain the corresponding cell presence region image.

[0080] 104. Input the images of the cell regions and cell clarity into the trained cell classification model to obtain the corresponding multiple cell regions and corresponding cell types.

[0081] Optionally, the cell clarity recognition model, the cell presence region recognition model, and the cell classification model are obtained based on joint training and fine-tuning.

[0082] As can be seen, the above-described embodiments of the invention can first identify the cell clarity of an image based on a trained cell clarity recognition model, then identify the corresponding prior high-probability cell-containing region image through a trained cell presence region recognition model, and finally obtain the corresponding cell region and cell type based on the cell-containing region image and cell clarity recognition using a trained cell classification model. This fully utilizes clarity factors and prior region recognition to accurately identify cell regions and types in images, improves the accuracy and efficiency of cell recognition, and provides an accurate data foundation for disease treatment or research.

[0083] As an optional embodiment, the cell clarity recognition model, cell presence region recognition model, and cell classification model in the above steps are jointly trained based on the same training dataset; the training dataset includes multiple training images and corresponding cell region annotations, cell type annotations, and cell clarity annotations.

[0084] As can be seen, through the above optional embodiments, three models can be jointly trained on the same training dataset to obtain recognition models responsible for different functions, so as to achieve accurate cell recognition, assist in making full use of sharpness factors and prior region recognition to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0085] As an optional embodiment, the cell clarity recognition model, cell presence region recognition model, and cell classification model in the above steps are trained through the following steps:

[0086] The training dataset is input into the cell clarity recognition model and the cell presence region recognition model for training until the first prediction accuracy of both reaches the preset first accuracy threshold.

[0087] The model weights of the cell clarity recognition model and the cell presence region recognition model are frozen, and the training dataset is input into the cell classification model for training until its second prediction accuracy reaches a preset second accuracy threshold; optionally, the second accuracy threshold is greater than the first accuracy threshold.

[0088] The model weights of the cell clarity recognition model and the cell presence region recognition model are unfrozen, and the training dataset is simultaneously input into the cell clarity recognition model, the cell presence region recognition model and the cell classification model for joint training until the third prediction accuracy of the three models reaches the preset third accuracy threshold, thus obtaining the trained cell clarity recognition model, cell presence region recognition model and cell classification model; the third accuracy threshold is greater than the second accuracy threshold.

[0089] As can be seen, through the above optional embodiments, three recognition model regions can be gradually trained based on step-by-step training steps and operations such as fine-tuning model weight freezing, so as to achieve accurate cell recognition. This helps to fully utilize sharpness factors and prior region recognition to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0090] As an optional embodiment, in the above steps, the cell clarity recognition model is a convolutional neural network, which uses L1 loss as the loss function during training; the cell presence region recognition model and the cell classification model are both fully convolutional neural networks, which use cross-entropy loss function as the loss function during training.

[0091] As can be seen, the model algorithm details and training details of the three recognition models are clarified through the above optional embodiments to achieve accurate cell recognition. This helps to fully utilize sharpness factors and prior region recognition to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0092] As an optional embodiment, in the above steps, when the training dataset is simultaneously input into the cell clarity recognition model, the cell presence region recognition model, and the cell classification model for joint training, the loss function used in the joint training is a weighted sum of the first loss function value, the second loss function value, and the third loss function value; wherein:

[0093] The first loss function value is the difference between the predicted sharpness output by the cell sharpness recognition model and the corresponding cell sharpness label, and its weight is proportional to the variance of the cell sharpness labels corresponding to all training images in the current input data.

[0094] The second loss function is the cross-entropy between the predicted cell presence region and the corresponding label of all cell regions output by the cell presence region recognition model. Its weight is proportional to the total area of ​​the union of all cell region labels corresponding to all training images in the current input data.

[0095] The third loss function is the cross-entropy between the predicted cell regions and types output by the cell classification model and the corresponding cell region and cell type labels. Its weight is proportional to the total number of cell type labels corresponding to all training images in the current input data.

[0096] As can be seen, through the above optional embodiments, the loss function settings in the joint training process can be comprehensively determined based on the preset weight rules and the loss function values ​​of the three models respectively, so as to obtain a recognition model with more correlation and cooperation among the three prediction results through joint training, thereby achieving accurate cell recognition. This helps to fully utilize the sharpness factor and prior region recognition to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0097] As an optional embodiment, the cell type in the above steps is PanCK+ cells, CD3+ cells, CD8- cells, or CD8+ cells.

[0098] As can be seen, the above optional embodiments define the types of cells to be identified for accurate cell identification related to tumor treatment, helping to fully utilize clarity factors and prior region identification to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell identification, and provide an accurate data foundation for disease treatment or research.

[0099] As an optional embodiment, the training dataset in the above steps is obtained through the following steps:

[0100] Acquire multiple training images;

[0101] Based on the operator's annotations, the cell nucleus location annotations and cell type annotations corresponding to each training image are determined;

[0102] For each nucleus location label, obtain the cell type outline corresponding to the cell type label corresponding to that nucleus location label;

[0103] Based on cell type contours, contours are generated with the cell nucleus location as the center to obtain the corresponding cell region labels;

[0104] Based on the time taken by the operator to annotate each training image, the cell sharpness annotation corresponding to each training image is determined.

[0105] As can be seen, through the above optional embodiments, cell region annotations can be automatically generated based on cell outlines of cell nucleus location and cell type, and cell clarity annotations can be determined based on annotation time. This comprehensively improves the work efficiency of annotators, reduces time costs, and lightens the burden on annotators. It also enables the full utilization of clarity factors and prior region recognition to accurately identify cell regions and types in images, improves the accuracy and efficiency of cell identification, and provides an accurate data foundation for disease treatment or research.

[0106] As an optional embodiment, the step of obtaining the training dataset described above, which involves determining the cell sharpness annotation corresponding to each training image based on the time taken by the operator to annotate each training image, includes:

[0107] For each training image, obtain the movement trajectory of the operator's cursor on that training image and the corresponding movement speed;

[0108] From multiple image regions of the training image, select multiple densely populated cell regions whose corresponding cell region annotations are greater than a preset threshold.

[0109] Calculate the product of the length of the trajectory of the cursor movement in each dense cell region and the corresponding movement speed to obtain the time taken to move the region label for each dense cell region.

[0110] Calculate the weighted average of the time taken to move the region labels corresponding to all densely populated cell regions to obtain the cell sharpness label for the training image; wherein, the weight corresponding to the time taken to move the region labels corresponding to each densely populated cell region is inversely proportional to the number of labels corresponding to that densely populated cell region.

[0111] As can be seen, through the above optional embodiments, a more accurate representation of the visibility of operator annotations on the image can be achieved based on the operator's annotation movement trajectory on the training image and the number of cell region annotations in different areas. The obtained calculation results can effectively represent whether the training image is easy to annotate and thus represent its cell clarity. In the subsequent training, a more accurate recognition model can be obtained, which can make full use of clarity factors and prior region recognition to accurately identify cell regions and types in the image, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0112] In one specific implementation scheme, a recognition model is constructed based on the present invention, primarily used to identify tumor cells (PanCK+), CD3+ / CD8- cells, and CD8+ cells in digital slices. Specifically, cell labeling in training images is performed using a combination of manual and machine methods. CD3+ and CD8+ cells are labeled manually. Since some CD8+ cells also express CD3+ positivity, to ensure that each cell in the label represents only one category, the two categories manually labeled are CD8+ cells and CD3+ / CD8- cells. Combining CD8+ and CD3+ / CD8- cells yields all CD3+ cells. To simplify the labeling workload for doctors, point labeling is used; doctors only need to mark the center point of the cell and its nuclear category as a dot.

[0113] Due to the large number of tumor cells in the images, manual annotation was too time-consuming; therefore, PanCK+ cell annotation was automated. This approach first uses a pre-trained Stardist network for cell nucleus segmentation based on the DAPI channel of the image. It was found that the model's segmentation performance in tumor epithelial regions with high cell density needs improvement. Therefore, based on the prediction results of existing cell nucleus segmentation models, this approach manually corrected the cell nucleus contours in the tumor epithelial region and retrained and fine-tuned the Stardist model.

[0114] On the other hand, this scheme calculates the percentage of PanCK-positive pixels within each cell nucleus contour region. PanCK-positive pixel identification is achieved using an adaptive thresholding method for the PanCK channel. If the percentage of PanCK-positive pixels within a cell nucleus contour is higher than 50%, the contour is automatically identified as a PanCK+ cell. If a cell identified as PanCK+ has a center point of a CD8+ or CD3+ / CD8- cell manually annotated by a doctor within its nucleus contour, the cell contour is preferentially classified according to the doctor's manually annotated category. Finally, for all cells determined to be PanCK+, center points are generated according to their nucleus contours, serving as the final annotation result. Specifically, this scheme further generates sharpness annotations for each training image based on the time taken by the doctor's annotation trajectory, but does not adopt a scheme where sharpness opinions are directly provided by the doctor. This is because sharpness opinions contain subjective components and cannot accurately represent the cell sharpness annotations of the image.

[0115] Based on the aforementioned combination of manual and machine annotation, this scheme obtained 66 images of 3000*3000 pixels each with relevant annotations, and used these images to train the cell recognition model. The 66 images were divided into 54 for training, 8 for validation, and 4 for testing. The images have four channels: DAPI, PanCK, CD3, and CD8. Each channel was truncated according to the 99th percentile of the pixel value distribution, and then the pixel values ​​were transformed to between 0 and 1 using the min-max normalization method.

[0116] The cell recognition model used in this scheme comprises three branches. The first is a sharpness recognition model, which is a convolutional network. The second is a region detection model, which is an 80-layer fully convolutional network (FCN) used to identify cell nucleus presence regions with high confidence, i.e., target prior regions. The third branch is a cell classification branch, whose input is the sharpness prediction output from the first branch and the high-confidence cell nucleus presence regions output from the second branch. This allows the classification model to focus on cell presence regions and adjust the cell region segmentation scheme based on the sharpness prediction of the image itself, without being affected by a large number of background regions.

[0117] During training, 128*128 pixels are randomly selected from each image and used as input to the network. Various image enhancement techniques are employed, including random flipping, Gaussian noise, brightness enhancement, and sharpening. For the three branches, a weighted cross-entropy loss function is used. The entire training process is divided into three phases. First, the first two branches are trained to maximize the performance of cell prior region detection and sharpness prediction. Next, the first two branches are frozen, and the third branch is trained. Finally, all three branches are jointly trained to achieve refinement detection and classification. The first and second phases are trained for 100 epochs each, and the third phase for 200 epochs, with a batch size of 500. The Adam optimizer is used, combined with a cosine warm-up learning rate strategy. After each training epoch, the validation set is evaluated, and the F1 score for each cell class and the average F1 score for all three classes are calculated. Finally, the model selects the optimal weights based on the highest average F1 score.

[0118] Figure 4 shows the recognition effect of the recognition model trained by this scheme. The left side of Figure 4 shows the segmentation result of the original pre-trained Stardist model, and the right side shows the segmentation result of the fine-tuned and jointly trained model. It can be seen that the new fine-tuned model has significantly improved the segmentation effect of cell nuclei in the tumor epithelial region.

[0119] Figure 5 shows the correlation coefficient between the manually counted number of cells and the machine-identified number of cells obtained from the prediction results of this scheme.

[0120] Finally, this method uses the above model to detect cells in the tumor region and the tumor invasion front region identified in each slice, and obtains the center point location of PanCK+, CD3+ / CD8- and CD8+ cells for the next step of feature extraction, which can effectively improve the accuracy of subsequent cell feature extraction and related research.

[0121] Example 2

[0122] Please refer to Figure 2, which is a schematic diagram of a model-based cell image recognition system disclosed in an embodiment of the present invention. The model-based cell image recognition system described in Figure 2 can be applied to a data processing system / data processing device / data processing server (wherein the server includes a local processing server or a cloud processing server). As shown in Figure 2, the model-based cell image recognition system may include:

[0123] The acquisition module 201 is used to acquire an image to be identified, which includes cell images.

[0124] The first recognition module 202 is used to input the image to be recognized into the trained cell sharpness recognition model in order to obtain the corresponding cell sharpness.

[0125] The second recognition module 203 is used to input the image to be recognized into the trained cell presence region recognition model to obtain the corresponding cell presence region image.

[0126] The third recognition module 204 is used to input the image of the cell region and the cell clarity into the trained cell classification model to obtain the corresponding multiple cell regions and the corresponding cell types.

[0127] Optionally, the cell clarity recognition model, the cell presence region recognition model, and the cell classification model are obtained based on joint training and fine-tuning.

[0128] As can be seen, the above-described embodiments of the invention can first identify the cell clarity of an image based on a trained cell clarity recognition model, then identify the corresponding prior high-probability cell-containing region image through a trained cell presence region recognition model, and finally obtain the corresponding cell region and cell type based on the cell-containing region image and cell clarity recognition using a trained cell classification model. This fully utilizes clarity factors and prior region recognition to accurately identify cell regions and types in images, improves the accuracy and efficiency of cell recognition, and provides an accurate data foundation for disease treatment or research.

[0129] As an optional embodiment, the cell clarity recognition model, the cell presence region recognition model, and the cell classification model are jointly trained based on the same training dataset; the training dataset includes multiple training images and corresponding cell region annotations, cell type annotations, and cell clarity annotations.

[0130] As can be seen, through the above optional embodiments, three models can be jointly trained on the same training dataset to obtain recognition models responsible for different functions, so as to achieve accurate cell recognition, assist in making full use of sharpness factors and prior region recognition to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0131] As an optional embodiment, the cell clarity recognition model, the cell presence region recognition model, and the cell classification model are trained through the following steps:

[0132] The training dataset is input into the cell clarity recognition model and the cell presence region recognition model for training until the first prediction accuracy of both reaches the preset first accuracy threshold.

[0133] The model weights of the cell clarity recognition model and the cell presence region recognition model are frozen, and the training dataset is input into the cell classification model for training until its second prediction accuracy reaches a preset second accuracy threshold; optionally, the second accuracy threshold is greater than the first accuracy threshold.

[0134] The model weights of the cell clarity recognition model and the cell presence region recognition model are unfrozen, and the training dataset is simultaneously input into the cell clarity recognition model, the cell presence region recognition model and the cell classification model for joint training until the third prediction accuracy of the three models reaches the preset third accuracy threshold, thus obtaining the trained cell clarity recognition model, cell presence region recognition model and cell classification model; the third accuracy threshold is greater than the second accuracy threshold.

[0135] As can be seen, through the above optional embodiments, three recognition model regions can be gradually trained based on step-by-step training steps and operations such as fine-tuning model weight freezing, so as to achieve accurate cell recognition. This helps to fully utilize sharpness factors and prior region recognition to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0136] As an optional embodiment, the cell clarity recognition model is a convolutional neural network, which uses L1 loss as the loss function during training; the cell presence region recognition model and the cell classification model are both fully convolutional neural networks, which use cross-entropy loss function as the loss function during training.

[0137] As can be seen, the model algorithm details and training details of the three recognition models are clarified through the above optional embodiments to achieve accurate cell recognition. This helps to fully utilize sharpness factors and prior region recognition to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0138] As an optional embodiment, when the training dataset is simultaneously input into the cell clarity recognition model, the cell presence region recognition model, and the cell classification model for joint training, the loss function used in the joint training is a weighted sum of the first loss function value, the second loss function value, and the third loss function value; wherein:

[0139] The first loss function value is the difference between the predicted sharpness output by the cell sharpness recognition model and the corresponding cell sharpness label, and its weight is proportional to the variance of the cell sharpness labels corresponding to all training images in the current input data.

[0140] The second loss function is the cross-entropy between the predicted cell presence region and the corresponding label of all cell regions output by the cell presence region recognition model. Its weight is proportional to the total area of ​​the union of all cell region labels corresponding to all training images in the current input data.

[0141] The third loss function is the cross-entropy between the predicted cell regions and types output by the cell classification model and the corresponding cell region and cell type labels. Its weight is proportional to the total number of cell type labels corresponding to all training images in the current input data.

[0142] As can be seen, through the above optional embodiments, the loss function settings in the joint training process can be comprehensively determined based on the preset weight rules and the loss function values ​​of the three models respectively, so as to obtain a recognition model with more correlation and cooperation among the three prediction results through joint training, thereby achieving accurate cell recognition. This helps to fully utilize the sharpness factor and prior region recognition to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0143] As an optional embodiment, the cell type is PanCK+ cells, CD3+ cells, CD8- cells, or CD8+ cells.

[0144] As can be seen, the above optional embodiments define the types of cells to be identified for accurate cell identification related to tumor treatment, helping to fully utilize clarity factors and prior region identification to accurately identify cell regions and types in images, improve the accuracy and efficiency of cell identification, and provide an accurate data foundation for disease treatment or research.

[0145] As an optional implementation, the training dataset is obtained through the following steps:

[0146] Acquire multiple training images;

[0147] Based on the operator's annotations, the cell nucleus location annotations and cell type annotations corresponding to each training image are determined;

[0148] For each nucleus location label, obtain the cell type outline corresponding to the cell type label corresponding to that nucleus location label;

[0149] Based on cell type contours, contours are generated with the cell nucleus location as the center to obtain the corresponding cell region labels;

[0150] Based on the time taken by the operator to annotate each training image, the cell sharpness annotation corresponding to each training image is determined.

[0151] As can be seen, through the above optional embodiments, cell region annotations can be automatically generated based on cell outlines of cell nucleus location and cell type, and cell clarity annotations can be determined based on annotation time. This comprehensively improves the work efficiency of annotators, reduces time costs, and lightens the burden on annotators. It also enables the full utilization of clarity factors and prior region recognition to accurately identify cell regions and types in images, improves the accuracy and efficiency of cell identification, and provides an accurate data foundation for disease treatment or research.

[0152] As an optional embodiment, the step of acquiring the training dataset, determining the cell sharpness annotation corresponding to each training image based on the time taken by the operator to annotate each training image, includes:

[0153] For each training image, obtain the movement trajectory of the operator's cursor on that training image and the corresponding movement speed;

[0154] From multiple image regions of the training image, select multiple densely populated cell regions whose corresponding cell region annotations are greater than a preset threshold.

[0155] Calculate the product of the length of the trajectory of the cursor movement in each dense cell region and the corresponding movement speed to obtain the time taken to move the region label for each dense cell region.

[0156] Calculate the weighted average of the time taken to move the region labels corresponding to all densely populated cell regions to obtain the cell sharpness label for the training image; wherein, the weight corresponding to the time taken to move the region labels corresponding to each densely populated cell region is inversely proportional to the number of labels corresponding to that densely populated cell region.

[0157] As can be seen, through the above optional embodiments, a more accurate representation of the visibility of operator annotations on the image can be achieved based on the operator's annotation movement trajectory on the training image and the number of cell region annotations in different areas. The obtained calculation results can effectively represent whether the training image is easy to annotate and thus represent its cell clarity. In the subsequent training, a more accurate recognition model can be obtained, which can make full use of clarity factors and prior region recognition to accurately identify cell regions and types in the image, improve the accuracy and efficiency of cell recognition, and provide an accurate data foundation for disease treatment or research.

[0158] Example 3

[0159] Please refer to Figure 3, which illustrates another model-based cell image recognition system disclosed in this embodiment of the invention. The model-based cell image recognition system described in Figure 3 is applied in a data processing system / data processing device / data processing server (wherein, the server includes a local processing server or a cloud processing server). As shown in Figure 3, this model-based cell image recognition system may include:

[0160] Memory 301 storing executable program code;

[0161] Processor 302 coupled to memory 301;

[0162] The processor 302 calls the executable program code stored in the memory 301 to execute the steps of the cell image recognition method based on model fine-tuning described in Embodiment 1.

[0163] Example 4

[0164] This invention discloses a computer read storage medium that stores a computer program for electronic data interchange, wherein the computer program causes a computer to execute the steps of the cell image recognition method based on model fine-tuning described in Embodiment 1.

[0165] Example 5

[0166] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of the model-based cell image recognition method described in Embodiment 1.

[0167] The foregoing has described specific embodiments of this specification. In some cases, the described actions or steps may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0168] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0169] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0170] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0174] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0175] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0176] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0177] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0178] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0179] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0180] Finally, it should be noted that the cell image recognition method and system based on model fine-tuning disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cell image recognition method based on model fine-tuning, characterized in that, The method includes: Acquire the image to be identified, which includes cell images; The image to be identified is input into the trained cell sharpness recognition model to obtain the corresponding cell sharpness. The image to be identified is input into a trained cell presence region recognition model to obtain the corresponding cell presence region image; The image of the cell-containing region and the cell clarity are input into the trained cell classification model to obtain multiple corresponding cell regions and corresponding cell types; the cell clarity recognition model, the cell-containing region recognition model, and the cell classification model are obtained based on joint training and fine-tuning.

2. The cell image recognition method based on model fine-tuning according to claim 1, characterized in that, The cell clarity recognition model, the cell presence region recognition model, and the cell classification model are jointly trained based on the same training dataset; the training dataset includes multiple training images and corresponding cell region annotations, cell type annotations, and cell clarity annotations.

3. The cell image recognition method based on model fine-tuning according to claim 2, characterized in that, The cell clarity recognition model, the cell presence region recognition model, and the cell classification model are trained through the following steps: The training dataset is input into the cell clarity recognition model and the cell presence region recognition model for training until the first prediction accuracy of both reaches a preset first accuracy threshold. The model weights of the cell clarity recognition model and the cell presence region recognition model are frozen, and the training dataset is input into the cell classification model for training until its second prediction accuracy reaches a preset second accuracy threshold. The second accuracy threshold is greater than the first accuracy threshold; The model weights of the cell sharpness recognition model and the cell presence region recognition model are unfrozen, and the training dataset is simultaneously input into the cell sharpness recognition model, the cell presence region recognition model, and the cell classification model for joint training until the third prediction accuracy of the three models reaches a preset third accuracy threshold, thus obtaining the trained cell sharpness recognition model, cell presence region recognition model, and cell classification model; the third accuracy threshold is greater than the second accuracy threshold.

4. The cell image recognition method based on model fine-tuning according to claim 3, characterized in that, The cell clarity recognition model is a convolutional neural network, which uses L1 loss as the loss function during training; the cell presence region recognition model and the cell classification model are both fully convolutional neural networks, which use cross-entropy loss function as the loss function during training.

5. The cell image recognition method based on model fine-tuning according to claim 3, characterized in that, When the training dataset is simultaneously input into the cell clarity recognition model, the cell presence region recognition model, and the cell classification model for joint training, the loss function used in the joint training is a weighted sum of the first loss function value, the second loss function value, and the third loss function value; wherein: The first loss function value is the difference between the predicted sharpness output by the cell sharpness recognition model and the corresponding cell sharpness label, and its weight is proportional to the variance of the cell sharpness labels corresponding to all training images in the current input data. The second loss function value is the cross-entropy between the predicted cell presence region output by the cell presence region recognition model and all the corresponding cell region labels, and its corresponding weight is proportional to the total area of ​​the union of all cell region labels corresponding to all training images in the current input data. The third loss function is the cross-entropy between the predicted cell region and predicted type output by the cell classification model and the corresponding cell region label and cell type label. Its weight is proportional to the total number of cell type labels corresponding to all training images in the current input data.

6. The cell image recognition method based on model fine-tuning according to claim 1, characterized in that, The cell type is PanCK+ cells, CD3+ cells, CD8- cells, or CD8+ cells.

7. The cell image recognition method based on model fine-tuning according to claim 2, characterized in that, The training dataset was obtained through the following steps: Acquire multiple training images; Based on the operator's annotations, the cell nucleus location annotations and cell type annotations corresponding to each training image are determined; For each nucleus location label, obtain the cell type outline corresponding to the cell type label corresponding to the nucleus location label; Based on the cell type contour, a contour is generated with the cell nucleus location marked as the center to obtain the corresponding cell region marking; The cell sharpness label corresponding to each training image is determined based on the time taken by the operator to annotate each training image.

8. The cell image recognition method based on model fine-tuning according to claim 7, characterized in that, The step of acquiring the training dataset, which involves determining the cell sharpness annotation corresponding to each training image based on the time taken by the operator to annotate each training image, includes: For each training image, obtain the movement trajectory of the operator's cursor on the training image and the corresponding movement speed; From multiple image regions of the training image, select multiple densely populated cell regions whose corresponding cell region annotations are greater than a preset threshold. Calculate the product of the length of the trajectory portion of the cursor movement path in each of the dense cell regions and the corresponding movement speed to obtain the region annotation movement time for each of the dense cell regions; Calculate the weighted average of the time taken to move the region labels corresponding to all the dense cell regions to obtain the cell sharpness label corresponding to the training image; wherein, the weight corresponding to the time taken to move the region labels corresponding to each dense cell region is inversely proportional to the number of labels corresponding to that dense cell region.

9. A cell image recognition system based on model fine-tuning, characterized in that, The system includes: The acquisition module is used to acquire an image to be identified, including cell images; The first recognition module is used to input the image to be recognized into the trained cell sharpness recognition model in order to obtain the corresponding cell sharpness. The second recognition module is used to input the image to be recognized into the trained cell presence region recognition model to obtain the corresponding cell presence region image; The third recognition module is used to input the image of the cell-containing region and the cell clarity into the trained cell classification model to obtain multiple corresponding cell regions and corresponding cell types; the cell clarity recognition model, the cell-containing region recognition model and the cell classification model are obtained based on joint training and fine-tuning.

10. A cell image recognition system based on model fine-tuning, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the cell image recognition method based on model fine-tuning as described in any one of claims 1-8.

Images