Medical image processing system

Abstract

A medical image processing system, comprising a communication module, an image processing module, and a three-dimensional modeling module. The communication module is communicatively connected to a user apparatus. A selection and comparison module allows a host to select conversion sites of a plurality of medical images and then compare the conversion sites with default site information to confirm that the conversion sites conform to corresponding organs or tissue sites in the default site information. The image processing module determines the types of the medical images, converts the medical images into a format of a right-anterior-superior (RAS) coordinate system, and performs automatic identification and segmentation of an organ or a lesion region of the converted medical images to generate a corresponding color segmentation mask. The three-dimensional modeling module generates a corresponding three-dimensional model on the basis of the color segmentation mask and generates a three-dimensional model file.

Description

Medical image processing system Technical Field

[0001] A medical image processing system, particularly a medical image processing system that performs standardized format conversion, automatic segmentation, three-dimensional geometric reconstruction and visualization processing based on medical images. Background Technology

[0002] This case asserts priority over U.S. provisional application numbers 63714875, 63714876, 63714878, and 63714879.

[0003] Currently, medical personnel primarily rely on patient medical records and imaging data displayed on computer screens as a reference when planning and executing surgeries. This imaging data, typically from examinations such as computed tomography (CT) or magnetic resonance imaging (MRI), provides high-resolution cross-sectional data, helping physicians determine the spatial location of lesions and their anatomical relationship with surrounding tissues, serving as an important reference source for preoperative planning and intraoperative navigation. Through these methods, the accuracy and safety of surgical procedures can be improved to a certain extent.

[0004] However, current technology still has several limitations and drawbacks. First, traditional images are mostly in the form of two-dimensional grayscale slices, requiring physicians to rely on experience to perform three-dimensional reconstruction and interpretation. For structurally complex or visually blurry areas, this can easily lead to interpretation difficulties or judgment errors. In addition, if it is necessary to further build a three-dimensional model or mark the location of lesions, it usually requires manual processing, such as manual delineation, segmentation, and modeling, which is not only time-consuming and labor-intensive, but may also lead to inconsistent results due to human factors.

[0005] Therefore, there is still an urgent need for a medical image processing system that integrates artificial intelligence automatic identification, 3D modeling and mixed reality display to solve the above problems, which has become a problem to be solved in the relevant technical field. Summary of the Invention

[0006] To address the aforementioned problems, according to one embodiment, the present invention provides a medical image processing system, comprising a host computer composed of multiple electrically connected hardware circuits and at least one user device having augmented reality functionality. The host computer includes a communication module, a selection and comparison module, an image processing module, and a three-dimensional modeling module.

[0007] The aforementioned communication module is communicatively connected to the aforementioned user device. The aforementioned selection and comparison module is communicatively connected to the aforementioned communication module. After receiving multiple medical images transmitted by the host through the aforementioned communication module, the host selects a conversion site from the multiple medical images. The selection and comparison module compares the conversion site with default site information to confirm that the conversion site matches the corresponding organ or tissue location in the default site information. The aforementioned image processing module is communicatively connected to the aforementioned selection and comparison module. After receiving the multiple medical images, it determines the type of the multiple medical images, converts the multiple medical images into RAS (Right-Anterior-Superior) coordinate system format, and performs corresponding preprocessing according to the type of the medical images. The image processing module uses a deep learning model to perform organ identification and segmentation on the preprocessed multiple medical images based on the conversion site information confirmed by the aforementioned selection and comparison module, generating corresponding multiple color segmentation masks. The aforementioned 3D modeling module is communicatively connected to the aforementioned image processing module, generates a corresponding 3D model based on the multiple color segmentation masks, and generates a 3D model file. The aforementioned 3D modeling module requires at least ten of the aforementioned medical images to establish the aforementioned 3D model with a specific depth of field, and transmits the aforementioned 3D model to the aforementioned user device through the aforementioned communication module, so that the aforementioned user device projects the aforementioned 3D model onto a desired physical location according to the aforementioned conversion part.

[0008] According to this embodiment, preferably, the selection and comparison module can identify the medical image as a computed tomography image or an magnetic resonance imaging image and then transmit it to the deep learning model for processing.

[0009] According to this embodiment, preferably, the above-mentioned image processing module performs lesion region identification and segmentation based on a lesion identification feature, and generates a corresponding lesion identification and segmentation mask.

[0010] According to this embodiment, preferably, the deep learning model is one of nnU-Net, nnTrans, or SUNETR.

[0011] According to this embodiment, preferably, for each of the above-mentioned transformation sites, a corresponding deep learning model is generated from at least 1000 of the above-mentioned medical images through preprocessing, model inference, and model verification.

[0012] According to this embodiment, preferably, the above-mentioned stereoscopic model file is in stereolithography (STL) format.

[0013] According to this embodiment, preferably, the medical image processing system further includes an image mask database, which is communicatively connected to the image processing module to receive and store the medical images and corresponding color segmentation masks from the image processing module, and to provide the training dataset and validation dataset required by the deep learning model.

[0014] According to this embodiment, preferably, the above-mentioned three-dimensional modeling module also marks and colors the different anatomical structures or lesion areas corresponding to the above-mentioned multiple color segmentation masks, so that each structure in the generated three-dimensional model is displayed with different colors.

[0015] The advantages that this invention can claim include: (1) automatically converting CT or MRI images into three-dimensional models with structural markers through artificial intelligence models, eliminating the tedious process of traditional manual drawing and reconstruction, and significantly reducing time and labor costs. (2) using deep learning models to identify and segment lesion areas, enabling accurate localization and three-dimensional reconstruction of microstructures, and assisting in clinical interpretation and surgical planning. Attached Figure Description

[0016] To make the above-described techniques and other objects, features, advantages, and embodiments of the present invention more apparent and understandable, the accompanying drawings are described below:

[0017] Figure 1 is a schematic diagram of the architecture of a medical image processing system according to one embodiment of the present invention.

[0018] Figure 2 is a schematic diagram of default part information according to one embodiment of the present invention.

[0019] Figure 3 is a schematic diagram of a medical image according to one embodiment of the present invention.

[0020] Figure 4 is a schematic diagram of the three-dimensional model made according to Figure 2.

[0021] Figure 5 is a schematic diagram of a medical image according to one embodiment of the present invention.

[0022] Figure 6 is a schematic diagram of the three-dimensional model made according to Figure 5.

[0023] Figure 7 is a schematic diagram of a medical-patient interaction scenario according to one embodiment of the present invention.

[0024] Figure 8 is a flowchart illustrating a medical image processing system according to one embodiment of the present invention. Detailed Implementation

[0025] To illustrate the various embodiments of the present invention in more detail, the following description is provided with reference to the accompanying drawings. It should be understood that when a component is referred to as "connected" or "set" on another component, it may mean that the component is directly located on the other component, or that there may be an intermediate component connecting the component to the other component. Conversely, when a component is referred to as "directly on another component" or "directly connected to another component," it is understood that this explicitly defines the absence of an intermediate component.

[0026] Referring to Figure 1, according to one embodiment, the present invention provides a medical image processing system 100, which specifically performs format parsing and data preprocessing on DICOM image format images obtained from computed tomography (CT) or magnetic resonance imaging (MRI), and converts them into a standardized spatial coordinate system (such as the RAS coordinate system) as the basis for subsequent image calculations and spatial correspondence. Furthermore, the system incorporates an image segmentation model to automatically identify and segment organ structures or lesion regions in the converted black-and-white medical image data, and generates corresponding masking marking information. Finally, the masking results are constructed into a three-dimensional geometric model using a triangular mesh reconstruction algorithm and exported in a three-dimensional stereolithography format to assist medical personnel in precise preoperative planning, thereby improving the efficiency and accuracy of clinical diagnosis and surgical navigation.

[0027] The medical image processing system 100 includes multiple hardware modules consisting of multiple electrically connected hardware circuits, including a communication module 110, a selection and comparison module 120, an image processing module 130, a three-dimensional modeling module 140, and an image mask database 150.

[0028] The communication module 110 is connected to the user device 160. The user can upload medical images obtained by computed tomography (CT) or magnetic resonance imaging (MRI) to this system through the host 170. The communication module 110 can also send the processed stereo model files back to the user device 160, so that the user can preview or download the model through the user device 160.

[0029] The selection comparison module 120 is communicatively connected to the aforementioned communication module 110. Upon receiving multiple medical images transmitted from the host 170 via the communication module 110, the user can select a conversion site through the host 170. These multiple medical images may include data from different imaging modalities (such as CT and MRI) and may be images of different scan areas of a single patient. When the user selects a conversion site from a specific image for modeling or analysis, the selection comparison module 120 automatically compares the conversion site with its internal default site information 122 to identify the corresponding organ or tissue structure. The default site information 122, as shown in Figure 2 and U.S. Provisional Application No. 63714878, includes 117 sites. Examples include the heart, lungs, colon, and bones in Figure 2.

[0030] The above default part information 122 includes the following 177 parts: spleen, kidney_right, kidney_left, gallbladder, liver, stomach, pancreas, adrenal_gland_right, adrenal_gland_left, lung_upper_lobe_left, lung_lower_lobe_left, lung_upper_lobe_right, lung_middle_lobe_right, lung_lower_lobe_right, esophagus, trachea, thyroid_gland, small_bowel, duodenum, colon, urinary_bladder, prostate, kidney_cyst_left, kidney_cyst_right, sacrum, vertebrae_S1, vertebrae_L5, vertebrae_L4, vertebrae_L3, vertebrae_L2, vertebrae_L1, vertebrae_T12, vertebrae_T11, vertebrae_T10, vertebrae_T9, vertebrae_T8, vertebrae_T7, vertebrae_T6, vertebrae_T5, vertebrae_T4, vertebrae_T3, vertebrae_T2, vertebrae_T1, vertebrae_C7, vertebrae_C6, vertebrae_C5, vertebrae_C4, vertebrae_C3, vertebrae_C2, vertebrae_C1, heart, aorta, pulmonary_vein, brachiocephalic_trunk, subclavian_artery_right, subclavian_artery_left, common_carotid_artery_right, common_carotid_artery_left, brachiocephalic_vein_left, brachiocephalic_vein_right, atrial_appendage_left, superior_vena_cava, inferior_vena_cava, portal_vein_and_splenic_vein,iliac_artery_left, iliac_artery_right, iliac_vena_left, iliac_vena_right, humerus_left, humerus_right, scapula_left, scapula_right, clavicula_left, clavicula_right, femur_left, femur_right, hi p_left, hip_right, spinal_cord, gluteus_maximus_left, gluteus_maximus_right, gluteus_medius_left, gluteus_medius_right, gluteus_minimus_left, gluteus_minimus_right, autochthon_left, autochth on_right, iliopsoas_left, iliopsoas_right, brain, skull, rib_left_1, rib_left_2, rib_left_3, rib_left_4, rib_left_5, rib_left_6, rib_left_7, rib_left_8, rib_left_9, rib_left_10, rib_left_11, rib_l eft_12,rib_right_1,rib_right_2,rib_right_3,rib_right_4,rib_right_5,rib_right_6,rib_right_7,rib_right_8,rib_right_9,rib_right_10,rib_right_11,rib_right_12,sternum,costal_cartilages. ,

[0031] The default site information 122 may include standard anatomical coordinates, structure name, shape features, typical location range, and possible boundary areas, etc., to improve the accuracy and flexibility of comparison. When the comparison result reaches a preset similarity threshold, it is determined that the converted site has been successfully identified as a specific organ or tissue site, which serves as the basis for subsequent segmentation and modeling by the image processing module 130.

[0032] The image processing module 130 is communicatively connected to the selection and comparison module 120. Upon receiving medical images transmitted by the selection and comparison module 120, it first determines the type of medical image to identify whether it is a computed tomography (CT) image or an magnetic resonance imaging (MRI) image. This determination step can be based on the imaging modality field in the DICOM file header information, which helps in selecting the most suitable image preprocessing and model inference parameters. After completing the image type identification, the image processing module 130 converts the received DICOM image into the standard medical image format NIfTI (Neuroimaging Informatics Technology Initiative). This conversion process includes reading the multi-section data series and spatial geometric information of the original DICOM image and merging them into a unified three-dimensional image file, while retaining the position and orientation information of each pixel in three-dimensional space. This NIfTI format not only supports efficient image processing and neural network model input, but is also compatible with subsequent coordinate transformation and segmentation masking operations.

[0033] Next, the image processing module 130 performs coordinate system transformation on the NIfTI format medical images, converting them to the RAS (Right-Anterior-Superior) spatial coordinate system format. This transformation step is based on the affine transformation matrix embedded in the NIfTI file. This matrix consists of pixel spacing, image orientation, and position information. Through matrix operations, the corresponding order and orientation of the image index axes are adjusted to correspond to the right, front, and top directions in the RAS coordinate system. This transformation unifies the positioning reference of medical images from different sources in three-dimensional space, ensuring structural consistency and orientation accuracy during subsequent automatic segmentation and 3D modeling, further improving the accuracy of deep learning model inferences and the feasibility of clinical applications.

[0034] After completing the RAS coordinate transformation, the image processing module 130 then uses a deep learning model to perform automatic segmentation on the transformed image. The deep learning model used in this invention is one of nnU-Net, nnTrans, or SUNETR, possessing the ability to dynamically configure the network architecture and adaptive training strategies, enabling the identification and segmentation of organ and lesion regions in medical images with different modalities and anatomical structures. In the modeling and identification process for each transformed site (e.g., liver, lung lobe, kidney, tumor lesion, etc.), the deep learning model is trained on at least 1000 preprocessed medical image data. This training process includes model inference, cross-validation, and optimization and iterative fine-tuning of the loss function. Performance is evaluated and adjusted using common medical image segmentation indicators such as Dice similarity coefficient and Hausdorff distance to ensure the stability and clinical application accuracy of the final model. The segmentation results are output in the form of a color segmentation mask. The color segmentation mask contains pixel or voxel labeling information for different tissue structures or lesion regions and is used by the subsequent 3D modeling and visualization modules.

[0035] Please refer to Figures 3 to 6. Figure 3 is a schematic diagram of a medical image according to one embodiment of the present invention. Figure 4 is a schematic diagram of a three-dimensional model made according to Figure 3. Specifically, Figure 3 is a medical image obtained from a full chest scan as described in U.S. Provisional Application No. 63714878; Figure 4 is a three-dimensional model 142 generated by three-dimensional modeling, i.e., identification marking, based on the obtained medical image as described in U.S. Provisional Application No. 63714878. Please refer to Figures 5 and 6. Figure 5 is a schematic diagram of a medical image according to one embodiment of the present invention. Figure 6 is a schematic diagram of a three-dimensional model made according to Figure 5. Specifically, Figure 5 is a medical image obtained from a full abdominal scan as described in U.S. Provisional Application No. 63714878; Figure 6 is a three-dimensional model 142 generated by three-dimensional modeling, i.e., identification marking, based on the obtained medical image as described in U.S. Provisional Application No. 63714878.

[0036] The 3D modeling module 140 is communicatively connected to the image processing module 130. After receiving the color segmentation mask output by the image processing module 130, it generates a corresponding 3D model based on the anatomical structure (i.e., organ) or lesion area marked by the color segmentation mask. The 3D modeling module 140 can apply volume reconstruction algorithms or surface reconstruction algorithms (e.g., Marching Cubes) to reconstruct the image slices into a 3D geometric structure and further generate a 3D model file, which is in stereolithography (STL) format.

[0037] After receiving the color segmentation mask output from the image processing module, the 3D modeling module 140 further marks and colors the different anatomical structures and lesion areas indicated by the mask. By assigning distinct colors to each tissue, organ, or lesion, the 3D model 142 generated by the 3D modeling module can more clearly present different structures, improving the intuitiveness and operational efficiency of clinical personnel in visual identification. The color configuration can also be customized according to user needs, enhancing its application flexibility. In Figure 4, the different shades of the 3D model 142 represent the anatomical structures or lesion areas represented by different colors.

[0038] The 3D modeling module 140 needs to integrate at least ten pre-processed medical images of the same transformation site in order to reconstruct a 3D model with sufficient depth of field and detail accuracy, so as to facilitate accurate presentation and interactive operation on the mixed reality device.

[0039] The completed 3D model, packaged in standard stereolithography (STL) format or an equivalent format, is transmitted to at least one user device via the aforementioned communication module. Upon receiving the model, the user device projects the 3D model onto the desired physical location based on the converted part information and the mixed reality positioning system, enabling applications such as preoperative simulation, surgical navigation, or doctor-patient interaction. Please refer to Figure 7, which illustrates a usage scenario of doctor-patient interaction according to one embodiment of the present invention. Figure 7 shows how medical personnel can project organ models onto physical space using a mixed reality device to assist patients in understanding the location of their symptoms and the proposed surgical strategy.

[0040] Furthermore, in surgical navigation scenarios, the user device, through a mixed reality positioning system and real-time tracking technology, accurately aligns the 3D model generated by the 3D modeling module 140 with the corresponding anatomical location on the patient's body surface or surgical site. In this way, medical personnel can refer to the 3D spatial distribution and morphological characteristics of organs, lesions, or important structures within the patient's body in real time within a mixed reality display during surgery, thereby improving positioning accuracy, avoiding critical tissues, and optimizing the surgical procedure. This application not only reduces surgical risks and time but also aids medical personnel in spatial judgment and clinical decision-making.

[0041] The image mask database 150 is communicatively connected to the image processing module 130. This image mask database 150 receives and stores medical images generated by the image processing module 130 and their corresponding color segmentation mask data. The stored data may include original DICOM images, converted NIfTI format images, RAS coordinate alignment information, and segmentation mask files inferred by a deep learning model. This image mask database 150 can serve as a data source for training and validating deep learning models, providing fully annotated and structurally consistent training and validation datasets to support the continuous optimization and automatic retraining mechanism of deep learning models (such as nnU-Net) in the image processing module 130, thereby improving the overall accuracy and generalization ability of the system in different organ and lesion identification tasks.

[0042] According to U.S. Provisional Application Nos. 63714875, 63714876, 63714878, and 63714879, the deep learning model of this invention uses DICOM format images from 2,500 CT / MRI cases of various body parts as training data. It is trained using the original image data of labeled body parts. After training, physicians make judgments, and the corrected results and adjustment parameters are recursively fed back into the model for further training. After training and adjustment, the accuracy of the deep learning model can reach over 85%.

[0043] Example 1

[0044] This embodiment 1 uses sarcopenia assessment as an example to illustrate a method for quantifying and judging muscle mass using a medical image processing system. First, the subject undergoes a CT or MRI scan of the whole body or a target area (such as the L3 lumbar spine, lower limb, or upper limb). The images are transmitted to the host computer via the system's communication module, and the selection and comparison module compares the converted area of ​​the image with the default area information to confirm the corresponding muscle region. Next, the image processing module performs preprocessing on the images (Z-score normalization for CT images; Z-score normalization for MRI images after removing extreme values), and uses a deep learning model (such as nnU-Net, nnTrans, or SUNETR) to identify and segment the muscle tissue, generating a color segmentation mask. Then, the 3D modeling module converts the color segmentation mask into a 3D model (STL format).

[0045] Muscle mass is calculated based on a three-dimensional model with a specific depth of field. Compared with the traditional method of estimating muscle mass using the cross-sectional area (CSA) of two-dimensional images, the muscle mass calculated by this system using a three-dimensional model can provide more complete and accurate muscle volume information, avoiding the deviation caused by a single slice.

[0046] Please refer to Figure 8, which is a flowchart of a medical image processing system 100 according to one embodiment of the present invention.

[0047] In step 200, after the comparison module 120 receives a plurality of medical images transmitted by the host 170, the host 170 selects a conversion site of the medical image.

[0048] In step 201, the comparison module 120 compares the converted site with a default site information to confirm that the converted site matches the corresponding organ or tissue site in the default site information.

[0049] In step 202, the image processing module 130 determines the type of the medical image, converts it into the format of the RAS coordinate system, and performs corresponding preprocessing according to the type of medical image.

[0050] In step 203, the image processing module 130 uses a deep learning model to automatically identify and segment the organ or lesion region of the converted medical image, and generates a corresponding color segmentation mask.

[0051] In step 204, the 3D modeling module 140 generates a corresponding 3D model based on the color segmentation mask.

[0052] In step 205, the 3D modeling module 140 marks and colors the different anatomical structures or lesion areas corresponding to the color segmentation mask, and generates a 3D model file.

[0053] In step 206, the 3D modeling module 140 transmits the 3D model file to the user device 160 through the communication module.

[0054] In step 207, the user device 160 projects the conversion part onto a desired physical location.

[0055] In summary, this invention performs standardized format conversion, deep learning segmentation, and geometric reconstruction and color labeling in the 3D modeling module for medical images, achieving fully automated operation. This allows medical personnel to obtain precisely labeled 3D visualizations of organs or lesions in a short time, improving the efficiency of medical image interpretation and enhancing the accuracy and real-time nature of preoperative planning, surgical guidance, and doctor-patient communication.

[0056] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0057] The above-described embodiments are merely illustrative examples and are not intended to limit the scope of the invention. Any equivalent modifications or alterations made to them shall not depart from the spirit and scope of the invention and shall be included in the claims of this application.

Claims

1. A medical image processing system, comprising a host computer composed of a plurality of electrically connected hardware circuits and at least one user device having augmented reality functionality, wherein the host computer is characterized in that... include: The communication module is used for communication connection to the at least one user device. The selection comparison module is connected to the communication module. After receiving multiple medical images transmitted by the host through the communication module, the host selects the conversion site of the medical images. The selection comparison module compares the conversion site with the default site information to confirm that the conversion site matches the corresponding organ or tissue site in the default site information. The image processing module, communicatively connected to the selection and comparison module, receives the medical images, determines the type of the medical images, converts the medical images into RAS (Right-Anterior-Superior) coordinate system format, and performs corresponding preprocessing according to the type of the medical images. The image processing module uses a deep learning model to perform organ identification and segmentation on the preprocessed medical images based on the transformation site information confirmed by the selection and comparison module, generating a plurality of corresponding color segmentation masks; and The 3D modeling module is connected to the image processing module and generates a corresponding 3D model based on the color segmentation masks, and generates a 3D model file. The 3D modeling module requires at least ten medical images to create a 3D model with a specific depth of field, and transmits the 3D model to the user device via the communication module, so that the user device can project the model onto the desired physical location based on the transformation area.

2. The medical image processing system as described in claim 1, wherein the selection and comparison module can identify the medical image as a computed tomography image or an magnetic resonance imaging image and then transmit it to the deep learning model for processing.

3. The medical image processing system as described in claim 1, wherein when the medical image is a computed tomography (CT) scan image, the standardization processing performed on the CT scan image by the image processing module is Z-score.

4. The medical image processing system as described in claim 1, wherein when the medical image is a magnetic resonance imaging (MRI) image, the standardization processing performed by the image processing module on the MRI image is in the following order: collecting all samples, removing extreme values, and Z-score.

5. The medical image processing system as described in claim 1, wherein the image processing module performs lesion region identification and segmentation based on a lesion identification feature, and generates a corresponding lesion identification and segmentation mask.

6. The medical image processing system of claim 1, wherein the deep learning model is one of nnU-Net, nnTrans, or SUNETR.

7. The medical image processing system of claim 1, wherein for each of the said conversion sites, a corresponding deep learning model is generated from at least 1000 of the said medical images through preprocessing, model inference, and model verification.

8. The medical image processing system as described in claim 1, wherein the stereo model file is in stereolithography (STL) format.

9. The medical image processing system of claim 1 further includes an image mask database, communicatively connected to the image processing module, for receiving and storing the medical image and the corresponding color segmentation mask from the image processing module, and providing the training dataset and validation dataset required by the deep learning model.

10. The medical image processing system as described in claim 1, wherein the three-dimensional modeling module further marks and colors the different anatomical structures or lesion areas corresponding to the color segmentation masks, so that each structure in the generated three-dimensional model is displayed with different colors.

Images