Loss function optimization system for 3D medical image segmentation deep learning model

Abstract

A loss function optimization system for a 3D medical image segmentation deep learning model, according to the present invention, comprises: an image input unit for receiving an input of a 3D medical image; a masking unit for masking a partial region of the input 3D medical image; a deep learning model unit for performing masked image modeling mechanism deep learning on the masked 3D medical image by using a deep learning model; a pixel area reconstruction unit that calculates a pixel-based loss function during learning of the deep learning model unit and performs backpropagation; a frequency domain reconstruction unit that calculates a frequency-based loss function during learning of the deep learning model unit and performs backpropagation; a loss function optimization unit that calculates a spectral distance scaling loss function in which the pixel-based loss function and the frequency-based loss function are combined, and performs backpropagation on the deep learning model of the deep learning model unit; and a pre-trained model generation unit for generating a pre-trained model that has been trained through the spectral distance scaling loss function.

Description

Loss Function Optimization System for 3D Medical Image Segmentation Deep Learning Models

[0001] The present invention relates to a loss function optimization system for a 3D medical image segmentation deep learning model, and more specifically, to a loss function optimization system for a 3D medical image segmentation deep learning model that enhances the encoder representation of the deep learning model and enables balanced learning of low-frequency and high-frequency details by using a spectral distance scaling loss function that integrates information in the frequency domain with a pixel-based method, instead of a general pixel-based loss function used when training a 3D medical image segmentation deep learning model.

[0002]

[0003] 3D medical imaging segmentation refers to a technology that identifies and separates specific structures, organs, lesions, etc., from 3D medical imaging data such as CT, MRI, or ultrasound.

[0004] Among segmentation tasks, medical imaging presents a unique challenge in accurately distinguishing anatomical structures.

[0005] These structures include blood vessels, lesions, small tumors, etc., and have complex details expressed primarily by high-frequency components.

[0006] Unlike natural images, medical images generally involve the task of analyzing a 3D volume (e.g., MRI and computed tomography [CT] scans) composed of stacks of 2D slices.

[0007] Each voxel (3D pixel) contains intensity values ​​corresponding to various tissues, making the data sparse and requiring significant computational resources to process.

[0008] Furthermore, advanced equipment and labor-intensive annotation work result in excessive costs, further limiting the utilization of this data.

[0009] One of the promising methods to solve the dataset shortage problem is self-supervised learning (SSL) using Masked Image Modeling (hereinafter referred to as 'MIM').

[0010] This approach involves training a model on a pre-text task using a large amount of unlabeled data, and then fine-tuning it for a downstream task of interest.

[0011] MIM masks parts of an image and trains the model to predict the hidden content.

[0012] Models trained with this method tend to generalize well to new data because they learn rich representations during the masking process.

[0013] The success of MIM is primarily attributed to vision transformers that can mask images by processing them into sequences of visual tokens.

[0014] This is similar to how masked language models process text.

[0015] However, an important difference between MIM and masked language modeling is that while linguistic tokens have meaning within a predefined vocabulary, visual tokens are essentially just a grid of local pixels with few restrictions.

[0016] Despite this complexity, pioneering research has demonstrated that a simple random masking strategy and a method of predicting the RGB values ​​of raw pixels are effective for learning transferable feature representations.

[0017] Nevertheless, the effectiveness of this approach often follows the neural scaling principle, where the performance of self-supervised learning (SSL) improves as the amount of training data increases.

[0018] Medical datasets generally consist of thousands to tens of thousands of 3D volumes, whereas natural image datasets consist of millions of images, so there is a significant difference in scale.

[0019] In addition, natural images are standardized to RGB with values ​​in the range of [0, 255], whereas medical images (e.g., CT scans) are generally single-channel images with values ​​in the range of [-1000, 1000].

[0020] Therefore, it is not suitable to use models pre-trained on natural images for this data.

[0021] Furthermore, while using only pixel reconstruction loss during pre-training may be beneficial in the spatial domain, it leads to biased learning in the frequency domain, hindering the ability to acquire enhanced representations.

[0022] In particular, Fourier phase information contains high-dimensional meaning and exhibits strong characteristics for transfer learning.

[0023] Low-dimensional spectra can change significantly without affecting high-dimensional meaning.

[0024]

[0025] This variability is influenced by factors such as sensors or lighting and affects spectral characteristics, so learning-based models must account for this variability.

[0026] In medical data, details such as lesions or small organs play an important role within the overall volume.

[0027] Therefore, learning biased toward a specific frequency band can have a negative impact on downstream segmentation tasks.

[0028] To overcome these limitations, the research community has developed techniques to improve the capabilities of models by manipulating frequency components to more accurately predict masked regions.

[0029] These efforts aim to compensate for small dataset sizes by improving pre-trained models to capture details.

[0030]

[0031] Focus Frequency Loss (FFL) measures the Euclidean distance between spectral vectors and enhances image reconstruction and synthesis through weighted frequency loss.

[0032] Reducing this difference improves representation learning capabilities, thereby enhancing the performance of downstream tasks.

[0033] However, FFL still has room for improvement.

[0034] Using FFL for pre-training yields sharper reconstruction results than using only pixel loss (Ll Loss), but contours or edges remain incomplete in some areas of the image.

[0035] This problem occurs because the magnitude of the low-frequency band, including the DC component, is very large when applying the Discrete Fourier Transform (DFT) to the reconstructed image.

[0036] This causes the dynamic range of the spectral distance to become excessively large, and the large size dominates the frequency loss value in backpropagation, resulting in spectral bias.

[0037] Convergence in neural networks exhibits frequency bias, which refers to the tendency of deep neural networks to align the objective function from low to high frequencies during training.

[0038] Recent research presents results that contradict but support this trend.

[0039] In other words, neural networks tend to focus on learning high-frequency components in semantic tasks such as recognition, but appear to prioritize low- or mid-frequency components in synthesis tasks.

[0040] This phenomenon is evident in the fact that while the model can easily outline low-frequency structures, capturing high-frequency details remains challenging.

[0041] As prior art, Korean Registered Patent Publication No. 10-2623979 (published on January 11, 2024) proposed "a masking-based deep learning image classification system and method thereof."

[0042] However, the above-mentioned prior art classifies masked product images by preprocessing the original image, and since the meaning of masking refers to a class, it is different from the masked image modeling (MIM) technique intended to be used in the present invention.

[0043] As another prior art, Korean Published Patent Application No. 10-2024-0123218 (published August 13, 2024) proposes "an electronic device for generating a pre-trained model for restoring a masked image and a method of operation thereof."

[0044] The aforementioned other conventional technology uses a masked image modeling (MIM) technique but adopts only a pixel-based loss function (L1 Loss), which leads to biased learning in the frequency domain as described above, thereby hindering the ability to acquire enhanced representations.

[0045]

[0046] The present invention was devised to solve the problems of the prior art as described above. The objective of the present invention is to provide a loss function optimization system for a 3D medical image segmentation deep learning model that applies a spectral distance scaling loss function (SDSL) to ensure a balance of frequency learning compared to the prior art that adopts only a pixel-based loss function (L1 Loss), and can improve the ability of the deep learning model to capture low-frequency and high-frequency details.

[0047] To achieve the above objectives, the loss function optimization system of a 3D medical image segmentation deep learning model according to the present invention comprises: an image input unit that receives a 3D medical image; a masking unit that masks a portion of the input 3D medical image; a deep learning model unit that performs deep learning of the masked 3D medical image using a deep learning model; a pixel region reconstruction unit that calculates a pixel-based loss function and performs backpropagation during the training of the deep learning model unit; a frequency domain reconstruction unit that calculates a frequency-based loss function and performs backpropagation during the training of the deep learning model unit; a loss function optimization unit that calculates a spectrum distance scaling loss function combining the pixel-based loss function and the frequency-based loss function and performs backpropagation to the deep learning model of the deep learning model unit; and a pre-training model generation unit that generates a pre-training model trained through the spectrum distance scaling loss function.

[0048] In addition, the loss function optimization system of the 3D medical image segmentation deep learning model according to the present invention is characterized by further including a downstream task unit that tunes and utilizes the pre-trained model.

[0049] Here, the image input unit comprises: an input module that is composed of a plurality of 2D slice stacks to form a 3D volume by combining the same, and receives a medical image without segmentation annotation labels as the 3D medical image; a preprocessing module that preprocesses the input 3D medical image; and a ground truth image conversion module that converts the preprocessed 3D medical image into a ground truth 3D medical image so that it can be used in the deep learning model unit.

[0050] Here, the preprocessing module is characterized by normalizing adjustable values ​​including the size, coordinate system, and brightness of the input 3D medical image to a defined standard, and if the size of the input 3D medical image is smaller than the defined standard, padding is performed to fill the insufficient size, and the image is converted into a tensor for deep learning.

[0051] Here, the masking unit is characterized by comprising: a patch segmentation module that divides the input 3D medical image into 3D patch units; a patch selection module that randomly selects a predetermined ratio of patches among all patches divided through the patch segmentation module; and a patch masking module that masks the values ​​of the patches selected by the patch selection module.

[0052] Here, the deep learning model unit is characterized by including: a prediction image generation module that generates a prediction 3D medical image by computing the input masked 3D medical image; an output channel number setting module that sets the final output channel number to a predetermined number; and a learning method application module that learns through dynamic learning rate change that reduces the learning rate at each step interval and performs mixed precision learning.

[0053] Here, the pixel region reconstruction unit is characterized by including a pixel loss function calculation module that calculates the pixel-based loss function through regression learning that calculates the mean absolute error for each pixel by comparing the correct answer 3D medical image converted from the image input unit and the predicted 3D medical image generated from the deep learning model unit.

[0054] Here, the frequency domain reconstruction unit is characterized by comprising: a frequency conversion module that converts the correct 3D medical image converted from the image input unit and the predicted 3D medical image generated from the deep learning model unit into a correct frequency value and a predicted frequency value, respectively; and a frequency loss function calculation module that calculates the frequency-based loss function through regression learning that calculates an error at the frequency position of the converted frequency value.

[0055] Here, the frequency conversion module is characterized by performing wavelet transform on the correct 3D medical image converted from the image input unit and the predicted 3D medical image generated from the deep learning model unit, respectively, to separate them into a predetermined number of low-frequency regions and high-frequency regions, and converting the outputs of the low-frequency regions and high-frequency regions separated into a predetermined number of regions into a correct frequency value and a predicted frequency value, respectively, through Fourier transform.

[0056] Here, the frequency loss function calculation module is characterized by calculating the average absolute error at each frequency position of the converted correct frequency value and the predicted frequency value, scaling the calculated average absolute error to a value suitable for 3D medical image segmentation through a hyperbolic tangent function, and calculating a frequency-based loss function through regression learning in which the sum of the scaled average absolute errors is the loss value.

[0057] Here, the loss function optimization unit calculates a spectrum distance scaling loss function that combines the pixel-based loss function and the frequency-based loss function through regression learning that calculates an error by comparing the pixel and frequency values ​​of the correct answer 3D medical image with the pixel and frequency values ​​of the predicted 3D medical image, respectively, and performs backpropagation to the deep learning model of the deep learning model unit.

[0058] Here, the pre-training model generation unit is characterized by saving the final deep learning model, which has completed 3D medical image reconstruction training from the deep learning model unit to the pixel region reconstruction unit, frequency region reconstruction, and loss function optimization unit, as the deep learning model of the deep learning model unit including weights and biases.

[0059] Here, the downstream task unit applies a deep learning model, which has completed 3D medical image reconstruction training including weights and biases of the pre-trained model generation unit, as a deep learning model for a downstream task by changing the head portion, wherein the number of layer outputs of the deep learning model for the downstream task can be set differently from the number of downstream tasks, and is characterized by training the deep learning model for the downstream task after preprocessing the input dataset.

[0060]

[0061] With the above configuration, the loss function optimization system of the 3D medical image segmentation deep learning model according to the present invention has the advantage of being able to provide a useful pre-trained model for various tasks such as classification, detection, and segmentation of medical images having three-dimensional volumes like CT and MRI.

[0062]

[0063] FIG. 1 is a configuration diagram of a loss function optimization system for a 3D medical image segmentation deep learning model according to an embodiment of the present invention.

[0064] FIG. 2 is a configuration diagram of an image input unit according to an embodiment of the present invention.

[0065] FIG. 3 is a configuration diagram of a masking unit according to an embodiment of the present invention.

[0066] FIG. 4 is a configuration diagram of a deep learning model unit according to an embodiment of the present invention.

[0067] FIG. 5 is a configuration diagram of a main part according to an embodiment of the present invention.

[0068] FIGS. 6 to 8 are conceptual diagrams of the mechanism of a loss function optimization system for a 3D medical image segmentation deep learning model according to an embodiment of the present invention.

[0069] FIGS. 9 to 11 are experimental graphs and experimental diagrams of a loss function optimization system for a 3D medical image segmentation deep learning model according to an embodiment of the present invention.

[0070]

[0071] The present invention provides a loss function optimization system for a 3D medical image segmentation deep learning model, characterized by comprising: an image input unit that receives a 3D medical image; a masking unit that masks a portion of the input 3D medical image; a deep learning model unit that learns the masked 3D medical image through a deep learning model; a pixel region reconstruction unit that calculates a pixel-based loss function and performs backpropagation during the learning of the deep learning model unit; a frequency domain reconstruction unit that calculates a frequency-based loss function and performs backpropagation during the learning of the deep learning model unit; a loss function optimization unit that calculates a spectrum distance scaling loss function combining the pixel-based loss function and the frequency-based loss function and performs backpropagation to the deep learning model of the deep learning model unit; and a pre-training model generation unit that generates a pre-training model learned through the spectrum distance scaling loss function.

[0072]

[0073] Hereinafter, a loss function optimization system for a 3D medical image segmentation deep learning model according to the present invention will be described with reference to an embodiment illustrated in the drawings.

[0074] FIG. 1 is a configuration diagram of a loss function optimization system for a 3D medical image segmentation deep learning model according to an embodiment of the present invention, FIG. 2 is a configuration diagram of an image input unit according to an embodiment of the present invention, FIG. 3 is a configuration diagram of a masking unit according to an embodiment of the present invention, FIG. 4 is a configuration diagram of a deep learning model unit according to an embodiment of the present invention, FIG. 5 is a configuration diagram of a main unit according to an embodiment of the present invention, FIG. 6 to FIG. 8 are conceptual diagrams of the mechanism of a loss function optimization system for a 3D medical image segmentation deep learning model according to an embodiment of the present invention, and FIG. 9 to FIG. 11 are experimental graphs and experimental diagrams of a loss function optimization system for a 3D medical image segmentation deep learning model according to an embodiment of the present invention.

[0075] Referring to FIG. 1, a loss function optimization system for a 3D medical image segmentation deep learning model according to an embodiment of the present invention includes an image input unit (10), a masking unit (20), a deep learning model unit (30), a pixel region reconstruction unit (40), a frequency region reconstruction unit (50), a loss function optimization unit (60), a pre-training model generation unit (70), and a downstream task unit (80).

[0076] The above image input unit (10) is configured to receive 3D medical images, and in one embodiment of the present invention, as shown in FIG. 2, it includes an input module (11), a preprocessing module (12), and a correct image conversion module (13).

[0077] The above input module (11) is configured to receive a medical image as a 3D medical image, which is composed of a plurality of 2D slice stacks and forms a 3D volume by combining them.

[0078] The above medical image may be a medical image composed of thousands to tens of thousands of 3D volumes, such as a CT or MRI image.

[0079] Meanwhile, in the present invention, since the deep learning model unit (30) performs self-supervised learning (SSL), segmentation annotation (ononotation) labels are not required for the medical image input to the image input unit (10).

[0080] The above preprocessing module (12) is configured to preprocess the input 3D medical image, and in one embodiment of the present invention, it is configured to normalize adjustable values ​​including the size, coordinate system, and brightness of the input 3D medical image to a predetermined standard.

[0081] General images have values ​​in the range [0, 255], so preprocessing is performed accordingly, but medical images are single-channel images with values ​​in the range [-1000, 1000], so preprocessing suitable for 3D medical images is used.

[0082] For example, the above preprocessing module (12) reduces the width, height, and volume of the input image by half, converts from the x, y, z axis coordinate system to the RAS coordinate system to convert to the RAS axis, changes the medical image brightness value (Hounsfield Unit) from [-1000, 1000] to [-160, 240] and then normalizes it to [0, 1].

[0083] Meanwhile, the above preprocessing module (12) performs padding to fill in the missing size when the size of the input 3D medical image is smaller than the specified standard, and converts it into a tensor for deep learning training.

[0084] For example, if the size (96,96,96) is insufficient, padding is performed, the result is converted into a tensor, and finally the data is loaded into main memory.

[0085] The above correct answer image conversion module (13) is configured to convert the above preprocessed 3D medical image into a correct answer 3D medical image so that it can be used in the deep learning model unit (30).

[0086] The converted 3D medical image as described above is used in the mask image modeling (MIM) mechanism of the deep learning model unit (30) to be described later.

[0087] The masking unit (20) is configured to mask a portion of the input 3D medical image, and in one embodiment of the present invention, as shown in FIG. 3, it includes a patch segmentation module (21), a patch selection module (22), and a patch masking module (23).

[0088] The patch splitting module (21) is configured to divide the input 3D medical image into 3D patch units.

[0089] Here, each patch can have a size of, for example, (16,16,16).

[0090] The patch selection module (22) is configured to randomly select a predetermined proportion of patches from the total patches divided through the patch division module (21).

[0091] For example, the patch selection module (22) can randomly select 40% of the total patches.

[0092] The patch masking module (23) is configured to mask the value of a patch selected by the patch selection module (22).

[0093] For example, the value of a randomly selected patch corresponding to 40% is changed to a random number value between 0 and 1 to mask it.

[0094] In the above case, 60% of the normal patches and 40% of the masked patches are used as inputs to the deep learning model part (30) described later.

[0095] The deep learning model unit (30) is configured to deep learn the masked 3D medical image using a deep learning model and a masked image modeling (MIM) mechanism.

[0096] In one embodiment of the present invention, the deep learning model unit (30) includes a prediction image generation module (31), an output channel number setting module (32), and a learning method application module (33), as shown in FIG. 4.

[0097] The above prediction image generation module (31) is configured to generate and output a prediction 3D medical image by performing calculations on the input masked 3D medical image.

[0098] The above output channel number setting module (32) is configured to set the final number of output channels to a predetermined number.

[0099]

[0100] In one embodiment of the present invention, the output channel number setting module (32) can set the final output channel number to 1.

[0101] The above-mentioned learning method application module (33) is configured to learn through dynamic change of learning rate that reduces the learning rate at each step interval and to perform mixed precision learning.

[0102] The above learning method application module (33) can improve learning efficiency by learning based on 16-bit rather than 32-bit, so that the activation function layer is computed based on 16-bit and the weight update based on gradient is computed based on 32-bit.

[0103] The pixel region reconstruction unit (40) is configured to calculate a pixel-based loss function (L1 Loss) and perform backpropagation during the training of the deep learning model unit (30).

[0104] In one embodiment of the present invention, the pixel region reconstruction unit (40) includes a pixel loss function calculation module (41) as shown in FIG. 5.

[0105] The pixel loss function calculation module (41) is configured to calculate the pixel-based loss function (L1 Loss) through regression learning that calculates the average absolute error for each pixel by comparing the correct answer 3D medical image converted from the image input unit (10) with the predicted 3D medical image generated from the deep learning model unit (30).

[0106] The above frequency domain reconstruction unit (50) is configured to calculate a frequency-based loss function (Frequency Loss) and perform backpropagation during the training of the above deep learning model unit (30).

[0107] In one embodiment of the present invention, the frequency domain reconstruction unit (50) includes a frequency conversion module (51) and a frequency loss function calculation module (52) as shown in FIG. 5.

[0108] The frequency conversion module (51) is configured to convert the correct 3D medical image converted from the image input unit (10) and the predicted 3D medical image generated from the deep learning model unit (30) into a correct frequency value and a predicted frequency value, respectively.

[0109] In one embodiment of the present invention, the frequency conversion module (51) is configured to perform a wavelet transform based on Haar wavelets on the correct 3D medical image converted from the image input unit (10) and the predicted 3D medical image generated from the deep learning model unit (30), respectively, to separate them into a predetermined number of low-frequency regions and high-frequency regions, and to convert the outputs of the low-frequency regions and high-frequency regions separated into a predetermined number of regions into correct frequency values ​​and predicted frequency values, respectively, through a Fourier transform.

[0110] For example, the frequency conversion module (51) can separate the correct 3D medical image and the predicted 3D medical image into one low-frequency region and seven high-frequency regions, respectively, and convert them into a correct frequency value and a predicted frequency value, respectively, through a Fourier transform.

[0111] The above frequency loss function calculation module (52) is configured to calculate the frequency-based loss function (Frequency Loss) through regression learning that calculates the error at the frequency position of the converted frequency value.

[0112] In one embodiment of the present invention, the frequency loss function calculation module (52) is configured to calculate the average absolute error at each frequency position of the converted correct frequency value and the predicted frequency value, scale the calculated average absolute error to a value suitable for 3D medical image segmentation through a hyperbolic tangent function, and calculate a frequency-based loss function (Frequency Loss) through regression learning in which the sum of the scaled average absolute errors is the loss value.

[0113] For example, the average absolute error is calculated at each frequency position of the above-mentioned total of 8 converted correct frequency values ​​and total of 8 predicted frequency values, the calculated average absolute error is scaled to a value between [0.1] using a hyperbolic tangent, and a frequency-based loss function (Frequency Loss) can be calculated by performing regression learning with the sum of the total of 8 scaled mean squared errors as the loss value.

[0114] FIG. 6 illustrates a conceptual diagram of a system in which the image input unit (10), masking unit (20), deep learning model unit (30), pixel area reconstruction unit (40), and frequency area reconstruction unit (50) are organically combined.

[0115] Meanwhile, FIG. 7 is a detailed illustration of the frequency domain reconstruction unit (50), illustrating the process of converting the deep learning model into the frequency domain when generating a reconstructed prediction image (from left to center) and the process of converting the original correct image into the frequency domain (from right to center), and the 2D representation of calculating the error between them when converting to the frequency domain.

[0116] Figure 8 is a 3D medical image that adds a detailed representation of the frequency domain reconstruction unit (50) to the system of Figure 6.

[0117] Referring to FIGS. 6 to 9, the masked image modeling (MIM)-based self-supervised learning (SSL) approach according to an embodiment of the present invention separates a reconstructed volume into high-frequency and low-frequency bands of multiple resolutions through a three-dimensional (3D) wavelet transform.

[0118] After that, this framework is converted into the frequency domain using the 3D Discrete Fourier Transform (3D-DFT).

[0119] The distance between the converted original and the reconstructed volume is calculated for each frequency and scaled by the tangent hyperbolic (tanh) function to ensure that specific frequency bands do not disproportionately affect the loss.

[0120] The above loss function optimization unit (60) calculates a Spectral Distance Scaling Loss (SDSL) that combines the pixel-based loss function (L1 Loss) and the frequency-based loss function (Frequency Loss), and performs backpropagation to the deep learning model of the deep learning model unit.

[0121] In one embodiment of the present invention, the loss function optimization unit (60) calculates a spectrum distance scaling loss function (SDSL) that combines the pixel-based loss function (L1 Loss) and the frequency-based loss function (Frequency Loss) through regression learning that calculates an error by comparing the pixel and frequency values ​​of the correct answer 3D medical image with the pixel and predicted frequency values ​​of the predicted 3D medical image, respectively, and is configured to perform backpropagation to the deep learning model of the deep learning model unit (30).

[0122] In the present invention, by using the spectral distance scaling loss function (SDSL) as described above in a deep learning model, the encoder representation of the deep learning model is enhanced, and low-frequency and high-frequency details can be learned in a balanced manner.

[0123] The above pre-training model generation unit (70) is configured to generate a pre-training model learned through the above-mentioned spectral distance scaling loss function (SDSL).

[0124] In one embodiment of the present invention, the pre-training model generation unit (70) stores the final deep learning model, which has completed 3D medical image reconstruction learning from the deep learning model unit (30) to the pixel area reconstruction unit (40), frequency area reconstruction (50), and loss function optimization unit (60), as the deep learning model of the deep learning model unit (30), including weights and biases.

[0125] Meanwhile, the above-mentioned pre-training model generation unit (70) can verify results in the 3D medical image reconstruction task itself and can be used as a pre-training model capable of performing various downstream tasks.

[0126] The above downstream task unit (80) is configured to tune and utilize the above pre-trained model to solve various problems thereafter.

[0127] In one embodiment of the present invention, the downstream task unit (80) applies the deep learning model, which has completed 3D medical image reconstruction learning including weights and biases of the pre-training model generation unit (70), as a deep learning model for downstream tasks by changing the head portion.

[0128] Here, the number of layer outputs of the deep learning model for the downstream task can be set differently from the number of downstream tasks, which can be configured by freezing the weights and biases of the pre-training model generation unit (70) as much as desired or initializing them to Gaussian random, and can be configured to train the deep learning model for the downstream task after preprocessing the input dataset.

[0129] For example, the downstream task unit (80) loads the weights of a pre-trained deep learning model into main memory, applies the weights to the encoder and decoder of the deep learning model, and the output of the final layer of the deep learning model may be set to be equal to or different from the number of classes of the downstream task.

[0130] Input image preprocessing can be performed by changing the medical image brightness values ​​(Hounsfield Unit) to [―1000,1000] ~ [-170,250] and normalizing them to [0,1], and removing regions with 0 values ​​from the outer edges of the input image.

[0131] Deep learning models can be computed using stochastic depth, which performs an affine transformation that mirrors a portion of the input image horizontally and vertically with a 10 percent probability and moves to the next layer without computing the inner layer of the deep learning model with a 10 percent probability.

[0132] The learning loss function can be trained using the Dice loss, which is the harmonic mean of sensitivity and precision.

[0133]

[0134] [Experimental Example]

[0135] FIGS. 9 to 11 illustrate experimental graphs and experimental diagrams of a loss function optimization system for a 3D medical image segmentation deep learning model according to an embodiment of the present invention.

[0136] To verify the performance of the Spectral Distance Scaling Loss Function (SDSL), the model was pre-trained using 3,647 CT volumes from 11 datasets, including the Abdomen lk and Word datasets.

[0137] The objective function in the frequency domain in experiments including masked image modeling (MIM) uses the Focal Frequency Loss (FFL).

[0138] w in FFL is the spectrum weight matrix w, which is a [0,1] weight matrix that is adaptively generated based on the spectrum distance.

[0139] The upper graph of Figure 9 visualizes the frequency distribution after preprocessing (96X96X96) the BTCV dataset and performing a Fourier transform on it.

[0140] The figure is 96x96, and each value represents the frequency magnitude of the corresponding area in the image. It was shifted so that the DC value is centered, and although three axes (H, W, and D) should originally be used, the D axis was averaged to visualize the data using only the H and W axes.

[0141] Looking at the results, it can be confirmed that the gap between the low frequency (DC value, middle) and the rest of the range decreases when scaled with tanh.

[0142] The lower graph of Figure 9 visualizes the difference (L1 Norm) between the pre-trained model and the original data in the frequency domain.

[0143] Since it is a difference value, a smaller value indicates a better result, and it can be confirmed that it shows a trend similar to the segmentation results (Fourier weight scaling > Wavelet > Focal Fourier Loss > L1).

[0144]

[0145] Meanwhile, looking at Figure 10, if loss function optimization is not used, the range is large as shown on the left, so the actual reconstruction task is not performed well, and if loss function optimization (SDSL) is used, it can be seen that the frequency domain distribution is even as shown on the right.

[0146] In other words, by minimizing the frequency distance [FFL] through FFL, it can be confirmed that improved performance is shown compared to previous reconstruction results and downstream tasks that were concentrated on the low-pass band.

[0147] Meanwhile, the Beyond the Cranial Vault (BTCV) [4], Medical Segmentation Decathlon (MSD) Spleen [2], Multimodality Whole Heart Segmentation MM-WHS

[0047] , and CT-ORG

[0048] datasets were used to evaluate the system according to the present invention.

[0148] The evaluation results of the system according to the present invention are as shown in Table 1 below.

[0149]

[0150]

[0151] Looking at Table 1, The Beyond the Cranial Vault (BTCV) abdomen challenge dataset consists of 30 abdominal scans, and 13 organs were labeled under the supervision of radiologists at Vanderbilt University Medical Center.

[0152] The BTCV Segmentation experiment recorded a dice score of 84.17%, showing better results than DAE, the current SOTA model for the BTCV challenge leaderboard, and achieved the highest performance compared to previous MIM-based SSL.

[0153] In particular, compared to SwinUNETR SSL [tang et al], which had the highest score among other methods, stomach recorded a dice score 2.78% higher.

[0154] In addition, the highest dice score was observed in the segmentation of the gallbladder, inferior vena cava, and left adrenal gland.

[0155] In summary, in the downstream segmentation task, it can be seen that the Spectral Distance Scaling Loss Function (SDSL) recorded the highest Dice score across all datasets compared to previous Masked Image Modeling (MIM) based methods.

[0156]

[0157] The loss function optimization system of the 3D medical image segmentation deep learning model described above and illustrated in the drawings is merely one embodiment for implementing the present invention and should not be interpreted as limiting the technical scope of the present invention. The scope of protection of the present invention is determined solely by the matters described in the following claims, and embodiments that are improved and modified without departing from the gist of the present invention shall be deemed to fall within the scope of protection of the present invention insofar as they are obvious to those skilled in the art to which the present invention belongs.

[0158] [Explanation of the symbol]

[0159] 10 Video Input Section

[0160] 11 Input Module

[0161] 12 Preprocessing Modules

[0162] 13 Answer Video Conversion Module

[0163] 20 Masking section

[0164] 21 Patch Split Module

[0165] 22 Patch Selection Modules

[0166] 23 Patch Masking Module

[0167] 30 Deep Learning Model Section

[0168] 31 Predictive Image Generation Module

[0169] 32 Output Channel Count Setting Module

[0170] 33 Learning Method Application Module

[0171] 40-pixel area reconstruction section

[0172] L1 Loss pixel-based loss function

[0173] 41 Pixel Loss Function Calculation Module

[0174] 50 Frequency Domain Reconstruction Unit

[0175] Frequency Loss, a frequency-based loss function

[0176] 51 Frequency conversion module

[0177] 52 Frequency Loss Function Calculation Module

[0178] 60 Loss function optimization section

[0179] SDSL Spectral Distance Scaling Loss Function

[0180] 70 Pre-training model generation section

[0181] 80 Downstream Task Force

Claims

1. Image input unit for receiving 3D medical images; A masking unit that masks a portion of the input 3D medical image; A deep learning model unit that trains the above-mentioned masked 3D medical image using a deep learning model and a masked image modeling mechanism; A pixel region reconstruction unit that calculates a pixel-based loss function and performs backpropagation during the training of the deep learning model unit; A frequency domain reconstruction unit that calculates a frequency-based loss function and performs backpropagation during the training of the deep learning model unit; A loss function optimization unit that calculates a spectrum distance scaling loss function by combining the pixel-based loss function and the frequency-based loss function, and performs backpropagation to the deep learning model of the deep learning model unit; and A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by including: a pre-training model generation unit that generates a pre-training model learned through the above-mentioned spectral distance scaling loss function.

2. In Paragraph 1, The loss function optimization system of the above 3D medical image segmentation deep learning model is, A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by further including a downstream task unit that tunes and utilizes the above-mentioned pre-trained model.

3. In Paragraph 1, The above image input unit is, An input module composed of a plurality of 2D slice stacks to form a 3D volume by their combination, and receiving a medical image without segmentation annotation labels as the 3D medical image; and A preprocessing module for preprocessing the input 3D medical image; and A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by including: a ground truth image conversion module that converts the preprocessed 3D medical image into a ground truth 3D medical image so that it can be used in the deep learning model unit.

4. In Paragraph 3, The above preprocessing module is, The adjustable values ​​of the above-mentioned input 3D medical image, including size, coordinate system, and brightness, are normalized to conform to the defined specifications, A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by performing padding to fill the insufficient size when the size of the input 3D medical image is smaller than a predetermined standard, and converting it into a tensor for deep learning training.

5. In Paragraph 1, The above masking part is, A patch segmentation module that divides the input 3D medical image into 3D patch units; A patch selection module that randomly selects a predetermined proportion of patches from the total patches divided through the patch splitting module; and A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by including a patch masking module that masks the value of a patch selected by the patch selection module.

6. In Paragraph 3, The above deep learning model unit is, A prediction image generation module that generates a predicted 3D medical image by performing calculations on the input masked 3D medical image; Output channel number setting module that sets the final number of output channels to a fixed number; A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by including a learning method application module that learns through dynamic learning rate change that decreases the learning rate at each step interval and performs mixed precision learning.

7. In Paragraph 6, The pixel area reconstruction unit above is, A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by including: a pixel loss function calculation module that calculates a pixel-based loss function through regression learning that calculates the mean absolute error for each pixel by mutually comparing the correct answer 3D medical image converted from the image input unit and the predicted 3D medical image generated from the deep learning model unit.

8. In Paragraph 6, The above frequency domain reconstruction unit is, A frequency conversion module that converts the correct answer 3D medical image converted from the image input unit and the predicted 3D medical image generated from the deep learning model unit into a correct answer frequency value and a predicted frequency value, respectively; and A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by including: a frequency loss function calculation module that calculates the frequency-based loss function through regression learning to find the error at the frequency position of the converted frequency value.

9. In Paragraph 8, The above frequency conversion module is, The correct answer 3D medical image converted from the above image input unit and the predicted 3D medical image generated from the above deep learning model unit are each subjected to wavelet transform to separate them into a predetermined number of low-frequency and high-frequency regions, and A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by converting the outputs of the low-frequency region and the high-frequency region, each separated into a predetermined number, into a correct frequency value and a predicted frequency value, respectively, through a Fourier transform.

10. In Paragraph 9, The above frequency loss function calculation module is, Calculate the average absolute error at each frequency position of the above-mentioned converted correct frequency value and predicted frequency value, and The above-mentioned average absolute error is scaled to a value suitable for 3D medical image segmentation using a hyperbolic function (hyperbolic tangent), and A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by calculating a frequency-based loss function through regression learning in which the sum of the scaled mean absolute errors is used as the loss value.

11. In Paragraph 9, The above loss function optimization unit is, A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by calculating a spectral distance scaling loss function that combines the pixel-based loss function and the frequency-based loss function through regression learning that calculates an error by mutually comparing the pixel and correct frequency values ​​of the correct answer 3D medical image with the pixel and predicted frequency values ​​of the predicted 3D medical image, respectively, and performing backpropagation on the deep learning model of the deep learning model unit.

12. In Paragraph 2, The above-mentioned pre-training model generation unit is, A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by storing the final deep learning model, which has completed 3D medical image reconstruction training from the deep learning model unit to the pixel region reconstruction unit, frequency region reconstruction, and loss function optimization unit, as the deep learning model of the deep learning model unit including weights and biases.

13. In Paragraph 12, The above-mentioned downstream task unit is, The deep learning model for 3D medical image reconstruction, including the weights and biases of the pre-trained model generation unit mentioned above, is applied as a deep learning model for downstream tasks by modifying the head part, The number of layer outputs of the deep learning model for the above downstream task can be set differently from the number of downstream tasks, and A loss function optimization system for a 3D medical image segmentation deep learning model, characterized by preprocessing an input dataset and then training a deep learning model for the downstream task.

Images