Image processing device, image processing method, and storage medium

The image processing apparatus adjusts inference image sizes based on mammary gland density indices to enhance the accuracy of lesion detection in chest images, addressing variability due to tissue density.

WO2026234788A1PCT designated stage Publication Date: 2026-11-12NEC CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2025/016990
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-09
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing image processing systems for chest images, such as mammographic images, struggle with varying lesion visibility due to tissue density, affecting the accuracy of computer-aided diagnosis.

Method used

An image processing apparatus and method that extracts mammary glands from chest images, calculates density indices for divided regions, and adjusts the size of inference images based on these indices to ensure accurate input into a lesion inference model.

Benefits of technology

This approach enhances the accuracy of computer-aided diagnosis by ensuring consistent input size and density for lesion inference, improving the reliability of lesion detection and diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2025016990_12112026_PF_FP_ABST
    Figure JP2025016990_12112026_PF_FP_ABST
Patent Text Reader

Abstract

An image processing device 1X includes a mammary gland extraction means 31X, an index calculation means 33X, and an inference image acquisition means 34X. The mammary gland extraction means 31X extracts a mammary gland from a chest image. The index calculation means 33X calculates an index related to the density of the mammary gland for each divided region obtained by dividing the chest image. The inference image acquisition means 34X cuts out an inference image from the chest image, which is input into a model that performs inference about lesions, on the basis of the index corresponding to the divided region where the inference image is cut out from the chest image.
Need to check novelty before this filing date? Find Prior Art

Description

Image Processing Apparatus, Image Processing Method, and Storage Medium

[0001] The present disclosure relates to the technical field of an image processing apparatus, an image processing method, and a storage medium that perform image processing.

[0002] Conventionally, image processing systems that process chest images such as mammographic images by computer-aided diagnosis are known. For example, Patent Document 1 discloses an information processing system that generates and generates oversight suppression data including image data indicating a region where a lesion is likely to be overlooked, based on a mammographic image and additional data.

[0003] Japanese Unexamined Patent Application Publication No. 2021-170174

[0004] In a region with dense tissue, the visibility of lesions is generally lower than in a region with sparse tissue, and even for images of the same size, the difficulty of inference regarding lesions varies depending on the density of the tissue. On the other hand, to ensure the reliability of computer-aided diagnosis, it is necessary to perform highly accurate inference regarding lesions regardless of the tissue density.

[0005] In view of the above-described problems, one of the objectives of the present disclosure is to provide an image processing apparatus, an image processing method, and a storage medium that perform processing suitable for ensuring the accuracy of computer-aided diagnosis related to chest images.

[0006] One aspect of the image processing apparatus is an image processing apparatus comprising: a mammary gland extraction unit configured to extract a mammary gland from a chest image; an index calculation unit configured to calculate an index related to the density of the mammary gland for each divided region obtained by dividing the chest image; and an inference image acquisition unit configured to cut out an inference image to be input to a model for performing inference regarding a lesion, from the chest image based on the index corresponding to the divided region from which the inference image is cut out from the chest image.

[0007] One aspect of the image processing method is an image processing method in which a computer extracts mammary glands from a chest image, calculates an index relating to the density of the mammary glands for each divided region of the chest image, and extracts an inference image from the chest image based on the index corresponding to the divided region from which the inference image is extracted, to be input into a model that performs inference about lesions.

[0008] One embodiment of the storage medium is a storage medium that stores a program that causes a computer to perform the process of extracting breast tissue from a chest image, calculating an index regarding the density of the breast tissue for each divided region of the chest image, and extracting an inference image from the chest image based on the index corresponding to the divided region from which the inference image is extracted, in order to input into a model that performs inference about lesions.

[0009] One example of the effects of this disclosure is that it becomes possible to obtain inference images suitable for ensuring the accuracy of computer-aided diagnosis of chest images.

[0010] This shows the schematic configuration of a chest imaging system. This diagram outlines the image processing performed by the image processing unit. This is an example of the functional blocks of the image processing unit's processor. This is an example of the display screen shown by the display device. This is an example of a flowchart showing the overview of the processing performed by the image processing unit. This is a block diagram of the image processing unit. This is an example of a flowchart executed by the image processing unit.

[0011] The following describes embodiments of the image processing apparatus, image processing method, and storage medium with reference to the drawings.

[0012] <First Embodiment> (1) System Configuration Diagram 1 shows the schematic configuration of the chest image processing system 100. The chest image processing system 100 performs computer-aided diagnosis (CAD: Computer-Aided Detection / Diagnosis) using chest images of the patient who is the subject of the examination. In this case, the chest image processing system 100 obtains highly accurate inference results from the model by cropping the chest image to an appropriate size for input into the model used for computer-aided diagnosis. The chest image processing system 100 mainly comprises an image processing device 1, a chest image generation device 2, a display device 3, and an operating device 4.

[0013] The image processing device 1 performs computer-aided diagnosis based on the patient's chest image supplied by the chest image generation device 2 and generates inference results regarding lesions. The image processing device 1 also performs display control to display information related to the inference results on the display device 3 and performs various processes based on operation signals received from the operation device 4.

[0014] The chest image generation device 2 generates a chest image of the patient's chest, which is the target area for imaging, and supplies the generated chest image to the image processing device 1. For example, the chest image generation device 2 generates a chest image (X-ray image of the breast) by irradiating the patient's breast with X-rays and detecting the X-rays that have passed through the breast. The chest image generation device 2 may be a mammography device based on the CR (Computed Radiography) method or a mammography device based on the FPD (Flat Panel Detector) method. The chest image generated by the chest image generation device 2 is digital data that can be analyzed by the image processing device 1. For example, pixels in the chest image are displayed as having larger pixel values ​​(brightness) the greater the X-ray absorption, and on the chest image, pixels with larger pixel values ​​are depicted as whiter. The chest images generated by the chest image generation device 2 are not limited to mammography images, but may also be ultrasound images, MRI (Magnetic Resonance Imaging) images, CT (Computed Tomography) images, chest X-ray images, angiography images, etc.

[0015] Furthermore, the chest image generation device 2 may supply the image processing device 1 with at least one of the patient information or examination information of the patient who is the subject of the chest image, in addition to the chest image. In this case, examples of patient information include patient identification information (patient ID) for identifying the patient, and other patient attribute information (name, gender, date of birth, etc.). Examples of examination information include examination identification information (examination ID) for identifying the examination, examination date and time information, and information regarding examination conditions (examination site, laterality (left, right), direction (e.g., vertical direction (CC), mediolateral oblique direction (MLO), compressed breast thickness)).

[0016] The display device 3 performs a predetermined display based on the display signal supplied from the image processing device 1. Examples of the display device 3 include displays such as CRTs (Cathode Ray Tubes) and LDCs (Liquid Crystal Displays), and projectors.

[0017] The operating device 4 generates operation signals based on operations performed by a user of the image processing device 1, such as a physician. Examples of the operating device 4 include buttons, keyboards, pointing devices such as mice, touch panels, remote controllers, voice input devices, and other arbitrary user interfaces.

[0018] Figure 1 also shows an example of the hardware configuration of the image processing device 1. The image processing device 1 mainly includes a processor 11, memory 12, and interface 13. Each of these elements is connected via a data bus 19.

[0019] The processor 11 executes predetermined processes by running programs and the like stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0020] Memory 12 is composed of various volatile memories used as working memory, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memory that stores information necessary for processing the image processing device 1. Memory 12 may also include an external storage device such as a hard disk connected to or built into the image processing device 1, or it may include a storage medium such as a removable flash memory. The memory 12 stores programs and other information necessary for the image processing device 1 to perform each of the processes in this embodiment.

[0021] In this embodiment, memory 12 contains parameters for constructing a model (also called the "inference model") that performs inferences related to computer-aided diagnosis based on chest images. Here, the inference model is a model (engine) that takes an image partially representing the chest as input and outputs an inference result regarding lesions in the input image. The inference result may be a detection result indicating whether or not a specific lesion has been detected, and / or the detected area, or it may be a qualitative diagnosis result of the lesion (including classification results regarding the type and malignancy of the lesion). The inference model is, for example, an inference model (including statistical models, the same applies hereinafter) having any architecture such as a neural network or a support vector machine. In this case, the inference model is a model that has learned the relationship between an image representing a part of the chest and a lesion in that image. The inference model is then trained using training data, and the parameters of the inference model obtained through training are pre-stored in memory 12. Here, the training data has, for example, multiple records that are pairs of an input image and ground truth data indicating the inference result that the inference model should output when that image is input. Representative examples of neural networks used in the architecture of inference models include, for example, Fully Convolutional Network, SegNet, U-Net, V-Net, Feature Pyramid Network, Mask R-CNN, and DeepLab. When the inference model is constructed using a neural network, memory 12 stores various parameters (including hyperparameters), such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weights of each element of each filter.

[0022] Hereafter, the images input to the inference model will also be referred to as "inference images." Inference images are partial images of the chest image extracted from the chest image generated by the chest image generation device 2. The inference model may be trained for each size of the inference image. In this case, "M" (where M is an integer of 2 or more) types of inference images of different sizes are generated, and M inference models are trained to take each of the M types of inference images as input, and the trained parameters are stored in memory 12.

[0023] Interface 13 performs interface operations between the image processing device 1 and external devices. For example, interface 13 is electrically connected to the chest image generation device 2, the display device 3, and the operating device 4. Interface 13 may be a communication interface such as a network adapter for wired or wireless communication with external devices, or it may be a hardware interface compliant with USB (Universal Serial Bus), SATA (Serial AT Attachment), etc. Interface 13 may also perform interface operations with external devices such as the chest image generation device 2, the display device 3, and the operating device 4 via a communication network such as the Internet.

[0024] The configuration of the chest image processing system 100 shown in Figure 1 is an example, and various modifications may be made.

[0025] For example, the image processing device 1 may be integrated with at least one of a chest image generation device 2, a display device 3, or an operating device 4. In another example, the image processing device 1 may include an audio output device that outputs information as sound. In yet another example, the image processing device 1 may be composed of multiple devices.

[0026] In another example, the chest image generation device 2 may be a storage device that stores pre-generated chest images of patients. In this case, the storage device is capable of data communication with the image processing device 1, and the image processing device 1 acquires, for example, a chest image of a patient specified by the operating device 4 from the storage device and executes processing according to this embodiment. The storage device may also be a server device that communicates data with the image processing device 1 via a communication network, or it may be built into the image processing device 1 as the memory 12 of the image processing device 1.

[0027] (2) Overview Diagram 2 is a diagram showing an overview of the image processing performed by the image processing device 1.

[0028] When the image processing device 1 acquires a chest image of a patient from the chest image generation device 2, it performs image analysis on the acquired chest image to extract (detect) the mammary glands (mammary gland tissue) in the chest image and generates an image representing the extracted mammary glands (also called a "mammary gland extraction image"). In this case, the mammary gland extraction image is, for example, a mask image of the mammary glands in which the pixels representing the mammary glands and the other pixels have different pixel values.

[0029] Next, the image processing device 1 divides the chest image and the extracted mammary gland image into equal parts "Nv" vertically and "Nh" horizontally (where Nv and Nh are integers of 2 or more) (i.e., divides them regularly in a grid pattern), setting up Nv × Nh congruent divided regions. The image processing device 1 then calculates the average mammary gland spacing in each of the Nv × Nh divided regions, which are divided by the vertical division number Nv and the horizontal division number Nh. In Figure 2, the vertical division number Nv is "4" and the horizontal division number Nh is "4", forming 16 divided regions. The image processing device 1 then calculates the average mammary gland spacing for each of the 16 divided regions. Here, as an example, the mammary gland spacing is represented by the number of pixels.

[0030] The image processing device 1 then determines the size (also called the "cut size") for each of the Nv × Nh divided regions to be cut from the chest image for inference, based on the average of the corresponding mammary gland spacing. In this case, the smaller the average mammary gland spacing (i.e., the denser the mammary glands), the smaller the cut size. Here, as an example, the image processing device 1 sets the cut size for each divided region to either a first size corresponding to "small" or a second size corresponding to "large," based on the average of the corresponding mammary gland spacing. Note that the cut size is not limited to a two-stage scale, but may also have three or more stages.

[0031] The image processing device 1 then inputs the inference image, which has been cut out from the chest image according to the cut size, into the inference model. The inference model then outputs an inference result regarding the lesion in the area corresponding to the input inference image. In this case, the image processing device 1 may obtain an inference image cut out from the entire chest image such that each pixel of the chest image is included in at least one of the inference images, or it may obtain an inference image corresponding to a part of the chest image.

[0032] In this way, the image processing device 1 acquires inference images cropped with a large crop size in areas where the mammary glands are widely spaced, and acquires inference images cropped with a small crop size in areas where the mammary glands are narrowly spaced. That is, the image processing device 1 determines the size of the inference images according to the degree of density of the mammary gland structure. By doing so, the image processing device 1 equalizes the number of mammary glands included in each inference image, thereby suitably ensuring the accuracy of the inference results output by the inference model.

[0033] (3) Functional block diagram 3 is an example of a functional block of the processor 11 of the image processing device 1. Functionally, the processor 11 of the image processing device 1 includes a chest image acquisition unit 30, a mammary gland extraction unit 31, a divided region setting unit 32, a mammary gland interval calculation unit 33, an inference image acquisition unit 34, an inference unit 35, and a display control unit 36. In Figure 3, blocks that exchange data are connected by solid lines, but the combination of blocks that exchange data is not limited to this. The same applies to the diagrams of other functional blocks described later.

[0034] The chest image acquisition unit 30 receives the patient's chest image generated by the chest image generation device 2 from the chest image generation device 2 via the interface 13. The chest image acquisition unit 30 then supplies the acquired chest image to the mammary gland extraction unit 31 and the display control unit 36.

[0035] The mammary gland extraction unit 31 extracts the mammary glands from the chest image supplied by the chest image acquisition unit 30 and generates a mammary gland extraction image representing the extracted mammary glands. The mammary gland extraction unit 31 then supplies the generated mammary gland extraction image and the chest image to the divided region setting unit 32. Alternatively, the mammary gland extraction unit 31 may supply the mammary gland extraction image to the display control unit 36.

[0036] Here, as an algorithm for extracting mammary gland structures from chest images, for example, the method described in the following paper can be used: Karla K Evans, Tamara Miner Haygood, Julie Cooper, Anne-Marie Culpan, Jeremy M Wolfe, "A half-second glimpse often lets radiologists identify breast cancer cases even when viewing the mammogram of the opposite breast", https: / / www.pnas.org / doi / full / 10.1073 / pnas.1606187113

[0037] The segmentation area setting unit 32 sets segmentation areas for the chest image and the mammary gland extract image. In this case, the segmentation area setting unit 32 sets Nv × Nh grid-like segmentation areas on the chest image and the mammary gland extract image using a predetermined number of vertical divisions Nv and a predetermined number of horizontal divisions Nh. The segmentation area setting unit 32 supplies the chest image and the mammary gland extract image with the segmentation areas set to the mammary gland interval calculation unit 33.

[0038] The mammary gland interval calculation unit 33 calculates the average mammary gland interval for each divided region and supplies the average mammary gland interval calculated for each divided region to the inference image acquisition unit 34. In this case, the mammary gland interval calculation unit 33 aggregates the number of mammary glands in the mammary gland extraction image for each of the Nv × Nh divided regions and calculates the average mammary gland interval based on the aggregation result.

[0039] For example, the mammary gland spacing calculation unit 33 approximates (replaces) the mammary glands in the mammary gland extraction image with longitudinal, transverse, or diagonal vectors of a predetermined length (unit length). Then, for each of the Nv × Nh divided regions, the mammary gland spacing calculation unit 33 calculates the average transverse mammary gland spacing based on the number of longitudinal vectors and the transverse length of the divided region, and calculates the average longitudinal mammary gland spacing based on the number of transverse vectors and the longitudinal length of the divided region. In this case, the mammary gland spacing calculation unit 33 may consider one diagonal vector as equivalent to 0.5 longitudinal vectors and 0.5 transverse vectors, and then aggregates the number of longitudinal and transverse vectors. In this way, the mammary gland spacing calculation unit 33 obtains the average transverse mammary gland spacing and the average longitudinal mammary gland spacing for each divided region. The average mammary gland spacing is an example of an indicator of the density of the mammary glands.

[0040] The inference image acquisition unit 34 determines the cut size for each divided region based on the average of the mammary gland intervals calculated by the mammary gland interval calculation unit 33, and acquires an inference image cut out from the chest image using the determined cut size. In this case, the inference image acquisition unit 34 determines the horizontal cut size based on the average of the horizontal mammary gland intervals, for example, and the vertical cut size based on the average of the vertical mammary gland intervals. The inference image acquisition unit 34 supplies the inference image to the inference unit 35. The inference image acquisition unit 34 also supplies information such as the determined cut size to the display control unit 36.

[0041] Here, we will explain a specific example of obtaining an inference image using the determined crop size.

[0042] In the first example, the inference image acquisition unit 34 extracts an inference image from each of the Nv × Nh divided regions according to the determined cut size, and inputs each extracted inference image into the inference model to obtain the inference result corresponding to each inference image. In this case, if the size of the divided region is a multiple of the cut size in both the vertical and horizontal directions, the inference image acquisition unit 34 acquires each image obtained by dividing each divided region according to the cut size as an inference image. If the size of the divided region is not a multiple of the cut size in at least one of the vertical or horizontal directions, the inference image acquisition unit 34 may resize the divided region or otherwise perform other actions to acquire the inference image based on the cut size. Methods for determining the resizing ratios in the vertical and horizontal directions to match the image size to a desired size are disclosed in International Publication WO2020 / 166247. In another example where the size of the divided region is not a multiple of the cut size in at least one of the vertical or horizontal directions, the inference image acquisition unit 34 may allow overlapping of the regions to be cut in the divided region and acquire the extracted inference image.

[0043] In the second example, the inference image acquisition unit 34 considers the designated region, which is specified as a region suspected of having a lesion, as the target for inference regarding the lesion, and acquires an inference image cut out from the designated region according to the determined cut size. The inference image acquisition unit 34 then inputs each of the cut-out inference images into the inference model to obtain the inference result corresponding to each inference image. The inference image acquisition unit 34 may also accept input from the operating device 4 to specify a region suspected of having a lesion on the chest image displayed on the display device 3, or it may detect a region suspected of having a lesion as a designated region using a lesion detection model different from the inference model described above. The latter lesion detection model is, for example, a machine learning model that acquires a chest image as input and outputs information indicating the lesion region of the chest image (e.g., a mask image or bounding box), and the learned parameters are stored in the memory 12, etc. Furthermore, if the designated region overlaps with multiple divided regions (i.e., spans multiple divided regions), the inference image acquisition unit 34 acquires an inference image for each divided region, cut out from the designated region according to the cut size set for that divided region. If the specified area is not a multiple of the cut size, the inference image acquisition unit 34 may, as in the first example, crop the inference image while allowing overlap of the cropping areas, or it may resize the specified area and then crop the inference image using the corresponding cut size.

[0044] The inference unit 35 inputs each inference image supplied from the inference image acquisition unit 34 to the inference model, thereby acquiring the inference result output by the inference model, and supplies the inference result for each inference image to the display control unit 36. In this case, in the first example, the inference unit 35 resizes (that is, normalizes) the size of the inference image supplied from the inference image acquisition unit 34 so as to reach a predetermined size (predetermined vertical and horizontal sizes) that matches the input format for the inference model, and inputs each inference image resized to the predetermined size matching the input format for the inference model into the inference model. In the second example, the inference unit 35 uses M inference models that have been trained for each size of input image. In this case, the inference unit 35 classifies each inference image supplied from the inference image acquisition unit 34 into one of the M inference models according to its size, and inputs each inference image into the classified inference model, thereby acquiring the inference result corresponding to each inference image. Note that the inference result corresponding to each inference image is associated with position information indicating the position of each inference image on the chest image.

[0045] The display control unit 36 controls display by the display device 3 based on information supplied from the chest image acquisition unit 30, the mammary gland extraction unit 31, the inference image acquisition unit 34, the inference unit 35, and the like. In this case, the display control unit 36 generates a display signal and supplies the generated display signal to the display device 3, thereby causing the display device 3 to display the inference results and the like related to the chest image. A display example displayed by the display device 3 under the control of the display control unit 36 will be described later.

[0046] Note that each component of the chest image acquisition unit 30, the mammary gland extraction unit 31, the divided region setting unit 32, the mammary gland interval calculation unit 33, the inference image acquisition unit 34, the inference unit 35, and the display control unit 36 can be realized, for example, by the processor 11 executing a program. Alternatively, the required programs may be recorded in any non-volatile storage medium in advance and installed as needed to implement each of the components. It should be noted that at least a part of each of these components is not limited to being implemented by program-based software, and may be implemented by any combination of hardware, firmware, and software, or the like. Further, at least a part of each of these components may be implemented using a user-programmable integrated circuit such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, a program composed of the above-described respective components may be implemented using this integrated circuit. Further, at least a part of each component may be constituted by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). As such, each component may be implemented by various types of hardware. The above also applies to other embodiments described later. Furthermore, each of these components may be implemented by the cooperation of a plurality of computers using, for example, cloud computing technology.

[0047] (4) Display Example FIG. 4 shows an example of a display screen that the display control unit 36 causes the display device 3 to display. The display control unit 36 generates display information based on the information supplied from the other processing blocks shown in FIG. 3, and outputs the display information to the display device 3. The display control unit 36 transmits the display information to the display device 3, thereby causing the display device 3 to display the display screen shown in FIG. 4.

[0048] The display control unit 36 provides a chest image display area 70, a mammary gland structure display area 71, an inference result display area 72, and an adjustment button 73 on the display screen.

[0049] Here, the display control unit 36 ​​displays the chest image acquired by the chest image acquisition unit 30 from the chest image generation device 2 in the chest image display area 70. The display control unit 36 ​​also refers to the inference result for each inference image supplied by the inference unit 35 and recognizes the inference result indicating the presence of a lesion area suspected to have a disease condition "○×". Therefore, the display control unit 36 ​​overlays a frame 75A on the chest image that highlights the area corresponding to the inference image for which an inference result indicating the presence of a lesion area suspected to have a disease condition "○×" was obtained.

[0050] Furthermore, the display control unit 36 ​​displays the extracted breast image in the breast structure display area 71, based on the extracted breast image supplied from the breast extraction unit 31 and the information supplied from the inference image acquisition unit 34, with the cutout lines (dashed lines) of the inference image clearly indicated. The cutout lines here correspond to the boundary lines between the inference images and are arranged at intervals based on the cut size determined by the inference image acquisition unit 34. In addition, the display control unit 36 ​​overlays a frame 75B on the extracted breast image that highlights the area corresponding to the inference image from which an inference result indicating the presence of a lesion area suspected to be diseased "○×" has been obtained.

[0051] Furthermore, the display control unit 36 ​​displays information based on the inference results obtained from each inference image in the inference result display area 72. Here, since the display control unit 36 ​​obtained an inference result indicating the presence of a lesion area suspected to have the disease condition "○×", the corresponding frames 75A and 75B display information indicating that the area is suspected to have the disease condition "○×".

[0052] Furthermore, when the display control unit 36 ​​detects that the adjustment button 73 has been selected, it accepts user input to change the cropping position (i.e., cut size) of the inference image. For example, when the display control unit 36 ​​detects that the adjustment button 73 has been selected, it accepts a change in the cropping position of the inference image by dragging and dropping the cropping line on the mammary gland structure display area 71. In another example, the display control unit 36 ​​accepts the selection of any inference image identified by the cropping line on the mammary gland structure display area 71 and a change in the cropping position (i.e., cut size) of the selected inference image. In yet another example, the display control unit 36 ​​displays a pull-down menu or the like that allows the user to specify the amount to increase or decrease the cut size in the vertical and horizontal directions, respectively.

[0053] Then, when the display control unit 36 ​​detects an input specifying a change in the cropping position of the inference image, the inference image acquisition unit 34 acquires the inference image again using the changed cropping position, and the inference unit 35 regenerates the inference result by executing the inference model using the newly acquired inference image. As a result, the inference result is obtained based on the cropping position of the inference image that has been accurately corrected based on user feedback, and the inference accuracy of the inference model is improved. Then, the display control unit 36 ​​updates the display of the chest image display area 70, the mammary gland structure display area 71, and the inference result display area 72 based on these inference images and inference results.

[0054] In this manner, the display control unit 36 ​​receives user input regarding the adjustment of the cut size (i.e., the cropping position of the inference image) on the display screen, and if a change in the cut size is specified, it displays the inference results based on the changed cut size. In addition, the display control unit 36 ​​may display the cropping lines superimposed on the chest image instead of superimposing them on the mammary gland extraction image, or in addition to the above.

[0055] (7) The processing flow diagram 5 is an example of a flowchart that shows an overview of the processing performed by the image processing device 1.

[0056] First, the image processing device 1 acquires a chest image of the patient's chest from the chest image generation device 2 (step S11). Next, the image processing device 1 extracts the mammary gland structure from the chest image acquired in step S11 (step S12). As a result, the image processing device 1 generates a mammary gland extract image representing the mammary gland structure.

[0057] The image processing device 1 then sets grid-like division regions for the chest image and the mammary gland extract image, and calculates the average spacing of mammary glands for each set division region (step S13). The image processing device 1 aggregates the mammary glands represented in the mammary gland extract image for each of the Nv × Nh division regions, and calculates the average of the mammary gland spacing in the vertical and horizontal directions for each division region.

[0058] Next, the image processing device 1 determines a cut size for each divided region based on the average spacing of the mammary glands (step S14). In this case, for example, the image processing device 1 determines a vertical cut size corresponding to the average spacing of the mammary glands in the vertical direction, and a horizontal cut size corresponding to the average spacing of the mammary glands in the horizontal direction.

[0059] Then, the image processing device 1 obtains an inference image cut out from the chest image based on the cut size (step S15). In this case, for example, the image processing device 1 considers the entire chest image as the region to be inferred, and obtains an inference image cut out according to the corresponding cut size for each divided region obtained by dividing the chest image. In another example, the image processing device 1 considers a specified region of a part of the chest image as the region to be inferred, and obtains an inference image by cutting out the specified region according to the cut size corresponding to the divided region that overlaps with the specified region.

[0060] Then, the image processing device 1 performs inference using the inference images (step S16). In this case, for example, the image processing device 1 uses an inference model corresponding to the size of the inference images to obtain the inference results output by the corresponding inference model from each inference image. In another example, the image processing device 1 may resize the inference images to a size that can be input to the inference model, and obtain the inference results output by the inference model from each inference image whose size has been standardized by the resizing.

[0061] Next, the image processing device 1 displays information regarding the inference results obtained in step S16 on the display device 3 (step S17). In this case, for example, the image processing device 1 displays the display screen shown in Figure 4 or the like on the display device 3 based on the information obtained in each step.

[0062] According to the first embodiment, the image processing device 1 acquires inference images cropped with a large crop size in areas where the mammary glands are widely spaced, and acquires inference images cropped with a small crop size in areas where the mammary glands are narrowly spaced. This equalizes the number of mammary glands included in each inference image, thereby suitably ensuring the accuracy of the inference results output by the inference model. Therefore, it is possible to improve the accuracy of lesion area estimation and the accuracy of mammary gland image diagnosis.

[0063] <Second Embodiment> Figure 6 is a block diagram of the image processing device 1X. The image processing device 1X includes a mammary gland extraction means 31X, an index calculation means 33X, and an inference image acquisition means 34X. The image processing device 1X may be composed of multiple devices.

[0064] The mammary gland extraction means 31X extracts mammary glands from a chest image. The mammary gland extraction means 31X can be, for example, the mammary gland extraction unit 31 in the first embodiment.

[0065] The index calculation means 33X calculates an index related to the density of mammary glands for each divided region obtained by dividing the chest image. The index calculation means 33X can be, for example, the mammary gland spacing calculation unit 33 in the first embodiment.

[0066] The inference image acquisition means 34X extracts an inference image from the chest image based on an index corresponding to a segmented region from which the inference image is extracted, which is then used as input to a model that performs inference about lesions. The inference image acquisition means 34X can be, for example, the inference image acquisition unit 34 in the first embodiment.

[0067] Figure 6 is an example of a flowchart showing the procedure of processing performed by the image processing device 1X. The mammary gland extraction means 31X extracts mammary glands from the chest image (step S21). Next, the index calculation means 33X calculates an index related to the density of mammary glands for each divided region obtained by dividing the chest image (step S22). The inference image acquisition means 34X extracts an inference image to be input to a model that performs inference about lesions from the chest image, based on the index corresponding to the divided region from which the inference image is extracted from the chest image (step S23).

[0068] According to the second embodiment, the image processing device 1X can extract an inference image from the chest image with an appropriate size according to the density of the mammary glands, thereby obtaining an inference image suitable for inferring about lesions.

[0069] In each of the embodiments described above, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer, such as a processor. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transient computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transient computer-readable media can supply programs to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0070] Furthermore, some or all of the above embodiments (including modified versions, hereinafter the same) may also be described as follows, but are not limited to the following. Also, some or all of the configurations described in the appendix dependent on Appendix 1 may be dependent on Appendix 11 and Appendix 12 in the same manner as the dependencies of the appendix dependent on Appendix 1. Moreover, not limited to the devices, methods, and storage media described in the appendix, some or all of the configurations described in the appendix may also be dependent on methods, various hardware, software, various recording means for recording software (including storage media), or systems, without departing from the above embodiments.

[0071] [Note 1] An image processing apparatus comprising: a mammary gland extraction means for extracting mammary glands from a chest image; an index calculation means for calculating an index relating to the density of the mammary glands for each divided region obtained by dividing the chest image; and an inference image acquisition means for extracting an inference image from the chest image based on the index corresponding to the divided region from which the inference image is extracted, to be input to a model for performing inference about lesions. [Note 2] The image processing apparatus according to Note 1, wherein the inference image acquisition means determines the size of the inference image to be extracted from the chest image based on the index corresponding to the divided region from which the inference image is extracted. [Note 3] The image processing apparatus according to Note 1, wherein the index calculation means calculates the average interval of the mammary glands for each divided region as the index. [Note 4] The image processing apparatus according to Note 3, wherein the inference image acquisition means increases the size of the inference image to be extracted from the chest image as the average corresponding to the divided region from which the inference image is extracted increases. [Note 5] The image processing apparatus according to Note 2, wherein the index calculation means calculates the index for the vertical and horizontal directions for each divided region obtained by dividing the chest image, and the inference image acquisition means determines the size in the vertical direction based on the vertical index and determines the size in the horizontal direction based on the horizontal index. [Note 6] The image processing apparatus according to Note 1, wherein the inference image acquisition means extracts the inference image from a specified region of the chest image based on the index corresponding to the divided region that overlaps with the specified region. [Note 7] The image processing apparatus according to Note 1, further comprising an inference means for acquiring an inference result output by the model by inputting the inference image to the model. [Note 8] The image processing apparatus according to Note 7, further comprising a display control means for displaying information regarding the inference result on a display device. [Note 9] The image processing apparatus according to Note 8, wherein the display control means causes the display device to display a cutout line indicating the position from which the inference image was cut out of the chest image on the chest image or an image representing the mammary glands extracted from the chest image.[Note 10] The image processing apparatus according to Note 9, wherein the display control means, when it detects an input instructing a change in the position of the cutting line, causes the display device to display the inference result based on the inference image cut out based on the position after the change has been reflected. [Note 11] An image processing method executed by a computer, comprising: extracting mammary glands from a chest image; calculating an index relating to the density of the mammary glands for each divided region obtained by dividing the chest image; and cutting out an inference image from the chest image based on the index corresponding to the divided region from which the inference image is cut out, to be input to a model that performs inference about lesions. [Note 12] A storage medium storing a program that causes a computer to execute the process of extracting mammary glands from a chest image; calculating an index relating to the density of the mammary glands for each divided region obtained by dividing the chest image; and cutting out an inference image from the chest image based on the index corresponding to the divided region from which the inference image is cut out, to be input to a model that performs inference about lesions.

[0072] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the structure and details of the present invention can be made as understood by those skilled in the art within the scope of the present invention. That is, the present invention includes the full disclosure, including the claims, and of course, various modifications and alterations that those skilled in the art could make in accordance with the technical idea. Furthermore, the above-mentioned patent and non-patent disclosures are incorporated herein by reference.

[0073] 1. 1X Image Processing Device 2. Chest Image Generation Device 3. Display Device 4. Operating Device 11. Processor 12. Memory 13. Interface 100. Chest Image Processing System

Claims

1. An image processing apparatus comprising: a mammary gland extraction means for extracting mammary glands from a chest image; an index calculation means for calculating an index relating to the density of the mammary glands for each divided region obtained by dividing the chest image; and an inference image acquisition means for extracting an inference image from the chest image based on the index corresponding to the divided region from which the inference image is extracted, to be input to a model for performing inference about lesions.

2. The image processing apparatus according to claim 1, wherein the inference image acquisition means determines the size of the inference image to be extracted from the chest image based on the index corresponding to the divided region from which the inference image is extracted.

3. The image processing apparatus according to claim 1, wherein the index calculation means calculates the average interval between mammary glands for each divided region as the index.

4. The image processing apparatus according to claim 3, wherein the inference image acquisition means increases the size of the inference image extracted from the chest image as the average corresponding to the divided region from which the inference image is extracted increases.

5. The image processing apparatus according to claim 2, wherein the index calculation means calculates the index for the vertical and horizontal directions for each divided region obtained by dividing the chest image, and the inference image acquisition means determines the size in the vertical direction based on the vertical index and determines the size in the horizontal direction based on the horizontal index.

6. The image processing apparatus according to claim 1, wherein the inference image acquisition means extracts the inference image from a designated region of the chest image based on the index corresponding to the divided region that overlaps with the designated region.

7. The image processing apparatus according to claim 1, further comprising inference means for obtaining an inference result output by the model by inputting the inference image to the model.

8. The image processing apparatus according to claim 7, further comprising display control means for displaying information relating to the inference result on a display device.

9. The image processing apparatus according to claim 8, wherein the display control means causes the display device to display a cutout line indicating the position from which the inference image was cut out of the chest image on the chest image or an image representing the mammary glands extracted from the chest image.

10. The image processing apparatus according to claim 9, wherein when the display control means detects an input instructing a change in the position of the cutting line, it causes the display device to display the inference result based on the inference image cut out based on the position after the change has been reflected.

11. An image processing method performed by a computer, comprising: extracting mammary glands from a chest image; calculating an index relating to the density of the mammary glands for each divided region of the chest image; and extracting an inference image from the chest image based on the index corresponding to the divided region from which the inference image is extracted, to be input into a model for inferring lesions.

12. A storage medium storing a program that causes a computer to perform the process of extracting breast tissue from a chest image, calculating an index regarding the density of the breast tissue for each divided region of the chest image, and extracting inference images from the chest image based on the index corresponding to the divided region from which the inference images are extracted, in order to input these inference images into a model for inferring lesions.