Image display device, method, and program

Abstract

In the present invention, a processor: derives a three-dimensional partial image that includes at least a bile duct from three-dimensional images of one or a plurality of types that include at least a liver; acquires a two-dimensional radiation image that includes at least the liver; divides the liver included in the three-dimensional image into a plurality of liver segments; aligns the two-dimensional radiation image and the three-dimensional partial image; and displays a superimposed image formed by superimposing the three-dimensional partial image, in which the liver segments are applied to the bile duct, on the two-dimensional radiation image.

Description

Image display apparatus, method and program

[0001] The present disclosure relates to an image display apparatus, method and program.

[0002] Endoscopic Retrograde Cholangiopancreatography (ERCP) is an examination and treatment method in which an endoscope is advanced from the mouth to the duodenum, a thin tube is inserted through the outlet (papilla) of the bile duct and pancreatic duct to inject a contrast agent, abnormalities of the gallbladder and bile duct are examined under X-ray images, a guide wire is inserted to remove stones, and a stent is inserted.

[0003] On the other hand, a technique has been proposed that displays tubular tissues contained in an MR image on an X-ray image by accurately aligning a pre-captured MR image on an X-ray image captured during an operation (see Patent Document 1 (Japanese Unexamined Patent Application Publication No. 2025-001462)). By using the technique described in Patent Document 1, the running state of a tubular tissue such as a bile duct can be confirmed on an X-ray image when performing ERCP.

[0004] On the other hand, in ERCP, a procedure is performed by inserting a tube or a guide wire from the papilla into the bile duct. However, with the technique described in Patent Document 1, it is difficult to recognize in which region of the liver the tube or guide wire is inserted.

[0005] The present disclosure has been made in view of the above circumstances, and an object thereof is to enable easy confirmation of which region of the liver a guide wire or the like has entered when performing ERCP.

[0006] An image display apparatus according to the present disclosure includes a processor, wherein the processor derives a three-dimensional partial image including at least a bile duct from one or more types of three-dimensional images including at least the liver, acquires a two-dimensional radiation image including at least the liver, divides the liver included in the three-dimensional image into a plurality of liver segments, aligns the two-dimensional radiation image and the three-dimensional partial image, and displays a superimposed image obtained by superimposing the three-dimensional partial image in which the liver segments are reflected on the bile duct onto the two-dimensional radiation image.

[0007] In the image display device according to this disclosure, the processor may derive an anatomical structure image including at least one anatomical structure other than the bile duct from one or more types of three-dimensional images, and superimpose the anatomical structure image onto a two-dimensional radiographic image.

[0008] In the image display device according to this disclosure, the anatomical image may include the pancreatic duct.

[0009] In the image display device according to this disclosure, the processor may detect the region of a surgical instrument in a two-dimensional radiographic image, distinguish the region of the surgical instrument from other regions, and display a superimposed image.

[0010] In the image display device according to this disclosure, the processor may extract abnormal tissue contained in the liver from a three-dimensional image and highlight the pathway along the bile duct from the papilla, which is the opening of the bile duct, to the abnormal tissue in the three-dimensional partial image.

[0011] In the image display device according to this disclosure, a two-dimensional radiation image is acquired by a radiation imaging device in which the radiation source position can be changed, and the processor may superimpose the three-dimensional partial image onto the two-dimensional radiation image by changing at least one of the viewpoint position, orientation and magnification of the three-dimensional partial image according to the radiation source position and radiation irradiation angle of the two-dimensional radiation image.

[0012] In the image display device according to this disclosure, the processor may extract corresponding feature points from one or more types of three-dimensional images and two-dimensional radiation images, and align one or more types of three-dimensional partial images and two-dimensional radiation images based on the positional relationship between the feature points extracted in one or more types of three-dimensional images and the feature points extracted in two-dimensional radiation images.

[0013] In the image display device according to this disclosure, the processor may derive a first superimposed image and a second superimposed image having parallax from the superimposed image, and display the first superimposed image and the second superimposed image on a stereoscopic display.

[0014] The image display method according to this disclosure involves a computer deriving a three-dimensional partial image including at least the bile duct from one or more types of three-dimensional images including at least the liver, obtaining a two-dimensional radiographic image including at least the liver, dividing the liver included in the three-dimensional image into multiple liver segments, aligning the two-dimensional radiographic image and the three-dimensional partial image, and displaying a superimposed image in which the three-dimensional partial image, in which the liver segments are reflected in the bile ducts, is superimposed on the two-dimensional radiographic image.

[0015] The image display program according to this disclosure causes a computer to perform the following steps: a procedure for deriving a three-dimensional partial image including at least the bile duct from one or more types of three-dimensional images including at least the liver; a procedure for acquiring a two-dimensional radiographic image including at least the liver; a procedure for dividing the liver included in the three-dimensional image into multiple liver segments; a procedure for aligning the two-dimensional radiographic image and the three-dimensional partial image; and a procedure for displaying a superimposed image obtained by superimposing the three-dimensional partial image, in which the liver segments are reflected in the bile ducts, onto the two-dimensional radiographic image.

[0016] Furthermore, the technology disclosed herein may be applied to program products.

[0017] According to this disclosure, when performing ERCP, it is possible to easily confirm which region of the liver the guidewire or the like is inserted into.

[0018] A diagram showing the schematic configuration of an image display system to which the image display device according to the embodiment of this disclosure is applied. A diagram showing the hardware configuration of the image display device according to this embodiment. A diagram showing the functional configuration of the image display device according to this embodiment. A diagram showing the functional configuration of the output unit. A schematic diagram showing a three-dimensional bile duct image. A diagram for explaining liver regions. A diagram showing an example of a bile duct image. A diagram showing a superimposed image. A diagram showing a superimposed image including an image of an endoscope inserted up to the papilla of the duodenum. A diagram showing a superimposed image with a path displayed. A diagram showing a superimposed image with a guide wire highlighted. A flowchart showing the processing performed in this embodiment. A diagram showing another example of the hardware configuration of the image display device according to this embodiment.

[0019] Embodiments of this disclosure will be described below with reference to the drawings. First, the configuration of an image display system to which the image display device according to this embodiment is applied will be described. Figure 1 is a diagram showing the schematic configuration of the image display system. In the image display system shown in Figure 1, the image display device 1, the shooting device 2, and the image storage device 3 according to this embodiment are connected in a manner that enables communication via a wired or wireless network 4. The network 4 is, for example, a LAN (Local Area Network) and a WAN (Wide Area Network).

[0020] The image display device 1 is, for example, a personal computer, on which the image display program according to this embodiment is installed. The image display device 1 may be directly connected to the imaging device 2.

[0021] The imaging device 2 acquires a two-dimensional X-ray image of the subject by photographing the subject during surgery. In this embodiment, for ERCP, an X-ray image of the abdomen, including the liver of the subject, is acquired by X-ray imaging of the abdomen. The imaging device 2 is equipped with an X-ray source and a radiation detector that generates a radiographic image of the subject. Furthermore, the imaging device 2 can move the X-ray source during surgery to change the direction of X-ray irradiation to the subject, making it possible to acquire X-ray images of the subject from different directions.

[0022] The image storage device 3 is a computer for storing and managing medical images, and is equipped with a storage device for storing medical images. In this embodiment, the image storage device 3 stores CT images acquired in advance by a CT (Computed Tomography) device and MR images acquired by an MRI (Magnetic Resonance Imaging) device for subjects undergoing EPCR. Both the CT images and MR images are three-dimensional images and include at least the liver of the subject. In this embodiment, the CT images are contrast-enhanced CT images acquired by injecting a contrast agent into the blood vessels of the liver.

[0023] Next, an image display device according to this embodiment will be described. Figure 2 is a diagram showing the hardware configuration of the image display device according to this embodiment. As shown in Figure 2, the image display device 1 includes a CPU (Central Processing Unit) 11, a memory 16 as a temporary storage area, and a non-volatile storage unit 13. The image display device 1 also includes a display 14 such as a liquid crystal display or an organic EL (Electro-Luminescence) display, and an input device 15 such as a keyboard. The display 14 and the input device 15 may be integrated as a touch panel display. The image display device 1 also includes a network I / F (Interface) 17 connected to a network 4. The CPU 11, memory 16, storage unit 13, and network I / F 17 are connected to a bus 19.

[0024] The storage unit 13 is implemented by an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory, etc. The image display program 12 is stored in the storage unit 13 as a storage medium. The CPU 11 reads the image display program 12 from the storage unit 13, expands it into memory 16, and executes the expanded image display program 12.

[0025] The image display program 12 is stored in a memory device of a server computer connected to the network, or in network storage, in a state that allows external access, and is downloaded and installed on the image display device 1 upon request.

[0026] Next, the functional configuration of the image display device according to this embodiment will be described. Figure 3 is a diagram showing the functional configuration of the image display device according to this embodiment. As shown in Figure 3, the image display device 1 includes an acquisition unit 21, an extraction unit 22, an output unit 23, an alignment unit 24, and a display control unit 25. When the CPU 11 executes the image display program 12, the CPU 11 functions as the acquisition unit 21, the extraction unit 22, the output unit 23, the alignment unit 24, and the display control unit 25.

[0027] The acquisition unit 21 acquires a two-dimensional X-ray image G0 from the imaging device 2 that includes at least the liver of the subject, and acquires a three-dimensional CT image G1 and an MR image G2 from the image storage device 3. The X-ray image G0 may be one frame of fluoroscopic imaging (video) taken during ERCP, or it may be an X-ray image (still image) taken during ERCP.

[0028] X-ray images G0 are generally known to depict the morphology (shape) of the subject, such as bones and contours, in detail, similar to CT images G1. Furthermore, because X-ray images G0 are acquired during surgery, if surgical instruments such as endoscopes and vascular catheters are inserted into the subject, these instruments will be depicted in the X-ray images G0. On the other hand, tubular structures such as bile ducts and pancreatic ducts in the subject are difficult to visualize in X-ray images G0 without the injection of contrast agent.

[0029] CT image G1 clearly depicts the morphology (shape) of the subject, such as bones and contours, but tubular structures such as blood vessels and pancreatic ducts are difficult to visualize without contrast agent injection. On the other hand, MR image G2 visualizes tubular tissues such as bile ducts and pancreatic ducts of the subject even without contrast agent, but the morphology (shape) of bones and contours is difficult to visualize.

[0030] The extraction unit 22 extracts abnormal tissue from the liver of the subject from the CT image G1 or MR image G2. For example, the extraction unit 22 extracts abnormal tissue from the CT image G1 or MR image G2 using an extraction model that has been machine-learned to extract abnormal tissue from the CT image G1 or MR image G2. In this embodiment, abnormal tissue is extracted from the MR image G2, but it is not limited to this.

[0031] Figure 4 is a block diagram showing the functional configuration of the derivation unit 23. As shown in Figure 4, the derivation unit 23 includes a first extraction unit 31, a second extraction unit 32, a division unit 33, a positioning unit 34, an image derivation unit 35, and a path derivation unit 36.

[0032] The extraction unit 23 extracts bile ducts and other anatomical structures from the CT image G1 and the MR image G2. In this embodiment, the first extraction unit 31 of the extraction unit 23 extracts bile ducts from the MR image G2 to derive a three-dimensional bile duct image. The second extraction unit 32 extracts other anatomical structures from the MR image G2 and the CT image G1 to derive a three-dimensional anatomical structure image. Examples of anatomical structures other than bile ducts include the pancreas, pancreatic duct, liver, hepatic blood vessels (arteries, veins, and portal veins), stomach, and duodenum. In this embodiment, the extraction unit 23 extracts at least the pancreas, pancreatic duct, liver, hepatic blood vessels, and stomach from the CT image G1.

[0033] The first extraction unit 31 extracts bile ducts from MR image G2 using an extraction model that has been machine-learned to extract bile ducts from MR image G2, and derives a three-dimensional bile duct image. The first extraction unit 31 extracts bile ducts, including the gallbladder, from the papilla in the duodenum, and derives a three-dimensional bile duct image.

[0034] Figure 5 is a schematic diagram showing a bile duct image. In Figure 5, the liver 43 containing the bile ducts extracted together with the bile duct image 41 is shown by a dashed line. The bile duct image 41 is an example of a three-dimensional partial image of this disclosure.

[0035] The second extraction unit 32 extracts the pancreatic duct from the MR image G2 using an extraction model that has been machine-learned to extract the pancreatic duct from the MR image G2. The second extraction unit 32 also extracts anatomical structures from the CT image G1 using an extraction model that has been machine-learned to extract the liver, hepatic blood vessels, and stomach. The second extraction unit 32 derives an anatomical structure image, which is a three-dimensional image of these extracted anatomical structures. In this embodiment, since the CT image G1 is a contrast-enhanced CT image, it is possible to extract the hepatic blood vessels from the CT image G1.

[0036] The division unit 33 divides the liver contained in CT image G1 into multiple liver segments. For example, the division unit 33 divides the liver extracted from CT image G1 by the second extraction unit 32 into multiple liver segments according to the Quinaud classification. The liver is divided into left and right lobes by the falciform ligament, and further divided into four segments: lateral segment, medial segment, anterior segment, and posterior segment. These segments are further divided into two segments each, resulting in eight segments: the caudate lobe segment S1, the posterolateral segment (superlateral segment) S2, the anterolateral segment (inferior lateral segment) S3, the medial superior and inferior segments (quadrature lobe) S4, the anterior inferior segment S5, the posterior inferior segment S6, the posterior superior segment S7, and the anterior superior segment S8 (see Figure 6). These segments can be divided based on the Canton line, the line connecting the vena cava and the gallbladder, the falciform ligament, the intermediate vein, the left hepatic vein, and the right hepatic vein. The division unit 33 divides the liver into the eight regions S1 to S8 based on the liver and hepatic vascular structure extracted by the second extraction unit 32.

[0037] The alignment unit 34 aligns the CT image G1 and the MR image G2 and determines the coordinate correspondence between the CT image G1 and the MR image G2. Since the CT image G1 and the MR image G2 are three-dimensional images, the alignment unit 34 aligns the CT image G1 and the MR image G2 in three-dimensional space. Known methods can be used for alignment. For example, conventional alignment methods using pixel values, alignment methods using feature points as landmarks, and alignment methods using mutual information can be used.

[0038] The image output unit 35 outputs a three-dimensional bile duct image B0 that reflects the liver regions of the liver divided by the division unit 33 from the bile duct image 41 output by the first extraction unit 31. The output bile duct image B0 is an example of a three-dimensional partial image that reflects the liver regions of this disclosure. In this embodiment, the image output unit 35 outputs a bile duct image B0 to which different colors are assigned according to the liver region in which the bile duct is located. Figure 7 shows an example of a bile duct image B0. In Figure 7, for illustrative purposes, the bile duct is divided into three regions, and different patterns are assigned to each region of the bile duct to show that the colors are different. Alternatively, instead of assigning different colors, the liver regions in the bile duct image B0 may be distinguished by changing the transparency, saturation and / or density. In addition, the pancreatic duct may be assigned a different color from the bile duct to distinguish it from the pancreatic duct.

[0039] The pathway derivation unit 36 ​​derivs the pathway from the papilla to the abnormal tissue extracted by the extraction unit 22 in the bile duct image B0. The derived pathway will be described later.

[0040] Returning to Figure 3, the alignment unit 24 of the image display device 1 aligns the X-ray image G0 with the bile duct image B0. Since the bile duct image B0 is a three-dimensional image, the alignment unit 24 derives a two-dimensional projected bile duct image B1 by projecting the bile duct image B0 onto the plane of the X-ray image G0 for alignment with the X-ray image G0. The projection direction is set to the direction of the X-ray irradiation that the imaging device 2 irradiates the subject with, i.e., the direction of the X-ray optical axis. As for the projection method, well-known methods such as the latsum method, which uses the sum of the pixel values ​​arranged along the ray as the pixel value of the projected image, or the maximum image projection (MIP) method can be used.

[0041] X-ray image G0 and CT image G1 clearly depict the morphology (shape) of the subject, such as bones and contours. On the other hand, while tubular tissues such as the bile ducts and pancreatic ducts of the subject are depicted in MR image G2, which is derived from projected bile duct image B1, the morphology (shape) of bones and contours is not easily depicted. Therefore, the alignment unit 24 aligns X-ray image G0 with CT image G1 in order to align X-ray image G0 with the two-dimensional projected bile duct image B1 after projection, and derives the coordinate correspondence between X-ray image G0 and CT image G1 as a parameter.

[0042] To align the X-ray image G0 and the CT image G1, the alignment unit 24 identifies feature points in the X-ray image G0 where the contrast of bone contours, etc., is higher than a predetermined value, and derives feature points in the CT image G1 that correspond to the feature points in the X-ray image G0. Then, the CT image G1 is aligned in the parallel, rotational, and scaling directions so that the feature points of the X-ray image G0 and the feature points of the CT image G1 coincide, and the correspondence between the coordinates in the parallel, rotational, and scaling directions is derived as parameters.

[0043] In this embodiment, the alignment unit 34 of the output unit 23 aligns the CT image G1 and the MR image G2, and the coordinate correspondence between the CT image G1 and the MR image G2 is determined. Therefore, the alignment unit 24 uses the parameters obtained from the alignment of the X-ray image G0 and the CT image G1 to translate, rotate, and enlarge / reduce the projected bile duct image B1, thereby aligning the projected bile duct image B1 with the X-ray image G0.

[0044] The display control unit 25 displays, on the display 14, a superimposed image G10 obtained by superimposing the projected bile duct image B1 aligned with the X-ray image G0. In this embodiment, the abnormal tissue extracted by the extraction unit 22 from the MR image G2 and the anatomical structure extracted by the second extraction unit 32 of the derivation unit 23 from the CT image G1 and the MR image G2 are also included and displayed in the superimposed image G10. It should be noted that both the abnormal tissue and the anatomical structure extracted from the CT image G1 are two-dimensional projected images projected onto the plane of the X-ray image G0 in the same manner as the projected bile duct image B1, and are aligned with the X-ray image G0 and included in the superimposed image G10 in the same manner as the bile duct image B0. In the superimposed image G10, the color of the anatomical structure may be different for each part.

[0045] FIG. 8 is a diagram showing the superimposed image G10. As shown in FIG. 8, in the superimposed image G10, the projected bile duct image B1 is superimposed on the X-ray image G0. Further, in the superimposed image G10, the projected images of the pancreatic duct 42 which is an anatomical structure extracted from the MR image G2, and the liver 51, the pancreas 52 and the stomach 53 which are anatomical structures extracted from the CT image G1 are superimposed on the X-ray image G0. In FIG. 8, the duodenum 50 depicted in the X-ray image G0 is shown by a broken line. Further, the liver 51 contains the abnormal tissue 54 extracted by the extraction unit 22.

[0046] In ERCP, an endoscope is inserted through the subject's mouth to the papilla of the duodenum. Therefore, the X-ray image G0 includes an image of the endoscope. FIG. 9 is a diagram showing a superimposed image including an image of an endoscope inserted to the papilla of the duodenum. As shown in FIG. 9, the superimposed image G10 includes an image 55 of the endoscope inserted into the subject. In this embodiment, in order to make the tip of the endoscope easily identifiable, the display control unit 25 may detect the tip of the endoscope included in the X-ray image G0, and attach a marker 60 to the detected tip of the endoscope.

[0047] The display control unit 25 may display the route from the papilla to the abnormal tissue 54 derived by the route derivation unit 36 of the derivation unit 23 on the superimposed image G10. FIG. 10 is a diagram showing the superimposed image G10 on which the route is displayed. As shown in FIG. 10, in the projected bile duct image B1, the route 62 from the papilla 61 to the abnormal tissue 54 is shown by a broken line.

[0048] During ERCP, a guide wire or tube is extended from the distal end of an endoscope and inserted into the bile duct. FIG. 11 is a diagram showing a superimposed image G10 obtained by extending the guide wire or tube. As shown in FIG. 11, the superimposed image G10 includes an image 63 of the guide wire or tube extended from the distal end of the endoscope 55.

[0049] Here, during ERCP, by changing the position of the X-ray source in the imaging apparatus 2, the X-ray irradiation angle to the subject may be changed, and X-ray images G0 from different angles may be acquired. In this case, the deriving unit 23 changes the projection angle of the bile duct image B0 in accordance with the irradiation angle of the X-ray source to derive a projected bile duct image B1, aligns the projected bile duct image B1 with the X-ray image G0, and derives the superimposed image G10. Accordingly, even when the X-ray irradiation angle to the subject is changed, the superimposed image G10 in which the X-ray image G0 and the projected bile duct image B1 are aligned can be displayed.

[0050] Next, processing performed in the present embodiment will be described. FIG. 12 is a flowchart showing processing performed in the present embodiment. It is assumed that the CT image G1 and the MR image G2 are acquired from the image storage device 3 and stored in the storage unit 13. First, the acquisition unit 21 acquires an X-ray image G0 (step ST1), and the extraction unit 22 extracts abnormal tissue from the CT image G1 or the MR image G2 (step ST2).

[0051] The first extraction unit 31 of the deriving unit 23 extracts a bile duct from the MR image G2 (step ST3), and the second extraction unit 32 extracts an anatomical structure from the CT image G1 and the MR image G2 (step ST4). The division unit 33 divides the liver into a plurality of hepatic segments (step ST5), and the alignment unit 34 aligns the CT image G1 with the MR image G2 (step ST6). The image deriving unit 35 derives a three-dimensional bile duct image B0 in which the hepatic segments are reflected in the bile duct (step ST7). The route deriving unit 36 derives a route from the papilla to the abnormal tissue extracted by the extraction unit 22 in the bile duct image B0 (step ST8).

[0052] The alignment unit 24 extracts the projected bile duct image B1 from the bile duct image B0 (step ST9), and aligns the X-ray image G0 with the projected bile duct image B1 (step ST10). The display control unit 25 displays the superimposed image G10 of the X-ray image G0 and the projected bile duct image B1 on the display 14 (step ST11), and terminates the process.

[0053] In this embodiment, a bile duct image B0 reflecting the liver segments is derived in the bile duct, and the X-ray image G0 is aligned with a two-dimensional projected bile duct image B1 derived from the bile duct image B0. A superimposed image G10 is then displayed by superimposing the projected bile duct image B1 onto the X-ray image G0. This makes it easy to confirm which liver segment the bile duct included in the superimposed image G10 belongs to. Consequently, when performing ERCP, it is easy to confirm which part of the liver the guidewire or the like is inserted into.

[0054] Furthermore, by including anatomical structures other than the bile duct, such as the pancreatic duct, stomach, and liver, in superimposed image G10, the positional relationship between the bile duct and other anatomical structures can be easily confirmed in superimposed image G10.

[0055] In particular, by including the pancreatic duct in the anatomical structure, the pancreatic duct is superimposed on superimposed image G10. This allows the position of the pancreatic duct to be confirmed in superimposed image G10, thus preventing the accidental insertion of a guidewire or other instrument into the pancreatic duct when inserting it into the bile duct from the tip of the endoscope during ERCP.

[0056] Furthermore, by attaching a marker 60 to the tip of the endoscope included in the X-ray image G0, the position of the endoscope tip in the superimposed image G10 can be easily confirmed, thus allowing the endoscope tip to be easily guided to the papilla.

[0057] Furthermore, by deriving the pathway from the papilla to the abnormal tissue and highlighting pathway 62 in the superimposed image G10, the pathway within the bile duct from the papilla to the abnormal tissue can be easily confirmed. Therefore, a guidewire or the like can be easily guided to the abnormal tissue.

[0058] In the above embodiment, the superimposed image G10 is displayed on the display 14, but the invention is not limited to this. A stereo image with parallax may be derived from the superimposed image G10, and the superimposed image may be displayed on a stereoscopic display to enable stereoscopic viewing. In this embodiment, since the bile duct image B0 derived from the MR image G2 and the anatomical structure image extracted from the CT image G1 are three-dimensional images, when a viewpoint for observing the superimposed image G10 is set, the distance from the starting point of each pixel in the bile duct image B0 and the anatomical structure image can be determined. On the other hand, the distance between the left and right eyes of a human is approximately constant. Therefore, a stereo image with parallax can be derived based on the distance between the left and right eyes of a human and the distance of each pixel from the starting point. The stereo image includes an image for the left eye and an image for the right eye. The image for the left eye and the image for the right eye are examples of the first superimposed image and the second superimposed image of this disclosure.

[0059] As shown in Figure 13, a stereoscopic display, such as a head-mounted display 18, can be provided in the image display device 1, and stereo images can be displayed on the head-mounted display 18. This makes it possible to grasp the distance to the endoscope tip, the guidewire extended from the endoscope tip, and the bile duct in three dimensions when performing ERCP.

[0060] Furthermore, in the above embodiment, both CT image G1 and MR image G2 are used to derive the bile duct image B0 and the anatomical structure image, but this is not limited to this. The bile duct image B0 and the anatomical structure image may also be derived using only MR image G2 or only CT image G1.

[0061] Furthermore, in the above embodiment, the superimposed image G10 is derived using the X-ray image G0 acquired during ERCP surgery, but this is not the only way to do so. The superimposed image G10 may also be derived using the X-ray image G0 acquired at a time other than during surgery, for example, before or after surgery.

[0062] In this embodiment, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor as hardware, a program as software, or a combination thereof. In that case, the processor is configured to work in cooperation with the program to execute the various processes in this embodiment, and can function as a unit or means in this embodiment. Also, the execution order of the processes by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process.

[0063] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a programmable logic device such as an FPGA (Field Programmable Gate Array), a dedicated circuit for executing a specific process such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Furthermore, the type of hardware may be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a processor, these multiple hardware components may reside in physically separate devices or in the same device. Also, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. Hardware is composed of electrical circuits (circuitry) that combine circuit elements such as semiconductor elements.

[0064] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a set of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located in physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0065] Furthermore, although the above embodiment describes an embodiment in which the image display program 12 is pre-stored (installed) in the storage unit 13, the invention is not limited to this. The image display program 12 may be provided in the form of a recording medium such as a CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the image display program 12 may be provided in the form of a download from an external device via a network.

[0066] The technology disclosed herein extends to all program products. Program products include all forms of products for providing programs. For example, program products include programs provided via networks such as the Internet, and non-temporary computer-readable recording media such as CD-ROMs, DVDs, and USB memory devices on which programs are stored.

[0067] The following are additional provisions of this disclosure. (Additional provision 1) An image display device comprising a processor, wherein the processor derives a three-dimensional partial image including at least a bile duct from one or more types of three-dimensional images including at least the liver, acquires a two-dimensional radiographic image including at least the liver, divides the liver included in the three-dimensional image into a plurality of liver regions, aligns the two-dimensional radiographic image and the three-dimensional partial image, and displays a superimposed image obtained by superimposing the three-dimensional partial image, in which the liver regions are reflected on the bile duct, onto the two-dimensional radiographic image. (Additional provision 2) The image display device according to Additional provision 1, wherein the processor derives an anatomical structure image including at least one anatomical structure other than the bile duct from one or more types of three-dimensional images, and superimposes the anatomical structure image onto the two-dimensional radiographic image. (Additional provision 3) The image display device according to Additional provision 2, wherein the anatomical structure includes a pancreatic duct. (Appendix 4) The image display device according to any one of Appendix 1 to 3, wherein the processor detects the region of a surgical instrument in the two-dimensional radiographic image and displays the superimposed image by distinguishing the region of the surgical instrument from other regions. (Appendix 5) The image display device according to any one of Appendix 1 to 4, wherein the processor extracts abnormal tissue contained in the liver from the three-dimensional image and highlights the path on the bile duct from the papilla, which is the exit of the bile duct, to the abnormal tissue in the three-dimensional partial image. (Appendix 6) The image display device according to any one of Appendix 1 to 5, wherein the two-dimensional radiographic image is acquired by a radiography device in which the source position can be changed, and the processor changes at least one of the viewpoint position, orientation and magnification of the three-dimensional partial image according to the source position and radiation irradiation angle of the acquisition of the two-dimensional radiographic image and superimposes it on the two-dimensional radiographic image. (Appendix 7) The image display device according to any one of Appendix 1 to 6, wherein the processor extracts corresponding feature points from each of the one or more types of three-dimensional images and the two-dimensional radiation image, and aligns the three-dimensional partial image and the two-dimensional radiation image based on the positional relationship between the feature points extracted in the one or more types of three-dimensional images and the feature points extracted in the two-dimensional radiation image.(Note 8) The image display device according to any one of Notes 1 to 7, wherein the processor derives a first superimposed image and a second superimposed image having parallax from the superimposed image, and displays the first superimposed image and the second superimposed image on a stereoscopic display. (Note 9) An image display method in which a computer derives a three-dimensional partial image including at least the bile duct from one or more types of three-dimensional images including at least the liver, obtains a two-dimensional radiographic image including at least the liver, divides the liver included in the three-dimensional image into a plurality of liver regions, aligns the two-dimensional radiographic image and the three-dimensional partial image, and displays a superimposed image obtained by superimposing the three-dimensional partial image, in which the liver regions are reflected on the bile ducts, onto the two-dimensional radiographic image. (Appendix 10) An image display program that causes a computer to perform the following steps: a procedure for deriving a three-dimensional partial image including at least the bile duct from one or more types of three-dimensional images including at least the liver; a procedure for acquiring a two-dimensional radiographic image including at least the liver; a procedure for dividing the liver included in the three-dimensional image into a plurality of liver regions; a procedure for aligning the two-dimensional radiographic image and the three-dimensional partial image; and a procedure for displaying a superimposed image obtained by superimposing the three-dimensional partial image, in which the liver regions are reflected on the bile duct, onto the two-dimensional radiographic image.

[0068] The disclosure of Japanese Patent Application No. 2025-047162, filed on 21 March 2025, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

Claims

1. An image display device comprising a processor, the processor derives a three-dimensional partial image including at least the bile duct from one or more types of three-dimensional images including at least the liver, acquires a two-dimensional radiographic image including at least the liver, divides the liver included in the three-dimensional image into a plurality of liver regions, aligns the two-dimensional radiographic image and the three-dimensional partial image, and displays a superimposed image obtained by superimposing the three-dimensional partial image, in which the liver regions are reflected on the bile duct, onto the two-dimensional radiographic image.

2. The image display device according to claim 1, wherein the processor derives an anatomical structure image including at least one anatomical structure other than the bile duct from the one or more types of three-dimensional images, and superimposes the anatomical structure image onto the two-dimensional radiographic image.

3. The image display device according to claim 2, wherein the anatomical structure includes a pancreatic duct.

4. The image display device according to claim 1 or 2, wherein the processor detects the region of a surgical instrument in the two-dimensional radiographic image, and displays the superimposed image by distinguishing the region of the surgical instrument from other regions.

5. The image display device according to claim 1 or 2, wherein the processor extracts abnormal tissue contained in the liver from the three-dimensional image, and in the three-dimensional partial image, highlights the pathway on the bile duct from the papilla, which is the opening of the bile duct, to the abnormal tissue.

6. The image display device according to claim 1 or 2, wherein the two-dimensional radiation image is acquired by a radiation imaging device capable of changing the radiation source position, and the processor changes at least one of the viewpoint position, orientation and magnification of the three-dimensional partial image according to the radiation source position and radiation irradiation angle used to acquire the two-dimensional radiation image, and superimposes it on the two-dimensional radiation image.

7. The image display device according to claim 1 or 2, wherein the processor extracts corresponding feature points from each of the one or more types of three-dimensional images and the two-dimensional radiation image, and aligns the three-dimensional partial image and the two-dimensional radiation image based on the positional relationship between the feature points extracted in the one or more types of three-dimensional images and the feature points extracted in the two-dimensional radiation image.

8. The image display device according to claim 1 or 2, wherein the processor derives a first superimposed image and a second superimposed image having parallax from the superimposed image, and displays the first superimposed image and the second superimposed image on a stereoscopic display.

9. An image display method comprising: a computer deriving a three-dimensional partial image including at least the bile duct from one or more types of three-dimensional images including at least the liver; obtaining a two-dimensional radiographic image including at least the liver; dividing the liver included in the three-dimensional image into a plurality of liver regions; aligning the two-dimensional radiographic image with the three-dimensional partial image; and displaying a superimposed image in which the three-dimensional partial image, in which the liver regions are reflected on the bile duct, is superimposed on the two-dimensional radiographic image.

10. An image display program that causes a computer to perform the following steps: a procedure for deriving a three-dimensional partial image including at least the bile duct from one or more types of three-dimensional images including at least the liver; a procedure for acquiring a two-dimensional radiographic image including at least the liver; a procedure for dividing the liver included in the three-dimensional image into a plurality of liver regions; a procedure for aligning the two-dimensional radiographic image and the three-dimensional partial image; and a procedure for displaying a superimposed image obtained by superimposing the three-dimensional partial image, in which the liver regions are reflected on the bile duct, onto the two-dimensional radiographic image.

Images