Program, information processing method, and information processing apparatus

Abstract

Provided is, for example, a program for associating a location in a contrast-enhanced image with a cross-sectional tomogram in which the location is shown.　This program causes a computer to execute a process for acquiring a plurality of cross-sectional tomograms that are generated via three-dimensional scanning using an image acquiring catheter and a plurality of angiograms (61) that depict the image acquiring catheter which is performing the three-dimensional scanning, generating a route (633) of the image acquiring catheter in a reference image that is selected from the acquired plurality of angiograms (61), acquiring the position of the image acquiring sensor from each of the angiograms (61), determining the position of the image acquiring sensor in the route (633) in each of the angiograms (61), and identifying the position of each of the cross-sectional tomograms in the route (633) on the basis of the frame rate of the angiograms (61) and the frame rate of the cross-sectional tomograms.

Description

Program, Information Processing Method, and Information Processing Apparatus

[0001] The present invention relates to a program, an information processing method, and an information processing apparatus.

[0002] A program or the like for associating a collateral in a tomographic image taken from inside a blood vessel using an image acquisition catheter with a collateral in a contrast image taken from outside the body has been proposed (Patent Document 1).

[0003] International Publication No. 2023 / 100838

[0004] According to Patent Document 1, it is possible to identify a tomographic image in which the same collateral as the collateral detected in the contrast image is photographed.

[0005] However, lesions such as thrombus or calcification can occur even in places where the collateral does not branch. In Patent Document 1, for places where the collateral does not branch, the place in the contrast image and the tomographic image in which the place is photographed cannot be associated.

[0006] In one aspect, an object is to provide a program or the like for associating a place in a contrast image with a tomographic image in which the place is photographed.

[0007] The present invention (1) acquires a plurality of transverse layer images generated by three-dimensional scanning using an image acquisition catheter that acquires images while moving a scanning plane passing through an image acquisition sensor in the axial direction, and a plurality of angiography images depicting the image acquisition catheter during the three-dimensional scanning; acquires a reference position in a reference image selected from the plurality of acquired angiography images and the position of a second landmark located within the three-dimensional scanning range of the image acquisition catheter; generates a path of the image acquisition catheter from the reference position to the second landmark in the reference image; acquires the position of the image acquisition sensor from each of the angiography images; calculates the moving average position of the image acquisition sensor in the angiography images arranged in the order of acquisition for each of the angiography images; and determines the position of the image acquisition sensor in the path for each of the angiography images by dividing the path based on the distance between the moving average positions in temporally adjacent angiography images. This program causes a computer to perform a process to determine the position of each of the transverse images along the path, based on the frame rate of the angiographic image and the frame rate of the transverse image.

[0008] Herein, in embodiments of the present invention, (2) the program of (1) above is further preferably such that the second landmark is the tip of the guiding catheter through which the image acquisition catheter is inserted.

[0009] (3) In the program described in (1) or (2) above, it is preferable that the reference position is the position of the image acquisition sensor at the start of the three-dimensional scan.

[0010] (4) In any of the programs described in (1) to (3) above, it is preferable that the reference position is the position of a first landmark located outside the three-dimensional scanning range of the image acquisition catheter.

[0011] (5) The program described in (4) above is further preferably such that the first landmark is a tip marker provided on the image acquisition catheter.

[0012] (6) The program described in (5) above further calculates a virtual frame number indicating which of the multiple transverse layer images corresponds to the first tomographic image in which the first landmark is depicted, assuming that the three-dimensional scan has been performed up to the position of the first landmark; determines the number of captured frames indicating which of the multiple transverse layer images corresponds to the second tomographic image in which the second landmark is depicted; and preferably uses the positions obtained by dividing the path from the tip side by the virtual frame number versus the captured frame number as the position of the image acquisition sensor at the start of the three-dimensional scan.

[0013] (7) The program described in any of (1) to (6) above preferably further displays a plurality of angiographic images and accepts the selection of the reference image.

[0014] (8) The program described in any of (1) to (7) above preferably further displays the reference image and accepts the reference position and the position of the second landmark in the displayed reference image.

[0015] (9) Preferably, the program described in any of (1) to (8) above further superimposes the generated path of the image acquisition catheter onto the reference image and accepts instructions to correct the path.

[0016] (10) The program described in any of (1) to (9) above further calculates the distance over which the position of the image acquisition sensor in the path changes in two angiography images taken in succession; calculates the movement speed of the image acquisition sensor based on the calculated distance and the angiography image acquisition interval determined based on the frame rate of the angiography images; calculates the movement interval of the image acquisition sensor for each transverse image acquisition interval by multiplying the calculated movement speed of the image acquisition sensor by the transverse image acquisition interval determined based on the frame rate of the transverse image; and preferably determines the position of each transverse image in the path by dividing the path based on the calculated movement interval of the image acquisition sensor.

[0017] (11) The program described in any of (1) to (10) above is further preferably an OCT (Optical Coherence Tomography) catheter for image acquisition.

[0018] (12) The program described in any of (1) to (10) above is further preferably an IVUS (Intravascular Ultrasound) catheter for image acquisition.

[0019] (13) Obtain multiple transverse layer images generated by three-dimensional scanning using an image acquisition catheter that acquires images while moving the scanning plane through the image acquisition sensor in the axial direction, and multiple angiography images in which the image acquisition catheter is performing the three-dimensional scanning, obtain the reference position and the position of a second landmark located within the three-dimensional scanning range of the image acquisition catheter in a reference image selected from the multiple acquired angiography images, generate a path of the image acquisition catheter from the reference position to the second landmark in the reference image, obtain the position of the image acquisition sensor from each of the angiography images, calculate the moving average position of the image acquisition sensor in the angiography images arranged in the order of acquisition for each of the angiography images, and determine the position of the image acquisition sensor in the path for each of the angiography images by dividing the path based on the distance between the moving average positions in temporally adjacent angiography images. Preferably, the information processing method involves a computer performing a process to identify the position of each of the transverse images along the path based on the frame rate of the angiographic image and the frame rate of the transverse image.

[0020] (14) An information processing apparatus having a control unit, the control unit acquires a plurality of transverse layer images generated by three-dimensional scanning using an image acquisition catheter that acquires images while moving a scanning plane through an image acquisition sensor in the axial direction, and a plurality of angiography images in which the image acquisition catheter is performing the three-dimensional scanning, the control unit acquires a reference position and the position of a second landmark located within the three-dimensional scanning range of the image acquisition catheter in a reference image selected from the plurality of acquired angiography images, respectively, the control unit generates a path of the image acquisition catheter from the reference position to the second landmark in the reference image, the control unit acquires the position of the image acquisition sensor from each of the angiography images, the moving average position of the image acquisition sensor in the angiography images arranged in the order of acquisition for each of the angiography images, and the control unit determines the position of the image acquisition sensor in the path for each of the angiography images by dividing the path based on the distance between the moving average positions in temporally adjacent angiography images. Preferably, the information processing device identifies the position of each of the transverse images along the path based on the frame rate of the angiographic image and the frame rate of the transverse image.

[0021] In one respect, it is possible to provide a program that associates a location in a contrast-enhanced image with the transverse layer image in which that location was captured.

[0022] This is an explanatory diagram illustrating the configuration of the information processing system. This is a time chart illustrating the relationship between respiration and heart rate and image acquisition timing. This is an explanatory diagram illustrating the relationship between angiographic images and each part of the image acquisition catheter. This is an explanatory diagram illustrating the relationship between angiographic images and each part of the image acquisition catheter. This is an explanatory diagram illustrating the relationship between a reference image selected from multiple angiographic images and a transverse image acquired by the image acquisition catheter. This is an explanatory diagram illustrating an overview of the information processing procedure. This is a detailed explanatory diagram of part VII in Figure 6. This is an enlarged view of part VIII in Figure 7. This is a detailed explanatory diagram of part IX in Figure 6. This is a detailed explanatory diagram of part X in Figure 6. This is an explanatory diagram illustrating the procedure for calculating the starting position of the imaging range. This is an explanatory diagram illustrating the GC judgment model. This is an example screen illustrating the operation procedure. This is an example screen illustrating the operation procedure. This is an example screen illustrating the operation procedure. This is an example screen illustrating the operation procedure. This is a flowchart illustrating the processing flow of the program. This is a flowchart illustrating the processing flow of the sensor position calculation subroutine. This is a flowchart illustrating the processing flow of the first frame position calculation subroutine. This is an example screen. This is an example screen. This is an explanatory diagram illustrating the configuration of the information processing system of Embodiment 2.

[0023] [Embodiment 1] Figure 1 is an explanatory diagram illustrating the configuration of the information processing system 10. In this embodiment, the information processing system 10 used to support endovascular treatments such as PCI (Percutaneous Coronary Intervention) will be described as an example. In the following description, tasks performed by paramedical staff such as nurses or clinical engineers based on the instructions of a physician may also be described as tasks performed by a physician.

[0024] The information processing system 10 includes a catheter system 30, an angiography device 41, and an information processing device 20. The catheter system 30 includes a catheter control device 31, an image acquisition catheter 32, and an MDU (Motor Driving Unit) 33. The image acquisition catheter 32 is connected to the catheter control device 31 via the MDU 33.

[0025] Details of the configuration of the image acquisition catheter 32 will be described later. In the following description, the side of the image acquisition catheter 32 furthest from the MDU 33 will be referred to as the proximal end, and the side closer to the MDU 33 will be referred to as the proximal end. The catheter control device 31 outputs catheter images such as a transverse view 71 (see Figure 5) and a longitudinal view 72 (see Figure 20) of the blood vessel.

[0026] The angiography apparatus 41 comprises a C-arm 421 and an X-ray tube 422 and a flat-panel detector 423 fixed to both ends thereof. X-rays emitted from the X-ray tube 422 pass through the patient's body and are detected by the flat-panel detector 423. Various image processing is performed in real time on the detected X-rays inside the angiography apparatus 41, and real-time images are output sequentially. In principle, real-time image acquisition is performed continuously during endovascular treatment.

[0027] The physician administers a contrast agent into the patient's blood vessels from an angiography device (not shown in the diagram if necessary). As the contrast agent flows into the blood vessels within the field of view of the angiography device 41, a real-time image clearly depicting the course of the blood vessels is captured. The real-time image during contrast administration is referred to as the angiography image 61 (see Figure 3).

[0028] In this embodiment, the transverse layers 71 are displayed in relation to the location of the cross-section of the blood vessel depicted in the angiographic image 61 through a process described later. This allows physicians to more easily understand the condition of the blood vessels than when observing the angiographic image 61 and the transverse layers 71 separately, enabling them to proceed with endovascular treatment appropriately.

[0029] The information processing device 20 comprises a control unit 21, a main memory 22, an auxiliary memory 23, a communication unit 24, a display unit 25, an input unit 26, and a bus. The control unit 21 is an arithmetic control device that executes the program of this embodiment. One or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), or multi-core CPUs are used in the control unit 21. The control unit 21 is connected to each hardware component of the information processing device 20 via the bus.

[0030] The main memory 22 is a storage device such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. The main memory 22 temporarily stores information necessary during processing performed by the control unit 21 and the program currently being executed by the control unit 21.

[0031] The auxiliary storage device 23 is a storage device such as SRAM, flash memory, hard disk, or magnetic tape. The auxiliary storage device 23 stores the GC (Guiding Catheter) determination model 55, the program to be executed by the control unit 21, and various data necessary for the execution of the program. The GC determination model 55 may be stored in an external mass storage device connected to the information processing device 20. The communication unit 24 is an interface for communication between the information processing device 20 and the network.

[0032] The display unit 25 is, for example, a liquid crystal display device or an organic EL (Electro-Luminescence) display device. The input unit 26 is, for example, an input device such as a keyboard, mouse, trackball, or microphone. The display unit 25 and the input unit 26 may be stacked together to form a touch panel.

[0033] The display unit 25 may be a connection interface that connects the information processing device 20 to an external display device. A communication unit 24 that connects data to an external display device via a network may realize the functions of the display unit 25 and the input unit 26.

[0034] The information processing device 20 in this embodiment is an information device such as a general-purpose personal computer, tablet, smartphone, or server computer. The information processing device 20 may also be a mainframe computer, a virtual machine operating on a mainframe computer, a cloud computing system, a quantum computer, or multiple personal computers performing distributed processing. The information processing device 20 may be configured integrally with, for example, a catheter control device 31, an angiography device 41, or a hospital information system (HIS) not shown in the figures.

[0035] In the following explanation, we will mainly use the case where the control unit 21 performs software-based processing as an example. The processes and models explained using flowcharts may each be implemented by dedicated hardware.

[0036] Figure 2 is a time chart illustrating the relationship between respiration, heart rate, and image acquisition timing. The horizontal axis represents time. The vertical lines indicate the timing of each event: respiration, heart rate, real-time image acquisition, and transverse layer image acquisition 71.

[0037] Respiration occurs approximately 12 to 20 times per minute. The duration of one respiration is indicated by Tb. Heartbeats occur approximately 60 to 100 times per minute. The duration of one heartbeat is indicated by Th. Tb and Th fluctuate constantly. The position and shape of the patient's heart change due to both respiration and heartbeat.

[0038] As mentioned above, real-time images are taken continuously during endovascular treatment. The frame rate of the real-time images is approximately 10 to 30 frames per second. The interval between real-time image acquisitions is indicated by Ta. Unless otherwise specified by the physician, the angiography device 41 takes real-time images at a constant frame rate.

[0039] The physician administers an angiographic agent as needed. The real-time images captured by the angiography device 41 become angiographic images 61. Approximately several dozen angiographic images 61 are captured during one heartbeat cycle. Due to the effects of respiration and pulsation, the arrangement and shape of the blood vessels depicted in each angiographic image 61 differ.

[0040] The physician observes the angiographic image 61 to confirm the course of the blood vessels and the insertion status of the image acquisition catheter 32, and then operates the MDU 33 as described later to start the pullback operation and perform a three-dimensional scan. During the pullback operation, multiple transverse images 71 are acquired in parallel with multiple angiographic images 61. The frame rate of the transverse images 71 is approximately 30 to 160 frames per second. The acquisition interval of the transverse images 71 is indicated by Tc. Ta and Tc may be the same or different.

[0041] Figure 3 is an explanatory diagram illustrating the relationship between the angiographic image 61 and the various parts of the image acquisition catheter 32. The physician inserts the guiding catheter 39 up to just before the treatment site. The guiding catheter 39 is a relatively thick tube and serves as a passage for the physician to safely insert the image acquisition catheter 32 and various treatment catheters, etc., up to the vicinity of the treatment site. The guiding catheter 39 is made of resin mixed with an X-ray contrast agent.

[0042] The left side of Figure 3 shows the tip of the image acquisition catheter 32 protruding from the tip of the guiding catheter 39. The image acquisition catheter 32 is either an OCT catheter or an IVUS catheter. The image acquisition catheter 32 may also be a dual-sensor type catheter that has the functions of both an OCT catheter and an IVUS catheter.

[0043] The image acquisition catheter 32 comprises a tip marker 321, a sensor 322, a shaft 323, and a sheath 324. The sheath 324 is a resin tube with its tip bent in a roughly crank shape. A guide wire insertion hole (not shown) is provided on the tip side of the bend.

[0044] Shaft 323 is inserted inside sheath 324. Sensor 322 is fixed to the tip of shaft 323. Sensor 322 is an image acquisition sensor used for taking tomographic images. When image acquisition catheter 32 is an OCT catheter, sensor 322, which is an OCT sensor, is a ball lens that emits light from a light source and receives reflected light. When image acquisition catheter 32 is an IVUS catheter, sensor 322, which is an IVUS sensor, is an ultrasonic sensor that transmits and receives ultrasonic waves.

[0045] Sensor 322 itself, or the housing that holds sensor 322, is made of a material with low X-ray transmittance, such as metal or ceramics. In the following description, sensor 322 may include the housing. Tip marker 321 is fixed to the tip of sheath 324. Tip marker 321 is made of a material with low X-ray transmittance, such as metal.

[0046] The description will continue using FIGS. 1 and 3. MDU 33 rotates sensor 322 and shaft 323 inside sheath 324. By rotating while sensor 322 transmits and receives light or ultrasonic waves, a radial scan is performed. Catheter control device 31 generates one image for each rotation of sensor 322. The generated image is a cross-sectional image 71 (see FIG. 5) centered on sensor 322 and substantially perpendicular to sheath 324. A series of processes from the transmission and reception of data by sensor 322 to the generation of one cross-sectional image 71 is described as the taking of cross-sectional image 71.

[0047] MDU 33 can also move forward and backward while rotating sensor 322 and shaft 323 inside sheath 324. The aforementioned pull-back operation is an operation of rotating while pulling sensor 322 toward the proximal end side at a constant speed. By the pull-back operation, catheter control device 31 realizes a three-dimensional scan of continuously taking a plurality of cross-sectional images 71 substantially perpendicular to sheath 324 at a predetermined interval.

[0048] That is, the catheter 32 for image acquisition is a so-called three-dimensional scanning catheter that acquires a tomographic image 71 while moving the scanning plane by radial scanning in the axial direction. In the following description, one tomographic image 71 may be described as one frame. A series of tomographic images 71 taken during the pull-back operation are numbered as the first frame, the second frame, the third frame, etc. in order from the first tomographic image 71. The three-dimensional scanning range in the axial direction is the range from the first tomographic image 71 taken by one pull-back operation to the last tomographic image 71. This range corresponds to the range in which the sensor 322 moves in the longitudinal direction of the sheath 324 by the pull-back operation.

[0049] The catheter control device 31 can also generate a longitudinal tomographic image 72 (see FIG. 20) based on a plurality of tomographic images 71 acquired by three-dimensional scanning. The catheter control device 31 can also automatically measure parameters such as the inner diameter of a blood vessel or the EEM (External Elastic Membrane) diameter. The catheter control device 31 can also generate tomographic images 71 and longitudinal tomographic images 72 in which various regions such as a plaque region, a calcified region, or a stent placement region are extracted and marked.

[0050] Since the automatic measurement by the catheter control device 31 and the automatic extraction of various regions have been conventionally used, the details will be omitted.

[0051] The functions and configuration of the catheter control device 31 are the same as those of conventionally used OCT or IVUS image diagnostic devices, so the details will be omitted. Note that the control unit 21 may realize the functions of the catheter control device 31.

[0052] An angiographic image 61 is shown on the right side of FIG. 3. In the angiographic image 61 in FIG. 3 and subsequent figures, the lower the X-ray transmittance, the finer the hatching. The region without hatching is the region with high X-ray transmittance. The region shown by the fine hatching corresponds to, for example, the tip marker 321, the tip of the guiding catheter 39 with a low radiation transmittance, the sensor 322, the region where the blood vessel wall is calcified, and the stent.

[0053] The guiding catheter 39 generally has low radiotransmittance throughout its entire length, and is equipped with a tip that has particularly low radiotransmittance to facilitate confirmation of its position under X-ray fluoroscopy. For illustrative purposes, in Figure 3, the tip is emphasized and shown with fine hatching, while the rest of the guiding catheter 39 is shown with coarse hatching.

[0054] In Figure 3, the location corresponding to the tip marker 321 is indicated by the first landmark 621, the location corresponding to the tip of the guiding catheter 39 is indicated by the second landmark 622, and the location corresponding to the sensor 322 is indicated by the sensor image 624.

[0055] Figure 4 is an explanatory diagram illustrating the relationship between the angiographic image 61 and the various parts of the image acquisition catheter 32. Figure 4 shows the state at a time slightly later than that shown in Figure 3. As shown on the left side of Figure 4, the sensor 322 has been slightly retracted towards the proximal end.

[0056] Due to the effects of heart rate, respiration, and body movement, the position and shape of the blood vessels depicted in the angiographic image 61 differ from those in Figure 3. In the angiographic image 61 of Figure 4, the first landmark 621 corresponding to the tip marker 321, the second landmark 622 corresponding to the tip of the guiding catheter 39, and the sensor image 624 corresponding to the sensor 322 are also depicted.

[0057] The position and shape of the blood vessels depicted in the angiographic image 61 differ between Figure 3 and Figure 4. Therefore, even for a physician, it is difficult to determine, by looking only at Figures 3 and 4, which position in the angiographic image 61 in Figure 3 corresponds to the position of the sensor image 624 in the angiographic image 61 in Figure 4.

[0058] Figure 5 is an explanatory diagram illustrating the relationship between a reference image 611 selected from multiple angiographic images 61 and a transverse image 71 taken with an image acquisition catheter 32. The method for selecting the reference image 611 will be described later. As explained using Figures 3 and 4, the shape of the blood vessels depicted in the angiographic image 61 changes moment by moment due to pulsation and respiration, and the shapes of the image acquisition catheter 32 and guiding catheter 39 inserted into the blood vessels also change.

[0059] During the pullback operation, the sensor 322 moves at a substantially constant speed inside the sheath 324, i.e., inside the blood vessel, from the tip to the proximal end. During the pullback operation, the relative positions of the tip marker 321 and the tip of the guiding catheter 39 and the blood vessel do not change.

[0060] With a single pullback operation, multiple transverse layer images 71 are captured, approximately perpendicular to the sheath 324, as shown on the left side of Figure 5. The distance ΔL between adjacent transverse layer images 71 is the distance the sensor 322 is pulled back during one rotation of the shaft 323. The movement speed of the sensor 322 during the pullback operation is approximately constant, and the distance ΔL between adjacent transverse layer images 71 is also approximately constant.

[0061] By integrating the movement of the sensor 322 along the blood vessels, using the blood vessels in the reference image 611 and the first landmark 621 and second landmark 622 as references, the positions where each transverse layer image 71 was captured can be mapped onto the reference image 611, as shown in Figure 5. The specific method is described below.

[0062] Figure 6 is an explanatory diagram illustrating the overview of the information processing procedure. The acquisition of multiple angiographic images 61 and multiple transverse images 71, enclosed by dashed lines, are performed in parallel as described above. However, the operation of the angiography device 41 and the operation of the catheter system 30 are not synchronized, and each image is acquired at a different time.

[0063] The control unit 21 displays multiple angiography images 61 on the display unit 25 and accepts the physician's selection of a reference image 611. The physician compares the multiple angiography images 61 and selects one angiography image 61 in which the treatment target area is clearly depicted as the reference image 611.

[0064] The control unit 21 may automatically select the reference image 611 using pattern recognition or a learning model generated by machine learning. Alternatively, the control unit 21 may extract several candidate reference images 611 from the angiography images 61, and the physician may select the reference image 611 from among these candidates.

[0065] The control unit 21 displays the selected reference image 611 on the display unit 25 in a manner that is easy to observe. The physician observes the reference image 611 and determines the positions of two landmarks from the angiographic image 61: a first landmark 621 corresponding to the tip marker 321 and a second landmark 622 corresponding to the tip of the guiding catheter 39. The control unit 21 acquires the positions of the two landmarks in the reference image 611 based on the physician's operation. The first landmark 621 is an example of a reference position in this embodiment.

[0066] The control unit 21 automatically generates a path for the image acquisition catheter 32 between the first landmark 621 and the second landmark 622. The control unit 21 superimposes the automatically generated path line 633 onto the angiography image 61 selected as the reference image 611, as illustrated in Figure 5.

[0067] An overview of the method for generating the path line 633 will be described. The control unit 21 generates a path line 633 that shows the shortest time to connect two points, for example, from the second landmark 622 to the first landmark 621, when the contrast-enhanced image is considered as a velocity field corresponding to the contrast intensity. The control unit 21 may also automatically generate the path line 633 based on any other algorithm.

[0068] The control unit 21 may accept user instructions to modify the route line 633. Specific examples of the user interface for processing such as landmark location acquisition and route line 633 modification will be described later.

[0069] Figure 7 is a detailed explanatory diagram of section VII in Figure 6. The control unit 21 detects the sensor images 624 depicted in each angiography image 61 and obtains the coordinates of representative points. Specifically, for example, the control unit 21 receives the input of an angiography image 61 and inputs the angiography image 61 into a learning model trained by machine learning to output sensor images 624 extracted from the angiography image 61, thereby obtaining the sensor images 624. The control unit 21 calculates the position of the sensor images 624, that is, the coordinates of representative points such as the centroid of the sensor images 624.

[0070] The control unit 21 may receive the angiography image 61 as input and input the angiography image 61 to a learning model trained by machine learning to output the position of the sensor image 624, thereby obtaining the position of the sensor image 624.

[0071] Pattern recognition may be used to detect the sensor image 624. Since methods for extracting known targets or representative points of targets from images are well known, a detailed explanation will be omitted. The control unit 21 may also display each angiography image 61 on the display unit 25 and accept input from the user for the position of the sensor image 624.

[0072] In Figure 7, each sensor image 624 is schematically shown by a black circle. The center of the black circle is the representative point of the sensor image 624. When the sensor images 624 detected from each angiography image 61 are combined into one image and connected by lines in the order in which the source angiography images 61 were acquired, a zigzag pattern is obtained, as shown in the second to last figure in Figure 7.

[0073] Figure 8 is an enlarged view of section VIII in Figure 7. The control unit 21 calculates a moving average of the positions of the sensor images 624 in the order in which the angiography images 61 were taken. The moving average position of the sensor image 624 in a single angiography image 61 is calculated by averaging the coordinates of the sensor images 624 in a total of (2n+1) angiography images 61, n images before and n images after the angiography image 61, with Gaussian weighting. The calculation formula is shown in equation (1).

[0074]

[0075] In this embodiment, σ = 4 and n = 16 were used. Other values ​​may be used for σ and n. The method for calculating the moving average position for the angiographic images 61 near both ends of the series of angiographic images 61 will be described later.

[0076] In the bottom figure of Figure 7, i.e., Figure 8, the averaged images 625 corresponding to each calculated averaging position are schematically shown by black circles. The center of the black circle corresponds to the averaging position. By taking the moving average explained using equations (1) and (2), the effects of heart rate, respiration, and body movement can be reduced. However, the averaged images 625 are still not aligned on the path line 633.

[0077] ΔA represents the distance between the averaged images 625 of two angiography images 61 taken adjacent to each other in time. ΔA indicates the distance the averaged image 625 has moved on the reference image 611 during the time Ta explained using Figure 2. ΔA also simultaneously indicates the inter-frame movement speed of the averaged image. Here, "inter-frame movement speed" refers to the movement speed of the averaged image 625, assuming that unit time is defined as the time interval between acquisitions of the angiography images 61, i.e., the inverse of the frame rate of the angiography images 61. As shown in Figure 8, the length of ΔA varies depending on the location.

[0078] Returning to Figure 6, the explanation continues. The control unit 21 calculates the position of the corrected image 626 by correcting the distance ΔA between each averaged image 625 based on the length of the path line 633. Figure 9 is a detailed explanatory diagram of the IX section in Figure 6. The upper part of Figure 9 is a schematic diagram showing the series of averaged images 625 explained using Figure 8 arranged in a straight line. The distance ΔA between the averaged images 625 is assigned sequential numbers. Figure 9 shows the state in which the position of the averaged image 625 has been calculated for each of the (N+1) angiography images 61.

[0079] The lower part of Figure 9 shows the state after correcting the position of the averaged image 625 based on the length of the path line 633. The corrected averaged image 625 is referred to as the corrected image 626. In Figure 9, the corrected image 626 is indicated by a white circle. The center of the white circle is located on the path line 633. The distance between the corrected images 626 is indicated by ΔB. The distance ΔB between the corrected images 626 is assigned sequential numbers. In Figure 9, the imaging range means the range in which the transverse layer image 71 was captured during the pullback operation.

[0080] The control unit 21 determines each ΔB such that the ratio of the distances from ΔA1 to ΔAN is equal to the ratio of the distances from ΔB1 to ΔBN. Equation (3) shows the formula for calculating ΔBk, which is the kth ΔB.

[0081]

[0082] The control unit 21 determines the position of the corrected image 626 by sequentially dividing the portion of the path line 633 on the reference image 611 that corresponds to the imaging range L4 using the calculated ΔBk. The distance ΔB between the corrected images 626 corresponds to the distance the sensor 322 moved while each angiography image 61 was acquired.

[0083] Similar to ΔA mentioned above, ΔB represents the movement speed of the corrected image 626, when unit time is defined as the time interval between acquisitions of the angiographic image 61, i.e., the inverse of the frame rate of the angiographic image 61.

[0084] Returning to Figure 6, the explanation continues. The control unit 21 calculates the movement speed of the sensor 322 on the path line 633 based on the position of the corrected image 626. Based on the movement speed of the sensor 322, the control unit 21 calculates the interval between the cross-sectional images 71.

[0085] Figure 10 is a detailed explanatory diagram of section X in Figure 6. The acquisition interval for the angiography image 61 is Ta, as explained using Figure 2. The control unit 21 calculates the movement speed of the sensor image 624 per unit time between adjacent corrected images 626 using equation (4). Here, "movement speed per unit time" means the movement speed of the corrected image 626 when the unit time is defined as a convenient time such as "one second", "one minute", or "one hour", i.e., "meters per second", "meters per minute", or "meters per hour". Vk = ΔBk / Ta ... (4) ΔBk is the distance between the k-th corrected image 626 and the (k+1)-th corrected image 626, i.e., the movement speed between frames. ΔVk is the movement speed of the sensor image 624 per unit time between the k-th corrected image 626 and the (k+1)-th corrected image 626.

[0086] During the pullback operation, the sensor 322 moves within the sheath 324 at a nearly constant speed. However, the sheath 324 is curved along the blood vessel and is not parallel to the planar detector 423. Therefore, the movement speed Vk of the sensor image 624 on the angiographic image 61 varies from location to location, as shown in Figure 10.

[0087] The acquisition interval for the transverse layer image 71 is Tc, as explained using Figure 2. The control unit 21 calculates the interval ΔC of the sensor image 624 for each Tc in the reference image 611 using equation (5). The interval ΔC represents the movement interval of the sensor image 624 during time Tc. ΔC = V × Tc ... (5) V is the movement speed Vk of the sensor image 624 corresponding to the position being calculated.

[0088] The control unit 21 calculates the interval ΔC using the moving speed Vk corresponding to the calculation position defined on the path line 633, and sequentially moves the calculation position. If the interval ΔC crosses a boundary line where the speed V changes, the control unit 21 calculates, for example, the distance C1 to the boundary line and the time T1 = C1 / Vk to reach the boundary line based on the speed Vk in that section. The control unit 21 then calculates the remaining time T2 = shooting interval Tc - T1.

[0089] The control unit 21 uses the remaining time T2 and the travel speed Vk+1 corresponding to the next section to calculate the distance C2 = Vk+1 × T2 after crossing the boundary line. The control unit 21 calculates the interval ΔC = C1 + C2 between the sensor images 624 that cross the boundary line. Through the above process, the control unit 21 calculates the interval between the sensor images 624 for each Tc along the path line 633, that is, the interval between the cross-sectional images 71, as shown at the bottom of Figure 10.

[0090] Returning to Figure 6, the explanation continues. Based on the path line 633, the interval ΔC between the sensor images 624 on the path line 633, and the first landmark 621 and the second landmark 622, the control unit 21 calculates the position on the reference image 611 for each sensor 322 corresponding to the cross-sectional image 71.

[0091] Figure 11 is an explanatory diagram illustrating the procedure for calculating the starting position of the imaging range. The upper part of Figure 11 shows the image acquisition catheter 32 in the pullback operation starting position, i.e., when the sensor 322 is at its furthest tip position. The position of the sensor 322 shown in Figure 11 is called the tip-side reference position 623 of the sensor 322.

[0092] The image acquisition catheter 32 captures transverse images 71 in a scanning range located from the tip-side reference position 623 toward the proximal end. As explained using Figure 5, the interval between transverse images 71 is ΔL, and the total number of transverse images 71 actually captured N1 is calculated by equation (6). Note that the decimal part of N1 is rounded up. N1 = L1 / ΔL ... (6) L1 is the length of the scanning range.

[0093] Beyond the pullback start position, the transverse layer image 71 is not captured. In the following explanation, the area between the pullback start position and the tip marker 321 is referred to as the non-scanning range. Assuming that transverse layer images 71 can be captured at intervals of ΔL even in the non-scanning range, the number of transverse layer images 71 virtually captured in the non-scanning range N2 is calculated by equation (7). Note that the decimal part of N2 is truncated. Note that N2 is called the number of virtual frames. N2 = L2 / ΔL ... (7) L2 is the length of the non-scanning range.

[0094] Here, the length L2 of the non-scanning range is determined by the specifications of the image acquisition catheter 32. If the effects of individual differences are to be considered, the user may measure and input the length of the non-scanning range. As mentioned above, when the first transverse image 71 taken during the pullback operation is counted as the first frame, the virtual transverse images 71 within the non-scanning range are counted sequentially from the proximal end of the non-scanning range as frame 0, frame -1, frame -2, and so on. The transverse image 71 at the very tip of the non-scanning range corresponds to frame (N2-1).

[0095] In the following explanation, the first tomographic image will refer to the hypothetical transverse image 71 at the very tip of the non-scanning range, that is, the image in which the tip marker 321 should be depicted if a transverse image 71 were actually obtainable. Similarly, the second tomographic image will refer to the transverse image 71 at the very tip of the GC range, that is, the image in which the tip of the guiding catheter 39 is depicted. The first tomographic image is the transverse image 71 corresponding to the position of the first landmark 621 in the transverse image 71, and the second tomographic image is the transverse image 71 corresponding to the position of the second landmark 622 in the transverse image 71. The number of transverse images 71 from the first to the second tomographic image will be referred to as the number of frames from the first to the second tomographic image.

[0096] In this embodiment, the tip of the guiding catheter 39 is located within the operating range. That is, near the end of the pullback operation, the transverse image 71 is captured from inside the guiding catheter 39. Since the guiding catheter 39 does not easily transmit light or ultrasound, the transverse image 71 captured from inside the guiding catheter 39 does not have an image quality of a usable level. Therefore, it is possible to determine whether the transverse image 71 was captured from inside the guiding catheter 39 or from outside the guiding catheter 39 based on the transverse image 71 itself. A specific example of the determination method will be described later.

[0097] If there are N3 transverse images 71 taken from inside the guiding catheter 39, the length L3 of the portion of the scanning range that is inside the guiding catheter 39 is calculated by equation (8): L3 = N3 × ΔL ‥‥‥(8)

[0098] The imaging range from the pullback start position to the tip of the guiding catheter 39 is the range from the start of the pullback operation to the tip of the guiding catheter 39. The length L4 of the imaging range and the number N4 of transverse images 71 within the imaging range are calculated using equations (9) and (10). Note that N4 is called the number of imaging frames. L4 = L1 - L3 ... (9) N4 = N1 - N3 ... (10)

[0099] A schematic representation of a linearly extended path line 633 is shown at the bottom of Figure 11. The length of the path line 633 in the angiographic image 61, i.e., the length between the first landmark 621 and the second landmark 622, is denoted as L5. L5 corresponds to the combined range of the non-scanning area L2 and the imaging area L4.

[0100] Assuming that the image acquisition catheter 32 is positioned on a plane substantially parallel to the plane detector 423, the positions obtained by dividing the path line 633 on the reference image 611 in the ratio of (virtual frame count N2 to acquired frame count N4) correspond to the position of the sensor image 624 at the start of the pullback operation, that is, the position of the sensor image 624 when the first frame of the transverse layer image 71 was acquired.

[0101] The control unit 21 sequentially divides the path line 633 in increments of ΔC, as explained using Figure 10, from the position of the sensor image 624 corresponding to the first frame toward the base end. Each divided point corresponds to the estimated position of the sensor image 624 in the reference image 611 when each cross-sectional image 71 was captured. With this, the correspondence between the reference image 611 and the cross-sectional image 71 is completed.

[0102] Figure 12 is an explanatory diagram illustrating the GC determination model 55. The GC determination model 55 receives a transverse image 71 and outputs whether the transverse image 71 was taken from inside the guiding catheter 39 or from outside the guiding catheter 39.

[0103] The GC judgment model 55 is a program that determines, for example, whether the inner surface of the guiding catheter 39 is depicted in the transverse layer image 71, using methods such as pattern recognition. The GC judgment model 55 may also be a learning model generated by supervised learning using training data recorded by associating the transverse layer image 71 with a flag indicating whether it was taken from inside or outside the guiding catheter 39.

[0104] The GC determination model 55 may output whether the transverse image 71 was taken from inside the guiding catheter 39 or from outside the guiding catheter 39. For example, as illustrated in Figure 12, if the probability of the image being taken from outside the guiding catheter 39 is greater than the probability of it being taken from inside the guiding catheter 39, the GC determination model 55 may be configured to output the conclusion that the image was taken from outside the guiding catheter 39.

[0105] According to this embodiment, any angiographic image 61 taken during or before / after the pullback operation can be selected as a reference image 611, and its correspondence with the acquisition position of each transverse image 71 can be performed. The necessary condition for the reference image 611 is that the relative positions of the tip of the guiding catheter 39 and the tip marker 321 and the blood vessel are the same as during the pullback operation.

[0106] The method for calculating the moving average position of the sensor image 624 for the angiography images 61 near both ends of a series of angiography images 61 will now be explained. As mentioned above, when calculating the moving average position of the sensor image 624 in each angiography image 61, the positions of the sensor image 624 in the n angiography images 61 before and after the angiography image 61 being processed are used.

[0107] For the angiography images 61 at the edges, the position of the sensor image 624 is virtually determined for the angiography images 61 that do not actually exist, and the moving average position is calculated. An example of how to determine the virtual position is shown in Table 1. Note that Table 1 shows an example where n=4, meaning there are four actual angiography images 61.

[0108]

[0109] In Table 1, the four columns labeled "Position in Angiographic Images" indicate the position of the sensor image 624 in each of the four angiographic images 61. That is, "P," "Q," "R," and "S" represent the position coordinates of the sensor image 624 detected from each angiographic image 61. The double lines in Table 1 indicate the boundary lines of the area where the angiographic images 61 actually exist.

[0110] The "virtual positions," which exist in four columns on each side, indicate the positions of the virtually defined sensor images 624 for angiography images 61 that were not actually captured. When calculating the moving average using equations (1) and (2), the coordinates shown in the "virtual positions" are used for the non-existent sensor images 624.

[0111] In "Reflection," the coordinates of the virtual position are defined so as to be symmetrical across the boundary line shown by the double line. In "Constant," the coordinates of the virtual position are defined as constant coordinates. In "Nearby," the virtual position is defined as being at the same position as the sensor image 624 in the angiography image 61 at both ends.

[0112] In the "mirror" view, a virtual position is defined so as to be symmetrical with respect to the angiography images 61 at both ends. In the example shown in Table 1, since there are four actual angiography images 61, the virtual position is defined for only three images. In this case, the averaged image 625 corresponding to the angiography images 61 at both ends cannot be calculated. The control unit 21 calculates the averaged image 625 corresponding to the angiography images 61 from the second image onward from the end.

[0113] In the "repetition" function, a virtual location is defined so that the actual angiographic image 61 is repeated even at the virtual location. According to the applicant's research, the most appropriate and best results were obtained when using the "neighborhood" function. However, other methods may also be used.

[0114] Figures 13 to 16 are example screens illustrating the operation procedure. After the user has finished capturing the transverse layer image 71 by the pullback operation, the user operates the input unit 26 to display the screen shown in Figure 13 on the display unit 25. The control unit 21 may detect the end of the pullback operation and automatically display the screen shown in Figure 13 on the display unit 25.

[0115] In the screen shown in Figure 13, the vascular image section 66 is located on the left, and the manual section 69 is located on the right. Below the vascular image section 66, a horizontal line and the first slider 671 are located.

[0116] The manual section 69 displays the operating procedure with illustrations and simple explanations. At the bottom center of the manual section 69, the "page number / total number of pages" is displayed, such as "1 / 4". In other words, Figure 13 is the screen for accepting the first of four steps of the operation. At the bottom right of the manual section 69, the "Next" button 682 is displayed.

[0117] The vascular image section 66 displays vascular images captured in real time by the angiography device 41 during or immediately before / after the pullback operation. Here, the vascular images include angiographic images 61 taken when the contrast agent is flowing through the blood vessels, and non-angiographic images taken when the contrast agent is not flowing through the blood vessels.

[0118] By moving the first slider 671 left or right, the vascular images displayed on the screen change according to the order in which they were captured. The first stage is the stage in which the selection of a reference image 611 to be used for subsequent processing is accepted.

[0119] The user operates the first slider 671 to select a vascular image, i.e., a reference image 611, in which the contrast agent is flowing through the blood vessels at an appropriate concentration and the lesion is clearly depicted. The user confirms the selection of the reference image 611 by selecting the next button 682. In subsequent processing, the angiography image 61 selected by the user is used.

[0120] Let's move to Figure 14 and continue the explanation. As indicated by "2 / 4" in the lower center of the manual section 69, Figure 14 is the screen for accepting the second of four steps of the operation. A back button 683 is displayed on the lower left side of the manual section 69. If the back button 683 is selected, the control unit 21 returns to the previous step, that is, the screen explained using Figure 13.

[0121] The vascular image section 66 displays the reference image 611 selected in the first stage. The user operates the cursor 619 to specify the position of the second landmark 622 corresponding to the tip of the guiding catheter 39. Specifically, the user places the cursor 619 at the desired position and performs an operation such as clicking. After that, the user selects the next button 682. The control unit 21 accepts the selection of the next button 682, and the position of the second landmark 622 is confirmed.

[0122] When the control unit 21 receives an operation such as a click, it is desirable to display a circular second landmark marker 632 (see Figure 15) centered on the position of the second landmark 622. The user can confirm whether or not the second landmark 622 has been designated to the desired position by visually observing the second landmark marker 632.

[0123] In Figure 14, an enlarged area 615 is displayed around the cursor 619, which magnifies a portion of the angiographic image 61 by approximately 1.5 to 2 times in both the vertical and horizontal directions. By using the enlarged area 615, the user can precisely specify the position of the second landmark 622.

[0124] The control unit 21 may use a known algorithm, such as a Voronoi diagram, to extract the location with the lowest X-ray transmittance near the user-specified location and use it as the location for the second landmark 622.

[0125] Although not shown in the diagram, in the third of the four steps, the user operates the cursor 619 to specify the position of the first landmark 621 corresponding to the tip marker 321. As a result, the control unit 21 completes acquiring the positions of the first landmark 621 and the second landmark 622. Note that the order of the second and third steps may be reversed.

[0126] Let's move to Figure 15 and continue the explanation. As indicated by "4 / 4" in the lower center of the manual section 69, Figure 15 is the screen for accepting the fourth and final stage of the four-stage process. Note that a back button 683 is displayed on the lower left of the manual section 69, and a confirm button 684 is displayed on the lower right.

[0127] The user visually inspects the path line 633 to determine whether the path for the image acquisition catheter 32 has been generated appropriately. If it is determined that it is not appropriate, the user modifies the path line 633 by adding points that the path line 633 passes through, as described in manual section 69. Figure 16 shows an example of a modified path line. The method of modifying the curve by adding points has been used conventionally, so a detailed explanation is omitted.

[0128] Subsequently, the user selects the confirmation button 684. The control unit 21 accepts the selection of the confirmation button 684, and the route line 633 is confirmed. The control unit 21 then performs subsequent processing based on the confirmed first landmark 621, second landmark 622, and route line 633.

[0129] Figure 17 is a flowchart illustrating the program's processing flow. The program shown in Figure 17 is activated after the pullback operation is completed, either based on user instructions or automatically. The control unit 21 acquires the catheter image taken in a single pullback from the catheter control device 31 (step S501). As mentioned above, the catheter image includes multiple transverse images 71 and longitudinal images 72. The catheter image may also include a three-dimensional model constructed based on the multiple transverse images 71.

[0130] The control unit 21 acquires real-time images captured in real time from the angiography apparatus 41 during and before / after the pullback operation (step S502). The control unit 21 displays the screen described using Figure 13 on the display unit 25 (step S503). The control unit 21 accepts the selection of the reference image 611 via the operation of the first slider 671 and the next button 682 (step S504).

[0131] The control unit 21 displays the screen described using Figure 14 on the display unit 25 and accepts the selection of the first landmark 621 and the second landmark 622 (step S505). As described using Figure 15, the control unit 21 generates and displays a route line 633 connecting the first landmark 621 and the second landmark 622 (step S506).

[0132] As explained using Figures 15 and 16, the control unit 21 receives a modification instruction from the user regarding the route line 633 (step S507). Upon receiving a modification instruction, the control unit 21 generates the modified route line 633. The control unit 21 then starts a sensor position calculation subroutine (step S508). The sensor position calculation subroutine calculates the position on the reference image 611 for each sensor 322 corresponding to each cross-sectional image 71, according to the procedure explained using Figures 7 to 9. The processing flow of the sensor position calculation subroutine will be described later.

[0133] The control unit 21 displays the relationship between the angiography image 61, the transverse image 71, and the longitudinal image 72 based on the calculation result of the sensor position calculation subroutine (step S509). A specific example of the display screen will be described later.

[0134] Figure 18 is a flowchart illustrating the processing flow of the sensor position calculation subroutine. The sensor position calculation subroutine calculates the position on the reference image 611 for each sensor 322 corresponding to each cross-sectional image 71, following the procedure described using Figures 7 to 9.

[0135] The control unit 21 starts a subroutine for calculating the position of the first frame (step S541). The subroutine for calculating the position of the first frame calculates the position of the sheath 324 when the transverse layer image 71 of the first frame was taken, according to the procedure described using Figure 11. The processing flow of the subroutine for calculating the position of the first frame will be described later.

[0136] The control unit 21 extracts one angiography image 61 from the angiography images 61 acquired in step S502 (step S542). The control unit 21 obtains the position of the sensor image 624 from the angiography image 61 (step S543). The control unit 21 determines whether or not the processing of all angiography images 61 has been completed (step S544). If it is determined that the processing has not been completed (NO in step S544), the control unit 21 returns to step S542.

[0137] If it is determined that the process is complete (YES in step S544), the position of the averaged image 625 is calculated by performing a moving average on the positions of the series of sensor images 624 acquired in step S543 using equations (1) and (2) (step S545). The control unit 21 calculates ΔA, which is the distance between the averaged images 625 of angiography images 61 that were acquired adjacent to each other in time, as explained with reference to Figure 8 (step S546).

[0138] The control unit 21 calculates ΔB based on equation (3) (step S547). The length of the shooting range L4 used in equation (3) is calculated by the subroutine for calculating the position of the first frame, which is started in step S541.

[0139] The control unit 21 calculates V based on equation (4) (step S548). The control unit 21 calculates ΔC based on equation (5) (step S549). The control unit 21 divides the path line 633 into ΔC units, starting from the position of the cross-sectional image 71 of the first frame (step S550). The position of the cross-sectional image 71 of the first frame is calculated by the subroutine for calculating the position of the first frame, which was started in step S541. The control unit 21 terminates processing.

[0140] Figure 19 is a flowchart illustrating the processing flow of the subroutine for calculating the position of the first frame. The control unit 21 extracts the last transverse image 71 taken from among the transverse images 71 taken by the pullback operation (step S521). The control unit 21 inputs the transverse image 71 to the GC determination model 55 and obtains the output of the GC determination model 55. Based on the obtained output, the control unit 21 determines whether or not the transverse image 71 being processed was taken from inside the guiding catheter 39 (step S522).

[0141] If it is determined that the image was taken from inside the guiding catheter 39 (YES in step S522), the control unit 21 extracts the previous image, i.e., the transverse layer image 71 from the tip side (step S523). The control unit 21 returns to step S522.

[0142] If it is determined that the image was taken from outside the guiding catheter 39 (NO in step S522), the control unit 21 calculates N4, which is the number of transverse images 71 taken within the imaging range, based on equation (10) (step S524). Specifically, the control unit 21 calculates N4 by subtracting the number of transverse images 71 taken inside the guiding catheter 39, N3, from N1, which is the number of transverse images 71 taken in a single pullback operation, based on equation (5).

[0143] The control unit 21 calculates the number N2 of transverse layer images 71 virtually captured in the non-scanning range based on equation (7) (step S525). The control unit 21 calculates the length of the path line 633 (step S526). Since the method for calculating the length of a curve drawn on the image is well known, a detailed explanation is omitted.

[0144] The control unit 21 calculates the position of the sensor image 624 when the first frame's cross-sectional image 71 was captured by dividing the length of the path line 633 calculated in step S526 into N4:N2 (step S527). N4 is on the side of the second landmark 622, and L2 is on the side of the first landmark 621. Since the method for calculating the position by dividing the length of the curve in a predetermined ratio is well known, a detailed explanation is omitted.

[0145] The control unit 21 calculates the length of the shooting range L4 (step S528). The length of the shooting range L4 is the length from the position calculated in step S527 along the path line 633 to the second landmark 622. After that, the control unit 21 terminates the process.

[0146] Figure 20 shows an example screen. The reference image 611 selected by the user in step S504 of the program, as explained using Figure 17, is displayed from the center to the upper left of the screen. The various indicators superimposed on the reference image 611 will be described later.

[0147] The OCT tomography image 711 is displayed in the upper right of the screen. The IVUS tomography image 712 is displayed below the OCT tomography image 711. The OCT tomography image 711 and the IVUS tomography image 712 are examples of transverse images 71. If the image acquisition catheter 32 is not a catheter for a dual sensor system, either the OCT tomography image 711 or the IVUS tomography image 712, or any one of the transverse images 71, will be displayed on the screen.

[0148] The control unit 21 may also superimpose indicators showing the shape of, for example, the vascular lumen, EEM, plaque area, or calcified area onto the OCT tomographic image 711 and the IVUS tomographic image 712.

[0149] A longitudinal section image 72 is displayed at the bottom of the screen. The left side is the tip of the image acquisition catheter 32. The hatched area is the area where a certain amount of plaque is present or greater. Two plaque areas are depicted in the longitudinal section image 72. A thick horizontal line, the first area marker 641, is displayed above the plaque area on the tip side. The first area marker 641 indicates the extent of the calcified area.

[0150] Based on the corresponding positions in the transverse image 71 and the positions in the transverse image 71 where the path line 633 was divided in step S550 as described using Figure 18, the control unit 21 displays a first region marker 641 indicating the location of the transverse image 71 including the calcified region on the outside of the blood vessel through which the image acquisition catheter 32 passed. The user can understand the status of the calcified region based on the first region marker 641 in the longitudinal image 72 and the first region marker 641 in the angiography image 61.

[0151] In the longitudinal tomographic image 72, a second slider 672 with a circle above the vertical line is displayed between two plaque regions. When the user operates the input unit 26 to move the second slider 672 left or right, the control unit 21 displays a transverse image marker 636 at the corresponding location on the angiographic image 61. The control unit 21 displays the transverse image 71 corresponding to the position of the second slider 672 on the right side of the angiographic image 61.

[0152] By sliding the second slider 672, the user can confirm which position on the longitudinal tomographic image 72 corresponds to which position on the angiographic image 61, and can also refer to the transverse tomographic image 71 at that position.

[0153] In the longitudinal section image 72, distal markers 638 and proximal markers 639 are shown outside the two plaque regions. The distal marker 638 has the letter "D" indicating distal at the lower right end of a vertical bar. The proximal marker 639 has the letter "P" indicating proximal at the lower left end of a vertical bar.

[0154] When the user operates the input unit 26 to move the tip marker 638 on the longitudinal tomographic image 72 left or right, the control unit 21 displays a roughly U-shaped tip marker 638 at the corresponding position in the angiographic image 61. Similarly, when the user operates the input unit 26 to move the proximal marker 639 on the longitudinal tomographic image 72 left or right, the control unit 21 displays the proximal marker 639 at the corresponding position in the angiographic image 61.

[0155] The control unit 21 displays the distance between the tip marker 638 and the proximal marker 639 at the bottom of the longitudinal tomographic image 72. The control unit 21 may also display values ​​such as the longitudinal and transverse inner diameter of the blood vessel at the position corresponding to the tip marker 638 on the lower left side of the tip marker 638. Similarly, the control unit 21 may also display values ​​such as the longitudinal and transverse inner diameter of the blood vessel at the position corresponding to the proximal marker 639 on the lower right side of the proximal marker 639.

[0156] The control unit 21 may accept user input on the angiographic image 61, such as the operation of the transverse tomographic marker 636, the proximal marker 638, and the proximal marker 639. If so, the control unit 21 displays the second slider 672, the proximal marker 638, and the proximal marker 639 at the corresponding positions on the longitudinal tomographic image 72.

[0157] For example, before implanting a stent, the user uses a screen illustrated in Figure 20 to examine the condition of the blood vessel and determine the specifications and implantation location of the stent to be implanted.

[0158] Figure 21 is an example screen. Figure 21 shows an example screen displaying data after a pullback operation has been performed following stent placement. Explanations of aspects common to Figure 20 are omitted.

[0159] In the longitudinal section image 72, the longitudinal section of the stent is depicted as a dashed line between the leading edge marker 638 and the proximal end marker 639. Second region markers 642 are displayed at three locations inside the stent. The second region markers 642 are markers displayed at locations where, for example, stent expansion failure or stent malposition occurs. The second region markers 642 may also be markers automatically generated by the control unit 21 based on predetermined criteria.

[0160] The control unit 21 displays a second region marker 642 at the corresponding location in the angiographic image 61. It is desirable that the control unit 21 use a different color for the second region marker 642 in the longitudinal tomographic image 72 and the angiographic image 61 than for other indicators, such as the first region marker 641. This allows the user to easily determine the correspondence between indicators based on color.

[0161] According to this embodiment, it is possible to provide an information processing method that can associate a location in a contrast-enhanced image with a transverse image in which that location is captured. Users can easily understand the correspondence between the catheter image, which includes the transverse image 71 and the longitudinal image 72, and the angiography image 61, thereby enabling them to smoothly perform treatment procedures such as PCI.

[0162] According to this embodiment, since the movement speed of the sensor image 624 in the reference image 611 is used to associate the transverse layer image 71 with the reference image 611, accurate association can be achieved even when the image acquisition catheter 32 is intricately bent relative to the detection surface of the planar detector 423.

[0163] [Variation] Before starting the pullback operation, the physician may operate the angiography device 41 to adjust the direction of acquisition of the angiographic image 61 and confirm the positional relationship between the blood vessels of the treatment site and the image acquisition catheter 32. In such cases, the angiographic image 61 acquired at the time of confirmation is suitable as the reference image 611.

[0164] If the reference image 611 is captured before the pullback operation begins, the control unit 21 in the screen described using Figure 14 may accept the tip-side reference position 623, i.e., the position of the sensor image 624 before the pullback operation begins, instead of the first landmark 621. The tip-side reference position 623 is an example of a reference position in this modified example.

[0165] The control unit 21 generates a path line 633 between the tip-side reference position 623 and the second landmark 622. Since the position of the tip-side reference position 623 is known, the control unit 21 does not need to calculate the position of the tip-side reference position 623 by the series of processes described using Figure 11. Therefore, accurate correspondence between the transverse layer image 71 and the reference image 611 can be achieved.

[0166] [Embodiment 2] This embodiment relates to a configuration in which an information processing system 10 is realized by operating a general-purpose computer 90 and a program 97 in combination. The parts that are common with Embodiment 1 will not be described.

[0167] Figure 22 is an explanatory diagram illustrating the configuration of the information processing system 10 of the second embodiment. The computer 90 includes the control unit 21, main memory 22, auxiliary memory 23, communication unit 24, display unit 25, input unit 26, and bus mentioned above, as well as a read unit 29.

[0168] The program 97 is recorded on a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and saves it to the auxiliary storage device 23. The control unit 21 may also read the program 97 stored in a semiconductor memory 98, such as a flash memory, installed in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown) and save it to the auxiliary storage device 23.

[0169] Program 97 is installed as a control program for the computer 90, loaded into the main memory 22, and executed. Thus, the information processing system 10 described in Embodiment 1 is realized. Program 97 in this embodiment is an example of a program product.

[0170] Program 97 may be provided on a recording medium or distributed from an external computer. The computer program can be deployed to run on a single computer or at a single site, or distributed across multiple sites and interconnected by a communication network.

[0171] The technical features (constituent elements) described in each embodiment are combinable with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims, not in the sense described above, and all modifications within the meaning and scope equivalent to the claims are intended to be included.

[0172] The independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. Furthermore, while the claims use a multi-claim format in which claims refer to two or more other claims (multi-claim format), this is not the only option. A multi-claim format in which at least one multi-claim is referenced (multi-multi-claim format) is also acceptable.

[0173] 10 Information Processing System 20 Information Processing Device 21 Control Unit 22 Main Memory Unit 23 Auxiliary Memory Unit 24 Communication Unit 25 Display Unit 26 Input Unit 29 Reading Unit 30 Catheter System 31 Catheter Control Device 32 Image Acquisition Catheter 321 Tip Marker 322 Sensor 323 Shaft 324 Sheath 33 MDU 39 Guiding Catheter 41 Angiography Device 421 C-Arm 422 X-ray Tube 423 Flat Panel Detector 55 GC Judgment Model 61 Angiographic Image 611 Reference Image 615 Magnified Area 619 Cursor 621 First Landmark 622 Second Landmark 623 Tip-side Reference Position 624 Sensor Image 625 Averaged Image 626 Modified Image 632 Second Landmark Marker 633 Path Line (Path) 636 Transverse image marker 638 Proximal marker 639 Proximal marker 641 First region marker 642 Second region marker 66 Vascular image section 671 First slider 672 Second slider 682 Next button 683 Back button 684 Confirm button 71 Transverse image 711 OCT tomography 712 IVUS tomography 72 Longitudinal image 90 Computer 96 Portable recording medium 97 Program 98 Semiconductor memory

Claims

1. A program that causes a computer to perform the following processes:

1. Acquire multiple transverse layer images generated by three-dimensional scanning using an image acquisition catheter that acquires images while moving a scanning plane through an image acquisition sensor in the axial direction, and multiple angiography images depicting the image acquisition catheter during the three-dimensional scanning; acquire a reference position in a reference image selected from the acquired multiple angiography images and the position of a second landmark located within the three-dimensional scanning range of the image acquisition catheter; generate a path of the image acquisition catheter from the reference position to the second landmark in the reference image; acquire the position of the image acquisition sensor from each of the angiography images; calculate the moving average position of the image acquisition sensor in the angiography images arranged in the order of acquisition for each of the angiography images; divide the path based on the distance between the moving average positions in temporally adjacent angiography images to determine the position of the image acquisition sensor in the path for each of the angiography images; and determine the position of each of the transverse layer images in the path based on the frame rate of the angiography images and the frame rate of the transverse layer images.

2. The program according to claim 1, wherein the second landmark is the tip of the guiding catheter through which the image acquisition catheter is inserted.

3. The program according to claim 1, wherein the reference position is the position of the image acquisition sensor at the start of three-dimensional scanning.

4. The program according to claim 1, wherein the reference position is the position of a first landmark located outside the three-dimensional scanning range of the image acquisition catheter.

5. The program according to claim 4, wherein the first landmark is a tip marker provided on the image acquisition catheter.

6. The program according to claim 5, which calculates a virtual frame number indicating which of the multiple transverse layer images corresponds to the first tomographic image in which the first landmark is depicted, assuming that the three-dimensional scan has been performed up to the position of the first landmark; determines the number of captured frames indicating which of the multiple transverse layer images corresponds to the second tomographic image in which the second landmark is depicted; and uses the positions obtained by dividing the path from the tip side by the virtual frame number versus the captured frame number as the positions of the image acquisition sensor at the start of the three-dimensional scan.

7. The program according to claim 1, which displays a plurality of angiographic images and accepts the selection of the reference image.

8. The program according to claim 1, which displays the reference image and receives the reference position and the position of the second landmark in the displayed reference image.

9. The program according to claim 1, which superimposes the generated path of the image acquisition catheter onto the reference image and accepts instructions for modifying the path.

10. The program according to claim 1, which calculates the distance over which the position of the image acquisition sensor changes in the path between two angiography images taken in succession; calculates the movement speed of the image acquisition sensor based on the calculated distance and the angiography image acquisition interval determined based on the frame rate of the angiography image; calculates the movement interval of the image acquisition sensor for each transverse image acquisition interval by multiplying the calculated movement speed of the image acquisition sensor by the transverse image acquisition interval determined based on the frame rate of the transverse image; and identifies the position of each transverse image in the path by dividing the path based on the calculated movement interval of the image acquisition sensor.

11. The program according to any one of claims 1 to 10, wherein the image acquisition catheter is an OCT (Optical Coherence Tomography) catheter.

12. The program according to any one of claims 1 to 10, wherein the image acquisition catheter is an IVUS (Intravascular Ultrasound) catheter.

13. Multiple transverse layer images generated by three-dimensional scanning using an image acquisition catheter that acquires images while moving the scanning plane through the image acquisition sensor in the axial direction, and multiple angiography images depicting the image acquisition catheter during the three-dimensional scanning are acquired. A reference position and the position of a second landmark located within the three-dimensional scanning range of the image acquisition catheter are acquired in a reference image selected from the acquired multiple angiography images. A path of the image acquisition catheter from the reference position to the second landmark is generated in the reference image. The position of the image acquisition sensor is acquired from each of the angiography images. The moving average position of the image acquisition sensor in the angiography images arranged in the order of acquisition is calculated for each of the angiography images. The position of the image acquisition sensor in the path is determined for each of the angiography images by dividing the path based on the distance between the moving average positions in temporally adjacent angiography images. An information processing method in which a computer performs a process to identify the position of each of the transverse images along the path based on the frame rate of the angiographic image and the frame rate of the transverse image.

14. An information processing apparatus having a control unit, wherein the control unit acquires a plurality of transverse layer images generated by three-dimensional scanning using an image acquisition catheter that acquires images while moving a scanning plane through an image acquisition sensor in the axial direction, and a plurality of angiography images in which the image acquisition catheter is being scanned in the three-dimensional scan, acquires a reference position and the position of a second landmark located within the three-dimensional scanning range of the image acquisition catheter in a reference image selected from the plurality of acquired angiography images, generates a path of the image acquisition catheter from the reference position to the second landmark in the reference image, acquires the position of the image acquisition sensor from each of the angiography images, calculates the moving average position of the image acquisition sensor in the angiography images arranged in the order of acquisition for each of the angiography images, and determines the position of the image acquisition sensor in the path for each of the angiography images by dividing the path based on the distance between the moving average positions in temporally adjacent angiography images. An information processing device that identifies the position of each of the transverse images along the path based on the frame rate of the angiographic image and the frame rate of the transverse image.

Images