Co-registration of fluoroscopy images with angiography and intravascular ultrasound images

The co-registration of fluoroscopy, angiography, and IVUS images using AI-driven landmark detection and path mapping improves procedural accuracy and safety in endovascular interventions by providing a comprehensive view of vascular structures.

WO2026241236A1PCT designated stage Publication Date: 2026-11-19BOSTON SCIENTIFIC SCIMED INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/027747
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-10-15
Filing Date
2026-05-12
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing vascular imaging modalities like fluoroscopy, angiography, and IVUS provide different levels of contrast and imaging perspectives, making it difficult to accurately align and register vascular structures without contrast, leading to variability in treatment device placement during endovascular procedures.

Method used

The co-registration of fluoroscopy, angiography, and IVUS images is achieved through automated landmark detection and path mapping using artificial intelligence, enabling real-time overlay of fluoroscopy images onto IVUS and angiographic images, with automated identification of guide entry and catheter tip locations.

Benefits of technology

This enhances procedural accuracy and safety by providing a comprehensive view of vascular structures, reducing reliance on physician memory and variability, and supporting informed decision-making during endovascular interventions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026027747_19112026_PF_FP_ABST
    Figure US2026027747_19112026_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure describes a method for displaying fluoroscopy images mapped to co-registered angiographic and intravascular ultrasound (IVUS) images. The mapped fluoroscopy images may be generated as part of a graphical user interface that is displayed on a display device such as a video display screen. The fluoroscopy images may be displayed in real time during a procedure being performed on a vessel, such as a cardiac artery, and may include one or more graphical indications of the location, orientation, and positioning of a treatment device or tool positioned within the vessel and being used as part of the treatment procedure being performed on the vessel.
Need to check novelty before this filing date? Find Prior Art

Description

Docket No. 3001.1378111BSC File No. 25-1102W001 CO-REGISTRATION OF FLUOROSCOPY IMAGES WITH ANGIOGRAPHY AND INTRAVASCULAR ULTRASOUND IMAGESCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of U.S. Patent Application Serial No.63 / 805,422, filed May 14, 2025, entitled "CO-REGISTRATION OF MULTI-MODALITY VASCULAR IMAGES”, and U.S. Patent Application Serial No. 63 / 899,461, filed October 15, 2025, entitled "CO-REGISTRATION OF FLUOROSCOPY IMAGES WITH ANGIOGRAPHY AND INTRAVASCULAR ULTRASOUND IMAGES”, which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The present disclosure generally relates to vascular imaging systems and image processing, such as intravascular ultrasound, angiography, and fluoroscopy processes and imaging procedures. Particularly, but not exclusively, the present disclosure relates to improved image processing across multiple imaging modalities to enhance procedural accuracy and safety during endovascular intervention procedures.BACKGROUND

[0003] Various types of imaging modalities may be used by medical professionals, such as cardiologist, to generate images and other information related to the configuration and condition of a patient’s vascular system, including cardiac vessels. Intravascular imaging (IVI) devices are insertable into a patient's vasculature and configured to capture images from within the vessel lumen. Two intravascular imaging modalities are intravascular ultrasound (IVUS) and optical coherence tomography (OCT). Angiography is a medical imaging procedure that uses a contrast dye to highlight blood vessels and improve their visibility on X-rays or other imaging techniques.

[0004] Medical fluoroscopy imaging involves capturing and replaying short, cyclical video sequences of moving structures to visualize motion and changes over time. This technique, available in modalities like ultrasound, fluoroscopy, and MRI, enhances diagnostic accuracy by allowing clinicians to observe subtle movements and physiological processes that are difficult to capture in single static images. In particular, fluoroscopy is an imaging technique that produces a real-time "video" of vascul r structures, and can provide real-time images ofDocket No. 3001.1378111BSC File No. 25-I102W001 positioning and movement of instruments, such as vascular treatment devices, which are inside a vascular structure of a patient. During a procedure, a medical professional may use a contrast agent to highlight specific areas and can guide instruments or monitor the progress of a procedure in real time using fluoroscopy imaging techniques.

[0005] While each of these imaging modalities have proven diagnostic capabilities for a variety of diseases and disorders, each of these imaging modalities provide different levels of contrast and imaging perspectives, with associated limitations as to the content and perspective provided by any given one of these different imaging modalities.BRIEF SUMMARY

[0006] This Brief Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to necessarily identify key features or essential features of the claimed subject matter, nor is it intended as an aid in determining the scope of the claimed subject matter.

[0007] The present disclosure relates to co-regi strati on and the mapping of images of different imaging modalities, in some embodiments in real time, of images of vascular structures, such as cardiac arteries. In various embodiments, images captured using intravascular ultrasound are co-registered with an angiographic image captured for the same vasculature structures using an angiography imaging procedure, in some embodiments performed in conjunction with the IVUS imaging procedure. Using the co-registered IVUS and angiographic images, fluoroscopy images captured at various time intervals or in real time, for example using fluoroscopy as an imaging technique, are mapped into the IVUS and the angiogram images. The mapping of the fluoroscopy images onto the IVUS and angiogram images may be performed in real time, and rendered as a visual display on a graphical user interface (GUI). The display is viewable by the medical professional as a treatment procedure, such as during a calcium modification or a stent placement procedure. Embodiments as described in this disclosure may be implemented through any of the example systems, apparatus, methods, and techniques, including the following examples.

[0008] Embodiments of the disclosure can be implemented as a method comprising: receiving, at a computing device, a sequence of fluoroscopy images provided by a fluoroscopy imaging device, wherein the sequence of fluoroscopy images includes images of a vessel of a patient; detecting in the fluoroscopy images the presence of a treatment device positioned within theDocket No. 3001.1378111BSC File No. 25-1102W001 vessel; and mapping a location of the treatment device detected in the fluoroscopy images onto at least one image of the vessel captured using at least one imaging modality that is different from the imaging modality provided by the fluoroscopy imaging device.

[0009] Alternatively or additionally, embodiments of the method further comprise displaying, on a display device and in real time, a graphical indication of the location of the treatment device in each one of the sequence of fluoroscopy images as each of the sequence of fluoroscopy images is received.

[0010] Alternatively or additionally, embodiments of the method comprise displaying, on a display device and in real time, a longview of the vessel including a graphical indication of the location of the treatment device as detected in the fluoroscopy images relative to a longitudinal position of the treatment device within the vessel.

[0011] Alternatively or additionally, embodiments of the method comprise wherein the at least one different imaging modality includes intravascular ultrasound (IVUS) imaging, and wherein the longview of the vessel is generated using a series of intravascular ultrasound image frames captured using IVUS imaging of the vessel.

[0012] Alternatively or additionally, embodiments of the method comprise wherein the mapping of the location of the treatment device detected in the fluoroscopy images is based on a pullback path utilized by the IVUS imaging to generate the series of intravascular ultrasound image frames.

[0013] Alternatively or additionally, embodiments of the method comprise wherein the at least one different imaging modality includes an angiographic image captured using an X-ray imaging device and captured while a contrast agent is present within the vessel.

[0014] Alternatively or additionally, embodiments of the method further comprise wherein the angiographic image is captured while an intravascular ultrasound imaging catheter is extended distally from a guide catheter positioned within the vessel, the intravascular ultrasound imaging catheter including a catheter tip positioned within the vessel and distally of a distal end of the guide catheter.

[0015] Alternatively or additionally, embodiments of the method further comprise: receiving, at the computing device, the angiographic image; determining within the angiographic image a location of the a guide entry point of the guide catheter; determining within the angiographic image a location of the catheter tip; determining a centerline extending through the vessel asDocket No. 3001.1378111BSC File No. 25-1102W001 imaged in the angiographic image, the centerline extending from the guide entry point of the guide catheter to the catheter tip; and mapping the location of the treatment device detected in the fluoroscopy or fluoroscopy images based on a path length determined from the centerline.

[0016] Alternatively or additionally, embodiments of the method comprise wherein the at least one at least one imaging modality that is different from the imaging modality provided by the fluoroscopy imaging device comprises intravascular ultrasound imaging and angiography imaging, and wherein the mapping of the location of the treatment device detected in the fluoroscopy images onto at least one image of the vessel comprises mapping the location the treatment device onto an angiographic image co-registered with a series of intravascular ultrasound images.

[0017] Alternatively or additionally, embodiments of the method comprise wherein the treatment device comprises an angioplasty balloon.

[0018] Alternatively or additionally, embodiments of the method comprise wherein the treatment device comprises a stent.

[0019] Alternatively or additionally, embodiments of the method comprise wherein the treatment device comprises a drug-eluting balloon.

[0020] Alternatively or additionally, embodiments of the method comprise wherein the graphical indication of the location of the treatment device in the fluoroscopy images is determined using artificial intelligence.

[0021] Embodiments of the disclosure can be implemented as a computer-readable storage device, comprising instructions executable by a processor of a computing device coupled to a fluoroscopy imaging device, wherein when executed the instructions cause the computing device to implement the method of any of the above embodiments.

[0022] Embodiments of the disclosure can be implemented as an apparatus comprising a processor arranged to be coupled to a fluoroscopy imaging device, the apparatus further comprising a memory comprising instructions, the processor arranged to execute the instructions to implement the method of any of the above embodiments.

[0023] Embodiments of the disclosure can be implemented as a fluoroscopy imaging system configured to be coupled to a fluoroscopy imaging device, the fluoroscopy imaging system comprising: processing circuitry and a memory comprising instructions, which when executed by the processing circuity cause the fluoroscopy imaging system to: receive, at the fluoroscopyDocket No. 3001.1378111BSC File No. 25-1102W001 imaging system, a sequence of fluoroscopy images provided by the fluoroscopy imaging device, wherein the sequence of fluoroscopy images includes images of a vessel of a patient; detect in the fluoroscopy images the presence of a treatment device positioned within the vessel; and map a location of the treatment device detected in the fluoroscopy images onto at least one image of the vessel captured using at least one imaging modality that is different from the imaging modality provided by the fluoroscopy imaging device.

[0024] Alternatively or additionally, embodiments of the fluoroscopy imaging system comprise wherein the instructions, when executed by the processing circuity cause the fluoroscopy imaging system further to: display, on a display device and in real time, a graphical indication of the location of the treatment device in each one of the sequence of fluoroscopy images as each of the sequence of fluoroscopy images is received; and display, on the display device and in real time, a longview of the vessel including a graphical indication of the location of the treatment device as detected in the fluoroscopy images relative to a longitudinal position of the treatment device within the vessel.

[0025] Alternatively or additionally, embodiments of the fluoroscopy imaging system comprise wherein the at least one different imaging modality includes intravascular ultrasound (IVUS) imaging, and wherein the longview of the vessel is generated using a series of intravascular ultrasound image frames captured using IVUS imaging of the vessel.

[0026] Alternatively or additionally, embodiments of the fluoroscopy imaging system comprise wherein the mapping of the location of the treatment device detected in the fluoroscopy images is based on a pullback path utilized by the IVUS imaging to generate the series of intravascular ultrasound image frames.

[0027] Alternatively or additionally, embodiments of the fluoroscopy imaging system comprise wherein the at least one different imaging modality includes an angiographic image captured using an X-ray imaging device and captured while a contrast agent was present within the vessel.

[0028] Alternatively or additionally, embodiments of the fluoroscopy imaging system comprise wherein the angiographic image was captured while an intravascular ultrasound imaging catheter was extended distally from a guide catheter positioned within the vessel, the intravascular ultrasound imaging catheter including a catheter tip positioned within the vessel and distally of a distal end of the guide catheter.Docket No. 3001.1378111BSC File No. 25-1102W001

[0029] Alternatively or additionally, embodiments of the fluoroscopy imaging system comprise instructions, which when executed by the processing circuity cause the fluoroscopy imaging system further to: receive, at the fluoroscopy processing system, the angiographic image; determine within the angiographic image a location of the a guide entry point of the guide catheter; determine within the angiographic image a location of the catheter tip; determine a centerline extending through the vessel as imaged in the angiographic image, the centerline extending from the guide entry point of the guide catheter to the catheter tip; and map the location of the treatment device detected in the fluoroscopy images based on a path length determined from the centerline.

[0030] Alternatively or additionally, embodiments of the fluoroscopy imaging system comprise wherein the at least one at least one imaging modality that is different from the imaging modality provided by the fluoroscopy imaging device comprises intravascular ultrasound imaging and angiography imaging, and wherein the mapping of the location of the treatment device detected in the fluoroscopy images onto at least one image of the vessel comprises mapping the location the treatment device onto an angiographic image co-registered with a series of intravascular ultrasound images.

[0031] Embodiments of the disclosure can be implemented as a non-transitory computer-readable storage device comprising computer executable instructors that, when executed by a processor in fluoroscopy imaging system cause the fluoroscopy imaging system to: receive a sequence of fluoroscopy images, wherein the sequence of fluoroscopy images are provided by a fluoroscopy imaging device and wherein the sequence of fluoroscopy images includes images of a vessel of a patient; detect in the fluoroscopy images the presence of a treatment device positioned within the vessel; and map a location of the treatment device detected in the fluoroscopy images onto at least one image of the vessel captured using at least one imaging modality that is different from the imaging modality provided by the fluoroscopy imaging device.

[0032] Alternatively or additionally, embodiments of the non-transitory computer-readable storage device include, instructions, which when executed by the processor further cause the fluoroscopy imaging system further to: render and display on a display device in real time a graphical user interface (GUI) including: a display of a first graphical indication of the location of the treatment device in each one of the sequence of fluoroscopy images as each of theDocket No. 3001.1378111BSC File No. 25-1102W001 sequence of fluoroscopy images is received; and a display of a longview of the vessel including a second graphical indication of the location of the treatment device as detected in the fluoroscopy images relative to a longitudinal position of the treatment device within the vessel.

[0033] Alternatively or additionally, embodiments of the non-transitory computer-readable storage device include wherein the at least one at least one imaging modality that is different from the imaging modality provided by the fluoroscopy imaging device comprises intravascular ultrasound imaging and angiography imaging, and wherein the mapping of the location of the treatment device detected in the fluoroscopy images onto at least one image of the vessel comprises instructions that when executed by the processor further cause the fluoroscopy imaging system to map the location of the treatment device onto an angiographic image that is co-registered with a series of intravascular ultrasound images.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0034] To easily identify the discussion of any element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced.

[0035] FIG. 1 illustrates a multi-modality co-registration and mapping system, in accordance with various embodiments of the present disclosure.

[0036] FIG. 2 illustrates a combined internal and external imaging system including both an endoluminal imaging system and an extravascular imaging system.

[0037] FIG. 3A illustrates an example cardiac vasculature structure and an IVUS catheter inserted into the cardiac vasculature structure.

[0038] FIG. 3B illustrates an IVUS imaging system including the IVUS catheter as shown in FIG. 3A, in accordance with various embodiments of the present disclosure.

[0039] FIG. 3C and FIG. 3D illustrate a series of IVUS image frames and a longitudinal view of the series of IVUS image frames.

[0040] FIG. 3E illustrates an angiographic image of an artery corresponding to the artery depicted in FIG. 3A.

[0041] FIG. 3F illustrates an annotated angiographic image, in accordance with various embodiments of the present disclosure.Docket No. 3001.1378111BSC File No. 25-I102W001

[0042] FIG. 4A illustrates an example of a treatment procedure being performed on the cardiac vasculature structure, in accordance with various embodiments of the present disclosure.

[0043] FIG. 4B illustrates a fluoroscopy image, in accordance with various embodiments of the present disclosure.

[0044] FIG. 4C illustrates a fluoroscopy image, which in various embodiments is an annotated variation of fluoroscopy image of FIG. 4B.

[0045] FIG. 4D illustrates a fluoroscopy image, in various embodiments illustrative of a further annotated version of the fluoroscopy image of FIG. 4C.

[0046] FIG. 4E illustrates a longview image, in accordance with various embodiments of the present disclosure.

[0047] FIG. 5A illustrates a first example graphical user interface (GUI), which may be generated according to various embodiments of the present disclosure.

[0048] FIG. 5B illustrates a second example GUI, which may be generated according to various embodiments of the present disclosure.

[0049] FIG. 5C illustrates a third example GUI, which may be generated according to various embodiments of the present disclosure.

[0050] FIG. 6 illustrates a logic flow, according to various embodiments as described in the present disclosure.

[0051] FIG. 7 illustrates computer-readable storage medium, according to various embodiments as described in the present disclosure.

[0052] FIG. 8 illustrates a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.DETAILED DESCRIPTION

[0053] The foregoing has broadly outlined the features and technical advantages of the present disclosure such that the following detailed description of the disclosure may be better understood. It is to be appreciated by those skilled in the art that the embodiments disclosedDocket No. 3001.1378111BSC File No. 25-1102W001 may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. The novel features of the disclosure, both as to its organization and operation, together with further objects and advantages will be better understood from the following description when considered in connection with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description and is not intended as a definition of the limits of the present disclosure.

[0054] In many cases, physicians rely on visual memory and anatomical landmarks when advancing a balloon or other treatment device to a lesion or treatment site without contrast being provided in the vessel. This process is highly dependent on their clinical experience and spatial awareness. However, it also introduces a degree of human variability. Differences in training, familiarity with the anatomy, and interpretation of fluoroscopic cues can all influence the accuracy of balloon placement during a treatment procedure being performed on a vessel or vessel structure.

[0055] Various imaging modalities, such as fluoroscopy, angiography, and IVUS each offer different levels of contrast and imaging perspectives. Fluoroscopy is a real-time X-ray imaging technique often used to visualize movement within the body, guide procedures, or position devices like catheters or balloons. It provides continuous imaging, where contrast is not typically used in fluoroscopy alone, especially for general imaging or guidance purposes. In fluoroscopy, without contrast, vascular landmarks are not visible, making it difficult to align or register them with angiographic images. To address these issues, embodiments as described herein utilize smart tools, such as artificial intelligent (Al) tools, to detect reliable landmarks — such as radiopaque markers and the guide catheter — and then map the advancement path using a centerline algorithm. Using these co-regi strati on techniques, an alignment of angiographic and IVUS images can be used to overlay an IVUS pullback path along a vessel onto an angiographic image. Once both the advancement path and pullback path are established, embodiments as disclosed herein can use either length-based or path-based correlation to map landmarks from the fluoroscopy non-contrast images onto both the angiographic image and the IVUS longitudinal view.

[0056] Automated co-regi strati on of IVUS images and / or OCT images with angiography images is clinically important because it enhances evaluation of coronary artery disease in aDocket No. 3001.1378111BSC File No. 25-1102W001 patient by providing detailed insights and a more comprehensive view into plaque characteristics, plaque compositions, severity, and vessel structure of the imaged vessel of the patient. The co-regi strati on of IVUS images and / or OCT images with angiography images improves efficiency and supports informed clinical decision making.

[0057] However, successful co-regi strati on of angiography with Intravascular Ultrasound (IVUS) image data or with Optical Coherence Tomography (OCT) image data requires the accurate identification of the guide entry (i.e., the distal endpoint location of a guide catheter), and the catheter tip of the imaging device on angiography images to ensure proper alignment of their start and end points. In various existing systems, the identification of the guide entry as the distal endpoint location of the guide catheter, (guide entry point) location, and the identification of the catheter tip of the imaging device, are performed manually by a clinician. As such, systems and methods for identifying guide entry point location and location of the catheter tip of the imaging device(s) are desirable. The term “guide entry,” and “guide entry point,” and “guide entry location” may be used interchangeably throughout this specification to refer to the distal endpoint location of a guide catheter.

[0058] The IVUS catheter or the OCT catheter, depending on which type of imaging device is being utilized, is configured to reenter the guide catheter at the guide entry point, i.e., at the distal end of the guide catheter through which the imaging device has been inserted in order to access the vessel being imaged. Although IVUS imaging devices and IVUS images captured using IVUS imaging devices are referred to throughout this disclosure, the system, apparatus, methods, and techniques of the various embodiments as described herein are equally applicable to the OCT imaging device and OCT images captured using OCT imaging devices.

[0059] The systems, apparatus, methods and techniques as described herein automate the coregistration process to accurately identify guide entry and IVUS catheter tip locations, using machine learning approaches, facilitating fully automated co-registration of IVUS and angiography images, and replacing the manual identification processes currently being utilized by clinicians. The systems, apparatus, methods and techniques as described herein are configured and arranged to automatically detect guide entry and IVUS catheter tips. The automatic identification of start and end points of the IVUS catheter on angiography images will facilitate fully automated co-registration of IVUS images and angiography images. As noted above, currently clinicians manually identify the start point (guide-entry) and end pointDocket No. 3001.1378111BSC File No. 25-1102W001 (catheter tip) on angiographic images. Automated identification of these two crucial points for centerline generation on angiographic images and co-regi strati on of the two image modalities will increase the accuracy of these identified locations, and thereby assist clinicians in making more efficient and informed decisions. IVUS and angiography co-regi stration enhances the assessment of coronary artery disease by offering a comprehensive understanding of plaque composition, severity, and vessel anatomy.

[0060] Embodiments as described in this disclosure provide for real-time co-regi strati on of intravascular treatment tools — such as balloon catheters — by mapping their position from noncontrast fluoroscopy onto both contrast-enhanced angiography and IVUS longview images. This approach enables physicians to:• Visualize the treatment tool in real time across multiple imaging modalities.• Confirm alignment with the target anatomy, even when contrast is not used in the fluoroscopy.• Enhance procedural accuracy and safety during endovascular interventions.

[0061] By integrating Al-based co-registration and fluoroscopy mapping, the embodiments as described herein not only enhance procedural accuracy but also extend the functional capabilities of current imaging devices. These innovations provide systems and methods configured to:• Enable landmark detection and path mapping without contrast, reducing reliance on physician memory and minimizing variability.• Overlay advancement and pullback paths across modalities like angiography and IVUS, improving spatial orientation and procedural planning.• Support more intuitive workflows, especially in complex or contrast-limited cases, which can lead to better clinical outcomes.

[0062] This advancement transforms presently available imaging systems from passive visualization tools into active procedural guidance platforms, and therefore represent improvement to the field of medical vascular imaging.

[0063] As noted, the disclosure uses intravascular ultrasound (IVUS) imaging, angiography, and fluoroscopy non-contrast images as the example imaging modalities. As such, an example IVUS imaging system, cardiac vasculature structure, and series of IVUS images, angiographicDocket No. 3001.1378111BSC File No. 25-1102W001 images, and angiography imaging systems, and fluoroscopy imaging systems are described prior to describing the improvements to IVUS guidance systems and the GUI of the present disclosure.

[0064] FIG. 1 illustrates a multi -modality co-regi strati on and mapping system (system) 100, in accordance with various embodiments of the present disclosure. In general, system 100 is a system configured perform co-regi strati on of intravascular images, such as IVUS images, angiography, and then map, in real time, fluoroscopy images onto the co-registered IVUS and angiography images (i.e., computer tomography angiography (CTA) images).

[0065] As shown in FIG. 1, system 100 includes, or can be coupled to, a vascular imaging system 102. The vascular imaging system 102 can be any of a variety of vascular imaging systems configured to capture images from multiple imaging modalities (e.g., angiography, CTA, IVUS, intravascular OCT, fluoroscopy imaging, or the like). As shown in FIG. 1, vascular imaging system 102 includes an IVUS imaging system 101, an angiography imaging system 103, and a fluoroscopy imaging system 105. Vascular imaging system 102 may include an intravascular ultrasound imaging system, such as IVUS imaging system 320 illustrated and described below with respect to FIG. 3A, and configured to perform a drawback imaging procedure when positioned within a vessel of a patient to capture a series of IVUS images, for example and illustrated and described below with respect to IVUS images 330 and FIG. 3C.

[0066] Referring back to FIG. 1, angiography imaging system 103 may be an imaging system configure to capture angiographic images of a vessel or vessel structure of a patient, using an X-ray imaging device and after a contract agent, such as iodinated contrast media or barium sulfate, has been injected into the patient and has arrived at the portion of the vessel or vessel structure to be imaged. Angiography imaging system 103 may be configured to generate and output one or more angiographic images captured using the angiographic imaging procedures performed by the fluoroscopy imaging system 105 in various embodiments is an imaging system configured to capture video images, in real time, of a vessel or vessel structure of a patient. In various embodiments, the fluoroscopy images may be provided in real time as a medical procedure, such as a calcium fracturing procedure, a balloon angioplasty treatment procedure, and / or a stent placemen procedure, is being performed on the vessel or vessel structure of the patient. As further described below, the fluoroscopy images being provided by the fluoroscopy imaging system 105 may be mapped, in real time, onto one or more imagesDocket No. 3001.1378111BSC File No. 25-1102W001 provided by the IVUS imaging system 101 and / or the one or more images provided by the angiography imaging system 103.

[0067] Example embodiments of a vascular imager system(s) configured to capture external (X-ray angiography, CTA) and internal (IVUS) vascular images, and in some embodiments configured to provide the fluoroscopy images, are described with reference to the combined internal and external imaging system 200 depicted in FIG. 2.

[0068] System 100 includes a computing device 104. The computing device 104 can be any of a variety of computing devices. In some embodiments, computing device 104 can be incorporated into and / or implemented by a console of the vascular imaging system 102. With some embodiments, computing device 104 can be a tablet, laptop, workstation, or server communicatively coupled to vascular imaging system 102. With still other embodiments, computing device 104 can be provided by a cloud-based computing device, such as, by a Computing as a Service (CaaS) system accessibly over a network (e g., the Internet, an intranet, a wide area network, or the like). Computing device 104 can include processor 106, memory 108, input and / or output (I / O) device 110, network interface 112 and imaging system acquisition circuitry 114.

[0069] The processor 106 may include circuity or processor logic, such as any of a variety of commercial processors. In some examples, processor 106 may include multiple processors, a multi-threaded processor, a multi-core processor (whether the multiple cores coexist on the same or separate dies), and / or a multi-processor architecture of some other variety by which multiple physically separate processors are in some way linked. Additionally, in some examples, the processor 106 may include graphics processing portions and may include dedicated memory, multiple-threaded processing and / or some other parallel processing capability. In some examples, the processor 106 may be an application specific integrated circuit (ASIC) or a field programmable integrated circuit (FPGA).

[0070] The memory 108 may include logic, a portion of which includes arrays of integrated circuits, forming non-volatile memory to persistently store data or a combination of nonvolatile memory and volatile memory. It is to be appreciated, that the memory 108 may be based on any of a variety of technologies. In particular, the arrays of integrated circuits included in memory 108 may be arranged to form one or more types of memory, such as, for example, dynamic random-access memory (DRAM), NAND memory, NOR memory, or the like.Docket No. 3001.1378111BSC File No. 25-1102W001

[0071] The I / O devices 110 can be any of a variety of devices to receive input and / or provide output. For example, I / O devices 110 can include, a keyboard, a mouse, a joystick, a foot pedal, a haptic feedback device, an LED, or the like. I / O devices may include or be configured to control a conventional display or a touch-enabled display, which can utilize a variety of display technologies, such as, liquid crystal display (LCD), light emitting diode (LED), or organic light emitting diode (OLED), or the like.

[0072] The network interface 112 can include logic and / or features to support a communication interface. For example, network interface 112 may include one or more interfaces that operate according to various communication protocols or standards to communicate over direct or network communication links. Direct communications may occur via use of communication protocols or standards described in one or more industry standards (including progenies and variants). For example, network interface 112 may facilitate communication over a bus, such as, for example, peripheral component interconnect express (PCIe), non-volatile memory express (NVMe), universal serial bus (USB), system management bus (SMBus), SAS (e.g., serial attached small computer system interface (SCSI)) interfaces, serial AT attachment (SATA) interfaces, or the like. Additionally, network interface 112 can include logic and / or features to enable communication over a variety of wired or wireless network standards (e.g., 802.11 communication standards). For example, network interface 112 may be arranged to support wired communication protocols or standards, such as, Ethernet, or the like. As another example, network interface 112 may be arranged to support wireless communication protocols or standards, such as, for example, Wi-Fi, Bluetooth, ZigBee, LTE, 5G, or the like.

[0073] The imaging system acquisition circuitry 114 may include circuity including custom manufactured or specially programmed circuitry configured to receive or receive and send signals with IVUS imaging system 101, including indications of intravascular images, intravascular image frames, or a series of intravascular image frames. For example, imaging system acquisition circuitry 114 can include image acquisition circuitry (not shown) and / or a pulse generator. In addition, the imaging system acquisition circuitry 114 may include circuity including custom manufactured or specially programmed circuitry configured to receive or receive and send signals with imaging system 102, including indications of intravascular images, intravascular image frames, or a series of intravascular image frames. For example, imaging system acquisition circuitry 114 can include image acquisition circuitry (not shown) toDocket No. 3001.1378111BSC File No. 25-1102W001 receive and process images captured by an angiographic imaging system, such as angiographic imaging system 103 and fluoroscopy imaging system 105.

[0074] Memory 108 can include instructions 116, series of IVUS image frames 120, angiographic data 122, fluoroscopy image data 124, IVUS / Angio co-regi strati on images 125, mapped fluoroscopy images 126, and graphical user interface (GUI) 128.

[0075] During operation, processor 106 can execute instructions 116 to cause computing device 104 to receive the series of IVUS image frames 120 from IVUS imaging system 102, angiographic image data 122 from angiographic imaging system 103, and fluoroscopy image data 124 from fluoroscopy imaging system 105. With some examples, processor 106 can execute instructions 116 to cause computing device 104 to receive the series of IVUS image frames 120 while they are captured, and the angiographic image data 122 as these images are being captured. In other examples, processor 106 can execute instructions 116 to receive the series of IVUS image frames 120 or the angiopathic image data 122 while the image data is captured, where the other image modality will have been captured previously. In yet other examples, processor 106 can execute instructions 116 to cause computing device 104 to receive the IVUS image frames 120 and the angiographic image data 122, which were both previously captured. In any of these examples, processor 106 can execute instructions 116 to cause computing device 104 to receive data structures of information comprising indications of the IVUS image frames 120 and the angiographic image data 122 (e.g., from vascular imaging system 102 directly, from network interface 112, from a memory location, or the like).

[0076] In general, the angiographic image data 122 can be a computer tomography (CT) image of a patient's heart (or portion of a patient's heart) captured after injection of a contrast agent into the patient's vasculature. With some examples, angiographic image data 122 can be a three-dimensional volume image. Similarly, IVUS image frames 120 can be a series of ultrasound images captured from within a vessel of the patient's heart as an ultrasound probe is pulled back through a portion of the vessel. With some embodiments, the angiographic image data 122 can be captured while IVUS image frames 120 is captured. In other embodiments, the angiographic image data 122 can be captured and then the IVUS image frames 120 can be captured, or vice versa.

[0077] As part of the co-regi strati on of the IVUS images with the angiographic image data, the instructions 116 may include programing that provides to “match” the side branches detected based on IVUS image frames 120 with side branches detected based on CTA imageDocket No. 3001.1378111BSC File No. 25-I102WO0I data for purposes of co-registration of the images captured by these different image modalities. In general, processor 106 can execute instructions 116 to cause computing device 104 to identify a side branch from CTA image side branch and 3D coronary artery tree information and a side branch from IVUS images side branch information to designate a reference side branch pair. Further, processor 106 can execute instructions 116 to identify candidate side branch pairs from CTA image side branch information and IVUS images side branch information. Candidate side branch pairs comprise several “pairs” of side branches where each pair references a side branch from CTA image side branch information and a side branch from IVUS images side branch information. It is noted that some side branches from either CTA image side branch information 122 or IVUS images side branch information may be in multiple “pairs." Said differently, candidate side branch pairs may not define pairs of side branches having a one side branch to one side branch (1:1) mapping between CTA image side branch information and IVUS images side branch information.

[0078] Where candidate side branch pairs comprise a pair that references a side branch in another pair, processor 106 can execute instructions 116 to cause computing device 104 to generate matched side branches based on candidate side branch pairs. Where candidate side branch pairs do not comprise a pair, and therefore reference a side branch in another pair, processor 106 can execute instructions 116 to cause computing device 104 to designate candidate side branch pairs and matched side branches. Said differently, where the candidate side branch pairs comprise a 1:N (n>l) relationship between paired side branches, for example, where a side branch from IVUS images side branch information is paired with two side branches from CTA image side branch information 122, or vice versa, processor 106 can execute instructions 116 to cause computing device 104 to remove one or more pairs from candidate side branch pairs to generate match side branches having a 1 : 1 pairing (or mapping) between side branches from CTA image side branch information 122 and IVUS images side branch information 134.

[0079] In various embodiments, features in angiography image data 122 and IVUS image frames 120 other than side branch features may be processed and identified as landmarks in the images. For example, features such as a distal end point of a stent located in the imaged portion of a vessel, and / or a proximal end point of the stent located in the imaged portion of the vessel, may be identified in both the angiography image data 122 and IVUS image frames 120, and utilized as CTA landmarks and IVUS landmarks, respectively. Other features, such as plaqueDocket No. 3001.1378111BSC File No. 25-1102W001 and / or lesions present in the imaged portion of the vessel may be utilized as landmarks in the angiographic landmarks and in the IVUS landmarks, respectively. In other words, any feature that can be identified in one or both the image data provided by angiographic image data 122 and / or by IVUS image frames 120 may be considered for use as landmarks in one or more of angiographic landmarks and IVUS landmarks.

[0080] With the IVUS / angio co-registered images 125 available, a fluoroscopy imaging procedure may be performed using fluoroscopy imaging system 105. The fluoroscopy imaging procedure is configured to provide a sequence of fluoroscopy images, in real time, which are provided to computing device 104. The fluoroscopy images provided to computing device 104 by the fluoroscopy imaging system 105 may be “mapped” in real time into the IVUS coregistered images 125 to generate the mapped fluoroscopy images 126. The mapped fluoroscopy images may then be displayed, in real time, as graphical user interface (GUI) 128 on a display device, such as display 129. The displayed fluoroscopy image may be displayed in real time and include imaging of a treatment device or tool being utilized to perform a medical procedure on the imaged portion of a vessel. The real-time fluoroscopy images captured and displayed are mapped onto the IVUS / angiographic co-registered images 125 for example to annotate the fluoroscopy images with graphical indications representing the positioning, orientation, and location with the vessel being images mapped onto one or more features of the vessel provided by the IVUS / angiographic co-registered images 125 that might not otherwise be visible in the fluoroscopy images themselves. The additional information provided by the mapped fluoroscopy images 126 being displayed in real time during a medical treatment procedure being performed on a vessel allows for enhanced information about the status of the procedure and regarding the vessel being treated to be provided to the practitioner performing the procedure on a real time basis.

[0081] FIG. 2 illustrates a combined internal and external imaging system 130 including both an endoluminal imaging system 132 (e.g., an IVUS imaging system, or the like) and an extravascular imaging system 134 (e.g., an angiographic imaging system, fluoroscopy imaging system). The combined internal and external imaging system 130 further includes a computing device 136, which includes circuitry, controllers, and / or processor(s) and memory and software as needed. With some embodiments, co-registration and mapping system 100 (FIG. 1) can be incorporated into computing device 136, or co-registration and mapping system 100 can incorporate computing device 136. In general, the endoluminal imaging system 132 can beDocket No. 3001.1378111BSC File No. 25-1102W001 arranged to generate intravascular imaging data (e.g., IVUS images, IVUS image frames 120, or the like) while the extravascular imaging system 134 can be arranged to generate extravascular imaging data (e.g., angiographic images, X-ray images, fluoroscopy imaging, or the like).

[0082] The extravascular imaging system 134 may include a table 133 that may be arranged to provide sufficient space for the positioning of an angiography / fluoroscopy unit c-arm 140 in an operative position in relation to a patient 131 on the drive unit. C-arm 140 can be configured to acquire X-ray or fluoroscopic images in the absence of contrast agent in the blood vessels of the patient 131 and / or acquire angiographic images while there is a presence of contrast agent in the blood vessels of the patient 131.

[0083] Raw radiological image data acquired by the c-arm 140 may be passed to an extravascular data input port 141 via a transmission cable 142. The input port 141 may be a separate component or may be integrated into or be part of the computing device 136. The input port 141 may include a processor that converts the raw radiological image data received thereby into extravascular image data (e.g., angiographic / fluoroscopic image data), for example, in the form of live video, DICOM, or a series of individual images. The extravascular image data may be initially stored in memory within the input port 141 or may be stored within memory of computing device 136. If the input port 141 is a separate component from the computing device 136, the extravascular image data may be transferred to the computing device 136 through a transmission cable 143 and into an input port (not shown) of the computing device 136. In some alternatives, the communication between the devices or processors may be carried out via wireless communication, rather than by cables as depicted.

[0084] The intravascular imaging data may be, for example, IVUS data or OCT data obtained by the endoluminal imaging system 132. The endoluminal imaging system 132 may include an intravascular imaging device such as an imaging catheter 151. The imaging catheter 151 is configured to be inserted within the patient 131 so that its distal end, including a diagnostic assembly or probe 152 (e.g., an IVUS probe), is in the vicinity of a desired imaging location of a blood vessel. A radiopaque material or marker 153 located on or near the probe 152 may provide indicia of a current location of the probe 152 in a radiological image. In some embodiments, imaging catheter 151 and / or probe 152 can include a guide catheter (not shown) that has been inserted into a lumen of the subject (e.g., a blood vessel, such as a coronaryDocket No. 3001.1378111BSC File No. 25-1102W001 artery) over a guidewire (also not shown). However, in some embodiments, the imaging catheter 151 and / or probe 152 can be inserted into the vessel of the patient 131 without a guidewire.

[0085] With some embodiments, imaging catheter 151 and / or probe 152 can include both imaging capabilities as well as other data-acqui sition capabilities. For example, FFR and / or IFR data, data related to pressure, flow, temperature, electrical activity, oxygenation, biochemical composition, or any combination thereof. In some embodiments, imaging catheter 151 and / or probe 152 can further include a therapeutic device, such as a stent, a balloon (e.g., an angioplasty balloon), a graft, a filter, a valve, and / or a different type of therapeutic endoluminal device.

[0086] Imaging catheter 151 is coupled to a proximal connector 155 to couple imaging catheter 151 to an image acquisition device 156. Image acquisition device 156 may be coupled to computing device 136 via a transmission cable 157, or a wireless connection. The intravascular image data may be initially stored in memory within the image acquisition device 156 or may be stored within memory of computing device 136. If the image acquisition device 156 is a separate component from computing device 136, the intravascular image data may be transferred to the computing device 136, via, for example, transmission cable 157.

[0087] The computing device 136 can also include one or more additional output ports for transferring data to other devices. For example, the computer can include an output port to transfer data to a data archive or memory device 160. The computing device 136 can also include a user interface (described in greater detail below) that includes a combination of circuitry, processing components and instructions executable by the processing components and / or circuitry to enable the image identification and vessel routing or pathfinding described herein and / or dynamic co-regi strati on of intravascular and extravascular images using the identified vessel pathway. In some embodiments, computing device 136 can include user interface devices, such as, a keyboard, a mouse, a joystick, a touchscreen device (such as a smartphone or a tablet computer), a touchpad, a trackball, a voice-command interface, and / or other types of user interfaces that are known in the art.

[0088] The user interface can be rendered and displayed on a display 162 coupled to computing device 136 via a display cable 163. Although the display 162 is depicted as separate from computing device 136, in some examples the display 162 can be part of computing deviceDocket No. 3001.1378111BSC File No. 25-1102W001 136. Alternatively, the display 162 can be remote and wireless from computing device 136. As another example, the display 162 can be part of another computing device different from computing device 136, such as, a tablet computer, which can be coupled to computing device 136 via a wired or wireless connection. For some applications, the display 162 includes a heads-up display and / or a head-mounted display. For some applications, the computing device 136 generates an output on a different type of visual, text, graphics, tactile, audio, and / or video output device, e.g., speakers, headphones, a smartphone, or a tablet computer. For some applications, the user interface rendered on display 162 acts as both an input device and an output device.

[0089] In various embodiments, display 162 may be configured to provide graphical displays showing the co-registrations of two or three different imaging modalities, such as coregistration of images generated by endoluminal imaging system 132 (e.g., an IVUS imaging system, or the like) and by extravascular imaging system 134 (e.g., an angiographic imaging system), wherein the co-regi strati on of the images may be performed using located guide entry positions and catheter tip positions determined by analysis of angiographic images using an artificial intelligence model or another trained model, as further described in this disclosure. The co-regi strati on may be further refined using landmarks identified in each of the images using any of the methods and techniques as described in this disclosure, and any equivalents thereof.

[0090] FIG. 3A illustrates an example cardiac vasculature structure 300 and an IVUS catheter inserted into the cardiac vasculature structure. FIG. 3A illustrates the cardiac vasculature structure 300 having a cardiac artery 302 (e.g., cardiac artery, vessel, or the like) with several side branches 304 (e.g., descending arteries, marginal arteries, vessels, or the like) branching off the cardiac artery 302. It is to be appreciated that various cardiac interventions, which may be referred to as percutaneous coronary intervention (PCI), can be guided by IVUS. For example, prior to performing a PCI procedure, during a PCI procedure, and / or after a PCI procedure, an intravascular imaging catheter (e.g., an IVUS imaging catheter 306) is inserted into the cardiac artery 302 via a guide catheter 308, and images of the cardiac artery 302 are captured. These images may be captured pre-PCI as well as post-PCI. During imaging, the IVUS imaging catheter 306 includes an ultrasound transducer 310 disposed on a distal end of the IVUS imaging catheter 306. The IVUS imaging catheter 306 can be pulled back from a distal point 312 in the cardiac artery 302 to a proximal point 314 in the cardiac artery 302, andDocket No. 3001.1378111BSC File No. 25-1102W001 intravascular image frames are captured while the IVUS imaging catheter 306 is pulled back. In various embodiments of the imaging procedure performed using the imaging catheter 306, the pullback of the imaging catheter 306 between distal point 312 and proximal point 314 may be performed automatically at a predetermined pullback speed through the cardiac artery 302 while the ultrasound transducer 310 captures images at a predetermined sampling rate.

[0091] In addition to the IVUS imaging process, angiography images may be captured in conjunction with the IVUS imaging process to allow for IVUS image and angiography image co-registration of the same vessels, for example co-regi strati on of the IVUS images with an angiographic image of artery 302. Angiographic imaging involves injecting a material comprising a contrast die into the vessel of interest, illustratively represented in FIG. 3A as wavy lines 315, and then capturing an image of the vessel of interest using an imaging modality such as X-ray imaging, for example using X-ray machine 317. The contrast die provides a level of reflection back of the X-ray radiation transmitted toward the vessel of interest, which can be captured as an angiographic image or images, and thereby providing a clear image of the structure of the vessel of interest. The captured angiographic image(s) may be transmitted from the X-ray machine 317 to imaging system acquisition circuitry, such as imaging system acquisition circuitry 114 of system 100 (FIG. 1).

[0092] In various embodiments, an angiographic image may be captured at some point in time when the guide catheter 308 is extended to the final position as shown in FIG. 3A, having the distal end of the guide catheter 308 located at the position within vessel 302 indicated by arrow 301, and having the imaging catheter 306 extending and distally out from the distal end of the guide catheter 308 so that the ultrasound transducer 310 is positioned at the distal point 312, as illustrated shown by ultrasound transducer 310A. By capturing an angiographic image at a time when the distal end 308A of the guide catheter 308 and when the ultrasound transducer 310 is positioned at the distal point 312 as illustratively shown as ultrasound transducer 310A, these two locational points can be analyzed in the captured images, as further described below, in order to accurately determine the centerline extending through the imaged artery 302. Based on this more accurately determined centerline for artery 302, co-registration between the angiographic image and the captured IVUS images of artery 302 may be performed, and the output images and other information available based on the co-registration can provide the clinician more accurate and more detailed information about the status and condition of the imaged vessel.Docket No. 3001.1378111BSC File No. 25-1102W001

[0093] As indicated in this figure, the cardiac artery 302 may include an obstruction 316 disposed in the vessel, which may comprise calcium or other types of obstructions such as plaque. In a PCI procedure, a physician can choose to break up calcium in the cardiac artery 302 (e.g., the obstruction 316, or the like). It is to be appreciated that by breaking up the calcium or causing a calcium “fracture” the efficacy or outcome of the PCI procedure is improved. For example, fractured calcium allows for better expansion (e.g., stent expansion, or the like). Further, the thickness of the calcium affects calcium fracture treatment selection as well as overall treatment outcomes. As described above, the ability to have co-registration images and information related to angiographic and IVUS images is important and helpful when determining treatment options and / or the efficacy of various treatment procedures that have been performed on a patient related to coronary conditions. Also, as further described below, a better understanding of this relationship between calcium and other issues related to the structure and condition of the vessel that may be determined and / or revealed based on the angiographic images, the IVUS images, and the ability to accurately co-register the angiographic and IVUS images can be helpful to determine the best treatment option(s) and / or treatment choice(s) for a given patient. Further, the co-registered angiographic and IVUS images allow for mapping of the positioning of treatment devices, such as angiopathy balloons and / or stents to fluoroscopy images being provided in real time while the treatment device or tool is positioned within and / or is being advanced within the vessel imaged in the fluoroscopy images, and made available as a visible display to the medical professional performing the procedure utilizing the treatment device on a real time basis. The capability to provide realtime imagery of the treatment tool using fluoroscopy images being mapped to co-registered angiographic and IVUS images provides additional real-time visual information regarding the relative positioning of the treatment device within the vessel while the treatment procedure is being performed, thus providing a level of enhanced procedural planning, including procedural safety, and as such provides improvement to the field of medical vascular imaging and cardiovascular treatment.

[0094] FIG. 3B illustrates an IVUS imaging system 320 including the IVUS catheter 306 as shown in FIG. 3A, in accordance with various embodiments. In various embodiments, IVUS imaging system 320 may comprise, in full or in part, IVUS imaging system 101 (FIG. 1) The IVUS imaging system 320 in FIG. 3B is depicted as including IVUS imaging catheter 306, which is couplable to a control system 322. The control system 322 may include, for example, aDocket No. 3001.1378111BSC File No. 25-1102W001 processor 324, a pulse generator 326, and a drive unit 328. In at least some embodiments, the pulse generator 326 forms electric pulses that may be input to one or more transducers (e.g., ultrasound transducer 310, or the like) disposed on IVUS imaging catheter 306. With some embodiments, ultrasound transducer 310 can be part of an imaging core and coupled to the proximal end of IVUS imaging catheter 306 via a drive cable or drive shaft. Mechanical energy from the drive unit 328 can be used to rotate the imaging core (and thus the ultrasound transducer 310).

[0095] The electrical pulses generated by pulse generator 326 can be delivered to the ultrasound transducer 310 while the ultrasound transducer 310 is rotated by the drive unit 328. The electrical signals are transformed by the ultrasound transducer 310 into acoustic pulses that are transmitted through the tissue of cardiac vasculature structure 300 (FIG. 3A). The tissue and / or structure (e.g., calcium, etc.) of the cardiac vasculature structure 300 reflects the acoustic pulses, which are absorbed by the ultrasound transducer 310 and transformed into electric pulses. The transformed electric pulses are delivered to the processor 324 and converted into IVUS images displayable on a monitor. As further described below, these IVUS images may be analyzed to determine guide entry point and catheter tip detection in an angiography image.

[0096] In general, IVUS imaging system 320 can detect obstruction 316 from images of the cardiac artery 302 captured by the IVUS imaging catheter 306. The calcium, and particularly the locations and calcium reverb, can be identified. For example, processor 324 can implement a calcium detection algorithm. Further, processor 324 can generate a GUI comprising indications of the detected calcium and locations and calcium reverb along the cardiac artery 302. As further described below, the IVUS images captured using IVUS imaging system 320 may be co-registered with images captured using a different imaging modality, including angiographic images captured using an angiographic imaging system.

[0097] Information related to guide entry point and catheter tip detection in an angiography image may be generated and displayed on a graphical user interface (GUI). A GUI display of the co-regi strati on between IVUS images and for example images captured using angiographic imaging techniques will provide additional information for a clinician regarding the status and condition of the imaged portion of a vessel, such as a cardiac artery. As such, embodiments of the GUIs as described and illustrated herein will facilitate quick identification of the guideDocket No. 3001.1378111BSC File No. 25-1102W001 entry point and catheter tip detection in an angiography image thereby allowing for more information to be available as part of the treatment decision making process regarding treatment options, and potentially providing improved patient outcomes and an increased level of patient safety.

[0098] FIG. 3C and FIG. 3D illustrate a series of IVUS image frames 330 and a longitudinal view 340 of the series of IVUS image frames 330. FIG. 3C illustrates an example series of IVUS image frames. FIG. 3D illustrates a longitudinal representation of the example series of IVUS image frames of FIG. 3C. As described above, several IVUS image frames can be captured while IVUS imaging catheter 306 is moved through cardiac artery 302 (FIG. 3A). For example, FIG. 3C depicts the series of IVUS image frames 330 including IVUS image frames 332-1, 332-2, 332-3 to 332-N, where N is a positive integer, in this case, greater than ten (10). In practice, N can be any positive integer greater than or equal to two (2) but may be greater than one hundred.

[0099] The series of IVUS image frames 330 can be stacked or grouped and depicted longitudinally to represent a longitudinal slice of the cardiac artery 302 from distal point 312 to proximal point 314. For example, FIG. 3D illustrates longitudinal view 340 that shows a longitudinal slice of cardiac artery 302 from distal point 312 to proximal point 314 formed from the series of IVUS image frames 330.

[0100] As noted above, calcium and / or calcium reverb may be represented in the series of IVUS image frames 330. The disclosure describes systems, methods, techniques to determine guide entry point and catheter tip detection in an angiography image based on an analysis of the series of IVUS image frames, such as IVUS image frames 330. Embodiments as further disclosed herein include to generate various graphical user interfaces (GUIs) comprising graphical depictions and displays of other information related to guide entry point and catheter tip detection in an angiography image determined through the analysis of the IVUS image frame. As further described in this disclosure, the series of IVUS image frames 330 may be coregistered with one or more angiographic images that have been captured for a corresponding portion of a vessel of a patient based on guide entry locations and catheter tip locations determined using a trained artificial intelligence model or another trained model performing analysis of the angiographic image(s). Fluoroscopy images captured in real time during a procedure being performed on the vessel imaged in the co-registered IVUS and angiographicDocket No. 3001.1378111BSC File No. 25-I102W001 images may be mapped onto the co-registered IVUS and angiographic images, which may be displayed graphically as GUIs available for viewing during the procedure being performed and imaged by the fluoroscopy imaging.

[0101] FIG. 3E illustrates an angiographic image 350 of an artery corresponding to artery 302 as depicted in FIG. 3 A. As shown in angiographic image 350, guide catheter 308 is inserted into artery 302 so that the distal end 308A of the guide catheter 308 is positioned within artery 302 at a location indicated by arrow 301. An imaging catheter 306 extends out of the distal end 308A of the guide catheter 308 so that the ultrasound transducer 310 of the imaging catheter 306 is positioned within artery 302 and extends distally away from the distal end 308 A so that the ultrasound transducer 310 is located within vessel 302 at a position indicated by arrow 303. As further described below, the angiographic image 350, may be analyzed, for example using a trained Artificial Intelligence tool, to generate an output providing an identification of the location of the guide entry and the location of the catheter tip within the image. Based on the analysis, a centerline for the vessel extending between the catheter tip and the guide entry point in the image may be calculated or determined, as further described below with respect to FIG.3F.

[0102] FIG. 3F illustrates an annotated angiographic image 360, according to various embodiments as described in this disclosure. As illustratively represented in FIG. 3F, a trained model using artificial intelligence has been utilized to analyze angiographic image 350 (FIG.3E) to detect the location of the guide entry, i.e., the distal end 308A of the guide catheter through which the imaging catheter 306 is extended, and to detect the location of the catheter tip within vessel 302. As shown in FIG. 3F, the trained model may include generating as an output the angiographic image 360 that has been annotated with a graphical indication of the detected location of the guide entry and a graphical indication of the detected location of the catheter tip. As shown in angiographic image 360, the detected location of the guide entry is indicated by the upper arrow in the figure, which is labeled “Guide entry,” and the detected location of the catheter tip is indicated by the lower arrow in the figure, which is labeled “Catheter tip.” It would be understood that the arrows, as depicted in FIG. 3F, are a nonlimiting example of the graphical depiction that may be used to annotate angiographic image 360, and that other formats for these graphical indications may be generated and displayed as a result of the analysis performed on the image by the trained model.Docket No. 3001.1378111BSC File No. 25-1102W001

[0103] Based on these determined locations provided using the trained model, an accurate calculation of the centerline 311 of the vessel 302 can be determined and displayed, as shown in angiographic image 360 as shown in FIG. 3F. Further, based on the centerline calculation, the co-regi strati on of the angiographic image to a series IVUS images may be performed, such as with the series of IVUS images 332-1 through 332-N, as illustrated and described with respect to FIGS. 3C and 3D. In addition, the co-registered IVUS and angiographic images with detected vessel centerlines may be utilized to allow mapping of real-time fluoroscopy images onto the IVUS and angiographic images during various treatment procedures that may be performed on the imaged vessel, as further described below with respect to FIGS. 4A-4F.

[0104] Some example IVUS imaging systems that may be used, for example with the methods disclosed herein, include, but are not limited to, those disclosed in, for example, U.S. Pat. Nos.7,246,959; 7,306,561; and 6,945,938; as well as U.S. Patent Application Publication Nos. US 2006 / 0100522; US 2006 / 0106320; US 2006 / 0173350; US 2006 / 0253028; US 2007 / 0016054; and US 2007 / 0038111; all of which are incorporated herein by reference. U.S. Patent Application Publication No. US 2025 / 0117962 and US 2025 / 0117931 are herein incorporated by reference as some examples of centerline detection and co-regi strati on, respectively, with regards to image processing and IVUS images.

[0105] FIG. 4A illustrates an example of a treatment procedure being performed on the cardiac vasculature structure 400, in accordance with various embodiments. In various embodiments, cardiac vasculature structure 400 corresponds to cardiac vasculature structure 300 of FIG. 3 A. As such, corresponding reference numbering is utilized in FIG 4A to represent corresponding elements from FIG. 3A. However, embodiments of treatment procedure being performed on cardiac vasculature structure 400 are not limited to corresponding to the same vasculature structure illustrated and described with respect to FIG. 3A, and may be representative of a treatment procedure being performed on any vasculature structure(s) that have been previously imaged to capture IVUS and angiographic images of the vessels depicted in cardiac vasculature structure 400.

[0106] As shown in FIG. 4A, cardiac vasculature structure 400 includes cardiac artery 302 (e.g., cardiac artery, vessel, or the like) with several side branches 304 (e.g., descending arteries, marginal arteries, vessels, or the like) branching off the cardiac artery 302. It is to be appreciated that various cardiac interventions, which may be referred to as percutaneousDocket No. 3001.1378111BSC File No. 25-1102W001 coronary intervention (PCI), can be guided by IVUS. For example, prior to performing the treatment procedure depicted in FIG. 4A, during a PCI procedure an intravascular imaging catheter (e.g., an IVUS imaging catheter 306, FIG. 3A) was inserted into the cardiac artery 302 via a guide catheter 308, and images of the cardiac artery 302 were captured. As part of the IVUS imaging procedure, the IVUS imaging catheter 306 was be pulled back from a distal point 312 in the cardiac artery 302 to a proximal point 314 in the cardiac artery 302, and intravascular image frames were captured while the IVUS imaging catheter 306 was pulled back.

[0107] In addition to the IVUS imaging process, angiography images were also captured in conjunction with the IVUS imaging process to allow for IVUS image and angiography image co-registration of the same vessels, for example co-regi strati on of the IVUS images with an angiographic image of artery 302. As described above, by capturing an angiographic image at a time when the distal end 308A of the guide catheter 308 and when the ultrasound transducer 310 is positioned at the distal point 312 as illustratively shown as ultrasound transducer 310A, these two locational points can be analyzed in the captured images, as further described above, in order to accurately determine the centerline extending through the imaged artery 302. Based on this more accurately determined centerline for artery 302, co-registration between the angiographic image and the captured IVUS images of artery 302 may be performed, and the output images and other information available based on the co-registration can provide the clinician more accurate and more detailed information about the status and condition on the imaged vessel.

[0108] As shown in FIG. 4A, the cardiac artery 302 includes obstruction 316 disposed in the vessel, which may comprise calcium or other types of obstructions such as plaque. As further shown in FIG. 4A, a treatment device 404 is positioned with vessel 302, and as part of a treatment procedure is being advanced over a guidewire 402 in the distal direction within vessel 302 toward obstruction 316. In various embodiments, treatment device 404 may be a tool configured to perform modifications of calcium, for example modifications on calcium forming obstruction 316, such as through use of an angioplasty balloon catheter. In various embodiments, treatment device 404 may be configured as a stent placement tool configured to advance and place a stent at the site of obstruction 316. In various embodiments, treatment device 404 may comprise an element or elements, such as a drug-eluting balloon, which incorporate(s) a drug intended to be applied within the vessel 302, for example at the site ofDocket No. 3001.1378111BSC File No. 25-1102W001 obstruction 316, for the purpose of providing some form of medical efficacy to the vessel 302 and / or at the site of the obstruction 316.

[0109] As part of the treatment procedure depicted in FIG. 4A, fluoroscopy images may be performed in real time, to capture images of the treatment device 404 positioned, being advanced, and / or being retracted within vessel 302. In various embodiments, fluoroscopy imaging device 410 may be operated to capture, in real time, a sequence of fluoroscopy images as the treatment procedure is being performed on cardiac vasculature structure 400. In some embodiments, fluoroscopy imaging device 410 is included as part of or comprises the entirety of fluoroscopy imaging system 105 of system 100 (FIG. 1). In various embodiments, fluoroscopy imaging device 410 as shown in FIG. 4A is configured as a fluoroscopic imager, which is configured to capture images of the vasculature structure 400 in real time without the need for a contrast agent being present with vessel 302 being imaged. In various embodiments, the images captured by fluoroscopy imaging device 410 are transmitted, in real time, to image system acquisition circuitry, such as imaging system acquisition circuitry 114 of system 100 (FIG. 1). Upon receiving the images provided by fluoroscopy imaging device 410, an imaging system, such as system 100 (FIG. 1) is configured to map the fluoroscopy images, again in some embodiments in real time, to the co-registered IVUS and angiographic images previously captured for the cardiac vasculature structure 400.

[0110] After mapping the fluoroscopy images onto the co-registered IVUS and angiographic images previously captured for the cardiac vasculature structure 400, a imaging system such as system 100 (FIG. 1) is further configured to render images of the fluoroscopy images mapped onto the IVUS and angiographic images, which are then displayed on a display device, such as display 129 of system 100, and which are available for viewing by one or more medical professionals preforming the procedure being performed on the cardiac vasculature structure 400. The viewable images of the fluoroscopy images mapped onto the IVUS and angiographic images, provided in real time, allow a practitioner performing the treatment procedure to view, in real time the locational positioning and orientation of the treatment device, such as treatment device 404, as the treatment procedure is being performed, but with the added benefit of mapping the location, orientation, and position of the treatment device to various features, such as side branch landmarks, that are not visible in the fluoroscopy images alone. Additional examples of the mapping of the fluoroscopy images to the co-registered IVUS and angiographic images is further illustrated and described below with respect to FIGS. 4B-4F.Docket No. 3001.1378111BSC File No. 25-1102W001

[0111] The combination of the fluoroscopy images mapped onto and displayed in real time as an overlay on to the IVUS and angiographic images, provides more information and a clearer picture of the structure of the vessel of interest, and the progress made during the treatment procedure itself, all on a real time basis during the procedure. The features of these combined images allow for a better patient outcome and provides additional patient safety to the procedure, and thus provide an improvement to the field of medical vessel imaging and cardiovascular treatment procedures. The capability to provide real-time imagery of the treatment tool using fluoroscopy images mapped to co-registered angiographic and IVUS images provides additional real-time visual information regarding the relative positioning of the treatment device within the vessel where the treatment procedure is being performed, thus providing a level of enhanced capability and procedural safety, and as such provides improvement to the field of medical vascular imaging and cardiovascular treatment procedures.

[0112] FIG. 4B illustrates a fluoroscopy image 420, in accordance with various embodiments of the present disclosure. Fluoroscopy image 420 may be one instance of a series of fluoroscopy images captured during a treatment procedure being performed on a cardiac vasculature structure, such as cardiac vasculature structure 400 as illustrated ad described with respect to FIG. 4A. As shown in FIG. 4B, fluoroscopy image 420 includes a vessel 423 having a treatment tool 426 positioned within the vessel and extending distally within vessel 423 beyond a distal end 424 of a guide catheter 422.

[0113] FIG. 4C illustrates image 430, which in various embodiments is an annotated variation of fluoroscopy image 420 of FIG. 4B. As shown in FIG. 4C, fluoroscopy image 430 has been annotated to include a graphical depiction of a centerline 425, which extends from the distal end 424 of guide catheter 422 to the treatment device 426 along vessel 423. The annotation of image 430 may be performed by mapping the fluoroscopy image 420 of FIG. 4B onto the coregistered IVUS and angiographic images previously captured for vessel 423.

[0114] FIG. 4D illustrates a fluoroscopy image 440, which in various embodiments is an illustration of a further annotated version of the fluoroscopy image 430 of FIG. 4C. As shown in FIG. 4D, a graphical indication 442 is rendered onto image 440, the graphical indication 442 is indicative of the position of the treatment device 426 within vessel 423. In various embodiments, image 440 and the graphical indication 442 indicating the positioning of the treatment device 426 within image 440 is rendered and displayed in real time, for example on aDocket No. 3001.1378111BSC File No. 25-1102W001 display device such as display 129 of system 100 (FIG. 1). In various embodiments, the rendering of the graphical indication 442 is updated in the fluoroscopy image 440 in real time, and therefore represents the actual positioning, orientation, and movements of the treatment device 426 within the vessel 423 real time.

[0115] In some embodiments, the positioning of the treatment device, and therefore the positioning of the graphical indication 442 is based on a length based correlation using the centerline determined for the angiographic image. Regarding the length based method, guide entry points from angiography and fluoroscopy will be used as endpoints. Since the total length of the catheter is known from angiography, an estimate the location of the treatment device can be made once the path is identified in fluoroscopy. In some embodiments, the positioning of the graphical indication 442 is determined using a path based correlation, or in the alternative using artificial intelligence. Regarding the path correlation method, given a series of paths from fluoroscopy and a reference path from angiography, a path correlation between the fluoroscopic paths and the angiographic path can be performed. Once a sufficient correlation is achieved, a map of the starting point — corresponding to the treatment device — onto the centerline in the X-ray angiography can be generated.

[0116] The overlaying of the graphical indication of the positioning and orientation of the treatment tool 426 in real time, and in perspective to various features such as landmark side branches and / or in relationship to obstructions such as the location and distribution of calcium within the vessel, which may not be visible in the fluoroscopy image itself, provides valuable information that the practitioner may view and utilize as the treatment device is being manipulated as part of the treatment being performed on the vessel 423.

[0117] In addition to providing real-time mapping of the fluoroscopy images to the angiographic image as shown in FIG. 4D, the graphical indication 442 of the positioning of the treatment device 426 may also be overlaid onto a longview image 452 (FIG. 4E) of the vessel 423.

[0118] FIG. 4E illustrates a longview image 450, in accordance with various embodiments of the present disclosure. Again, by providing the graphical indication 442 of the treatment device in a longview image 452 as shown in image 450 in real time, a practitioner performing the treatment procedure on vessel 423 may be provided with particular information related to the positioning and orientation of the treatment tool within the vessel 423 relative to variousDocket No. 3001.1378111BSC File No. 25-1102W001 landmark features, such as side branch and obstruction features, which would not necessarily be visible in the fluoroscopy images alone.

[0119] FIG. 5A illustrates a first example graphical user interface (GUI) 500, which can be generated according to various embodiments of the present disclosure. In some embodiments, GUI 500 can be generated by imaging system 100 as GUI 128 and displayed on display 129. As depicted, GUI 500 includes several graphical components or visual indications, such as, menus 502A and 502B, cross-sectional view 504, and vessel longview 506, vessel profile view 508, and assessments 510 and 512.

[0120] With some embodiments, a processor (e.g., processor 106 of system 100, or the like) can be configured to execute instructions (e.g., instructions 116, or the like) to generate GUI 500. In some examples, GUI 500 can be generated responsive to mapping, in real time, fluoroscopy images onto co-registered IVUS and angiographic images. As part of this mapping, one or more graphical indications, such as graphical indication 442 (FIGS. 4D and 4E) depicting the position of a treatment device within the vessel being imaged, may be incorporated into the fluoroscopy images and for example into a longview of the vessel, both of which may be updated in real time to correspond to the real-time fluoroscopy image(s) being displayed a part of GUI 500.

[0121] Menu 502A can comprise GUI inputs such as buttons, drop down menus, selection icons, or the like. Menu 502A can include GUI input options to select measurement and annotation tools, length tools, modification reset buttons, or the like. Menu 502B can comprise GUI inputs such as buttons, drop down menus, selection icons, or the like. Menu 502B can include GUI input options to select views related to views of the processed angiographic images, IVUS images, fluoroscopy images, layout options, annotations, navigation, dynamic review options, status of the computing device, or the like.

[0122] Cross-sectional view 504 can comprise a cross-sectional view, or on-axis view, of one (e.g., a frame, or the like) of the processed angiographic image(s) and / or a series of IVUS image frames 330. For example, cross-sectional view 504 can include IVUS image frame 332-1 (or IVUS image frame 332-2 to 332-N). Further, cross-sectional view 504 can include assessments 510. With some examples, the assessments can include visual indications of vessel and / or lumen borders, indications of vessel and / or lumen area, indications of a plaque burden, indications of a calcium arc length, and / or a calcium reverb indication, and / or informationDocket No. 3001.1378111BSC File No. 25-I102W001 related to image processing of the IVUS images. In addition, cross-sectional view 504 may include images of unprocessed angiographic images, processed angiographic images, including images annotated to display a graphical indication of the determined centerline for a vessel in an angiographic image, and / or images related to the co-regi strati on of the processed angiographic image(s) with the IVUS images for a same corresponding portion of a vessel. Cross-sectional view 504 may include fluoroscopy images, including fluoroscopy images that are captured in real time, and mapped onto an angiographic image of the vessel, which may include a graphical indication of the position of the treatment device, such as an angioplasty balloon or a stent, within the vessel.

[0123] The vessel longview 506 can include the series of IVUS image frames 330 represented longitudinally, for example, as depicted in longitudinal view of FIG. 3D. The vessel profile view 508 can include a visual depiction of the longitudinal profile of the vessel. For example, vessel profile view 508 can include indications of the lumen border and vessel border, indications of plaque, indications of a stent, indications of key frames, indications, or the like. Further, vessel long view 506 and vessel profile view 508 can include a navigation slider (see e.g., slider 532, FIG. 5B and FIG. 5C), which can be selected and used by a user (e.g., physician, or the like) to navigate through the series of IVUS image frames 300. In some examples, the slider can be linked to the cross-sectional view 504 such that the frame depicted in cross-sectional view 504 corresponds to the location of the slider along long view 506. As the slider is moved, the image displayed in cross-sectional view 504 can change to match the location of the slider.

[0124] Further, the vessel longview 506 and the vessel profile view 508 may include assessments 512. With some examples, the assessments 512 may include visual indications of plaque, plaque burden, calcium, detected calcium reverb, detected vulnerable plaque, stent expansion, and the like.

[0125] FIG. 5B illustrates a second example GUI 530, which may be generated according to various embodiments of the present disclosure. For example, the GUI 530 can be generated by the image system 100 (FIG. 1) as the GUI 128 and displayed on the display 129. In various embodiments, GUI 530 can include the menu 502A and menu 502B, the cross-sectional view 504 including annotated angiographic image 531, and a vessel long view 506. In the vessel longview 506, a slider 532 may be represented. The GUI 530 provides the slider 532 such that aDocket No. 3001.1378111BSC File No. 25-1102W001 user can scrub across the series of IVUS image frames when cross-sectional images are displayed as part of view area 504. Additionally, the GUI 530 may include assessments 512 in vessel long view 506. With some examples, assessments 512 can include visual indications of key frames (e.g., brackets, or the like), a visual indication of the location of the minimum lumen area, etc.

[0126] With some embodiments, responsive to receiving a fluoroscopy image, processor 106 can execute instructions to process the image to generate annotated fluoroscopy image 531. In various embodiments, the annotated fluoroscopy image is mapped onto an angiographic image of the same vessel imaged in the fluoroscopy image 531, but with the added graphical indication 542 representing the positioning of a treatment device positioned within the vessel 535. In various embodiments, a trained Al model that has been trained to detect the guide entry and catheter tip locations within the angiographic image, and to produce an output angiographic image 531 which includes graphical indication 542 indicative of the location of the treatment device located within the vessel 535.

[0127] In addition, responsive to receiving the fluoroscopy images, processor 106 can execute instructions 116 to generate the longview 506 depicting the imaged vessel included in the series of IVUS image frames 330 corresponding to the vessel imaged in angiographic image 531, but in a longitudinal depiction. Further, processor 106 can execute instructions 116 to a generate one or more graphical indications, such as graphical indication 544, representative of the location of the treatment device positioned within vessel 535, the positioning of graphical indication 544 provided in real time and updated in real time to correspond to the graphical depiction 542 of the positioning of the treatment device as provided in the angiographic image 531.

[0128] FIG. 5C illustrates a third example GUI 540, which may be generated according to various embodiments of the present disclosure. For example, the GUI 540 can be generated by the system 100 (FIG. 1) as the GUI 128 and displayed on the display 129. In various embodiments, GUI 540 can include menu 502A and menu 502B, the cross-sectional view 504 including annotated fluoroscopy image 531 and cross-sectional image 551, and a vessel longview 506. In the vessel long view 506, the slider 532 may be represented. The GUI 540 provides the slider 532 such that a user can scrub across the series of IVUS image frames when cross-sectional images are displayed as part of view area 504. Additionally, the GUI 540 mayDocket No. 3001.1378111BSC File No. 25-I102W001 include assessments 512 in vessel long view 506. With some examples, assessments 512 can include visual indications of key frames (e.g., brackets, or the like), a visual indication of the location of the minimum lumen area, etc.

[0129] As shown in FIG. 5C, cross-sectional view 504 includes fluoroscopy image 531, as illustrated and described above with regards to FIG. 5B, alongside a cross-sectional image 551 of a portion of the vessel imaged for example as part of an IVUS drawback imaging procedure. In various embodiments, the portion of the vessel imaged in the cross-sectional image 551 corresponds to the portion of the vessel at the longitudinal position along the vessel indicated by the positioning of slider 532. In various embodiments, GUI 540 is configured so that when slider 532 is actuated to a different position within long view 506, the image of the cross-sectional image 541 will be updated to reflect the image captured for that portion of the vessel indicated by the repositioning of the slider 532.

[0130] As shown in FIG. 5C, assessments 510 includes the X-Y coordinate information and calcium arc information related to a calcium arc 552 annotated onto the cross-sectional image 551. The calcium arc represents the presence of detected calcium in the portion of the vessel imaged in cross-sectional image 541.

[0131] The ability to graphically illustrate the relationship between fluoroscopy image 531 and cross-sectional image 541 in a single GUI, such as GUI 540, can provide information to a user, such as clinicians and physicians, which is useful in making real-time determinations about what treatments, and / or what types of procedures may be best utilized in view of this information provided by the GUI, such as the GUI 540, as well as real-time images of the status of a treatment procedure being performed using a treatment device on the vessel imaged in both fluoroscopy image 531, the cross-sectional view 551, as well as in longview 506.

[0132] FIG. 6 illustrates a logic flow 600, according to various embodiments as described in the present disclosure. The logic flow 600 can be implemented by a multi-modality coregistration and mapping system 100 (system 100), and will be described with reference to system 100 for clarity of presentation. However, it is noted that logic flow 600 could also be implemented by an imaging or guidance system different than the image visualization system 300.

[0133] The logic flow 600 begins at block 602. At block 602 logic flow 600 comprises, “receiving an angiographic image of a vessel having an IVUS catheter positioned within theDocket No. 3001.1378111BSC File No. 25-1102W001 vessel.” System 100 is configured to receive, for example at imaging system acquisition circuitry 114, the angiographic image captured for the vessel. In various embodiments, the angiographic image is generated by angiographic imaging system 103, and transmitted to computing device 104. The angiographic image may be stored in memory 108 as angiographic image data 122. In various embodiments, the angiographic image is captured at a time when an IVUS catheter, such as catheter 306 (FIGS. 3A, 3B), is positioned within the vessel, for example as illustrated and described above with respect to FIG. 3A. In various embodiments, the angiographic image is captured at a time when a contrast agent, such as an iodinated contrast media or barium sulfate, is present in the area of the vessel that is being imaged.

[0134] In various embodiments, the angiographic image includes an image of at least a portion of the vessel, such as a cardiac artery, having an imaging device inserted into the vessel and having a distal portion of the imaging device extending distally outward into the vessel from a distal end (guide entry) of a guide catheter positioned within the vessel. The imaging device includes a catheter tip positioned at the distal end of the imaging device, and having the catheter tip located within the vessel and distally from the guide entry point, and at a location representing a distal starting point for initiating an imaging procedure configured to image the vessel from the distal starting point in a direction toward the guide entry point. In various embodiments, the imaging device is an IVUS imaging catheter, and the imaging procedure comprises an IVUS imaging procedure. In addition, processor 106 of system 100 may be configured to analyze and annotate the angiographic image to indicate the location of the guide entry point within the angiographic image, and to further annotate the angiographic image to indicate the location of the catheter tip within the image. The annotated version of the angiographic image may be stored back in memory 108 as angiographic image data 122.

[0135] Logic flow continues at block 604, “receiving a series of IVUS images captured during a pullback imaging procedure performed on the vessel.” System 100 is configured to receive, for example at imaging system acquisition circuitry 114, the series of IVUS images that is captured during the pullback imaging procedure performed on the vessel. In various embodiments, the series of IVUS images generated by IVUS imaging system 101, and transmitted to computing device 104. The captured IVUS images may be stored in memory 108 as the series of IVUS image frames 120. In various embodiments, the series of IVUS images is captured using the imaging devices that was positioned within the vessel at the time theDocket No. 3001.1378111BSC File No. 25-1102W001 angiographic image as described at block 602 was captured, and for example as illustrated and describe above with respect to FIG. 3A.

[0136] Logic flow continues at block 606, “co-registering the angiographic image with the series of IVUS images.” At block 606, processor 106 can execute the instructions 116 to coregister the angiographic image of the vessel received at block 602 with the series of IVUS images captured for the vessel and received at block 604. In some embodiments, artificial intelligence, machine learning, neural networks and / or some combination thereof, can process one or more angiographic images to determine the location of the guide entry and the catheter tip of the imaging device within the angiographic image and thereby determine a centerline for the vessel extending between the guide entry and the catheter tip. In addition, landmarks, such as side branch landmarks, may be determined for both the angiographic image and for the IVUS images included in the series of IVUS images.

[0137] In various embodiments, utilizing the determined centerline detected based on the locations of the guide entry and the catheter tip of the imaging device in the angiographic image and the corresponding landmarks detected in the angiographic image and in the series of IVUS images, processor 106 executes instructions 116 to co-register the endpoints and the landmarks of the angiographic image with the corresponding IVUS images. The co-regi strati on of the angiographic image with the series of IVUS images may be stored in memory 108 as IVUS / angio co-registered images 125. In various embodiments, the co-regi strati on may be performed using software tool(s), which may utilize artificial intelligence, machine learning, neural networks and / or some combination thereof, including but not limited to convolutional neural networks, transformer-based neural networks, diffusion models, attention-based networks, recurrent neural networks, fully convolutional networks, graph-based neural networks, or combinations thereof.

[0138] Logic flow continues at block 608, “receiving in real time a series of fluoroscopy images captured while a treatment procedure is being performed on the vessel.” System 100 is configured to receive, for example at imaging system acquisition circuitry 114, the series of fluoroscopy images captured while a treatment procedure is being performed on the vessel. In various embodiments, the series of fluoroscopy images is generated by fluoroscopy imaging system 105, and transmitted to computing device 104. The fluoroscopy images may be stored in memory 108 as fluoroscopy image data 124. In various embodiments, the fluoroscopy imagesDocket No. 3001.1378111BSC File No. 25-1102W001 comprise a sequence of images capture while a treatment device, such as treatment device 404 (FIG. 4A) is positioned with the vessel and is being utilized as part of a treatment procedure being performed on the vessel. The treatment device is not limited to a particular type or form of a treatment device, and may be for example a balloon angioplasty catheter, a stent, or any other type of treatment device that may be utilized to perform a medical treatment procedure on the vessel. The fluoroscopy images may be provided to system 100 at the start of and throughout the duration of the medical treatment procedure being performed on the vessel.

[0139] Logic flow 600 continues at block 610, “mapping in real time the series of fluoroscopy images to the co-registered angiographic and IVUS images.” At block 610, processor 106 executes instructions 116 that map, in some embodiments in real time, the received series of fluoroscopy images to the IVUS / angio co-registered images 125. Mapping of the fluoroscopy images includes mapping of the centerline of the vessel as determined for the IVUS / angio coregistered images to the images path of the vessel presented in each of the fluoroscopy images, wherein the imaged pathway of the vessel may be varying from one fluoroscopy image to the next fluoroscopy image, for example in width and / or in position within the patient due to various factors such and blood flow and / or other disturbances, such as the introduction and movements of the treatment device now present within the vessel.

[0140] Mapping of the fluoroscopy images to the co-registered angiographic and IVUS images may be performed by tools such as an artificial intelligence model or another trained model performing analysis of the fluoroscopy images, which may include the use of machine learning, neural networks and / or some combination thereof. Mapping of the fluoroscopy images includes detecting the location, orientation, and positioning of the treatment device, such as treatment device 404 (FIG. 4A) within the vessel, and mapping the treatment device into the corresponding positioned of the vessel as imaged in the co-registered IVUS and angiographic images. The mapping of the treatment device as detected in the fluoroscopy images into the co-registered IVUS and angiographic images includes generating a graphical indication, such as graphical indication 442 (FIG. 4D), which is rendered onto the fluoroscopy images in real time. In various embodiment, the mapping of the treatment device as detected in the fluoroscopy images into the co-registered IVUS and angiographic images includes generating a graphical indication, such as graphical indication 442 (FIG. 4E), which is rendered in real time onto a long view of the vessel generated based on the co-registered IVUS images and the fluoroscopy images. As described above, the placement of the graphical indicatorDocket No. 3001.1378111BSC File No. 25-I102W001 within the mapped fluoroscopy image may be based on length-based correlation, path-based correlation, or by utilization of an artificial intelligence model or tool.

[0141] Logic flow 600 continues at block 612, “displaying in real-time the fluoroscopy images including graphical indications of the positioning of the treatment device within the vessel and mapped to the co-registered IVUS and angiographic images.” At block 612, processor 106 executes instructions to display, in some embodiments in real time, the mapped fluoroscopy images generated at block 610 and provided as mapped fluoroscopy images 126 from memory 108. The mapped fluoroscopy images may be generated as GUI 128, and displayed on a display device, such as display 129 of system 100. FIGS. 5A-5C as described above illustrate non-limiting example of the graphical user interfaces that may be generated and displayed, for example in real time, and that include fluoroscopy images that have be coregistered with IVUS and angiographic images of the vessel and including one or more graphical inhiations of the current location, orientation, and positioning of a treatment device positioned within the vessel being imaged in the fluoroscopy images.

[0142] FIG. 7 illustrates computer-readable storage medium 700, according to various embodiments as described in the present disclosure. Computer-readable storage medium 700 may comprise any non-transitory computer-readable storage medium or machine-readable storage medium, such as an optical, magnetic or semiconductor storage medium. In various embodiments, computer-readable storage medium 700 may comprise an article of manufacture. In some embodiments, computer-readable storage medium 700 may store computer executable instructions 702 with which circuitry (e.g., processor 106, and the like) can execute. For example, computer executable instructions 702 can include instructions to implement operations described with respect to instructions 116, GUI 128, GUI 500, GUI 530, GUI 540 and logic flow 600. Computer executable instructions 702 can include image processing instructions 704, IVUS / angio co-regi strati on instructions 706, fluoroscopy image coregistration instructions 708, which are configured to perform, when executed by processor 106, the function ascribed to various embodiments of system 100 as described herein, including the logic flow 600, and any equivalents thereof.

[0143] Examples of computer-readable storage medium 700 or machine-readable storage medium may include any tangible media capable of storing electronic data, including volatile memory or non-volatile memory, removable or non-removable memory, erasable or nonDocket No. 3001.1378111BSC File No. 25-1102W001 erasable memory, writeable or re-writeable memory, and so forth. Examples of computer executable instructions 702 may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, object-oriented code, visual code, and the like.

[0144] FIG. 8 illustrates a diagrammatic representation of a machine 800 in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein. More specifically, FIG.8 shows a diagrammatic representation of the machine 800 in the example form of a computer system, within which instructions 808 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 800 to perform any one or more of the methodologies discussed herein may be executed. For example, the instructions 808 may cause the machine 800 to execute instructions 116, or routine of logic flow 600 of FIG. 6, or the like. More generally, the instructions 808 may cause the machine 800 to co-register IVUS and angiography images as outlined herein, and to map, in some embodiments in real time, fluoroscopy images to the co-registered IVUS and angiographic images for display on a display device, such as display 129.

[0145] The instructions 808 transform the general, non-programmed machine 800 into a particular machine 800 programmed to carry out the described and illustrated functions in a specific manner. In alternative embodiments, the machine 800 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 800 may operate in the capacity of a server machine or a client machine in a serverclient network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 800 may comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 808, sequentially or otherwise, that specify actions to be taken by the machine 800. Further, while only a single machine 800 is illustrated, the term “machine” shall also be taken to include a collection of machines 800 that individually or jointly execute the instructions 808 to perform any one or more of the methodologies discussed herein.Docket No. 3001.1378111BSC File No. 25-1102W001

[0146] The machine 800 may include processors 802, memory 804, and I / O components 842, which may be configured to communicate with each other such as via a bus 844. In an example embodiment, the processors 802 (e.g., a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an ASIC, a Radio-Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 806 and a processor 810 that may execute the instructions 808. The term “processor” is intended to include multi-core processors that may comprise two or more independent processors (sometimes referred to as “cores”) that may execute instructions contemporaneously. Although FIG. 8 shows multiple processors 802, the machine 800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiples cores, or any combination thereof.

[0147] The memory 804 may include a main memory 812, a static memory 814, and a storage unit 816, both accessible to the processors 802 such as via the bus 844. The main memory 804, the static memory 814, and storage unit 816 store the instructions 808 embodying any one or more of the methodologies or functions described herein. The instructions 808 may also reside, completely or partially, within the main memory 812, within the static memory 814, within machine-readable medium 818 within the storage unit 816, within at least one of the processors 802 (e.g., within the processor’s cache memory), or any suitable combination thereof, during execution thereof by the machine 800.

[0148] The I / O components 842 may include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I / O components 842 that are included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I / O components 842 may include many other components that are not shown in FIG. 8. The I / O components 842 are grouped according to functionality merely for simplifying the following discussion and the grouping is in no way limiting. In various example embodiments, the I / O components 842 may include output components 828 and input components 830. The output components 828 may include visual components (e.g., a display such as a plasmaDocket No. 3001.1378111BSC File No. 25-1102W001 display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The input components 830 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or another pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0149] In further example embodiments, the I / O components 842 may includebiometric components 832, motion components 834, environmental components 836, or position components 838, among a wide array of other components. For example, the biometric components 832 may include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like. The motion components 834 may include acceleration sensor components (e.g., accelerometer), gravitation sensor components, rotation sensor components (e.g., gyroscope), and so forth. The environmental components 836 may include, for example, illumination sensor components (e.g., photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detection concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to a surrounding physical environment. The position components 838 may include location sensor components (e.g., a GPS receiver component), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude may be derived), orientation sensor components (e.g., magnetometers), and the like.Docket No. 3001.1378111BSC File No. 25-1102W001

[0150] Communication may be implemented using a wide variety of technologies. The I / O components 842 may include communication components 840 operable to couple the machine 800 to a network 820 or devices 822 via a coupling 824 and a coupling 826, respectively. For example, the communication components 840 may include a network interface component or another suitable device to interface with the network 820. In further examples, the communication components 840 may include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components to provide communication via other modalities. The devices 822 may be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a USB).

[0151] Moreover, the communication components 840 may detect identifiers or include components operable to detect identifiers. For example, the communication components 840 may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect onedimensional bar codes such as Universal Product Code (UPC) bar code, multi-dimensional bar codes such as Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar code, and other optical codes), or acoustic detection components (e.g., microphones to identify tagged audio signals). In addition, a variety of information may be derived via the communication components 840, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting an NFC beacon signal that may indicate a particular location, and so forth.

[0152] The various memories (i.e., memory 804, main memory 812, static memory 814, and / or memory of the processors 802) and / or storage unit 816 may store one or more sets of instructions and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. These instructions (e.g., the instructions 808), when executed by processors 802, cause various operations to implement the disclosed embodiments as described herein, and any equivalents thereof.

[0153] As used herein, the terms “machine-storage medium,” “device-storage medium,” “computer-storage medium” mean the same thing and may be used interchangeably in this disclosure. The terms refer to a single or multiple storage devices and / or media (e.g., aDocket No. 3001.1378111BSC File No. 25-1102W001 centralized or distributed database, and / or associated caches and servers) that store executable instructions and / or data. The terms shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media, including memory internal or external to processors. Specific examples of machine-storage media, computer-storage media and / or device-storage media include non-volatile memory, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine-storage media,” “computer-storage media,” and “device-storage media” specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term “signal medium” discussed below.

[0154] In various example embodiments, one or more portions of the network 820 may be an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, the Internet, a portion of the Internet, a portion of the PSTN, a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, the network 820 or a portion of the network 820 may include a wireless or cellular network, and the coupling 824 may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling 824 may implement any of a variety of types of data transfer technology, such as Single Carrier Radio Transmission Technology (IxRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3 GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standard, others defined by various standardsetting organizations, other long range protocols, or other data transfer technology.

[0155] The instructions 808 may be transmitted or received over the network 820 using a transmission medium via a network interface device (e.g., a network interface component included in the communication components 840) and utilizing any one of several well-known transfer protocols (e.g., hypertext transfer protocol (HTTP)). Similarly, the instructions 808Docket No. 3001.1378111BSC File No. 25-1102W001 may be transmitted or received using a transmission medium via the coupling 826 (e.g., a peer-to-peer coupling) to the devices 822. The terms “transmission medium” and “signal medium” mean the same thing and may be used interchangeably in this disclosure. The terms “transmission medium” and “signal medium” shall be taken to include any intangible medium that can store, encoding, or carrying the instructions 808 for execution by the machine 800, and includes digital or analog communications signals or other intangible media to facilitate communication of such software. Hence, the terms “transmission medium” and “signal medium” shall be taken to include any form of modulated data signal, carrier wave, and so forth. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a matter as to encode information in the signal.

[0156] Terms used herein should be accorded their ordinary meaning in the relevant arts, or the meaning indicated by their use in context, but if an express definition is provided, that meaning controls.

[0157] Herein, references to "one embodiment" or "an embodiment" do not necessarily refer to the same embodiment, although they may. Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise," "comprising," and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to." Words using the singular or plural number also include the plural or singular number respectively, unless expressly limited to one or multiple ones. Additionally, the words "herein," "above," "below" and words of similar import, when used in this application, refer to this application as a whole and not to any portions of this application. When the claims use the word "or" in reference to a list of two or more items, that word covers all the following interpretations of the word: any of the items in the list, all the items in the list and any combination of the items in the list, unless expressly limited to one or the other. Any terms not expressly defined herein have their conventional meaning as commonly understood by those having skill in the relevant art(s). The terms “real time” and “real-time” refer to a period of time for a processor, such as a computer processor or microprocessor, to receive data related to an image, to analyze the image, and to output a required result, such as an annotation or other information related to the analyzed image. In general a non-limiting example of the time frame is fractions of a second, for example a timeframe of less than 100 milliseconds, in some examples less than 10 milliseconds.

Claims

Docket No. 3001.1378111BSC File No. 25-1102W001 CLAIMSWhat is claimed is:

1. A method, comprising:receiving, at a computing device, a sequence of fluoroscopy images provided by a fluoroscopy imaging device, wherein the sequence of fluoroscopy images includes images of a vessel of a patient;detecting in the fluoroscopy images the presence of a treatment device positioned within the vessel; andmapping a location of the treatment device detected in the fluoroscopy images onto at least one image of the vessel captured using at least one imaging modality that is different from the imaging modality provided by the fluoroscopy imaging device.

2. The method of claim 1, further comprising:displaying, on a display device and in real time, a graphical indication of the location of the treatment device in each one of the sequence of fluoroscopy images as each of the sequence of fluoroscopy images is received.

3. The method of claims 1 or 2, further comprising:displaying, on a display device and in real time, a longview of the vessel including a graphical indication of the location of the treatment device as detected in the fluoroscopy images relative to a longitudinal position of the treatment device within the vessel.

4. The method of claim 3, wherein the at least one different imaging modality includes intravascular ultrasound (IVUS) imaging, and wherein the longview of the vessel is generated using a series of intravascular ultrasound image frames captured using IVUS imaging of the vessel.

5. The method of claim 4, wherein the mapping of the location of the treatment device detected in the fluoroscopy images is based on a pullback path utilized by the IVUS imaging to generate the series of intravascular ultrasound image frames.Docket No. 3001.1378111BSC File No. 25-1102W0016. The method of any one of claims 1 to 5, wherein the at least one different imaging modality includes an angiographic image captured using an X-ray imaging device and captured while a contrast agent is present within the vessel.

7. The method of claim 6, wherein the angiographic image is captured while an intravascular ultrasound imaging catheter is extended distally from a guide catheter positioned within the vessel, the intravascular ultrasound imaging catheter including a catheter tip positioned within the vessel and distally of a distal end of the guide catheter.

8. The method of claim 7, further comprising:receiving, at the computing device, the angiographic image;determining within the angiographic image a location of a guide entry point of the guide catheter;determining within the angiographic image a location of the catheter tip; determining a centerline extending through the vessel as imaged in the angiographic image, the centerline extending from the guide entry point of the guide catheter to the catheter tip; andmapping the location of the treatment device detected in the fluoroscopy images based on a path length determined from the centerline.

9. The method of any one of claim 1 to 8,wherein the at least one at least one imaging modality that is different from the imaging modality provided by the fluoroscopy imaging device comprises intravascular ultrasound imaging and X-ray angiography imaging, andwherein the mapping of the location of the treatment device detected in the fluoroscopy images onto at least one image of the vessel comprises mapping the location the treatment device onto an X-ray angiographic image co-registered with a series of intravascular ultrasound images.

10. The method of any of claims 1 to 9, wherein the treatment device comprises an angioplasty balloon.Docket No. 3001.1378111BSC File No. 25-1102W00111. The method of any one of claims 1 to 9, wherein the treatment device comprises a stent.

12. The method of any one of claims 1 to 9, wherein the treatment device comprises a drug¬ eluting balloon.

13. The method of any one of claims 1 to 12, wherein the graphical indication of the location of the treatment device in the fluoroscopy images is determined using artificial intelligence.

14. A computer-readable storage device, comprising instructions executable by a processor of a computing device coupled to a fluoroscopy imaging device, wherein when executed the instructions cause the computing device to implement the method of any one of claims 1 to 13.

15. An apparatus comprising a processor arranged to be coupled to a fluoroscopy imaging device, the apparatus further comprising a memory comprising instructions, the processor arranged to execute the instructions to implement the method of any one of any one of claims 1 to 13.