Artificial intelligence-based system for three-dimensional visualization from two-dimensional images

Abstract

An example medical system includes one or more memories configured to store fluoroscopy imaging data of a patient of a plurality of vessel locations. For each of the plurality of vessel locations, the fluoroscopy imaging data includes at least three images. Each of the at least three images are captured from an angle different from a capture angle of each other of the at least three images. The system includes processing circuitry configured to obtain the fluoroscopy imaging data. The processing circuitry is configured to execute one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information. The processing circuitry is configured to output a 3D visualization of the plurality of vessel locations including a visualization of the lesion.

Description

ARTIFICIAL INTELLIGENCE-BASED SYSTEM FOR THREE-DIMENSIONAL VISUALIZATION FROM TWO-DIMENSIONAL IMAGES

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 736,313, filed on December 19, 2024, the content of which is hereby incorporated by reference.TECHNICAL FIELD

[0002] This disclosure relates to lesion visualization.BACKGROUND

[0003] During a medical procedure, a clinician may use an imaging system to be able to visualize internal anatomy of a patient. Such an imaging system may display anatomy, medical instruments, or the like, and may be used to diagnose a patient condition or assist in guiding a clinician in navigating and / or placing a device inside a patient, such as moving a medical instrument to an intended location inside the patient and / or placing a stent at a particular location. Imaging systems may use sensors to capture image data which may be displayed during the medical procedure. Imaging systems include angiography systems, fluoroscopic systems (e.g., isocentric C-arm fluoroscopic systems), computed tomography (CT) scan systems (including coronary computed tomography angiography (CCTA) systems), intravascular ultrasound (IVUS) systems, other ultrasound imaging systems, optical coherence tomography (OCT), magnetic resonance imaging (MRI) systems, positron emission tomography (PET) systems, near-infrared spectroscopy (NIRS), as well as other imaging systems.

[0004] Coronary artery disease is a leading cause of death globally. Accurate detection and characterization of composition and morphology of arterial lesions are important for effective diagnosis and treatment. Clinicians may prefer to use different treatments for lesions of different compositions and / or morphologies.SUMMARY

[0005] Coronary artery disease is the leading cause of death globally. The accurate detection and characterization of the composition and morphologies of arterial lesions are important for effective diagnosis and treatment. Current techniques for interpreting coronary lesions involve manual reading of fluoroscopic images, for example, by a clinician. Manual reading of fluoroscopic images may be subjective and prone to error. Intravascular imaging (e.g., IVUS) can be used to provide additional information, but is invasive, costly, and time consuming.Additionally, the use of an intravascular imaging as an additional information source and correlating data from both an intravascular imager and a fluoroscopic imager may have a steep a learning curve.

[0006] Pre-procedure CT imaging (e.g., diagnostic CT imaging) may also be used for case preplanning and peri-case support, but such CT imaging is generally outside the workflow of interventional cardiologists. As such, there is a growing demand for non-invasive, automated, workflow integrated tools that assist clinicians in identifying and classifying plaques, defining lesion boundaries, and measuring key morphological and spatial parameters.

[0007] This disclosure describes an artificial intelligence (Al)-driven tool that addresses this need by using machine learning and / or neural network techniques to enhance the interpretation of angiography images, such as coronary C-arm fluoroscopic images, improving both accuracy and efficiency in the clinical setting. As such, the techniques of this disclosure may result in improved patient outcomes and the reduction of the usage and cost of medical resources.

[0008] In one example, the disclosure describes a medical system comprising: one or more memories configured to store fluoroscopy imaging data of a patient of a plurality of vessel locations, for each of the plurality of vessel locations the fluoroscopy imaging data comprising at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images; and processing circuitry communicatively coupled to the memory, the processing circuitry being configured to: obtain the fluoroscopy imaging data; execute one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information; and output a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

[0009] In another example, the disclosure describes a method comprising: obtaining, by processing circuitry of a medical system, fluoroscopy imaging data of a patient of a plurality of vessel locations, for each of the plurality of vessel locations the fluoroscopy imaging data comprising at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images; executing, by the processing circuitry, one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information; and outputting, by the processing circuitry, a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

[0010] In yet another example, the disclosure describes non-transitory computer readable media comprising instructions, which, when executed, cause processing circuitry to: obtain fluoroscopy imaging data of a patient of a plurality of vessel locations, for each of the plurality of vessel locations the fluoroscopy imaging data comprising at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images; execute one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information; and output a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

[0011] These and other aspects of the present disclosure will be apparent from the detailed description below. In no event, however, should the above summaries be construed as limitations on the claimed subject matter, which subject matter is defined solely by the attached claims.

[0012] This summary is intended to provide an overview of the subject matter described in this disclosure. It is not intended to provide an exclusive or exhaustive explanation of the apparatus and methods described in detail within the accompanying drawings and description below. Further details of one or more examples are set forth in the accompanying drawings and the description below.BRIEF DESCRIPTION OF DRAWINGS

[0013] FIG. 1 is a schematic perspective view of one example of a system for lesion identification, classification, and / or morphological analysis according to one or more aspects of this disclosure.

[0014] FIG. 2 is a schematic view of one example of a computing system of the system of FIG. 1.

[0015] FIG. 3 is a conceptual diagram of a cross-section of a vessel including a lesion.

[0016] FIG. 4 is a conceptual diagram of a cross-section of a vessel including a lesion and the measured calcium thickness across the cross-section of the vessel.

[0017] FIG. 5 is a conceptual diagram of cross-sections of a vessel having a non-uniform lesion with images captured from different three angles and the measured calcium thickness across the cross-section.

[0018] FIG. 6 is a conceptual diagram of a 3D visualization of a vessel according to one or more aspects of this disclosure.

[0019] FIG. 7 is a flow diagram illustrating example techniques for lesion classification according to one or more techniques of this disclosure.

[0020] FIG. 8 is a conceptual diagram illustrating an example machine learning model according to one or more aspects of this disclosure.

[0021] FIG. 9 is a conceptual diagram illustrating an example training process for a machine learning model according to one or more aspects of this disclosure.

[0022] FIG. 10 is a conceptual diagram illustrating another example training process for a machine learning model according to one or more aspects of this disclosure.DETAILED DESCRIPTION

[0023] This disclosure describes an Al-driven system for three-dimensional (3D) visualization of coronary artery lesions based on two-dimensional (2D) images, such as those of an imager, such as a C-arm fluoroscopy imager. The system integrates machine learning models, neural networks, and / or shape recognition technologies to generate a 3D model and visualization(s) thereof based on 2D images. By generating the 3D model and 3D visualizations, the techniques of this disclosure may provide a clinician with a better visualization of lesion location, morphology, and / or characteristics, which may better identify and demarcating the border between healthy vessel and diseased lesion. This improved visualization may inform the diagnosis and / or treatment to be used to treat the lesion.

[0024] In some examples, the system may perform lesion detection, grayscale interpretation, plaque classification, lesion boundary identification, lesion characteristics determination, and / or visualization. For example, the system may identify lesions (acute and diffuse) by recognizing specific grayscale patterns and shapes associated with coronary artery plaques in captured image data. The system may interpret all greyscale values in an image, which may provide high contrast resolution than possible with the human eye analyzing an 8-bit or higher greyscale image. This higher contrast resolution may better identify disease than may be possible by a clinician simply viewing the image data. The system may classify plaque using Al-driven algorithms. For example, the system may identify the three types of plaque (calcified, fatty, fibrous) based on the grayscale values and density of plaque in the image data. The system may delineate the boundaries between the lesion and healthy tissue, highlighting the proximal and distal boundaries of the lesion. The system may determine lesion characteristics, such as lesion density, volume, and arc angle. The system may convert the greyscale values of the image data into colors which can be overlayed onto a fluoroscopic image. In some examples, the system may visually highlight the lesion’s boundaries and plaque composition using distinct colors for each plaque type and display the lesion on a three-dimensional (3D) model of the vessel. Techniques for determining types of plaque based on greyscale values are described in co-pending U.S. Provisional Patent Application No. 63 / 736,294, entitled ARTIFICIAL INTELLIGENCE-BASED SYSTEM FOR LESION IDENTIFICATION, CLASSIFICATION, AND MORPHOLOGICAL ANALYSIS, and filed on December 19, 2024, the entire content of which is incorporated herein by reference.

[0025] The techniques of this disclosure may provide a technical solution to a technical problem. For example, current techniques may require invasive or time intensive measures, such as IVUS, OCT, and / or CT imaging to accurately identify lesions, determine the morphology of lesions, and / or to determine the characteristics of lesions, such as calcium, fibrous, and / or fatty. By utilizing one or more machine learning models to generate a 3D model based on a sparse number of 2D images from each location of a vessel of a patient, the techniques of this disclosure, may increase the speed of a medical procedure and efficient use of medical equipment. In some examples, the one or more machine learning models include or form a multimodal model that is a unification of a plurality of machine learning models. In some examples, each machine learning model may be trained on its own training data. For example, one machine learning model may be trained on images and another machine learning model may be trained on metadata. Such machine learning models may be unified to form one machine learning model. The techniques of this disclosure may improve accuracy of a diagnosis and better inform treatment options, thereby increasing a likelihood of a successful treatment procedure and improving patient outcomes.

[0026] FIG. 1 is a schematic perspective view of one example of a system for lesion identification, classification, and / or morphological analysis according to one or more aspects of this disclosure. System 100 includes a display device 110, a table 120, an imager 140, and a computing device 150. System 100 may be an example of a system for use in an emergency room or a Cath lab. In some examples, system 100 may include other devices, not shown for simplicity purposes. In some examples, system 100 may also include server 160, which may be co-located with the other devices of system 100 or may be located elsewhere. System 100 may be used during a medical procedure, such as a medical procedure to identify and / or treat lesions.

[0027] Computing device 150 may include, for example, an off-the-shelf device such as a laptop computer, desktop computer, tablet computer, smart phone, or other similar device or may include a specific purpose device. Computing device 150 may perform various control functions with respect to imager 140. In some examples, computing device 150 may include a guidance workstation. Computing device 150 may control the operation of imager 140 and receive the output of imager 140 and may receive imaging data from imager 140. Computing device 150 may execute one or more machine learning models to determine procedural recommendations to provide to a clinician.

[0028] Display device 110 may be configured to output instructions, images, visualizations, messages relating to the medical procedure(s), and representations identifying lesions, their characteristics, and / or morphologies. For example, display device 110 may display imaging data obtained through imager 140, lesion data, a 3D model of a portion of the vasculature of a patient, and / or the like. In some examples, lesion data may include data relating to lesion identification, classification, morphological analysis, and / or the like. In some examples, display device 110 may display image data overlaid by lesion data. Table 120 may be, for example, an operating table or other table suitable for use during a medical procedure.

[0029] In the example of FIG. 1, imager 140, such as C-arm fluoroscopy imager, an angiography imager, other fluoroscopy imager, or other imaging device, may be used to image relevant portions of the patient’s anatomy during a medical procedure, such as a medical procedure to identify and determine characteristics and morphologies of lesions in the vasculature of a patient. While primarily described herein as a C-arm fluoroscopy imager, imager 140 may be any type of imaging device, such as an angiography device, a fluoroscopy device, a CT device, a CCTA device, an IVUS device, an OCT device, an MRI device, a PET device, an ultrasound device, or the like. In some examples, imager 140 may represent more than one imaging device, such as a plurality of any of the aforementioned devices. In some examples, the system may include C-arm video processing equipment or a connected video capture system.

[0030] Imager 140 may image a region of interest in the patient’s body. The particular region of interest may be dependent on anatomy, the medical procedure, patient symptoms, and / or the like. For example, when performing a medical procedure to identify and / or treat any potential lesions in the vasculature of a patient, the region of interest may include a portion of the vasculature of the patient.

[0031] Computing device 150 may be communicatively coupled to imager 140, display device 110 and / or server 160, for example, by wired, optical, or wireless communications. Server 160 may be a hospital server which may or may not be located in an emergency room or Cath lab of a hospital, a cloud-based server, or the like. Server 160 may be configured to store patient imaging data (such as angiography data), electronic healthcare or medical records, or the like. In some examples, server 160 may be configured to execute the machine learning model(s) and / or perform one or more of, or a portion of one or more of, the determinations associated therewith.

[0032] Any of, or any combination of, computing device 150, imager 140, and / or server 160 may include one or more machine learning model(s). For example, computing device 150, imager 140, and / or server 160 may obtain fluoroscopy imaging data, e.g., via imager 140. The fluoroscopy imaging data may include, for each of a plurality of vessel locations, at least three images, each of the at least three images being captured from an angle different from a capture angle of each otherof the at least three images. Computing device 150, imager 140, and / or server 160 may execute one or more machine learning models to determine, based on the fluoroscopy imaging data, a 3D characteristic of a lesion, a 3D lesion volume, and 3D spatial information. Computing device 150, imager 140, and / or server 160 may output a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

[0033] By outputting a 3D visualization, which may include an identification, classification, and morphological representation of one or more lesions, system 100 may assist clinicians in more effectively determining a treatment plan for such lesions. As such, the techniques of this disclosure may reduce risk of adverse outcomes of patients, improve patient outcomes, and / or improve medical facility efficiency.

[0034] FIG. 2 is a schematic view of one example of a computing device of the system of FIG.1. Computing device 150 may include a workstation, a desktop computer, a laptop computer, a smart phone, a tablet, a dedicated computing device, or any other computing device capable of performing the techniques of this disclosure.

[0035] Computing device 150 may be configured to perform processing, control and other functions associated with imager 140. In some examples, computing device 150 may represent multiple instances of computing devices, each of which may be associated with imager 140. Computing device 150 may include, for example, a memory 202, processing circuitry 204, a display 206, a network interface 208, input device(s) 210, and / or output device(s) 212, each of which may represent any of multiple instances of such a device within the computing system, for ease of description.

[0036] While processing circuitry 204 appears in computing device 150 in FIG. 2, in some examples, features attributed to processing circuitry 204 may be performed by processing circuitry of any of computing device 150, imager 140, or server 160, or combinations thereof. In some examples, one or more processors associated with processing circuitry 204 in computing system may be distributed and shared across any combination of computing device 150, imager 140, and server 160. Computing device 150 may be used to perform any of the techniques described in this disclosure, and may form all or part of devices or systems configured to perform such techniques, alone or in conjunction with other components, such as components of computing device 150, imager 140, server 160, or a system including any or all of such systems / devices.

[0037] Memory 202 of computing device 150 includes any non-transitory computer-readable storage media for storing data or software that is executable by processing circuitry 204 and that controls the operation of computing device 150 and / or imager 140, as applicable. In one or more examples, memory 202 may include one or more solid-state storage devices such as flash memorychips. In one or more examples, memory 202 may include one or more mass storage devices connected to the processing circuitry 204 through a mass storage controller (not shown) and a communications bus (not shown).

[0038] Although the description of computer-readable media herein refers to a solid-state storage, it should be appreciated by those skilled in the art that computer-readable storage media may be any available media that may be accessed by the processing circuitry 204. That is, computer readable storage media includes non-transitory, volatile and non-volatile, removable and nonremovable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. For example, computer-readable storage media includes RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information and that may be accessed by computing device 150. In one or more examples, computer-readable storage media may be stored in the cloud or remote storage and accessed using any suitable technique or techniques through at least one of a wired or wireless connection.

[0039] Memory 202 may store imaging data 214, lesion data 216, and / or 3D model 228. Imaging data 214 may include C-arm fluoroscopy data and / or other imaging data obtained, for example, from imager 140 during a medical procedure.

[0040] During the medical procedure, imaging data may be obtained from imager 140 and stored in imaging data 214. Such imaging data may be displayed via display 206 and / or display device 110 and may be used by a clinician during performance of the diagnostic medical procedure or thereafter, when considering or determining potential treatments for a patient.

[0041] Imaging data 214 may be generated by imager 140 of anatomy of the patient and obtained by computing device 150 via network interface 208 which may be communicatively coupled to imager 140. In some examples, imager 140 may generate other types of imaging data, such as when imager 140 represents more than one imaging device. For example, imaging data 214 may be captured by imager 140. Processing circuitry 204 may obtain imaging data 214 from imager 140 and store imaging data 214 in memory 202. Imaging data 214 may include imaging data of a plurality of locations of a vessel. For example, imaging data 214 may include data representing different locations along a longitudinal axis of a vessel. For each of the plurality of locations, imaging data 214 may include at least three images captured from different angles around the vessel. The different angles are further discussed with respect to FIG. 5, hereinafter.

[0042] Lesion data 216 may include lesion identification, classification, and morphological analysis data concerning lesions within the vasculature of a patient. Lesion data 216 may bedetermined by processing circuitry 204 based on imaging data 214. For example, processing circuitry 204 may execute one or more machine learning model(s) 222 to determine lesion data 216. Processing circuitry 204 may store such lesion data 216 in memory 202. For example, lesion identification data may include data indicative of a lesion existing. Classification data may include data indicative of characteristics of a lesion, such as calcium, fat, and / or fibrous. Morphology data may include data about the location, shape, size, and / or the like of the lesion.

[0043] 3D model 228 may include a 3D model of a portion of the vasculature of a patient. In some examples, processing circuitry 204 may generate 3D model 228 based on imaging data 214 and / or lesion data 216. In some examples, processing circuitry 204 may execute one or more machine learning model(s) 222 to generate 3D model 228 or portions thereof.

[0044] Memory 202 may also store one or more machine learning model(s) 222. Machine learning model(s) 222 may be configured to determine, when executed by processing circuitry 204, lesion data 216 and / or 3D model 228.

[0045] In some examples, imaging data 214 includes C-arm fluoroscopy data. In some examples, processing circuitry 204 may use imaging data 214 to generate 3D model 228 of the anatomy of the patient which processing circuitry 204 may cause display 206 and / or display device 110 to visually display.

[0046] Processing circuitry 204 may execute machine learning model(s) 222, which may include a deep learning model, such as a convolutional neural network (CNN), a 3D U-Net, a linear regression, a logistical regression, a decision tree, a recurrent neural network (RNN), a hybrid model, or other type of machine learning model, to determine lesion data 216. Machine learning model(s) 222 may be trained based on Digital Imaging and Communications in Medicine (DICOM) data, intravascular imaging datasuch as OCT data or IVUS data, and / or CT data. The DICOM data may include fluoroscopy imaging data and metadata. The OCT data may include imaging data, pullback trajectory, and catheter position.

[0047] In some examples, the training data may further include patient data associated with the DICOM data, intravascular imaging data, and / or CT data, such as patient gender, patient body mass index or other measure of patient body mass, portion of patient’s body being imaged, etc. Training data is further discussed herein with respect to FIG. 10.

[0048] Processing circuitry 204 may obtain fluoroscopy imaging data (e.g., imaging data 214), for example, from imager 140. Imaging data 214 may include, for each of a plurality of vessel locations, at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images. Processing circuitry 204 may execute one or more machine learning model(s) 222 to determine, based on the based on the fluoroscopy imaging data, a 3D characteristic of a lesion, a 3D lesion volume, and 3D spatialinformation. Processing circuitry 204 may output a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

[0049] In some examples, machine learning model(s) may not be continuously trained, but may be trained on a closed data set that does not change over time. In some examples, machine learning model(s) 222 may be continuously trained, using data, such as imaging data 214 from procedures occurring after initial deployment of machine learning model(s) 222, for example, when there is also intravascular imaging data captured, to further train machine learning model(s) 222, thereby continuously improving machine learning model(s) 222.

[0050] Processing circuitry 204 may be implemented by one or more processors, which may include any number of fixed-function circuits, programmable circuits, or a combination thereof. In various examples, control of any function by processing circuitry 204 may be implemented directly or in conjunction with any suitable electronic circuitry appropriate for the specified function. Fixed-function circuits refer to circuits that provide particular functionality and are preset on the operations that may be performed. Programmable circuits refer to circuits that may programmed to perform various tasks and provide flexible functionality in the operations that may be performed. For instance, programmable circuits may execute software or firmware that cause the programmable circuits to operate in the manner defined by instructions of the software or firmware. Fixed-function circuits may execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed-function circuits perform are generally immutable. In some examples, the one or more of the units may be distinct circuit blocks (fixed- function or programmable), and in some examples, the one or more units may be integrated circuits.

[0051] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), graphics processing units (GPUs) or other equivalent integrated or discrete logic circuitry. Accordingly, the term processing circuitry 204 as used herein may refer to one or more processors having any of the foregoing processor or processing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0052] Display 206 may be touch sensitive or voice activated, enabling display 206 to serve as both an input and output device. Alternatively, a keyboard (not shown), mouse (not shown), or other data input devices (e.g., input device(s) 210) may be employed.

[0053] Network interface 208 may be adapted to connect to a network such as a local area network (LAN) that includes a wired network or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, or the internet. For example, computing device 150 may obtain imaging data 214 from imager 140 during a medical procedure. Computing device 150 may receive updates to its software, for example, application(s) 217, via network interface 208. Computing device 150 may also display notifications on display 206 that a software update is available.

[0054] Input device(s) 210 may include any device that enables a user to interact with computing device 150, such as, for example, a mouse, keyboard, foot pedal, touch screen, augmented-reality input device receiving inputs such as hand gestures or body movements, or voice interface.

[0055] Output device(s) 212 may include any connectivity port or bus, such as, for example, parallel ports, serial ports, universal serial busses (USB), or any other similar connectivity port known to those skilled in the art.

[0056] Application(s) 217 may be one or more software programs stored in memory 202 and executed by processing circuitry 204 of computing device 150. Processing circuitry 204 may execute user interface 218, which may display imaging data 214, lesion data 216, and / or 3D model 228 on display 206 and / or display device 110.

[0057] In some examples, more than one image (e.g., three to five images or three to ten images) may be obtained that are taken from different angles to build up a 3D picture of the vessel. Density data (e.g., which may correspond to characteristics of lesions in lesion data 216) may be displayed on 3D model 228 of the vessel on a display 206 and / or display device 110. In some examples, processing circuitry 204 may execute one or more machine learning model(s) on imaging data 214 to generate 3D model 228.

[0058] Clinicians such as interventional cardiologists may desire to know the longitudinal position of plaque of a lesion, and the plaque’s radial position and thickness. Both intravascular imaging (IVUS, OCT, NIRS, etc.) and CT imaging can provide this data. However, intravascular imaging an invasive device to be posited inside the vessels and CT takes hundreds of images every 1-3 degrees around the circumference of patient, and thus is a time-consuming process and may require additional visits to medical facilities to obtain such imaging data. For example, CT imaging may be a separate procedure outside of the procedural workflow of an interventional cardiologist and may require an additional visit to a medical facility prior to treatment of a lesion by an interventional cardiologist.

[0059] As such, it may be desirable to determine calcium radial position and thickness, for examples, from cines taken from as few as 3-5 different positions. While this example may notachieve the same resolution or functionality as intravascular imaging or CT it will provide significant more information for the 70% of interventional cardiologists who currently rely on manual interpretation of C-arm fluoroscopy images, while not greatly extending the time of a medical procedure, using more medical resources, or employing invasive imaging equipment.

[0060] FIG. 3 is a conceptual diagram of a cross-section of a vessel including a lesion. In the example of FIG. 3, vessel 300 includes lesion 302 which has a vessel coverage angle of 304 (e.g., less than 180 degrees) at the location of the cross-section. Lesion 302 also has a maximum thickness of X.X millimeters at the location of the cross-section. It should be noted that in a crosssection of the same vessel at a different angle or different location, lesion 302 may have a different vessel coverage angle and / or a different maximum thickness because lesion 302 is a 3D object being viewed in a 2D manner in FIG. 3. By capturing images from different angles, processing circuitry 204 may be better able to generate 3D model 228 and output a visualization of 3D model 228 for display.

[0061] According to the techniques of this disclosure, processing circuitry 204 may train machine learning model(s) 222 based on morphology data, multiple images of rotational angiography image data, and / or intravascular imaging or CT imaging data to generate a 3D morphology analysis, 3D lesion volume, and / or 3D spatial information of 3D model 228.

[0062] FIG. 4 is a conceptual diagram of a cross-section of a vessel including a lesion and the measured calcium thickness across the cross-section of the vessel. X-rays 420 will transmit though vessel 400 and detect, e.g., calcium, but may not be able to detect what side of vessel 400 the calcium is located on. For example, calcium measurement 402 seems to show that calcium is thicker on the left side of vessel 400 than on the right side of vessel 400. However, this is due to the combination of the upper calcium deposit 410 and the lower calcium deposit 412, not necessarily because the upper calcium deposit 410 is thicker on the left side of vessel 400 than on the right side of vessel 400.

[0063] However, if imager 140 captures a plurality of images from different angles around the cross-section of vessel 400, processing circuitry 204 may calculate or determine an estimation of the calcium position within vessel 400. For example, initially imager 140 captures three images at approximated 45 degrees apart. From these three images, processing circuitry 204 may determine an initial assessment of vessel 400. If additional information is required by processing circuitry 204 to meet a programmed or predetermined level of accuracy, processing circuitry 204 may control output device(s) 212, display 206, and / or display device 110 to prompt the clinician or imager 140 to capture one or more additional images from predetermined angle(s) to complete the assessment. In some examples, five images may be used to cover 180 degrees around the crosssection of vessel 400. In some examples, the number of the plurality of images around the cross-section of vessel 400 is sparse, for example, between three and five, inclusive (3-5). In some examples, the number of the plurality of images is between three and ten (3-10). In some examples, the number of the plurality of images is between seven and ten (7-10). In some examples, imager 140 captures rotational angiography data.

[0064] In some examples, processing circuitry 204 may perform a 360° morphology analysis from 2D images and intravascular imaging data (OCT, IVUS, and / or the like) and / or CT data. For example, intravascular imaging data and / or CT data may be used to train machine learning model(s) 222 to provide ground truth morphology data as well as spatial information. Processing circuitry 204 may use spatial information to ensure accuracy within 3D model 228 in modeling the shape of the vessels. Spatial information may include minimal lumen diameter (MLD), minimal lumen area (MLA), length, lesion thickness, calcium percentage of the lesion, etc. In some examples, processing circuitry 204 may also use lesion volume. Processing circuitry 204 may use lesion volume to ensure lesion volume and shape are accurate within 3D model 228 to not oversize the lesion and verify any determined lesion composition volumes (e.g., calcium, fat, fibrous).

[0065] FIG. 5 is a conceptual diagram of cross-sections of a vessel having a non-uniform lesion with images captured from different three angles and the measured calcium thickness across the cross-section. The cross-section of the vessels is intended to be the same cross-section of the same vessel shown as taken at different angles. In example 500, imager 140 captures the image from angle C. In this case, processing circuitry 204 may determine the calcium thickness measurement 502 as shown. In example 510, imager 140 captures the image from angle A. In this case, processing circuitry 204 may determine the calcium thickness measurement 512 as shown. In example 530, imager 140 captures the image from angle B. In this case, processing circuitry 204 may determine the calcium thickness measurement 522 as shown. As can be seen, calcium thickness measurements 502, 512, and 522 are very different. This difference depends on the angle from which an image is captured.

[0066] Some calcium (or fat or fibrious) configurations may look the same from certain angles. Additional angled images may be required to determine the configuration and the thickness of the lesions. In such examples, processing circuitry 204 prompt a clinician (e.g., via output device 212, display 206, and / or display device 110) or may control imager 140 to capture the additional angled images.

[0067] FIG. 6 is a conceptual diagram of a 3D visualization of a vessel according to one or more aspects of this disclosure. 3D visualization 600 may include a visual representation of 3D model 228 shown in panel 602 and a key or legend shown in panel 604. For example, panel 604 includes a key that shows colors which are used in panel 602 to represent different compositions of identified lesions. For example, calcium may be represented by a 1stcolor (e.g., blue), fibrousmay be represented by a 2ndcolor (e.g., green), and fat may be represented by a 3rdcolor (e.g., yellow). Such colors may be overlaid on a visual representation of the vessel(s) to visually indicate the location, size, composition, etc. of any lesions. It should be noted that any colors may be used to represent compositions and / or other visually distinguishing markings, such as cross-hatching or other fill.

[0068] FIG. 7 is a flow diagram illustrating example techniques 3D visualization of lesions from 2D images according to one or more aspects of this disclosure. Processing circuitry 204 may obtain fluoroscopy imaging data of a patient of a plurality of vessel locations (700). For each of the plurality of locations, the fluoroscopy imaging data may include at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images. For example, processing circuitry 204 may obtain imaging data 214 which may include fluoroscopy images captured from three, four, five, or up to ten different angles around a cross-section of a vessel for a particular location on the vessel. By limiting the number of angles for which imager 140 captures images to between three and five (or between three and ten), the length of a medical procedure may be reduced compared to capturing a greater number of images. As such, limiting the number of images to between three and five (or between three and ten) may conserve medical resources.

[0069] Processing circuitry 204 may execute one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information (702). For example, processing circuitry may generate 3D model 228 which may include 3D characteristic(s) of a lesion, a 3D volume of a lesion, and / or 3D spatial information.

[0070] Processing circuitry 204 may output a 3D visualization of the plurality of vessel locations including a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information (704). For example, processing circuitry 204 may output 3D visualization 600 (FIG. 6).

[0071] In some examples, the at least three images include between three and ten images. For example, the at least three images may include three images, four images, five images, or up to ten images, each captured from a different angle around a vessel, such as the angles A, B, and C shown in FIG. 5.

[0072] In some examples, processing circuitry 204 may determine to include at least one additional image of a first vessel location of the plurality of vessel locations as input to the at least one machine learning model. For example, processing circuitry 204 may determine that a resolution or a confidence level of the output of machine learning model(s) 222 does not satisfy a predetermined resolution or confidence level. For example, the resolution or confidence level maybe less than or less than or equal to a predetermined resolution threshold or a predetermined confidence level threshold. In such examples, processing circuitry 204 may output at least one of a prompt to a clinician to capture the at least one additional image of the first vessel location at a different angle than each other of the at least three images or a command to an imager to capture the at least one addition image at the different angle than each other of the at least three images.

[0073] In some examples, the 3D characteristic of the lesion includes at least one of fat, calcium, or fibrous. In some examples, the 3D visualization includes a representation of the 3D characteristic by a color overlayed on a representation of at least a portion of a vessel.

[0074] In some examples, machine learning model(s) 222 are trained using Digital Imaging and Communications in Medicine (DICOM) data and at least one of intravascular imaging data or computed tomography imaging data. In some examples, the intravascular imaging data includes optical coherence tomography (OCT) imaging data or intravascular ultrasound (IVUS) imaging data. In some examples, the DICOM data includes training fluoroscopy imaging data and training DICOM metadata.

[0075] In some examples, processing circuitry 204 may preprocess the fluoroscopy imaging data and execute the one or more machine learning models on preprocessed fluoroscopy imaging data. In some examples, processing circuitry may preprocess the training data and train the one or more machine learning models using the preprocessed training data. In some examples, to preprocess the training data, processing circuitry 204 may preprocess the DICOM data and preprocess the intravascular imaging data. In some examples, to preprocess the DICOM data, processing circuitry 204 may normalize pixel densities of the training fluoroscopy imaging data. In some examples, to preprocess the intravascular imaging data, processing circuitry 204 may at least one of normalize pixel values, segment features, or map spatial measurements in a 3D space.

[0076] In some examples, processing circuitry 204 may align the training fluoroscopy imaging data with the intravascular imaging data in a 3D space. Processing circuitry 204 may relate the intravascular imaging data to the training fluoroscopy imaging data. Processing circuitry 204 may generate training pair inputs comprising a main input and an auxiliary input. The main input may include two-dimensional (2D) fluoroscopy projections and the auxiliary input may include intravascular imaging derived features mapped onto fluoroscopy views.

[0077] To train the one or more machine learning models, processing circuitry 204 may apply a main loss associated with the training fluoroscopy imaging data. Processing circuitry 204 may apply an auxiliary loss associated with the intravascular imaging data. Processing circuitry 204 may reduce a weight of the auxiliary loss over time during the training.

[0078] In some examples, processing circuitry 204 may preprocess imaging data 214 (e.g., angiograms) and intravascular image data (e.g., OCT data). In some examples, processing circuitry204 may perform feature extraction from the OCT data and project the features onto the angiograms. In some examples, processing circuitry 204 may create training pairs of the angiograms and the OCT data. Processing circuitry 204 may use the training pairs to train machine learning model(s) 222. Machine learning model(s) 222 may include a CNN, such as an encoderdecoder CNN or a U-Net 3D model. Machine learning model(s) 222 may include multiple input branches for OCT data and for angiogram data and combine features in a fused representation for 3D construction (e.g., construction of 3D model 228). Machine learning model(s) 222 may include a main branch for angiogram 3D and an auxiliary branch for OCT feature prediction. Processing circuitry 204 may use a main loss function for angiograms and an auxiliary loss function for the OCT data. Processing circuitry 204 may regularize machine learning model(s) 222 to rely on DICOM features by applying dropout to OCT during input training and reducing the weight of the auxiliary OCT loss gradually.

[0079] FIG. 8 is a conceptual diagram illustrating an example machine learning model according to one or more aspects of this disclosure. Machine learning model 800 may be an example of machine learning model(s) 222. Machine learning model 800 may be an example of a deep learning model, or statistical machine learning model, trained to generate 3D model 288. 3D model 288 may include lesion identification, classification, and / or morphological data, including the location of the lesion in a 3D space. One or more of computing device 150 and / or server 160 may train, store, and / or utilize machine learning model 800, but other devices of system 100 may apply inputs to machine learning model 800 in some examples. In some examples, various types of machine learning and deep learning models or algorithms may be utilized. For examples, a CNN, e.g., ResNet-18, may be used. Some non-limiting examples of machine learning techniques include 3D U-Net, Linear Regression, Logistical Regression, Decision Trees, RNN, Support Vector Machines, K-Nearest Neighbor algorithm, Multi-layer Perceptron, and hybrid techniques.

[0080] As shown in the example of FIG. 8, machine learning model 800 may include three types of layers. These three types of layers include input layer 802, hidden layers 804, and output layer 806. Output layer 806 comprises the output from the transfer function 805 of output layer 806. Input layer 802 represents each of the input values XI through X4 provided to machine learning model 800. In some examples, the input values may include any of the values input into the machine learning model, as described above. For example, the input values may include imaging data 214, as described above. In addition, in some examples input values of machine learning model 800 may include additional data, such as other data that may be collected by or stored in system 100. While three types of layers are shown in FIG. 8, it should be understood that, in some examples, a machine learning model of this disclosure, such as machine learningmodel(s) 222 may include specialized layers, separate encoders (e.g., for each modality), a fusion layer, a joint layer, an output layer, and / or the like, to form a multimodal machine learning model.

[0081] Each of the input values for each node in the input layer 802 is provided to each node of a first layer of hidden layers 804. In the example of FIG. 8, hidden layers 804 include two layers, one layer having four nodes and the other layer having three nodes, but fewer or greater number of nodes may be used in other examples. Each input from input layer 802 is multiplied by a weight and then summed at each node of hidden layers 804. During training of machine learning model 800, the weights for each input are adjusted to establish a relationship between imaging data 214 and 3D model 228. In some examples, one hidden layer may be incorporated into machine learning model 800, or three or more hidden layers may be incorporated into machine learning model 800, where each layer includes the same or different number of nodes.

[0082] The result of each node within hidden layers 804 is applied to the transfer function of output layer 806. The transfer function may be linear or non-linear, depending on the number of layers within machine learning model 800. Example non-linear transfer functions may be a sigmoid function or a rectifier function. The output 807 of the transfer function may be a classification of a 3D characteristic, a 3D lesion volume, and / or 3D spatial information. For example, the output 807 may include the morphology of a lesion in a 3D space, the shape of a vessel in a 3D space, and / or a lesion or portion thereof as being calcium, fat, and / or fibrous at various points in the 3D space.

[0083] As shown in the example above, by applying machine learning model 800 to input data such as imaging data 214, processing circuitry 204 is able to identify lesions, determine characteristics of the lesions, and / or determine a morphology of the lesions. From such information, processing circuitry 204 may generate 3D model 228 and output a 3D visualization including a visualization of a lesion. This may improve the ability of a clinician to successfully treat lesions, thereby improving patient outcomes.

[0084] FIG. 9 is a conceptual diagram illustrating an example training process for a machine learning model according to one or more aspects of this disclosure. Process 900 may be used to train machine learning model(s) 222 or machine learning model 800. A machine learning model 974 (which may be an example of machine learning model 800 and / or machine learning model(s) 222) may be implemented using any number of models for semi-supervised, supervised, and / or reinforcement learning, such as but not limited to, a CNN, a 3D U-Net, Linear Regression, Logistical Regression, Decision Trees, RNN, Support Vector Machines, K-Nearest Neighbor algorithm, Multi-layer Perceptron, naive Bayes network, long short-term memory (LSTM), ensemble network, to name only a few examples.

[0085] In some examples, one or more of computing device 150 and / or server 160 or other computing device initially trains machine learning model 974 based on a corpus of training data972. Training data 972 may include, for example, any of, or all of, training data discussed herein. For example, training data 972 may include DICOM imaging data, IVUS data, OCT data, CT data, and / or patient data. In some examples, training data 972 may be annotated to identify vessel walls, lesions, lesion characteristics, lesion morphology, and / or the like.

[0086] While training machine learning model 974, processing circuitry 204 may compare a prediction or classification with a target output 978. Processing circuitry 204 may utilize an error signal from the comparison to train (leaming / training 980) machine learning model 974. Processing circuitry 204 may generate machine learning model weights or other modifications which processing circuitry 204 may use to modify machine learning model 974. For examples, processing circuitry 204 may modify the weights of machine learning model 974 based on the leaming / training 980. For example, processing circuitry 204 may, for each training instance in training data 972, modify, based on training data 972, the manner in which a 3D characteristic, 3D lesion volume, and / or 3D spatial information is determined. Learning / training 980 may be supervised, semi-supervised, and / or reinforcement learning. In some examples, machine learning model 974 may output, during learning / training 980, a 3D reconstructed volume of coronary arteries and a lesion density map (e.g., calcium, fat, fibrous) and / or segmentation within the arteries.

[0087] FIG. 10 is a conceptual diagram illustrating another example training process for a machine learning model according to one or more aspects of this disclosure. Process 1000 may be used to train machine learning model(s) 222 or machine learning model 800. A machine learning model 1008 (which may be an example of machine learning model 800 and / or machine learning model(s) 222) may include a neural network or other type of machine learning model. In the example of FIG. 10, intravascular imaging data is described using the example of OCT imaging data. It should be understood that other types of intravascular imaging data may be used.

[0088] In some examples, machine learning model 1008 may include a base network for volume reconstruction. Examples of the base network include a 3D U-Net, an encoder-decoder CNN, or the like. In some examples, for auxiliary OCT supervision, machine learning model(s) 222 may include a secondary output branch for predicting OCT-like features from DICOM input. For example, this secondary output branch may only be used during training and removed during deployment of machine learning model 1008. Multi-task loss design for machine learning model 1008 may include a primary task of reconstructing a 3D volume and an auxiliary task of predicting OCT-derived features, such as vessel wall boundaries and lesion characteristics, such as calcium, fat, and / or fibrous intensities.

[0089] Training data 1002 may include imaging data, such as DICOM images sparse images (e.g., three to five or up to ten or so images) taken at known angles. The DICOM images mayinclude metadata indicating imaging geometry (e.g., source-to-detector distance, pixel spacing, power used during image capture, angle, bit size of the imager, pixel size, etc.). Training data 1002 may also include OCT pullback data. The OCT pullback data may include high-resolution cross- sectional images of vessels, such as a coronary artery, pullback trajectory, and catheter position. The DICOM images and the OCT pullback data may be from a plurality of patients taken during the same procedures. In other words, for a given procedure, there may be both DICOM images and OCT pullback data that are used as part of training data 1002. The pullback trajectory and catheter position information from the OCT pullback data may be used, for example, for spatial alignment between the OCT images and the DICOM images.

[0090] In some examples, training data 1002 may include patient data, such as patient BMI or other measure of patient body mass, patient gender, or other patient information. In some examples, training data 1002 may include density data of physical objects represented in imaging data as calibration artifacts. In some examples, training data 1002 may be annotated to identify vessel walls, a calibration artifact, density of the physical object causing the calibration artifact, lesions, lesion characteristics, and / or lesion morphology.

[0091] In some examples, training data 1002 may include synthetic data, real data, augmented data, and / or regularization. Synthetic data may include generated realistic 3D phantom images with the synthetic DICOM and OCT pairs. Processing circuitry 204 may generate synthetic data using forward models and / or an OCT simulator. Real data may include datasets with paired DICOM and OCT data and / or validated 3D reconstructions (e.g. CT reconstructions). For data augmentation, processing circuitry 204 may introduce variability in angular spacing and noise in DICOM projections and / or introduce variability distributions and artifacts in OCT data. Processing circuitry 204 may regularize machine learning model 1008 to rely on DICOM features by applying dropout to OCT during input training and reducing the weight of the auxiliary OCT loss gradually.

[0092] The input training data 1002 may be pre-processed 1004. Processing circuitry 204 may preprocess angiogram data and OCT data individually. For the angiogram data, processing circuitry 204 may extract 2D images and associated metadata and normalize pixel densities across the angiogram data. For the OCT data, processing circuitry 204 may normalize intensity values, segment features, such as healthy / disease boundary and calcium / plaque regions, map spatial measurements in 3D space, and / or the like.

[0093] Processing circuitry 204 may align angiograms with corresponding OCT data in 3D space. For example, processing circuitry 204 may reconstruct 3D vessel geometry from 2D angiograms. For example, processing circuitry 204 may apply epipolar geometry and / or iterative reconstruction techniques for 3D models (e.g., Simultaneous Algebraic Reconstruction Technique(SART), Maximum Likelihood Expectation Maximization (MLEM), Total Variation (TV) Minimization, Iterative Back Projection (IBP), Compressed Sensing-based Techniques).

[0094] Processing circuitry 204 may use known catheter pullback trajectory from OCT data and map the pullback trajectory into 3D space. Processing circuitry 204 may align each OCT frame to its corresponding position along the vessel. Processing circuitry 204 may perform spatial registration so as to align the two data sets (angiography data and OCT data) to a common coordinate system. This aligns the angiography data with the OCT data for consistent interpretation in the same spatial context.

[0095] Processing circuitry 204 may relate the OCT data to angiographic projections. For example, processing circuitry 204 may simulate OCT projections in angiograms. Processing circuitry 204 may use the 3D positions from the OCT data to show their appearance in 3D. For example, processing circuitry 204 may, from each OCT frame, calculate its projection in an angiogram. Processing circuitry 204 may extract features from OCT data (e.g., calcium, boundaries, etc.).

[0096] Processing circuitry 204 may create training pair inputs. Each training sample may include a main input of 2D angiogram projections (e.g., 3-5 images captured at different angles) and an auxiliary input of OCT derived features mapped onto the angiographic views. Processing circuitry 204 may fuse the two types of data using early fusion, late fusion, or hybrid fusion techniques.

[0097] Processing circuitry 204 may perform training 1006 of a machine learning model 1008 using the pre-processed data. For example, training 1006 may generate a transfer function (Fx) that may be applied to input data to make a prediction regarding boundaries of lesions and / or characteristics of lesions, such as lesion composition as being calcium, fat, and / or fibrous. It should be noted that output of machine learning model 1008 may be used for further training 1006.

[0098] In some examples, a primary loss function, 3D reconstruction, may measure the error between the reconstructed 3D volume and the ground-truth 3D structure which is represented in the OCT data. The auxiliary loss function, OCT feature prediction, may penalize deviations between the predicted OCT-like feature and the true OCT data. For example, L2 loss ((least squared errors loss) which may be used with pixel intensity or pixel values), Dice loss (which may be used for segmentation tasks). A physics-based loss function may also be used. The physicsbased loss function may force a consistency between forward-projections of the reconstructed volume and the input DICOM images.

[0099] Once machine learning model 1008 is deployed for operational use, processing circuitry 204 may obtain single DICOM images or sparce DICOM projections from imager 140 as input data. Processing circuitry 204 may pre-process the DICOM images to match training datadistribution. In some examples, processing circuitry 204 may normalize the input DICOM image(s) to make or ensure that the input DICOM image(s) are all the same size and may apply gaussian filtering to make or ensure that the images look relatively similar so that the images can be compared and calibrated. In some examples, processing circuitry 204 may perform such preprocessing across the training data set, and processing circuitry 204 may perform the preprocessing to alter the input DICOM image(s) to match the size, noise level, etc. of the training data set.

[0100] Processing circuitry 204 may pre-process the DICOM images to match training data distribution. In some examples, processing circuitry 204 may normalize input images to make or ensure that the input images are all the same size and may apply gaussian filtering to make or ensure that the images look relatively similar so that the images can be compared and calibrated. In some examples, processing circuitry 204 may perform such pre-processing across the training data set, and processing circuitry 204 may perform the pre-processing to alter the input images to match the size, noise level, etc. of the training data set.

[0101] Processing circuitry 204 may input the pre-processed DICOM images to machine learning model(s) 222. Machine learning model(s) 222 may output a reconstructed 3D artery volume and calcium density map or segmentation.

[0102] The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware or any combination thereof. For example, various aspects of the described techniques may be implemented within one or more processors or processing circuitry, including one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuitry, as well as any combinations of such components. The terms “controller”, “processor”, or “processing circuitry” may generally refer to any of the foregoing logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit comprising hardware may also perform one or more of the techniques of this disclosure. Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various operations and functions described in this disclosure. In addition, any of the described units, circuits or components may be implemented together or separately as discrete but interoperable logic devices. Depiction of different features as circuits or units is intended to highlight different functional aspects and does not necessarily imply that such circuits or units must be realized by separate hardware or software components. Rather, functionality associated with one or more circuits or units may be performed by separate hardware or software components or integrated within common or separate hardware or software components.

[0103] The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. Instructions embedded or encoded in a computer-readable storage medium may cause a programmable processor, or other processor, to perform the method, e.g., when the instructions are executed. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), or electronically erasable programmable read only memory (EEPROM), or other computer readable media.

[0104] This disclosure includes the following non-limiting examples.

[0105] Example 1. A medical system comprising: one or more memories configured to store fluoroscopy imaging data of a patient of a plurality of vessel locations, for each of the plurality of vessel locations the fluoroscopy imaging data comprising at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images; and processing circuitry communicatively coupled to the memory, the processing circuitry being configured to: obtain the fluoroscopy imaging data; execute one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information; and output a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

[0106] Example 2. The medical system of example 1, wherein the at least three images comprise between three and ten images.

[0107] Example 3. The medical system of any of examples 1-2, wherein the processing circuitry is further configured to: determine to include at least one additional image of a first vessel location of the plurality of vessel locations as input to the one or more one machine learning models; and output at least one of a prompt to a clinician to capture the at least one additional image of the first vessel location at a different angle than each other of the at least three images or a command to an imager to capture the at least one addition image at the different angle than each other of the at least three images.

[0108] Example 4. The medical system of any of examples 1-3, wherein the 3D characteristic of the lesion comprises at least one of fat, calcium, or fibrous.

[0109] Example 5. The medical system of example 4, wherein the 3D visualization comprises a representation of the 3D characteristic by a color overlayed on a representation of at least a portion of a vessel.

[0110] Example 6. The medical system of any of examples 1-5, wherein the one or more machine learning models are trained using Digital Imaging and Communications in Medicine(DICOM) data and at least one of intravascular imaging data or computed tomography imaging data.

[0111] Example 7. The medical system of example 6, wherein the intravascular imaging data comprises optical coherence tomography (OCT) imaging data or intravascular ultrasound (IVUS) imaging data.

[0112] Example 8. The medical system of example 6 or example 7, wherein the DICOM data comprises training fluoroscopy imaging data and training DICOM metadata.

[0113] Example 9. The medical system of any of examples 6-8, wherein the processing circuitry is configured to preprocess the fluoroscopy imaging data and execute the one or more machine learning models on preprocessed fluoroscopy imaging data.

[0114] Example 10. The medical system of example 9, wherein the processing circuitry is further configured to: preprocess training data; and train the one or more machine learning models using the preprocessed training data.

[0115] Example 11. The medical system of example 10, wherein to preprocess the training data, the processing circuitry is configured to: preprocess the DICOM data; and preprocess the intravascular imaging data.

[0116] Example 12. The medical system of example 11, wherein to preprocess the DICOM data, the processing circuitry is configured to normalize pixel densities of the training fluoroscopy imaging data.

[0117] Example 13. The medical system of example 11 or example 12, wherein to preprocess the intravascular imaging data, the processing circuitry is configured to at least one of normalize pixel values, segment features, or map spatial measurements in a 3D space.

[0118] Example 14. The medical system of any of examples 10-13, wherein the processing circuitry is further configured to: align the training fluoroscopy imaging data with the intravascular imaging data in a 3D space; relate the intravascular imaging data to the training fluoroscopy imaging data; and generate training pair inputs comprising a main input and an auxiliary input, wherein the main input comprises two-dimensional (2D) fluoroscopy projections and the auxiliary input comprises intravascular imaging derived features mapped onto fluoroscopy views.

[0119] Example 15. The medical system of any of examples 10-14, wherein to train the one or more machine learning models, the processing circuitry is configured to: apply a main loss associated with the training fluoroscopy imaging data; and apply an auxiliary loss associated with the intravascular imaging data, wherein a weight of the auxiliary loss is reduced over time during the training.

[0120] Example 16. A method comprising: obtaining, by processing circuitry of a medical system, fluoroscopy imaging data of a patient of a plurality of vessel locations, for each of the plurality of vessel locations the fluoroscopy imaging data comprising at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images; executing, by the processing circuitry, one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information; and outputting, by the processing circuitry, a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

[0121] Example 17. The method of example 16, wherein the at least three images comprise between three and ten images.

[0122] Example 18. The method of any of examples 16-17, further comprising: determining, by the processing circuitry, to include at least one additional image of a first vessel location of the plurality of vessel locations as input to the at least one machine learning model; and outputting, by the processing circuitry, at least one of a prompt to a clinician to capture the at least one additional image of the first vessel location at a different angle than each other of the at least three images or a command to an imager to capture the at least one addition image at the different angle than each other of the at least three images.

[0123] Example 19. The method of any of examples 16-18, wherein the 3D characteristic of the lesion comprises at least one of fat, calcium, or fibrous.

[0124] Example 20. Non-transitory computer-readable storage media storing instructions, which when executed cause processing circuitry to: obtain fluoroscopy imaging data of a patient of a plurality of vessel locations, for each of the plurality of vessel locations the fluoroscopy imaging data comprising at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images; execute one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information; and output a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

[0125] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

What is claimed is:

1. A medical system comprising: one or more memories configured to store fluoroscopy imaging data of a patient of a plurality of vessel locations, for each of the plurality of vessel locations the fluoroscopy imaging data comprising at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images; and processing circuitry communicatively coupled to the memory, the processing circuitry being configured to: obtain the fluoroscopy imaging data; execute one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information; and output a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

2. The medical system of claim 1, wherein the at least three images comprise between three and ten images.

3. The medical system of any of claims 1-2, wherein the processing circuitry is further configured to: determine to include at least one additional image of a first vessel location of the plurality of vessel locations as input to the one or more one machine learning models; and output at least one of a prompt to a clinician to capture the at least one additional image of the first vessel location at a different angle than each other of the at least three images or a command to an imager to capture the at least one addition image at the different angle than each other of the at least three images.

4. The medical system of any of claims 1-3, wherein the 3D characteristic of the lesion comprises at least one of fat, calcium, or fibrous.

5. The medical system of claim 4, wherein the 3D visualization comprises a representation of the 3D characteristic by a color overlayed on a representation of at least a portion of a vessel.

6. The medical system of any of claims 1-5, wherein the one or more machine learning models are trained using Digital Imaging and Communications in Medicine (DICOM) data and at least one of intravascular imaging data or computed tomography imaging data.

7. The medical system of claim 6, wherein the intravascular imaging data comprises optical coherence tomography (OCT) imaging data or intravascular ultrasound (IVUS) imaging data.

8. The medical system of claim 6 or claim 7, wherein the DICOM data comprises training fluoroscopy imaging data and training DICOM metadata.

9. The medical system of claim 8, wherein the processing circuitry is configured to preprocess the fluoroscopy imaging data and execute the one or more machine learning models on preprocessed fluoroscopy imaging data.

10. The medical system of claim 9, wherein the processing circuitry is further configured to: preprocess training data; and train the one or more machine learning models using the preprocessed training data.

11. The medical system of claim 10, wherein to preprocess the training data, the processing circuitry is configured to: preprocess the DICOM data; and preprocess the intravascular imaging data.

12. The medical system of claim 11, wherein to preprocess the DICOM data, the processing circuitry is configured to normalize pixel densities of the training fluoroscopy imaging data, and wherein to preprocess the intravascular imaging data, the processing circuitry is configured to at least one of normalize pixel values, segment features, or map spatial measurements in a 3D space.

13. The medical system of any of claims 10-12, wherein the processing circuitry is further configured to: align the training fluoroscopy imaging data with the intravascular imaging data in a 3D space; relate the intravascular imaging data to the training fluoroscopy imaging data; andgenerate training pair inputs comprising a main input and an auxiliary input, wherein the main input comprises two-dimensional (2D) fluoroscopy projections and the auxiliary input comprises intravascular imaging derived features mapped onto fluoroscopy views.

14. The medical system of any of claims 10-13, wherein to train the one or more machine learning models, the processing circuitry is configured to: apply a main loss associated with the training fluoroscopy imaging data; and apply an auxiliary loss associated with the intravascular imaging data, wherein a weight of the auxiliary loss is reduced over time during the training.

15. A method comprising: obtaining, by processing circuitry of a medical system, fluoroscopy imaging data of a patient of a plurality of vessel locations, for each of the plurality of vessel locations the fluoroscopy imaging data comprising at least three images, each of the at least three images being captured from an angle different from a capture angle of each other of the at least three images; executing, by the processing circuitry, one or more machine learning models to determine, based on the fluoroscopy imaging data, a three-dimensional (3D) characteristic of a lesion, a 3D lesion volume, and 3D spatial information; and outputting, by the processing circuitry, a 3D visualization of the plurality of vessel locations comprising a visualization of the lesion based on the 3D characteristic, the 3D lesion volume, and the 3D spatial information.

Images