Systems and methods for generating pseudo-fluoroscopic views and auto view selection
The generation of pseudo-fluoroscopic views and multiple view angles from ultrasound data addresses the limitations of current ultrasound systems by providing radiation-free imaging and efficient navigation in interventional procedures, improving spatial orientation and procedural guidance.
Patent Information
- Application Number
- PCT/IB2026/000250
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-07
- Filing Date
- 2026-05-06
- Publication Date
- 2026-11-12
AI Technical Summary
Current ultrasound imaging systems face challenges in stabilizing internal imaging devices due to dynamic body movements, leading to inconsistent and inaccurate image capture, particularly in interventional procedures like cardiac ablation, where real-time catheter guidance and multiple view angles are crucial for effective treatment.
Systems and methods generate pseudo-fluoroscopic views from ultrasound data, providing radiation-free imaging and multiple view angles in 3D volumetric imaging, using catheter-based ultrasound data combined with tracking data to create intuitive 2D projections and interactive digital models, enhancing spatial orientation and navigation.
This approach improves navigation and visualization in interventional procedures by offering radiation-free imaging and efficient multiple view angles, reducing reliance on fluoroscopy and streamlining image manipulation, thus enhancing procedural guidance and spatial awareness.
Smart Images

Figure 00000060_0000 
Figure 00000061_0000 
Figure 00000062_0000
Abstract
Description
[0001] Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0002] PCT APPLICATION SYSTEMS AND METHODS FOR GENERATING PSEUDO-FLUOROSCOPIC VIEWS AND AUTO VIEW SELECTION
[0003] Cross-Reference to Related Applications
[0004] This application claims priority to, and the benefit of, U.S. Provisional Application No.
[0005] 63 / 801,394, filed May 7, 2025, the content of which is incorporated by reference herein in its entirety.
[0006] Field of the Invention
[0007] The invention generally relates to ultrasound imaging, and, more particularly, to systems and methods for providing ultrasound-derived pseudo-fluoroscopic views and multiple view angles for improved navigation in cardiac procedures.
[0008] Background
[0009] Ultrasound imaging is a medical imaging technique for imaging organs and soft tissues in a human body. An ultrasound image is produced based on the reflection of high-frequency sound waves off of body structures. The strength (amplitude) of the sound signal in conjunction with the time it takes for the wave to travel through the body provides the information necessary to produce the image.
[0010] For example, catheter-based endovascular ultrasound imaging technology employed within the vasculature (e.g., intravascular ultrasound (IVUS) or intracardiac echocardiography (ICE)) is commonly performed with two-dimensional (2D)-ultrasound imaging. Tn IVUS / TCE imaging systems, an ultrasonic transducer assembly is attached to a distal end of a catheter. The catheter is carefully maneuvered through a patient's body to an area of interest, such as within a coronary artery (for the case of IVUS), or within the right atrium (for the case of ICE). The transducer assembly transmits ultrasound waves and receives echoes from those waves. The received echoes are then converted to electrical signals and transmitted to processing equipment, in which a resulting ultrasound image of the area of interest may be displayed.
[0011] In typical ultrasound systems configured to visualize inner body regions, dynamic forces are often employed, resulting in a dynamic movement of the body regions over time. These dynamic forces and movements make it difficult to stabilize internal imaging devices and to generate consistent and accurate images. As a result, the captured images often lack the necessary quality required to prescribe appropriate treatment or therapy. Because of the dynamicAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0012] PCT APPLICATION
[0013] forces and movements in play, internal real-time imaging is limited to small two-dimensional areas or limited three-dimensional volumetric regions respectively. For certain interventional treatments, such as cardiac ablation, real-time catheter guidance and accurately capturing a visual representation of the anatomy of interest in multiple views is paramount for a successful procedure. Accordingly, improved systems and methods for real-time visual understanding the anatomical environment for delivering effective therapy are needed.
[0014] Summary
[0015] The present invention recognizes the limitations of currently utilized strategies for realtime catheter guidance and the live visualization of complex anatomy in interventional procedures, namely the information overload, and the lack of a standardized view orientation for practitioners such as electrophysiologists, as well as the limited capacity of interventional surgeons to manipulate imaging software / consoles while performing an intervention. The invention addresses these challenges by providing systems and methods for generating simulated views, such as pseudo-fluoroscopic views from ultrasound data, thus providing radiation-free imaging for improved navigation and visualization. The invention further provides systems and methods for generating multiple view angles in 3D volumetric imaging on a specific target for improved real-time assessment of anatomical conditions and navigation, thus improving spatial orientation without requiring multiple user actions to create the view angles.
[0016] In interventional procedures, electrophysiologists rely on fluoroscopy for real-time catheter guidance due in part because the standardized orientation of C-arms and patient positioning in relation to the practitioner provides for efficient anatomical orientation in relation to the interventional room. However, the drawbacks of fluoroscopy guidance include poor soft tissue contrast and radiation dosage for operator and patient alike. The invention recognizes that, despite advancements in ultrasound such as 3D Intracardiac Echocardiography (ICE), information overload and the lack of a standardized view orientation can impede effective orientation, hindering the transition to zero fluoroscopy procedures. The invention solves these problems by providing systems and methods for generating pseudo-fluoroscopy data from ultrasound data. Utilizing ultrasound information, optionally augmented with tracking data, and leveraging the known catheter dynamics, to provide practitioners such as electrophysiologists, with an intuitive 2D projection. Thus, the pseudo-fluoroscopic views provided by the systems and methods of the invention, align with the accustomed visual reference, enhancingAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0017] PCT APPLICATION
[0018] comprehension of the internal structures and tracked tools without the need for fluoroscopy. Accordingly, the systems and methods of the invention enable a radiation-free alternative for catheter navigation and optimize procedural guidance with streamlined and improved live visualization of complex structures.
[0019] Further, the invention recognizes that interventional surgeons are limited in their capacity to manipulate imaging software / consoles while performing an interventional procedure, while at the same time needing to quickly understand the anatomical environment to safely deliver the therapy. The invention addresses the limitations of current systems by providing multiple view angles in a 3D volumetric image of a specific target to improve both assessment of anatomical conditions and device navigation in the area without requiring multiple user actions to create the views. In particular, the systems and methods of the invention provides a simple two-step procedure for generating a first view of a desired landmark and a second 3D view of the desired landmark from an opposite viewing direction. In this way, the systems and methods of the invention streamline the control and manipulation of imaging during the interventional procedure thus allowing for quick and efficient understanding of the anatomical environment and improved spatial orientation with limited user actions required.
[0020] Aspects of the invention provide systems for generating a simulated view from three-dimensional (3D) ultrasound data. The systems include a console configured to be operably associated with an ultrasound imaging device and to exchange data therewith. The console includes a hardware processor coupled to non-transitory, computer-readable memory containing instructions executable by the processor to cause the console to receive data associated with one or more anatomical regions of interest, the data comprising at least one of catheter-based ultrasound imaging data, catheter position data, and catheter orientation data. The processor further causes the console to process and combine the received data; generate, based on the processing and combining of data, simulated two-dimensional (2D) projection image data; and reconstruct, based on the simulated 2D data, a simulated image of the one or more anatomical regions of interest, wherein the simulated image comprises an estimated catheter position within the simulated image.
[0021] In some embodiments, the simulated 2D projection image data comprises simulated fluoroscopy data, wherein a simulated fluoroscopic image of the one or more anatomical regions of interest is generated.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0022] PCT APPLICATION
[0023] In some embodiments, the simulated image corresponds to a standardized orientation of a view of the anatomical region of interest. For example, the standardized orientation includes at least one of an anterior-posterior (AP) view, a posterior-anterior (PA) view, a left anterior oblique (LAO) view, a right anterior oblique (RAO) view, a left lateral (LL) view, and a right lateral (RR) view, in some embodiments.
[0024] In some embodiments of the system, the catheter-based ultrasound imaging data is received from an intracardiac echocardiogram (ICE) two-dimensional (2D) and / or 3D catheter. Further, in some embodiments, the one or more anatomical regions of interest comprise at least one of cardiac anatomy and vascular anatomy. In some embodiments, the received data further comprises at least one of intracardiac echocardiography (ICE) data, data associated with one or more anatomical landmarks, data associated with an accessory pathway, cardiac pulse phase data, and a dynamic of a structure of the catheter. In some embodiments, the system is configured to calculate a catheter position and a catheter orientation within the one or more anatomical regions of interest in relation to at least one of a cardiac anatomy, vascular anatomy, and an anatomical landmark.
[0025] In some embodiments, the catheter position data and catheter orientation data include 3D tracking data. In further embodiments, the 3D tracking data comprises at least one of electromagnetic (EM)-based tracking data, impedance tracking data, image-based tracking data, and fiber optic shape sensing.
[0026] In some embodiments, the simulated image includes a projection of 3D ultrasound data of the one or more anatomical regions of interest onto a 2D projection plane with the catheter overlay ed onto the 2D projection such that the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
[0027] In some embodiments, the simulated image includes at least one of a sectional reconstruction and a multiplanar reconstruction of the anatomical region of interest onto one or more 2D projection planes visualized in conjunction with the catheter position such that the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
[0028] In some embodiments, the simulation includes at least one of a raytracing-based simulation, a physics-based simulation, and a machine learning-based simulation.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0029] PCT APPLICATION
[0030] In some embodiments the simulated image includes an interactive digital model of the one or more anatomical regions of interest, wherein the interactive digital model comprises a common view generated beginning from a catheter access site to one or more targets within a heart to provide an interactive visualization of patient-specific structure and / or morphology of intra-cardiac structures.
[0031] In some embodiments, the catheter-based ultrasound imaging data is received from a catheter-based ultrasound comprising an endovascular and / or intravascular ultrasound configured for in-body applications. In further embodiments, the ultrasound imaging data is received from a plurality of cylindrical fields-of-view acquired via a rotating transducer array and / or a cylindrical folded transducer matrix.
[0032] Aspects of the invention provide methods for generating a simulated view from three-dimensional (3D) ultrasound data. The methods include the steps of providing a console configured to be operably associated with an ultrasound imaging device and exchange data therewith; receiving, via the console, data associated with one or more anatomical regions of interest, the data comprising at least one of catheter-based ultrasound imaging data, catheter position data, and catheter orientation data; processing and combining, via the console, the received data; generate, based on the processing and combining of data, simulated two-dimensional (2D) projection image data; and reconstruct, based on the simulated 2D data, a simulated image of the one or more anatomical regions of interest, wherein the simulated image comprises an estimated catheter position within the simulated image.
[0033] In some embodiments, the simulated 2D projection image data comprises simulated fluoroscopy data, wherein a simulated fluoroscopic image of the one or more anatomical regions of interest is generated.
[0034] In some embodiments, the simulated image corresponds to a standardized orientation of a view of the anatomical region of interest. For example, the standardized orientation includes at least one of an anterior-posterior (AP) view, a posterior-anterior (PA) view, a left anterior oblique (LAO) view, a right anterior oblique (RAO) view, a left lateral (LL) view, and a right lateral (RR) view, in some embodiments.
[0035] In some embodiments of the method, the catheter-based ultrasound imaging data is received from an intracardiac echocardiogram (ICE) two-dimensional (2D) and / or 3D catheter. In some embodiments, the one or more anatomical regions of interest comprise at least one ofAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0036] PCT APPLICATION
[0037] cardiac anatomy and vascular anatomy. In further embodiments, the received data further includes at least one of intracardiac echocardiography (ICE) data, data associated with one or more anatomical landmarks, data associated with an accessory pathway, cardiac pulse phase data, and a dynamic of a structure of the catheter. For example, the system calculates a catheter position and a catheter orientation within the one or more anatomical regions of interest in relation to at least one of a cardiac anatomy, vascular anatomy, and an anatomical landmark, in some embodiments.
[0038] In some embodiments of the method, the catheter position data and catheter orientation data include 3D tracking data. The 3D tracking data includes at least one of electro-magnetic (EM)-based tracking data, impedance tracking data, image-based tracking data, and fiber optic shape sensing, in some embodiments.
[0039] In some embodiments of the method the simulated image includes a projection of 3D ultrasound data of the one or more anatomical regions of interest onto a 2D projection plane with the catheter overlay ed onto the 2D projection such that the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
[0040] In some embodiments of the method, the simulated image comprises at least one of a sectional reconstruction and a multiplanar reconstruction of the anatomical region of interest onto one or more 2D projection planes visualized in conjunction with the catheter position such that the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
[0041] In some embodiments, the simulation includes at least one of a raytracing-based simulation, a physics-based simulation, and a machine learning-based simulation.
[0042] In some embodiments of the method, the simulated image includes an interactive digital model of the one or more anatomical regions of interest, wherein the interactive digital model comprises a common view generated beginning from a catheter access site to one or more targets within a heart to provide an interactive visualization of patient-specific structure and / or morphology of intra-cardiac structures.
[0043] In some embodiments of the method, the catheter-based ultrasound imaging data is received from a catheter-based ultrasound comprising an endovascular and / or intravascular ultrasound configured for in-body applications. The ultrasound imaging data is received from aAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0044] PCT APPLICATION
[0045] plurality of cylindrical fields-of-view acquired via a rotating transducer array and / or a cylindrical folded transducer matrix, in some embodiments.
[0046] Aspects of the invention include systems for providing a plurality of view angles of an anatomical region of interest. The systems include a console configured to be operably associated with an ultrasound imaging device and to exchange data therewith. The console includes a hardware processor coupled to non-transitory, computer-readable memory containing instructions executable by the processor to cause the console to receive and analyze three-dimensional (3D) ultrasound image data obtained from an ultrasound imaging device for a target anatomical region, the 3D ultrasound image data being associated with a 3D volume of the target anatomical region oriented in a selected view angle; generate a first image of the target anatomical region at the selected view angle; calculate, via one or more algorithms, data associated with a second image of the target anatomical region; and generate the second image of the target region comprising a second view angle different from the selected view angle.
[0047] In some embodiments, the second view angle is opposite of the selected view angle. In some embodiments, the second view angle is a defined angle of interest, wherein the angle of interest comprises one or more of an opposite of the selected view angle and an orthogonal view. In some embodiments, any defined angle may be used to yield an improved dual view.
[0048] In some embodiments, the 3D ultrasound image data comprises coordinate position data of an original camera and a viewing direction at the target anatomical region. In some embodiments, the algorithm is configured to define a clip plane splitting the first image of the target anatomical region into an intersecting mirror plan, wherein the algorithm mirrors the coordinate position and viewing angle in the second image. Further, the a location of the intersecting mirror plane is defined by adding a defined buffer offset from the clip plane to the clip plane coordinates in the 3D volume, in some embodiments. In some embodiments, the defined buffer offset is based on at least one of known averages from one or more anatomical databases and known sizes of one or more anatomical landmarks. In some embodiments, a position of a mirrored camera source is calculated based on the distance of the original camera to the clip plane, the defined buffer offset, and by adding one or more same distances to the mirror plane in a same direction as the viewing direction such that the mirrored camera source is placed at a same distance from the mirror plane with an opposed viewing direction to the original camera to thereby provide a second 3D view from an opposite view.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0049] PCT APPLICATION
[0050] In some embodiments of the systems, a dual view on a landmark improves spatial orientation without creating a new view either manually or via a multi-step process.
[0051] In some embodiments, the system automatically creates a second 3D view from the opposite viewing direction.
[0052] In some embodiments, the desired 3D view is oriented in at least one of a particular rotation and a particular zoom.
[0053] Aspects of the invention provide methods for providing a plurality of view angles of an anatomical region of interest. The method includes the steps of providing a console configured to be operably associated with an ultrasound imaging device and exchange data therewith; receiving and analyzing, via the console, three-dimensional (3D) ultrasound image data obtained from an ultrasound imaging device for a target anatomical region, the 3D ultrasound image data being associated with a 3D volume of the target anatomical region oriented in a selected view angle; generating, via the console, a first image of the target anatomical region at the selected view angle; calculating, via one or more algorithms, data associated with a second image of the target anatomical region; and generating, via the console, the second image of the target region comprising a second view angle different from the selected view angle.
[0054] In some embodiments of the method, the second view angle is opposite of the selected view angle. In some embodiments, the second view angle is a defined angle of interest, wherein the angle of interest comprises one or more of an opposite of the selected view angle and an orthogonal view. In some embodiments, any defined angle may be used to yield an improved dual view.
[0055] In some embodiments of the method, the 3D ultrasound image data comprises coordinate position data of an original camera and a viewing direction at the target anatomical region. In particular embodiments, the algorithm is configured to define a clip plane splitting the first image of the target anatomical region into an intersecting mirror plan, wherein the algorithm mirrors the coordinate position and viewing angle in the second image. Further, a location of the intersecting mirror plane is defined by adding a defined buffer offset from the clip plane to the clip plane coordinates in the 3D volume, in some embodiments. In some embodiments of the methods, the defined buffer offset is based on at least one of known averages from one or more anatomical databases and known sizes of one or more anatomical landmarks. In some embodiments of the methods, a position of a mirrored camera source is calculated based on theAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0056] PCT APPLICATION
[0057] distance of the original camera to the clip plane, the defined buffer offset, and by adding one or more same distances to the mirror plane in a same direction as the viewing direction such that the mirrored camera source is placed at a same distance from the mirror plane with an opposed viewing direction to the original camera to thereby provide a second 3D view from an opposite view.
[0058] In some embodiments of the methods, a dual view on a landmark improves spatial orientation without creating a new view either manually or via a multi-step process.
[0059] In some embodiments of the methods, the method automatically creates a second 3D view from the opposite viewing direction.
[0060] In some embodiments of the methods, the desired 3D view is oriented in at least one of a particular rotation and a particular zoom.
[0061] Brief Description of the Drawings
[0062] FIG. 1A illustrates an animated depiction of a procedure in which an ICE catheter transverses the Inferior Vena Cava (IVC) to access the Right Atrium (RA).
[0063] FIG. IB illustrates the view of FIG. 1A using 3D ultrasound data in which the images are generated via intracardiac echocardiography (ICE) using ultrasound waves for imaging.
[0064] FIG. 1C illustrates a pseudo-fluoroscopic view of this data according to one embodiment of the invention.
[0065] FIGS. 2A and 2B are diagrammatic illustrations of an exemplary ultrasound system for providing visualization and characterization of vasculature within a patient with which devices of the invention may be used.
[0066] FIG. 3 is a perspective view of an imaging catheter with which devices of the invention may be coupled.
[0067] FIG. 4 illustrates a system for generating a 4D dynamic model according to one embodiment of the invention.
[0068] FIG. 5 Illustrates a block diagram of a method 500 for generating a simulated fluoroscopic view from 3D ultrasound data according to one embodiment of the invention.
[0069] FIG. 6 illustrates calculation of a dual view according to one embodiment of the systems and methods of the invention.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0070] PCT APPLICATION FIG. 7 illustrates a block diagram of a method of providing a plurality of view angles of an anatomical region of interest according to one embodiment of the invention.
[0071] Detailed Description
[0072] The present invention recognizes the limitations of conventional strategies for real-time catheter guidance and the live visualization of complex anatomy in interventional procedures, namely the information overload, and the lack of a standardized view orientation for practitioners such as electrophysiologists, as well as the limited capacity of interventional surgeons to manipulate imaging software / consoles while performing an intervention. The invention provides systems and methods for generating simulated views, such as pseudo-fluoroscopic views, from ultrasound data, thus providing radiation-free imaging for improved navigation and visualization. The invention further provides systems and methods for efficiently generating multiple view angles in 3D volumetric imaging on a specific target for improved real-time assessment of anatomical conditions and navigation, thus improving spatial orientation without requiring multiple user actions to create the view angles.
[0073] Systems and Methods for Generating Simulated Views
[0074] Overview
[0075] In interventional procedures, electrophysiologists rely on fluoroscopy for real-time catheter guidance due in part because the standardized orientation of C-arms and patient positioning in relation to the practitioner provide for efficient anatomical orientation in relation to the interventional room. Fluoroscopy, i.e. 2D projections generated on demand via C-Arm X-ray systems, is used in almost all interventions as it provides an “anatomical context” as well as clinical familiarity due to its inclusion in electrophysiology training. As such, while it suffers from poor soft tissue contrast, it is still the cornerstone of procedures, particularly in single shot ablations because of a lack of a better alternative. However, fluoroscopy is a radiation-based modality and the necessity of cumbersome personal protective equipment (e.g. lead vests) can lead to chronic neck or back injuries for many clinicians as well as cataract and other ocular diseases for interventional staff in the long term. Thus, the drawbacks of fluoroscopy guidance include poor soft tissue contrast and radiation dosage for operator and patient alike. Despite advancements in ultrasound such as 3D Intracardiac Echocardiography (ICE), informationAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0076] PCT APPLICATION
[0077] overload and the lack of a standardized view orientation can impede effective orientation for catheter guidance during interventional procedures, hindering the transition to zero fluoroscopy procedures.
[0078] As an example, FIG. 1 A illustrates an animated depiction of a procedure in which the ICE catheter transverses the inferior vena cava (IVC) to access the right atrium, depicted as “RA” in FIG. 1. In FIG. 1, the tricuspid valve is labeled as “TV”, the right ventricle is labeled as “RV”, the left ventricle is labeled as “LV”, the mitral valve is labeled as “MV”, the left atrium is labeled as “LA”, the pulmonary artery is labeled as “PA”, and the superior vena cava is labeled as “SVC”.
[0079] FIG. IB illustrates this view using 3D ultrasound data. The images are generated via intracardiac echocardiography (ICE) using ultrasound waves for imaging. ICE, i.e. 2D and 3D catheter-based ultrasound imaging, provides high soft tissue contrast and enables real-time, close-up imaging of cardiac structures. ICE is a catheter-based form of echocardiography that gathers images from within the heart, rather than by gathering images of the heart by sending sound waves through the chest wall. ICE is used as a standard step in many procedures for transseptal crossings, and generally in procedural steps where dynamic anatomical information may be required in real-time. This traditional 3D ultrasound data, while comprehensive, can be overwhelming for electrophysiologists striving for optimal orientation during interventional procedures. The displayed gray-scale ultrasound data may be challenging to interpret as it lacks the anatomical context of surrounding structures and is susceptible to speckle, clutter, and other artefacts inherent to ultrasound as an imaging modality. This is rendered more complex as probes / catheters require manual guidance while imaging the anatomy, requiring operators to have an anatomical context, hence its general use is in tandem with fluoroscopy to provide a wider context. The invention addresses this challenge by generating pseudo-fluoroscopy data from ultrasound data. In particular, systems and methods of the invention utilize ultrasound information, which may be optionally augmented with tracking data, in addition to leveraging the known catheter dynamics, to provide practitioners such as electrophysiologists with an intuitive 2D projection.
[0080] FIG. 1C illustrates a pseudo-fluoroscopic view of this data according to one embodiment of the invention. The novel approach involves translating the intricate 3D ultrasound data into a more digestible format by overlaying the catheter’s position onto a 2D projection of the heart toAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0081] PCT APPLICATION
[0082] create a familiar and easily interpretable visual for electrophysiologists. This pseudo-fluoroscopic view aligns with the accustomed fluoroscopic view with the accustomed visual references of electrophysiologists, thus providing an enhanced comprehension of internal structures and tracked tools in a radiation-free manner. The pseudo-fluoroscopic representation enhances spatial awareness, thus facilitating precise navigation and reducing reliance on conventional fluoroscopy. Accordingly, the systems and methods of the invention provide practitioner, such as electrophysiologists, a radiation-free alternative, diminishing reliance on conventional fluoroscopy and optimizing procedural guidance with streamlined visualization.
[0083] Further, the invention recognizes that interventional surgeons are limited in their capacity to manipulate imaging software / consoles while performing an interventional procedure, while at the same time needing to quickly understand the anatomical environment to safely deliver the therapy. The invention addresses the limitations of current systems by providing multiple view angles in a 3D volumetric image of a specific target to improve both assessment of anatomical conditions and device navigation in the area without requiring multiple user actions to create the views. In particular, the systems and methods of the invention provides a simple two-step procedure for generating a first view of a desired landmark and a second 3D view of the desired landmark from an opposite viewing direction. In this way, the systems and methods of the invention streamline the control and manipulation of imaging during the interventional procedure thus allowing for quick and efficient understanding of the anatomical environment and improved spatial orientation with limited user actions required.
[0084] Systems and methods of the invention use imaging data and various other data combined with image segmentation and registration-based or mapping-based techniques for real-time or near-real-time interactive reconstruction of a patient-specific digital anatomical model. In particular, the invention uses catheter-based ultrasound imaging data combined with, for example, three-dimensional (3D) position / orientation data, and / or pulse phase data, to reconstruct a patient-specific real-time and / or interactive digital anatomical model. The catheterbased ultrasound data may be obtained via endovascular and intravascular ultrasound systems for in-body applications. This includes, for example, systems capable of imaging large cavities of the heart, coronary and peripheral arteries, as well as other organs such as the liver, kidneys, and the like.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0085] PCT APPLICATION
[0086] In particular embodiments, systems and methods of the invention may use a 3D intracardiac echocardiography (ICE) catheter to acquire ultra-fast 4D ultrasound image data of a considered cardiac and / or vascular anatomy. Information on cardiac pulse phase and / or 3D position data of the ICE catheter may be utilized to map the ultrasound image data to a patient coordinate system. The systems also provide for integrating 3D position data of interventional tools for live visualization of the interventional tools within the anatomy.
[0087] As is disclosed in more detail herein, the 3D position data may be obtained through, for example, electromagnetic (EM) tracking, impedance tracking, image-based tracking, fiber optic shape sensing, or a combination of these or other modalities. Cardiac pulse phase data may be obtained via electrocardiography (ECG), pulse oximetry, intracardiac activity data from catheters or image-based methods. As such, the invention provides systems and methods for interactive imaging and flexible and intuitive intra-operation navigation that enables interventional localization and tracking of catheters and tools in relation to the patient anatomy. This allows for real-time and / or interactive model generation of complex anatomy, for example, interactive generation of a patient-specific model of endocardial and / or myocardial anatomy, starting from the catheter access site, e.g. inferior vena cava, and up to targets within the heart (e.g. right atrium).
[0088] Based on the limitations of current systems, and the need for intuitive and efficient navigation for operators, the present invention provides systems and methods for imaging complex anatomy, such as cardiac structures, with real-time, near-real-time, and / or interactive anatomical context. Thus, in particular embodiments, the systems and methods of the invention provide for detailed live / interactive inspection of structures of interest for interventionalists performing complex catheter-based procedures.
[0089] Systems for generating simulated fluoroscopic (pseudo-fluoroscopic) views
[0090] The invention provides systems for generating a simulated fluoroscopic view from three-dimensional (3D) ultrasound data.
[0091] Aspects for the invention provide systems for generating a simulated fluoroscopic view from 3D ultrasound data. The system includes a console configured to be operably associated with an ultrasound imaging device and exchange data therewith. The console includes a hardware processor coupled to non-transitory, computer-readable memory containing instructions executable by the processor. The instructions cause the processor to receive dataAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0092] PCT APPLICATION
[0093] associated with one or more anatomical regions of interest, the data comprising at least one of catheter-based ultrasound imaging data, catheter position data, and catheter orientation data; process and combine the received data; generate, based on the processing and combining of data, simulated two-dimensional (2D) projection image data; and reconstruct, based on the simulated 2D data, a simulated image of the one or more anatomical regions of interest, wherein the simulated image comprises an estimated catheter position within the simulated image.
[0094] In some embodiments, the simulated 2D projection image data comprises simulated fluoroscopy data, wherein a simulated fluoroscopic image of the one or more anatomical regions of interest is generated.
[0095] The catheter-based imaging device may be an ultrasound system with certain hardware and software for providing image reconstruction and imaging assembly control, for example as described in International PCT Application No. PCT / IB2019 / 000963 (Published as WO 2020 / 044117) to Hennersperger et al., U.S. Application Publication No. US 2022-0287679A1 to Hennersperger et al., and U.S. Patent No. 11,382,599 to Hennersperger et al., the contents of each which are incorporated by reference herein in their entirety.
[0096] The systems may include a console configured to be operably associated with one or more devices such as an ultrasound imaging device and to exchange data therewith. The console may comprise a hardware processor coupled to non-transitory, computer-readable memory containing instructions executable by the processor to cause the console to receive data associated with a complex anatomy. For example, the console may be operable to receive a plurality of data, process and combine the data, and reconstruct, based on the processing and combining of the data, an interactive digital model of an imaged anatomy. In some embodiments, the complex anatomy may be cardiac and / or vascular anatomy. The data may be, for example, catheter-based ultrasound imaging data, pulse phase data, and / or catheter position data.
[0097] The console may be in active communication with a computing system configured to communicate across a network. The computing system or computing device may include one or more processors and memory, as well as an input / output mechanism (i.e., a keyboard, knobs, scroll wheels, or the like) with which a user can interact so as to operate the console, including making adjustments to the ultrasound imaging system, saving images, initialization, continuous 3D image registration, filtering, optimization, 3D fusion, and panoramic image reconstruction.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0098] PCT APPLICATION
[0099] The console may generally include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both) and storage, such as main memory, static memory, or a combination of both, which communicate with each other via a bus or the like. The memory according to embodiments of the invention can include a machine-readable medium on which may be stored one or more sets of instructions (e g., software) embodying any one or more of the methodologies or functions described herein. The software may also reside, completely or at least partially, within the main memory and / or within the processor during execution thereof by the computer system, the main memory and the processor also constituting machine-readable media. The software may further be transmitted or received over a network via the network interface device.
[0100] During operation, the CPU and / or GPU may control the transmission and receipt of electrical currents, subsequently controlling the emission and receipt of sound waves from the probe. The CPU and / or GPU may also analyze electrical pulses that the probe makes in response to reflected waves coming back and convert this data into images (i.e., ultrasound images) that can then be viewed on a display, which may be an integrated monitor. Such images may also be stored in memory and / or printed via a printer.
[0101] As disclosed in more detail herein, the console may be operable to receive available ultrasound imaging data, pulse phase data, and / or position data, to process and combine the received data, to generate, based on the processing and combining of the data, simulated 2D image data, and to reconstruct, based on the simulated 2D image data, a simulated view of one or more anatomical regions of interest. The simulated view may encapsulate a representation of the catheter position within the simulated image.
[0102] As disclosed in more detail herein, systems of the invention may reconstruct a simulated view of one or more anatomical regions of interest. In exemplary embodiments, systems of the invention may generate, based on the processing and combining of the data, simulated fluoroscopic image data, and reconstruct, based on the simulated fluoroscopic data, a simulated fluoroscopic image / view of one or more anatomical regions of interest. The simulated fluoroscopic view may encapsulate a representation of the catheter position within the fluoroscopic image.
[0103] The systems may be configured to receive three-dimensional (3D) ultrasound image data from an imaging device. In some embodiments, the systems may receive, process, and combineAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0104] PCT APPLICATION
[0105] real-time catheter-based ultrasound imaging data with other data such as pulse-phase data and / or 3D pose data (position and orientation data) to reconstruct a representation of the digital anatomy.
[0106] In some embodiments, the invention provides for pseudo-fluoroscopic view for use in minimally invasive procedures in the vasculature. Accordingly, ultrafast ultrasound imaging techniques, such as planewave or diverging wave imaging, may be required to enable imaging within the constraints of the application, particularly for intravascular and / or intracardiac tissue assessment and analysis. These constraints could be posed due to the high temporal update rate as required for effects observed in visualization and tissue characterization, where plane and diverging wave methods enable high imaging rates, commonly also referred to ultrafast imaging approaches. Systems and methods of the invention allow for the direct utilization of all native ultrafast imaging techniques.
[0107] For example, for intracardiac imaging, planewave imaging may refer to an ultrasound imaging modality where, through a flat transmit of all transducer elements (at different angles) from the angular imaging aperture, a plane wave front may traverse the tissue and may be partially scattered back to the transducer. From the received radio frequency (RF) (i.e. channel) data the overall image may be reconstructed at once in parallel by dynamically beamforming the received RF data for each target position.
[0108] Ultrafast ultrasound methods offer imaging at thousands of frames per second limited only by the physical propagation speed of sound waves in tissue, and enable ultrasensitive bloodflow tracking, shear-wave imaging, super-resolution imaging, and other applications. For example, achieving optimal spatial resolution while enabling artifact-free imaging of dynamic cardiac structures requires a careful balance between spatial sampling and volumetric update rate which can only be achieved using ultrafast imaging techniques. Thus, the 3D ultrasound image data received by systems of the invention may be real-time 3D ultrasound data. For example, the data may be full circumferential, three-dimensional (3D) image data.
[0109] As disclosed in more detail herein, in some embodiments, the ultrasound imaging data may be intracardiac ultrasound image data. As such, the systems may use available 3D ICE catheters for acquiring ultra-fast 4D ultrasound image data of a considered cardiac and / or vascular anatomy for processing and combining with other data to generate the simulated fluoroscopic image in conjunction with a digital anatomical model.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0110] PCT APPLICATION
[0111] In some embodiments, systems of the invention utilize multiple fields-of-view, e.g. cylindrical fields-of-view acquired with a rotating transducer array or a cylindrical folded transducer matrix. Thus, cylindrical / 360-degree imaging may be applied to provide an extended fteld-of view (FOV). Any fteld-of view implementations may be used to achieve imaging of the whole vascular geometry and its surroundings for each volume such that imaging may be 360 degrees around the catheter. In some embodiments, the ultrasound imaging data may be received from a plurality of cylindrical fields-of-view. For example, in some embodiments, the plurality of cylindrical fields of view are acquired via a rotating transducer array and / or a cylindrical folded transducer matrix.
[0112] Generally, a whole considered anatomical region is not covered by a single 3D ultrasound image. Therefore, in some embodiments, systems of the invention provide for selecting an appropriate trajectory of the ICE catheter such that the whole region may be covered over time by multiple partly overlapping 3D images. Systems of the invention may provide a trajectory starting from the catheter access site up to targets of interest, for example targets within the heart and thus reconstructing a digital anatomy from different acquisitions for use in generating the simulated fluoroscopic image. A clinically important protocol is a trajectory starting from the catheter access site up to the targets within the heart for reconstruction of the intracardiac digital anatomy. In some embodiments, data is acquired during the advancement of the catheter from the access site up into the right atrium. Further data may be acquired during pull back (retraction) from the superior vena cava into the inferior vena cava for reconstruction of a right atrial map and left atrial map. Data may be collected from a pullback from the left-sided pulmonary veins into the septal wall for reconstructing a high resolution left atrial map.
[0113] To increase the field-of-view, the catheter’s motion may be perturbed slightly from a smooth trajectory in a defined way. For example, the catheter tip may be disturbed or “wobbled” using a built-in steering mechanism, which changes the viewing angle. By doing so, the image contrast of interfaces that are not well aligned with the ultrasound wave front in the smooth trajectory is improved. Thus, in some embodiments, the ultrasound imaging data may be received from a field of view whereby a motion of the catheter may be perturbed from a smooth trajectory to change the viewing angle. Further, in some embodiments, the speed of the ICE catheter’s trajectory may be limited by the field-of-view of the ultrasound image data such that subsequent phase-gated, e.g. ECG-gated, 3D images have sufficient overlap.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0114] PCT APPLICATION
[0115] While exemplary embodiments describe ultrasound imaging data received from 3D ICE catheters, catheter-based ultrasound imaging as described herein is not limiting. Imaging data may be received from catheter-based ultrasound systems that may include endovascular and / or intravascular ultrasound for in-body applications, in some embodiments. The ultrasound imaging data may be received from a plurality of cylindrical ftelds-of-view acquired via a rotating transducer array and / or a cylindrical folded transducer matrix. This may include systems and imaging data for imaging in large cavities of the heart, coronary and peripheral arteries, as well as other organs such as liver, kidney, and the like. Systems and methods of the invention may be configured to receive information on ablation electrodes, e.g. spline electrodes on basket catheters, or single electrodes on a point by point catheter.
[0116] FIGS. 2A and FIG. 2B are diagrammatic illustrations of an exemplary ultrasound system 100 for providing catheter-based ultrasound imaging data specific to a patient 12. The system 100 may include an imaging device 101 equipped with an imaging assembly 104 and a console 106 to which the imaging device 101 is to be connected. The imaging device may be an imaging catheter 102. Accordingly, systems for generating pseudo-fluoroscopic views may use a 4D intracardiac echocardiogram (ICE) system that captures the anatomy of interest.
[0117] As disclosed in detail herein, the imaging device may generally be in the form of an imaging catheter capable of providing imaging and mapping capabilities. Accordingly, such a device may be useful for ultrasound visualization of intravascular and / or intracardiac tissue, which may be particularly useful for catheter-based interventional procedures for assessing the anatomy as well as functional data in relation to a target volume of interest.
[0118] As generally understood, the systems and methods of the present invention may be used for ultrasound visualization of tissue of any kind with respect to any kind of procedure in which imaging analysis is used and / or preferred. The imaging device may be useful in carrying out catheter ablation to treat a cardiac condition, such as atrial fibrillation (AF) or the like. For example, in some embodiments, the catheter may include components providing associated capabilities. For example, portions of the catheter may include sensors (e.g., localization and / or tracking sensors) and / or energy delivery elements (e.g., ablation elements).
[0119] FIG. 3 is a perspective view of an imaging catheter 102 with which the device may be coupled. The catheter 102 may include a catheter body 108, including proximal and distal portions. The imaging assembly 104 may be provided at the distal portion, for example,Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0120] PCT APPLICATION
[0121] generally defining a distal end 103 of an imaging catheter. The system may include a collar 110 rotatably coupled to a portion of the catheter-based imaging device. An interface member 112 may be provided at a proximal portion of the catheter body 108. The interface member 112 generally provides a connection between the imaging catheter 102, including the imaging assembly 104 and collar 110, and the console 106 for transmission of signals therebetween.
[0122] In some embodiments, the simulated 2D image may correspond to a standardized orientation of a view of an anatomical region of interest. In some embodiments, the simulated fluoroscopic image may correspond to a standardized orientation of a fluoroscopic view of the anatomical region of interest. In non-limiting examples, the standardized orientation of a view, such as a fluoroscopic view, may be defined in relation to the subject facing an observer and standing upright, i.e. via their attitudinal position. As such, structures closer to the observer may be described as anterior and those relatively farther away as posterior. Components lying closer to the head may be superior (i.e., cranial) and those toward the feet may be said to be inferior (i.e., caudal). Structures on the right-hand side of the subject may be right-sided, and vice-versa for the left. Alternatively, the standard orientation may label structures according to their relative position within the heart, (e g., septal, lateral) as opposed to their attitudinal orientation. The standardized orientation of the fluoroscopic view may be a viewing angle or projection such that the source-to-detector direction is orthogonal to the axis of symmetry of the anatomic feature of interest. In addressing cardiac procedures, the standardized orientation of the fluoroscopic view may be standardized views of different cardiac structures such as right- and left-sided chamber views. The standardized orientation of the simulated view may be a standard catheter in a supraventricular tachycardia, atrial flutter, atrial fibrillation, and ventricular tachycardia. The standardized orientation of the fluoroscopic view may be a standard fluoroscopic catheter position in supraventricular tachycardia, atrial flutter, atrial fibrillation, and ventricular tachycardia.
[0123] In some embodiments, the standardized orientation comprises at least one of an anterior-posterior (AP) view, a posterior-anterior (PA) view, a left anterior oblique (LAO) view, a right anterior oblique (RAO) view, a left lateral (LL) view, and a right lateral (RR) view.
[0124] In some embodiments, the catheter-based ultrasound imaging data may be received from an intracardiac echocardiogram (ICE) two-dimensional (2D) and / or 3D catheter. In someAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0125] PCT APPLICATION
[0126] embodiments, the one or more anatomical regions of interest comprise at least one of cardiac anatomy and vascular anatomy.
[0127] For intracardiac procedures, the reconstructed image, such as the reconstructed fluoroscopic image, may be generated based on received ultrasound image data obtained from an ICE catheter and combined with, for example, electrocardiography (ECG) data and 3D position data, such as EM tracking data. This provides real-time 3D localization and tracking of the ICE catheter and interventional tools within the generated fluoroscopic or other simulated view, enabling navigation of the ICE catheter, and tool tracking and navigation including contact assessment.
[0128] In some embodiments, the received data may comprise at least one of intracardiac echocardiography (ICE) data, data associated with one or more anatomical landmarks, data associated with an accessory pathway, cardiac pulse phase data, and a dynamic of a structure of the catheter.
[0129] For example, the console may be operable to automatically detect and localize anatomical landmarks for visualization within the image thus providing advanced visualizations of the structure (and morphology) of cardiac chambers and a patient’s cardiac anatomy. The data associated with one or more anatomical landmarks may include data associated with inspection of the of trabeculae as well as other intracardiac structures such as tendons, chordae, and valves / leaflets. The anatomical landmarks may be, in non-limiting examples, one or more of the aortic valve in short axis and the interatrial septum, the mitral valve, left ventricle (LV), right ventricle (RV), and the left atrial appendage (LAA).
[0130] The data may be associated with an accessory pathway, i.e. an abnormal electrical connection in the heart that allows electrical signals to travel from the atria to the ventricles more quickly than normal, thus bypassing the atrioventricular (AV) node, where electrical signals normally travel more slowly. The data may be physiological data and / or electrophysiological characteristics of an accessory pathway.
[0131] Cardiac pulse phase data may include information about the different stages of the cardiac cycle capturing the timing and characteristics of the heart's contractions and relaxations within a single heartbeat. The cardiac pulse phase data may be represented as a waveform with distinct phases that may be analyzed to extract one or more cardiovascular parameters. In particular embodiments, separate 3D models for each cardiac phase may be combined along theAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0132] PCT APPLICATION
[0133] trajectory of the catheter to generate a temporal dynamic model that represents the patientspecific anatomy over time, wherein the dynamic model visualizes the heartbeat as a sequence of animated 3D models. An extended dynamic model may be created by considering, in part, different phases of the heart cycle, thus generating a combination of different models for each ECG-gated phase.
[0134] The data may include data associated with a dynamic of a structure of the catheter. For example, the data may be associated with rotational movement of the transducer. Data related to movement of the catheter and / or transducer may be combined with data associated with dynamic movement of imaged body regions over time. The structure of the catheter may be, in nonlimiting examples, a rotational ICE catheter, a phased array ICE catheter, and / or a mechanical ultrasound tipped catheter. Thus, in some embodiments, the systems of the invention provide dynamic anatomical information in real time via the pseudo-fluoroscopic image to provide a wider anatomical context.
[0135] In some embodiments, the system may calculate a catheter position and a catheter orientation within the one or more anatomical regions of interest in relation to at least one of a cardiac anatomy, vascular anatomy, and an anatomical landmark. The catheter position and / or catheter orientation data may be calculated from three-dimensional (3D) position data. For example, in some embodiments, the position data may be calculated from a spatial position and an orientation of a catheter and / or an interventional tool. The position data may be calculated from 3D pose data comprising six degrees of freedom. The 3D pose data may be obtained through electromagnetic (EM) tracking, impedance tracking, image-based tracking, fiber-optic shape sensing, and / or a combination of these modalities.
[0136] Thus, systems of the invention may provide for real-time 3D localization and tracking of the catheter, for example an ICE catheter, and interventional tools within the patient coordinate system. Systems of the invention may receive 3D position and orientation data of the catheter, and process and combine this data with the ultrasound imaging data and / or the pulse phase data for use in generating the simulated (e.g. pseudo-fluoroscopic) image. It is noted however, that in some embodiments, the simulated image is generated without measured 3D position data.
[0137] Tracking data may be used by the console to map the ultrasound image data to a patient coordinate system and / or to the simulated image, such as the pseudo-fluoroscopic image. In some embodiments EM tracking data is used in generating the pseudo-fluoroscopic image.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0138] PCT APPLICATION
[0139] Electromagnetic tracking generates a defined EM field in which EM micro sensors are tracked.
[0140] 6 degree-of-freedom tracking information (spatial position and orientation) may be acquired, for example, by embedding micro sensors into rigid or flexible instruments, where they serve as localization points for the instrument in space. The micro sensors can be embedded, for example, in a coil in the catheter tip. This allows for tracking the catheter tip position inside an electromagnetic field. The EM field generator emits a low intensity, varying EM field that establishes a measurement volume. Small currents are induced inside the sensors when they enter the EM field. The currents are relayed to the sensor interface unit where they are amplified and digitized as signals. The signals are transmitted to the console which calculates each sensor’s position and orientation as a transformation. In some embodiments EM tracking of interventional tools may be also incorporated. Thus, in some embodiments, the 3D position data comprises a spatial position and an orientation of a catheter and / or an interventional tool.
[0141] In some embodiments, the 3D pose data may be obtained through optical fiber shape sensing. For example, a low reflectance strain sensor may be positioned in a multi-core fiber within the catheter to determine how a point along the fiber is positioned in space. By sensing the relative change of the sensors in each of three or more fiber cores, the three-dimensional position can be determined.
[0142] In some embodiments, the simulated image may include a projection of 3D ultrasound data of the one or more anatomical regions of interest onto a 2D projection plane with the catheter overlay ed onto the 2D projection such that the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
[0143] In some embodiments, the fluoroscopic image may include at least one of a sectional reconstruction and a multiplanar reconstruction of the anatomical region of interest onto one or more 2D projection planes visualized in conjunction with the catheter position. In some embodiments the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest. Thus, by utilizing the ultrasound data, augmented with tracking data in some cases, the systems provide users with an intuitive and familiar 2D projection aligning with accustomed visual references. The simulated view in conjunction with the 2D projection of the catheter position enhances comprehension of the internal structures and tracked tools.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0144] PCT APPLICATION
[0145] Referring back to FIG. 1C, multiple factors enable the presentation of intuitive projected image datasets from which the illustrated image is generated. As discussed herein, based on the known accessory pathways (through the Inferior Vena cava), as well as the catheter dynamics (tubular, flexible structure) in combination with imaging information provided by a 3D-Ultrasound catheter, the correct anatomical position and orientation in relation to the cardiac anatomy (and specific landmarks) can be retrieved by matching image information with anatomical landmarks. Optionally, through 3D-Tracking of the catheter, its position and orientation with respect to the patient’s chest and heart can be visualized, whereas 3D-Tracking can be either Electromagnetic-Based, Image-Based, or a combination thereof. Optionally, from acquired 3D ultrasound data, fluoroscopic pseudo-images can be reconstructed through simulation, where simulation can be done through various approaches (including raytracingbased, physics-based, machine-learning based) estimating a certain simulated image, such as a fluoroscopy image, from a known anatomy imaged by 3D-ultrasound. Alternatively, a common fluoroscopic view (“Patient atlas”) can be considered and optionally tailored to the specific anatomy or view orientation. Using, for example, these data points, a standard projection view of 3D data onto a 2D projection plane, or sectional or multiplanar reconstructions on certain 2D projection planes may be visualized in conjunction with the underlying catheter position.
[0146] Thereby, the use of standardized orientations (e.g. Anterior-Posterior view - AP-View, or Right Anterior Oblique - RAO) allows EPs to work in familiar orientations / views familiar and thus perform interventional navigation and guidance of catheters more intuitively.
[0147] In some embodiments, the simulation comprises at least one of a ray tracing-based simulation, a physics-based simulation, and a machine learning-based simulation. The ray tracing simulation may be used to simulate / render, for example, reflections, refractions, and shadows typical of a fluoroscopic view. The physics-based simulation may include mathematical models of the physical phenomena associated with the anatomical region of interest, the catheter, the interventional tool, and the like by applying the laws of physics. The machine-learning based simulation may include, for example, one or more of supervised, unsupervised, and reinforcement learning. The machine learning simulation may be combined with traditional modeling techniques to leverage the strengths of multiple simulation modalities.
[0148] In some embodiments, the simulated fluoroscopy image may include an interactive digital model of the one or more anatomical regions of interest, wherein the interactive digitalAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0149] PCT APPLICATION
[0150] model comprises a common fluoroscopic view generated beginning from a catheter access site to one or more targets within a heart to provide an interactive visualization of patient-specific structure and / or morphology of intra-cardiac structures.
[0151] For example, the system may generate a 4D dynamic (spatio-temporal) digital model. FIG. 4 illustrates a system for generating a 4D dynamic model according to one embodiment of the invention. As disclosed herein, in some embodiments, the systems 400 may include a console 401 in active communication with a computing system 403 configured to communicate across a network. The console 401 may be configured to be operably associated with an ultrasound imaging device 409, as well as with other devices and / or means for receiving data 411, and to exchange data therewith. The console may comprise a hardware processor 403 coupled to non-transitory, computer-readable memory 405 containing instructions executable by the processor to cause the console to receive data associated with at least one of cardiac and vascular anatomy the data comprising catheter-based ultrasound imaging data, and, in some embodiments, pulse phase data and / or 3D position data 411. The console may be operable to process and combine the received data, and reconstruct, based on said processing and combining of data, an interactive digital model of an imaged anatomy, wherein the digital model encapsulates a representation of the anatomy in a spatial topology for live visualization of one or more anatomical regions of interest.
[0152] The computing system 403 or computing device may include one or more processors and memory, as well as an input / output mechanism (i.e., a keyboard, knobs, scroll wheels, or the like) with which a user can interact so as to operate the console, including making adjustments to the ultrasound imaging system 409, saving images, initialization, continuous 3D image registration, filtering, optimization, 3D fusion, and panoramic image reconstruction.
[0153] The computing system 403 may include a computer program comprising one or more algorithms 407 for image registration, filtering, segmentation, optimization, and reconstruction of the digital anatomical model. For example, the algorithm 407 may be part of a computer program executable by the computing system 403 and in communication with the console 401 of the systems 400. The systems may be in communication with the imaging device 409 to receive 3D ultrasound image data from the imaging device 409.
[0154] As disclosed in detail herein, imaging protocols and algorithms may be used to dynamically reconstruct properties of the anatomy. The systems may include one or moreAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0155] PCT APPLICATION
[0156] algorithms for dynamically reconstructing multiple images from the 3D image data to provide a 3D visualization of the anatomical region of interest and targeted tissue site.
[0157] The digital model may be achieved by registration and fusion of ultrasound volumes as described herein. By applying the steps for generating a patient-specific 3D digital anatomical model for different cardiac phases, a separate 3D model may be generated for each cardiac phase. These separate models may be combined into a temporal dynamic model that represents the patient-specific anatomy over time. In some embodiments the temporal dynamic model visualizes the heartbeat as a sequence of animated 3D models. The 3D model may be expanded piece by piece along the trajectory of the ICE catheter in the vessels of the heart. The 4D view visualizes the heartbeat as a sequence of animated models.
[0158] To generate the interactive patient-specific 3D digital anatomical model, the received data may be processed and combined. Systems of the invention may combine multiple image data into a large anatomical representation. The digital anatomical model may be generated in real-time for anatomy that is within the field-of-view.
[0159] For example, to generate a patient-specific 3D digital anatomical model based on the 4D image data acquired along the ICE catheter’s trajectory over time, as well as to navigate the ICE catheter, the systems of the method utilize medical image registration, computer vision (CV)-based approaches, and / or simultaneous localization and mapping (SLAM). As disclosed in more detail herein, the methods of combination and reconstruction may be, for example, image-based registration, (SLAM), and other computer vision (CV)-based approaches.
[0160] Image registration is the process of aligning multiple data, i.e. images, volumes, or surfaces to the patient coordinate system. Systems of the invention utilize one or more registration algorithms to, for example, combine images of the patient and / or data from different modalities and to align temporal sequences of images to generate the interactive and / or real-time patient-specific digital anatomical model. The one or more algorithms find an optimal spatial transformation that best aligns the underlying anatomical structures for reconstruction of the digital anatomical model. Thus, the systems of the invention may use image registration techniques to combine multiple image data into an anatomical representation that combines the multiple image data into a large anatomical representation.
[0161] Simultaneous localization and mapping (SLAM) is a computational method that constructs or updates a map of an unknown environment while simultaneously keeping track ofAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0162] PCT APPLICATION
[0163] an agent’s location within it. In some embodiments, systems of the invention adapt and utilize one or more algorithms for SLAM applications to ultrasound image data acquired to generate the patient-specific anatomical model as well as to navigate the catheter. For example, SLAM processing techniques may be adapted for use in conjunction with the catheter utilizing various sensors to incrementally build the map of a patient anatomical environment and simultaneously determine the location of the catheter within the map. Thus, the systems of the invention may use adapted SLAM techniques to combine multiple image data into an anatomical representation that combines the multiple image data into a large anatomical representation.
[0164] CV-based image reconstruction approaches utilized by systems of the invention may include approaches based on deep neural networks (DNNs) such as autoencoders (AEs), convolutional neural networks (CNNs), and generative adversarial networks (GANs). The computer vision approaches utilized by systems of the invention aim to detect, interpret and reconstruct data in a way that mimics the intricacy of the human visual system thus providing for intuitive navigation to precisely target an anatomical region.
[0165] Further, in some embodiments, the system may adapt and utilize one or more image segmentation algorithms for generating the patient-specific digital anatomical model. Image segmentation is the process of dividing an image into multiple meaningful and homogeneous regions or objects based on their inherent characteristics, such as color, texture, shape, or brightness. In some embodiments, each pixel is labeled, and all pixels belonging to the same category have a common label assigned to them. Segmentation may be achieved via one or more algorithms. For example, in some embodiments, segmentation is achieved via instance image segmentation in which each object in an image is detected and segmented, via one or more algorithms to separate overlapping objects. In some embodiments, segmentation may be achieved via semantic segmentation in which one or more algorithms are used to label each pixel. In some embodiments, segmentation may be achieved via panoptic segmentation in which one or more machine learning algorithms are used to label each pixel with a class label and identifies each object instance in the image to provide for detection and interaction of the object within the environment. Thus, the systems of the invention may use CV-based techniques to combine multiple image data into an anatomical representation that combines the multiple image data into a large anatomical representation.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0166] PCT APPLICATION
[0167] In some embodiments, the segmentation may be performed by a vision algorithm such as thresholding, connected component analysis, or a neural network based segmentation. In some embodiments, segmentation of the full panoramic volumes may be performed, for example, via a deep learning algorithm.
[0168] In some embodiments, receiving, processing, and combining the received data to reconstruct the interactive digital model of an imaged anatomy includes initialization, continuous 3D image registration, segmentation, fdtering, optimization, 3D fusion, and panoramic image reconstruction.
[0169] In some embodiments, combining the received data comprises selecting a 3D ultrasound image in the catheter-based ultrasound imaging data and mapping the 3D ultrasound image to the physical patient coordinate system using, for example, EM tracking data. In this way, a registration of each 3D ultrasound image in the physical patient’s coordinate system may be initialized. The ultrasound image selected for initialization may be the first or any subsequent ultrasound image. 3D pose data may be used for initialization of the registration. In some embodiments, EM tracking data may be used for initialization of the registration.
[0170] As disclosed herein, systems of the invention may use multimodal image registration using one or more algorithms to correlate morphologic and / or functional features between images. The registration volume may be the entire imaged volume or only a subset of the available image set. A suitable spatial overlap may be necessary to enable registration. Utilizing a rigid registration scheme is beneficial, while a deformable registration scheme can cope with deformations caused by inaccurate ECG filtering and breathing effects. A deformable image registration takes into account non-linear deformation and displacement, and may include geometrically deforming one image into another by applying one or more algorithms, for example deep learning-based models. For example, the deformable registration scheme may be a computational process in which an image similarity measure function and a transformation model are defined for the images of interest, then an optimization algorithm may be used to adjust the transformation model in a way that maximizes the similarity function. The transformation models may include spline and demons, elastic, fluid, finite element model, and free form deformations. Rigid registration is a global match between image sets that preserves the relative distance between every pair of points from the patient’s anatomy. Rigid-bodyAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0171] PCT APPLICATION
[0172] registration may include a combination of rotation and translation in order to bring the images into the same coordinate system.
[0173] 3D ultrasound images, subsequent to the initialization, along the ICE catheter’s trajectory may be filtered using pulse phase-gating. For example, the pulse-phase gating may be ECG-gating in some embodiments. 3D ultrasound images of the same cardiac phase may be registered against the respective previous 3D ultrasound image or the current fused 3D image.
[0174] Subsequent optimization steps increase the accuracy of the localization of each image in the patient’s coordinate system. The optimization steps also ensure robustness of the system. For example, wherein 3D pose data comprises EM tracking data, optimization ensures robustness of the systems against missing EM tracking data, for instance where the ICE catheter leaves the EM tracking field. To provide for improved robustness of registration in the case of inaccurate EM data used for initialization, or in the case of limited image features (e.g. when the ICE catheter is in the inferior vena cava (IVC) next to the lung), an additional tracking scheme may be used. The tracking scheme may use the EM tracking data and the previous registration results to predict the next 3D position. The predicted next 3D position may then be used for initialization of the continuous registration. To further provide for improved registration in regions with few features, prior knowledge of the imaged anatomy may be applied.
[0175] In some embodiments, combining the received data further comprises continuously registering subsequent 3D ultrasound images within the physical patient’s coordinate system. In some embodiments, the 3D pose data may be combined with one or more previous registration results to predict a next 3D position. For example, where EM tracking data is used, the EM tracking data may be combined with one or more previous registration results to predict a next 3D position.
[0176] In some embodiments, the predicted next 3D position may be used for initialization of the continuous registration. As disclosed in more detail herein, in some embodiments, the continuously registered subsequent 3D ultrasound images are partly overlapping such that the continuously registered and partly overlapping subsequent 3D ultrasound images are fused into one panoramic digital model.
[0177] The continuous 3D ultrasound image registration may include fdtering, using pulse phase-gating, the subsequent 3D ultrasound images along the catheter’s trajectory, such that a separate digital model may be generated for each cardiac phase. Thus, the 3D ultrasound imagesAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0178] PCT APPLICATION
[0179] of a same cardiac phase may be registered against a respective previous 3D ultrasound image and / or a fused 3D image. Further, in some embodiments, the filtered subsequent 3D ultrasound images are segmented using a supervised or unsupervised deep-learning algorithm.
[0180] The continuously registered subsequent 3D ultrasound images may be partly overlapping such that the continuously registered and partly overlapping subsequent 3D ultrasound images are fused into a large 3D image representing a patient-specific panoramic reconstruction of the anatomy. This 3D panoramic reconstruction may be updated sequentially over time as the catheter is moved through the anatomical region or regions of interest, such that the patientspecific digital 3D anatomical model is continuously updated with subsequent registration and segmentation results.
[0181] In some embodiments, the digital anatomical model encapsulates a representation of the anatomy in a spatial topology for live visualization of one or more anatomical regions of interest. The spatial topology comprises a topological map, such that each point in the topological map represents tissue and / or one or more specific anatomical properties. Thus, ultrasound imaging data from one or more directions and / or one or more views may be combined to provide the representation of the anatomy. This representation may be a panoramic image reconstruction (i.e. an intensity volume), a (segmented) surface model, and / or a mesh. The digital anatomy may also be a more advanced representation of the anatomy, where for each point in the topological map, a representation of tissue or specific anatomical properties are encapsulated, such that different views or information from different directions can be combined for a complete representation. Thus, by reconstructing an interactive digital anatomical model, based on the processing and combining of the received data, the systems of the invention provide a digital anatomical model similar to computed tomography (CT), magnetic resonance imaging (MRI), or ultrasound tomographic reconstructions. The digital model encapsulates a representation of the anatomy in a spatial topology for live visualization of one or more anatomical regions of interest.
[0182] The digital model may be at least one of a panoramic image reconstruction, a segmented surface model, and a mesh. In some embodiments, the panoramic digital model may be segmented using a deep learning algorithm.
[0183] As disclosed herein, the continuous 3D ultrasound image registration may include filtering, using pulse phase-gating, the subsequent 3D ultrasound images along the catheter’s trajectory, such that a separate digital model may be generated for each cardiac phase. Thus, theAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0184] PCT APPLICATION
[0185] 3D ultrasound images of a same cardiac phase may be registered against a respective previous 3D ultrasound image and / or a fused 3D image. Further, in some embodiments, the filtered subsequent 3D ultrasound images are segmented using a supervised or unsupervised deeplearning algorithm.
[0186] Systems of the invention provide for 3D segmentation of the ultrasound imaging data. In some embodiments, relevant structures of the cardiac and vascular anatomy are segmented in ECG-gated 3D ultrasound images. For example, a main focus may be the segmentation of the inner shell (surface) of the right atrium (RA), left atrium (LA), and the traversed veins. The inner shell may be defined as the tissue-blood interface of the blood pool and the cardiac or vascular wall. Because segmentation of ICE data is challenging, systems of the invention include using supervised or unsupervised machine-learning approaches in some embodiments. In some embodiments, the filtered subsequent 3D ultrasound images are segmented using a supervised or unsupervised deep-learning algorithm.
[0187] The systems of the invention provide for adaptive 3D model generation. By fusing the segmented 3D shells using the 3D registration results, a patient-specific digital 3D anatomical model may be obtained. The 3D model may be continuously updated with the newest registration and segmentation results.
[0188] In some embodiments, the continuously updated patient-specific digital 3D anatomical model may be visualized simultaneously with the fused ultrasound image data to generate the simulated fluoroscopic view and to enable 3D navigation of the catheter in real-time. The systems of the invention provide for generating 3D anatomical landmarks. To aid the navigation of the ICE catheter and interventional tools, a detection and localization step may be added to automatically find prominent anatomical landmarks. These landmarks may be visualized in 3D along with the model.
[0189] Systems of the invention provide for tracking of interventional tools within the 3D model. The console may be further configured to detect, localize, and segment in the 3D ultrasound images, relevant interventional tools. For initialization of the detection step, 3D pose data, such as EM tracking data, may be used as available. The segmented tools may be visualized along with the 3D digital anatomical model. In addition to physical tracking and navigation, the invention provides for contact assessment between the tool tip and cardiac or vascular wall.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0190] PCT APPLICATION
[0191] In some embodiments, the model may be generated once when all of the 3D ultrasound image data is available. After model generation, navigation of the ICE catheter as well as tracking and navigation of interventional tools is possible. Thus, the anatomical context generated with the model enables safer navigation of catheter tools, for example during septal crossing procedures. In some embodiments, the console may be further operable to automatically detect and localize anatomical landmarks for visualization within the digital model to aid navigation of the catheter and interventional tools.
[0192] As disclosed herein, in some embodiments, the interactive digital model may be generated beginning from a catheter access site to one or more targets within a heart to provide an interactive visualization of patient-specific structure and / or morphology of intra-cardiac structures in a simulated fluoroscopic view. For example, the intra-cardiac structures may be one or more of trabeculae, tendons, chordae, and valves / leaflets.
[0193] Methods for generating simulated fluoroscopic (pseudo-fluoroscopic) views
[0194] Aspects of the invention provide methods for generating a simulated fluoroscopic view from three-dimensional (3D) ultrasound data.
[0195] FIG. 5 illustrates a block diagram of a method 500 for generating a simulated fluoroscopic view from 3D ultrasound data according to one embodiment of the invention. The method includes the steps of providing 502 a console configured to be operably associated with an ultrasound imaging device and to exchange data therewith; 504 receiving, via the console, data associated with one or more anatomical regions of interest, the data comprising at least one of catheter-based ultrasound imaging data, catheter position data, and catheter orientation data; processing and combining 506, via the console, the received data; generating 508, based on said processing and combining of data, simulated 2-dimensional (2D) projection image data; and reconstructing 510, based on the 2D projection image data, a simulated image of the one or more anatomical regions of interest, wherein the simulated image comprises an estimated catheter position within the image.
[0196] In some embodiments, the simulated 2D projection image data comprises simulated fluoroscopy data, wherein a simulated fluoroscopic image of the one or more anatomical regions of interest is generated.
[0197] As disclosed herein, the catheter-based imaging device may be an ultrasound system with certain hardware and software for providing image reconstruction and imaging assembly control.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0198] PCT APPLICATION
[0199] The console may be in active communication with a computing system configured to communicate across a network. The computing system or computing device may include one or more processors and memory, as well as an input / output mechanism (i.e., a keyboard, knobs, scroll wheels, or the like) with which a user can interact so as to operate the console, including making adjustments to the ultrasound imaging system, saving images, initialization, continuous 3D image registration, filtering, optimization, 3D fusion, and panoramic image reconstruction.
[0200] The console may generally include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both) and storage, such as main memory, static memory, or a combination of both, which communicate with each other via a bus or the like. The memory according to embodiments of the invention can include a machine-readable medium on which may be stored one or more sets of instructions (e.g., software) embodying any one or more of the methodologies or functions described herein. The software may also reside, completely or at least partially, within the main memory and / or within the processor during execution thereof by the computer system, the main memory and the processor also constituting machine-readable media. The software may further be transmitted or received over a network via the network interface device.
[0201] During operation, the CPU and / or GPU may control the transmission and receipt of electrical currents, subsequently controlling the emission and receipt of sound waves from the probe. The CPU and / or GPU may also analyze electrical pulses that the probe makes in response to reflected waves coming back and convert this data into images (i.e., ultrasound images) that can then be viewed on a display, which may be an integrated monitor. Such images may also be stored in memory and / or printed via a printer. As disclosed herein, the console may be operable to receive available ultrasound imaging data, pulse phase data, and / or position data, to process and combine the received data, to generate, based on the processing and combining of the data, simulated fluoroscopic data, and to reconstruct, based on the simulated fluoroscopic data, a simulated fluoroscopic image of one or more anatomical regions of interest. The simulated fluoroscopic view may encapsulate a representation of the catheter position within the fluoroscopic image.
[0202] In some embodiments, methods of the invention utilize multiple fields-of-view, e.g. cylindrical fields-of-view acquired with a rotating transducer array or a cylindrical folded transducer matrix. Thus, cylindrical / 360-degree imaging may be applied to provide an extendedAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0203] PCT APPLICATION
[0204] fteld-of view (FOV). Any fteld-of view implementations may be used to achieve imaging of the whole vascular geometry and its surroundings for each volume such that imaging may be 360 degrees around the catheter. In some embodiments, the ultrasound imaging data may be received from a plurality of cylindrical ftelds-of-view. For example, in some embodiments, the plurality of cylindrical fields of view are acquired via a rotating transducer array and / or a cylindrical folded transducer matrix.
[0205] As disclosed herein, generally, a whole considered anatomical region is not covered by a single 3D ultrasound image. Therefore, in some embodiments, methods of the invention provide for selecting an appropriate trajectory of the ICE catheter such that the whole region may be covered over time by multiple partly overlapping 3D images. Methods of the invention may provide a trajectory starting from the catheter access site up to targets of interest, for example targets within the heart and thus reconstructing a digital anatomy from different acquisitions for use in generating or with the generated simulated image. A clinically important protocol is a trajectory starting from the catheter access site up to the targets within the heart for reconstruction of the intracardiac digital anatomy. In some embodiments, data is acquired during the advancement of the catheter from the access site up into the right atrium. Further data may be acquired during pull back (retraction) from the superior vena cava into the inferior vena cava for reconstruction of a right atrial map and left atrial map. Data may be collected from a pullback from the left-sided pulmonary veins into the septal wall for reconstructing, for example a high resolution left atrial map, and / or the simulated fluoroscopic image, such as the simulated fluoroscopic image.
[0206] To increase the field-of-view, the catheter’s motion may be perturbed slightly from a smooth trajectory in a defined way. For example, the catheter tip may be disturbed or “wobbled” using a built-in steering mechanism, which changes the viewing angle. By doing so, the image contrast of interfaces that are not well aligned with the ultrasound wave front in the smooth trajectory may be improved. Thus, in some embodiments, the ultrasound imaging data may be received from a field of view whereby a motion of the catheter may be perturbed from a smooth trajectory to change the viewing angle. Further, in some embodiments, the speed of the ICE catheter’s trajectory may be limited by the field-of-view of the ultrasound image data such that subsequent phase-gated, e.g. ECG-gated, 3D images have sufficient overlap.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0207] PCT APPLICATION
[0208] While exemplary embodiments describe ultrasound imaging data received from 3D ICE catheters, catheter-based ultrasound imaging as described herein is not limiting. Imaging data may be received from catheter-based ultrasound systems that may include endovascular and / or intravascular ultrasound for in-body applications, in some embodiments. The ultrasound imaging data may be received from a plurality of cylindrical ftelds-of-view acquired via a rotating transducer array and / or a cylindrical folded transducer matrix. This may include systems and imaging data for imaging in large cavities of the heart, coronary and peripheral arteries, as well as other organs such as liver, kidney, and the like. Systems and methods of the invention may be configured to receive information on ablation electrodes, e.g. spline electrodes on basket catheters, or single electrodes on a point by point catheter.
[0209] In some embodiments of the method, the simulated image, such as a simulated fluoroscopic image, may correspond to a standardized orientation of a view of the anatomical region of interest. In non-limiting examples, the standardized orientation of a view may be defined in relation to the subject facing an observer and standing upright, i.e. via their attitudinal position. As such, structures closer to the observer may be described as anterior and those relatively farther away as posterior. Components lying closer to the head may be superior (i.e., cranial) and those toward the feet may be said to be inferior (i.e., caudal). Structures on the right-hand side of the subject may be right-sided, and vice-versa for the left. Alternatively, the standard orientation may label structures according to their relative position within the heart, (e g., septal, lateral) as opposed to their attitudinal orientation. The standardized orientation of the fluoroscopic view may be a viewing angle or projection such that the source-to-detector direction is orthogonal to the axis of symmetry of the anatomic feature of interest. In addressing cardiac procedures, the standardized orientation of the fluoroscopic view may be standardized views of different cardiac structures such as right- and left-sided chamber views. The standardized orientation of the fluoroscopic view may be a standard fluoroscopic catheter position in supraventricular tachycardia, atrial flutter, atrial fibrillation, and ventricular tachycardia. In some embodiments, the standardized orientation comprises at least one of an anterior-posterior (AP) view, a posterior-anterior (PA) view, a left anterior oblique (LAO) view, a right anterior oblique (RAO) view, a left lateral (LL) view, and a right lateral (RR) view.
[0210] In some embodiments of the method, the catheter-based ultrasound imaging data may be received from an intracardiac echocardiogram (ICE) two-dimensional (2D) and / or 3D catheter.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0211] PCT APPLICATION
[0212] In some embodiments, the one or more anatomical regions of interest comprise at least one of cardiac anatomy and vascular anatomy.
[0213] For intracardiac procedures, the reconstructed image, such as the reconstructed fluoroscopic image, may be generated based on received ultrasound image data obtained from an ICE catheter and combined with, for example, electrocardiography (ECG) data and 3D position data, such as EM tracking data. This provides real-time 3D localization and tracking of the ICE catheter and interventional tools within the generated fluoroscopic view, enabling navigation of the ICE catheter, and tool tracking and navigation including contact assessment.
[0214] In some embodiments of the method, the received data may comprise at least one of intracardiac echocardiography (ICE) data, data associated with one or more anatomical landmarks, data associated with an accessory pathway, cardiac pulse phase data, and a dynamic of a structure of the catheter.
[0215] For example, the console may be operable to automatically detect and localize anatomical landmarks for visualization within the image thus providing advanced visualizations of the structure (and morphology) of cardiac chambers and a patient’s cardiac anatomy. The data associated with one or more anatomical landmarks may include data associated with inspection of the of trabeculae as well as other intracardiac structures such as tendons, chordae, and valves / leaflets. The anatomical landmarks may be, in non-limiting examples, one or more of the aortic valve in short axis and the interatrial septum, the mitral valve, left ventricle (LV), right ventricle (RV), and the left atrial appendage (LAA).
[0216] The data may be associated with an accessory pathway, i.e. an abnormal electrical connection in the heart that allows electrical signals to travel from the atria to the ventricles more quickly than normal, thus bypassing the atrioventricular (AV) node, where electrical signals normally travel more slowly. The data may be physiological data and / or electrophysiological characteristics of an accessory pathway.
[0217] Cardiac pulse phase data may include information about the different stages of the cardiac cycle capturing the timing and characteristics of the heart's contractions and relaxations within a single heartbeat. The cardiac pulse phase data may be represented as a waveform with distinct phases that may be analyzed to extract one or more cardiovascular parameters. In particular embodiments, separate 3D models for each cardiac phase may be combined along the trajectory of the catheter to generate a temporal dynamic model that represents the patient-Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0218] PCT APPLICATION
[0219] specific anatomy over time, wherein the dynamic model visualizes the heartbeat as a sequence of animated 3D models. An extended dynamic model may be created by considering, in part, different phases of the heart cycle, thus generating a combination of different models for each ECG-gated phase.
[0220] The data may include data associated with a dynamic of a structure of the catheter. For example, the data may be associated with rotational movement of the transducer. Data related to movement of the catheter and / or transducer may be combined with data associated with dynamic movement of imaged body regions over time. The structure of the catheter may be, in nonlimiting examples, a rotational ICE catheter, a phased array ICE catheter, and / or a mechanical ultrasound tipped catheter. Thus, in some embodiments, the methods of the invention provide dynamic anatomical information in real time via the pseudo-fluoroscopic image to provide a wider anatomical context.
[0221] In some embodiments, the methods may calculate a catheter position and a catheter orientation within the one or more anatomical regions of interest in relation to at least one of a cardiac anatomy, vascular anatomy, and an anatomical landmark. The catheter position and / or catheter orientation data may be calculated from three-dimensional (3D) position data. For example, in some embodiments, the position data may be calculated from a spatial position and an orientation of a catheter and / or an interventional tool. The position data may be calculated from 3D pose data comprising six degrees of freedom. The 3D pose data may be obtained through electromagnetic (EM) tracking, impedance tracking, image-based tracking, fiber-optic shape sensing, and / or a combination of these modalities.
[0222] Thus, methods of the invention may provide for real-time 3D localization and tracking of the catheter, for example an ICE catheter, and interventional tools within the patient coordinate system. Methods of the invention may receive 3D position and orientation data of the catheter, and process and combine this data with the ultrasound imaging data and / or the pulse phase data for use in generating the simulated fluoroscopic image. It is noted however, that in some embodiments, the simulated fluoroscopic image, also referred to herein as the pseudo-fluoroscopic image, is generated without measured 3D position data.
[0223] Tracking data may be used by the console to map the ultrasound image data to a patient coordinate system and / or to the simulated fluoroscopic image. In some embodiments EM tracking data is used in generating the pseudo-fluoroscopic image. Electromagnetic trackingAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0224] PCT APPLICATION
[0225] generates a defined EM field in which EM micro sensors are tracked. 6 degree-of-freedom tracking information (spatial position and orientation) may be acquired, for example, by embedding micro sensors into rigid or flexible instruments, where they serve as localization points for the instrument in space. The micro sensors can be embedded, for example, in a coil in the catheter tip. This allows for tracking the catheter tip position inside an electromagnetic field. The EM field generator emits a low intensity, varying EM field that establishes a measurement volume. Small currents are induced inside the sensors when they enter the EM field. The currents are relayed to the sensor interface unit where they are amplified and digitized as signals. The signals are transmitted to the console which calculates each sensor’s position and orientation as a transformation. In some embodiments EM tracking of interventional tools may be also incorporated. Thus, in some embodiments, the 3D position data comprises a spatial position and an orientation of a catheter and / or an interventional tool.
[0226] In some embodiments of the method, the 3D pose data may be obtained through optical fiber shape sensing. For example, a low reflectance strain sensor may be positioned in a multicore fiber within the catheter to determine how a point along the fiber is positioned in space. By sensing the relative change of the sensors in each of three or more fiber cores, the three-dimensional position can be determined.
[0227] In some embodiments of the method, the simulated fluoroscopic image may include a projection of 3D ultrasound data of the one or more anatomical regions of interest onto a 2D projection plane with the catheter overlayed onto the 2D projection such that the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
[0228] In some embodiments, the simulated fluoroscopic image may include at least one of a sectional reconstruction and a multiplanar reconstruction of the anatomical region of interest onto one or more 2D projection planes visualized in conjunction with the catheter position. In some embodiments the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest. Thus, by utilizing the ultrasound data, augmented with tracking data in some cases, the methods provide users with an intuitive and familiar 2D projection aligning with accustomed visual references. The simulated fluoroscopic view in conjunction with the 2D projection of the catheter position enhances comprehension of the internal structures and tracked tools.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0229] PCT APPLICATION
[0230] As disclosed in detail herein, multiple factors enable the presentation of intuitive projected image datasets from which the simulated fluoroscopic image is generated. As discussed herein, based on the known accessory pathways (through the Inferior Vena cava), as well as the catheter dynamics (tubular, flexible structure) in combination with imaging information provided by a 3D-Ultrasound catheter, the correct anatomical position and orientation in relation to the cardiac anatomy (and specific landmarks) can be retrieved by matching image information with anatomical landmarks. Optionally, through 3D-Tracking of the catheter, its position and orientation with respect to the patient’s chest and heart can be visualized, whereas 3D-Tracking can be either Electromagnetic-Based, Image-Based, or a combination thereof. Optionally, from acquired 3D ultrasound data, Fluoroscopic pseudoimages can be reconstructed through simulation, where simulation can be done through various approaches (including raytracing-based, physics-based, machine-learning based) estimating a certain fluoroscopy image from a known anatomy imaged by 3D-ultrasound. Alternatively, a common fluoroscopic view (“Patient atlas”) can be considered and optionally tailored to the specific anatomy or view orientation. Using, for example, these data points, a standard projection view of 3D data onto a 2D projection plane, or sectional or multiplanar reconstructions on certain 2D projection planes may be visualized in conjunction with the underlying catheter position. Thereby, the use of standardized orientations (e.g. Anterior-Posterior view - AP-View, or Right Anterior Oblique - RAO) allows practitioners to work in familiar orientations / views familiar and thus perform interventional navigation and guidance of catheters more intuitively.
[0231] In some embodiments of the method, the simulation comprises at least one of a ray tracing-based simulation, a physics-based simulation, and a machine learning-based simulation. The ray tracing simulation may be used to simulate / render, for example, reflections, refractions, and shadows typical of a fluoroscopic view. The physics-based simulation may include mathematical models of the physical phenomena associated with the anatomical region of interest, the catheter, the interventional tool, and the like by applying the laws of physics. The machine-learning based simulation may include, for example, one or more of supervised, unsupervised, and reinforcement learning. The machine learning simulation may be combined with traditional modeling techniques to leverage the strengths of multiple simulation modalities.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0232] PCT APPLICATION
[0233] In some embodiments of the method, the simulated fluoroscopy image may include an interactive digital model of the one or more anatomical regions of interest, wherein the interactive digital model comprises a common fluoroscopic view generated beginning from a catheter access site to one or more targets within a heart to provide an interactive visualization of patient-specific structure and / or morphology of intra-cardiac structures.
[0234] As disclosed herein, the system may generate a 4D dynamic (spatio-temporal) digital model. The disclosed imaging protocols and algorithms may be used to dynamically reconstruct properties of the anatomy. The methods may include one or more algorithms for dynamically reconstructing multiple images from the 3D image data to provide a 3D visualization of the anatomical region of interest and targeted tissue site. The digital model may be achieved by registration and fusion of ultrasound volumes as described herein. By applying the steps for generating a patient-specific 3D digital anatomical model for different cardiac phases, a separate 3D model may be generated for each cardiac phase. These separate models may be combined into a temporal dynamic model that represents the patient-specific anatomy over time. In some embodiments the temporal dynamic model visualizes the heartbeat as a sequence of animated 3D models. The 3D model may be expanded piece by piece along the trajectory of the ICE catheter in the vessels of the heart. The 4D view visualizes the heartbeat as a sequence of animated models.
[0235] To generate the interactive patient-specific 3D digital anatomical model, the received data may be processed and combined. Systems of the invention may combine multiple image data into a large anatomical representation. The digital anatomical model may be generated in real-time for anatomy that is within the field-of-view. For example, and as disclosed herein, to generate a patient-specific 3D digital anatomical model based on the 4D image data acquired along the ICE catheter’s trajectory over time, as well as to navigate the ICE catheter, the systems of the method may utilize medical image registration, computer vision (CV)-based approaches, and / or simultaneous localization and mapping (SLAM).
[0236] Further, as disclosed herein, the method may adapt and utilize one or more image segmentation algorithms for generating the patient-specific digital anatomical model. Image segmentation is the process of dividing an image into multiple meaningful and homogeneous regions or objects based on their inherent characteristics, such as color, texture, shape, or brightness. In some embodiments, each pixel is labeled, and all pixels belonging to the sameAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0237] PCT APPLICATION
[0238] category have a common label assigned to them. Segmentation may be achieved via one or more algorithms. For example, in some embodiments, segmentation is achieved via instance image segmentation in which each object in an image is detected and segmented, via one or more algorithms to separate overlapping objects. In some embodiments, segmentation may be achieved via semantic segmentation in which one or more algorithms are used to label each pixel. In some embodiments, segmentation may be achieved via panoptic segmentation in which one or more machine learning algorithms are used to label each pixel with a class label and identifies each object instance in the image to provide for detection and interaction of the object within the environment. Thus, the systems of the invention may use CV-based techniques to combine multiple image data into an anatomical representation that combines the multiple image data into a large anatomical representation.
[0239] In some embodiments of the method, the segmentation may be performed by a vision algorithm such as thresholding, connected component analysis, or a neural network based segmentation. In some embodiments, segmentation of the full panoramic volumes may be performed, for example, via a deep learning algorithm. As disclosed herein the receiving, processing, and combining the received data to reconstruct the interactive digital model of an imaged anatomy includes initialization, continuous 3D image registration, segmentation, filtering, optimization, 3D fusion, and panoramic image reconstruction. Combining the received data may include selecting a 3D ultrasound image in the catheter-based ultrasound imaging data and mapping the 3D ultrasound image to the physical patient coordinate system using, for example, EM tracking data. In this way, a registration of each 3D ultrasound image in the physical patient’s coordinate system may be initialized. The ultrasound image selected for initialization may be the first or any subsequent ultrasound image. 3D pose data may be used for initialization of the registration. In some embodiments, EM tracking data may be used for initialization of the registration.
[0240] As disclosed herein, methods of the invention may use multimodal image registration using one or more algorithms to correlate morphologic and / or functional features between images. The registration volume may be the entire imaged volume or only a subset of the available image set. A suitable spatial overlap may be necessary to enable registration. Utilizing a rigid registration scheme is beneficial, while a deformable registration scheme can cope with deformations caused by inaccurate ECG filtering and breathing effects. A deformable imageAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0241] PCT APPLICATION
[0242] registration takes into account non-linear deformation and displacement, and may include geometrically deforming one image into another by applying one or more algorithms, for example deep learning-based models. For example, the deformable registration scheme may be a computational process in which an image similarity measure function and a transformation model are defined for the images of interest, then an optimization algorithm may be used to adjust the transformation model in a way that maximizes the similarity function. The transformation models may include spline and demons, elastic, fluid, finite element model, and free form deformations. Rigid registration is a global match between image sets that preserves the relative distance between every pair of points from the patient’s anatomy. Rigid-body registration may include a combination of rotation and translation in order to bring the images into the same coordinate system.
[0243] 3D ultrasound images, subsequent to the initialization, along the ICE catheter’s trajectory may be filtered using pulse phase-gating. For example, the pulse-phase gating may be ECG-gating in some embodiments. 3D ultrasound images of the same cardiac phase may be registered against the respective previous 3D ultrasound image or the current fused 3D image.
[0244] Subsequent optimization steps increase the accuracy of the localization of each image in the patient’s coordinate system. The optimization steps also ensure robustness of the system. For example, wherein 3D pose data comprises EM tracking data, optimization ensures robustness of the systems against missing EM tracking data, for instance where the ICE catheter leaves the EM tracking field. To provide for improved robustness of registration in the case of inaccurate EM data used for initialization, or in the case of limited image features (e.g. when the ICE catheter is in the inferior vena cava (IVC) next to the lung), an additional tracking scheme may be used. The tracking scheme may use the EM tracking data and the previous registration results to predict the next 3D position. The predicted next 3D position may then be used for initialization of the continuous registration. To further provide for improved registration in regions with few features, prior knowledge of the imaged anatomy may be applied.
[0245] In some embodiments, combining the received data further comprises continuously registering subsequent 3D ultrasound images within the physical patient’s coordinate system. In some embodiments, the 3D pose data may be combined with one or more previous registration results to predict a next 3D position. For example, where EM tracking data is used, the EMAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0246] PCT APPLICATION
[0247] tracking data may be combined with one or more previous registration results to predict a next 3D position.
[0248] As disclosed herein, the digital anatomical model may encapsulate a representation of the anatomy in a spatial topology for live visualization of one or more anatomical regions of interest. The spatial topology comprises a topological map, such that each point in the topological map represents tissue and / or one or more specific anatomical properties. Thus, ultrasound imaging data from one or more directions and / or one or more views may be combined to provide the representation of the anatomy. This representation may be a panoramic image reconstruction (i.e. an intensity volume), a (segmented) surface model, and / or a mesh. The digital anatomy may also be a more advanced representation of the anatomy, where for each point in the topological map, a representation of tissue or specific anatomical properties are encapsulated, such that different views or information from different directions can be combined for a complete representation. Thus, by reconstructing an interactive digital anatomical model, based on the processing and combining of the received data, the systems of the invention provide a digital anatomical model similar to computed tomography (CT), magnetic resonance imaging (MRI), or ultrasound tomographic reconstructions. The digital model encapsulates a representation of the anatomy in a spatial topology for live visualization of one or more anatomical regions of interest.
[0249] As disclosed herein, methods of the invention may provide for 3D segmentation of the ultrasound imaging data. In some embodiments, relevant structures of the cardiac and vascular anatomy are segmented in ECG-gated 3D ultrasound images. For example, a main focus may be the segmentation of the inner shell (surface) of the right atrium (RA), left atrium (LA), and the traversed veins. The inner shell may be defined as the tissue-blood interface of the blood pool and the cardiac or vascular wall. Because segmentation of ICE data is challenging, systems of the invention include using supervised or unsupervised machine-learning approaches in some embodiments. In some embodiments, the filtered subsequent 3D ultrasound images are segmented using a supervised or unsupervised deep-learning algorithm.
[0250] The methods of the invention may provide for adaptive 3D model generation. By fusing the segmented 3D shells using the 3D registration results, a patient-specific digital 3D anatomical model may be obtained. The 3D model may be continuously updated with the newest registration and segmentation results.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0251] PCT APPLICATION
[0252] In some embodiments of the methods, the continuously updated patient-specific digital 3D anatomical model may be visualized simultaneously with the fused ultrasound image data to generate the simulated fluoroscopic view and to enable 3D navigation of the catheter in realtime. The systems of the invention provide for generating 3D anatomical landmarks. To aid the navigation of the ICE catheter and interventional tools, a detection and localization step may be added to automatically find prominent anatomical landmarks. These landmarks may be visualized in 3D along with the model.
[0253] Methods of the invention provide for tracking of interventional tools within the 3D model. The console may be further configured to detect, localize, and segment in the 3D ultrasound images, relevant interventional tools. For initialization of the detection step, 3D pose data, such as EM tracking data, may be used as available. The segmented tools may be visualized along with the 3D digital anatomical model. In addition to physical tracking and navigation, the invention provides for contact assessment between the tool tip and cardiac or vascular wall.
[0254] In some embodiments of the methods, the model may be generated once when all of the 3D ultrasound image data is available. After model generation, navigation of the ICE catheter as well as tracking and navigation of interventional tools is possible. Thus, the anatomical context generated with the model enables safer navigation of catheter tools, for example during septal crossing procedures. In some embodiments, the console may be further operable to automatically detect and localize anatomical landmarks for visualization within the digital model to aid navigation of the catheter and interventional tools.
[0255] As disclosed herein, in some embodiments of the methods, the interactive digital model may be generated beginning from a catheter access site to one or more targets within a heart to provide an interactive visualization of patient-specific structure and / or morphology of intracardiac structures in a simulated fluoroscopic view. For example, the intra-cardiac structures may be one or more of trabeculae, tendons, chordae, and valve s / leaflets.
[0256] Systems and methods for providing multiple view angles of an anatomical region of interest Overview
[0257] Aspects of the invention include systems and methods for providing a plurality of view angles of an anatomical region of interest. Interventional surgeons are limited in their capacity to manipulate imaging software / consoles while performing an intervention. It is necessary toAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0258] PCT APPLICATION
[0259] quickly understand the anatomical environment and for delivering therapy safely. In this regard, the invention provides systems and methods for generating multiple view angles in 3D volumetric imaging on a specific target. Multiple view angle may be used in quickly assessing anatomical conditions and navigating devices in that area. Additionally, the provided dual view of a landmark improves spatial orientation without requiring multiple user actions to create the additional views. The systems and methods of the invention automatically create a second 3D view from an opposite viewing direction once a desired landmark is within the field of view, thus limiting user actions to navigating to the desired landmark and selecting a create dual view option within the system.
[0260] In conventional systems, a user must manually set the location of the camera (e.g. atrium) and the desired viewing direction and distance (e.g. towards ventricle) by clicking, dragging and clicking again in a 2D plane first. This creates two planes around the landmark that crop the 3D volume at that position and creates two cameras attached to each respectively looking at the center between the two plane. If the user wants to move the crop plane, they must clear the selection and start a new one. The systems and methods of the invention provide for automatically generating a dual view which does not require the user to set these points but rather uses the present view angle in the 3D and a single command button that runs the auto-dual view generation process to automatically provide a view from the opposite of the viewed target. Both views include camera clipping, thus allowing the user to move the clip plane by simply zooming in and out of the view. This eliminates the need of the multiple steps of setting new crop planes of conventional systems when needing to modify the volume in view.
[0261] Systems for generating multiple views
[0262] The system includes a console configured to be operably associated with an ultrasound imaging device and exchange data therewith, the console comprising a hardware processor coupled to non-transitory, computer-readable memory. The memory contains instructions executable by the processor to cause the console to receive and analyze three-dimensional (3D) ultrasound image data obtained from an ultrasound imaging device for a target anatomical region, the 3D ultrasound image data being associated with a 3D volume of the target anatomical region oriented in a selected view angle; generate a first image of the target anatomical region at the selected view angle; calculate, via one or more algorithms, data associated with a secondAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0263] PCT APPLICATION
[0264] image of the target anatomical region; and generate the second image of the target region comprising a second view angle different from the selected view angle.
[0265] In some embodiments, the second view angle is opposite of the selected view angle. In some embodiments, the second view angle is a defined angle of interest, wherein the angle of interest comprises one or more of an opposite of the selected view angle and an orthogonal view. In some embodiments, any defined angle may be used to yield an improved dual view.
[0266] In some embodiments, the 3D ultrasound image data comprises coordinate position data of an original camera and a viewing direction at the target anatomical region. The user orients a 3D volume to a desired rotation and / or zoom. The system is configured to create the second 3D view angle from the opposite viewing angle. The second view angle may be selected by the user via an input from the user.
[0267] In some embodiments, the one or more algorithms may be configured to define a clip plane splitting the first image of the target anatomical region into an intersecting mirror plan. Accordingly, the one or more algorithms may mirror the coordinate position and viewing angle in the second image. In some embodiments, the clip plane is associated with the camera at a preset offset. In some embodiments, the location of the intersecting mirror plane may be defined by adding a defined buffer offset from the clip plane to the clip plane coordinates in the 3D volume. In some embodiments, the defined buffer offset may be based on at least one of known averages from one or more anatomical databases and known sizes of one or more anatomical landmarks.
[0268] FIG. 6 illustrates calculation of a dual view according to one embodiment of the systems and methods of the invention. The algorithm uses the coordinates of the camera and its viewing direction and mirrors the coordinates based on a mirror plane (blue). The location of the mirror plane is defined by taking the clip plane coordinates in the 3D scene and adding a defined buffer offset (d2) from the clip plane based on averages collected from anatomical datasets and sizes of anatomical landmarks. The position of the mirrored camera (Cm) source is created based on the distance of the original camera (C ) to the clip plane (dl), the defined buffer offset (d2) (= location of mirror plane) and adding the same distances to the opposite side of the plane in the same direction (defined by the viewing direction). Thus, the mirrored camera is placed at the same distance from the mirror plane with the opposed viewing direction to the first, resulting in a second 3D view from the opposite angle with a camera clip function. By zooming in and out ofAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0269] PCT APPLICATION
[0270] the view, the camera moves the clip plane and the clip on the respective view may be adjusted as part of a natural image inspection workflow.
[0271] The following formula may be used:
[0272] dl = Distance (C, clip plane)
[0273] d2 = Distance (clip plane, mirror plane)
[0274] Mirrored camera position: Cm(x,y,z) = C(x,y,z) + 2*dl + 2*d2 along the viewing direction vector
[0275] Mirrored camera viewing angle: -(viewing vector of C)
[0276] In some embodiments, the position of the mirrored camera source is calculated based on the distance of the original camera to the clip plane, the defined buffer offset, and by adding one or more same distances to the mirror plane in a same direction as the viewing direction such that the mirrored camera source is placed at a same distance from the mirror plane with an opposed viewing direction to the original camera to thereby provide a second 3D view from an opposite view.
[0277] In some embodiments, the dual view on a landmark improves spatial orientation without creating a new view, either manually or via a multi-step process. In some embodiments, the system may automatically create a second 3D view from the opposite viewing direction. In some embodiments, the desired 3D view may be oriented in at least one of a particular rotation and a particular zoom.
[0278] Methods for generating multiple views
[0279] The invention provides methods for generating a plurality of view angles of an anatomical region of interest.
[0280] FIG. 7 illustrates a block diagram of a method 700 of providing a plurality of view angles of an anatomical region of interest according to one embodiment of the invention. The method includes the steps of providing 702 a console configured to be operably associated with an ultrasound imaging device and exchange data therewith; receiving and analyzing 704, via the console, three-dimensional (3D) ultrasound image data obtained from an ultrasound imaging device for a target anatomical region, the 3D ultrasound image data being associated with a 3D volume of the target anatomical region oriented in a selected view angle; generating 706, via theAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0281] PCT APPLICATION
[0282] console, a first image of the target anatomical region at the selected view angle; calculating 708, via one or more algorithms, data associated with a second image of the target anatomical region; and generating 710, via the console, the second image of the target region comprising a second view angle different from the selected view angle.
[0283] The method includes a console configured to be operably associated with an ultrasound imaging device and exchange data therewith, the console comprising a hardware processor coupled to non-transitory, computer-readable memory. The memory contains instructions executable by the processor to cause the console to receive and analyze three-dimensional (3D) ultrasound image data obtained from an ultrasound imaging device for a target anatomical region, the 3D ultrasound image data being associated with a 3D volume of the target anatomical region oriented in a selected view angle; generate a first image of the target anatomical region at the selected view angle; calculate, via one or more algorithms, data associated with a second image of the target anatomical region; and generate the second image of the target region comprising a second view angle different from the selected view angle.
[0284] In some embodiments of the method, the second view angle is opposite of the selected view angle. In some embodiments, the second view angle is a defined angle of interest, wherein the angle of interest comprises one or more of an opposite of the selected view angle and an orthogonal view. In some embodiments, any defined angle may be used to yield an improved dual view.
[0285] In some embodiments of the method, the 3D ultrasound image data comprises coordinate position data of an original camera and a viewing direction at the target anatomical region. The user orients a 3D volume to a desired rotation and / or zoom. The method may be configured to create the second 3D view angle from the opposite viewing angle. The second view angle may be selected by the user via an input from the user.
[0286] In some embodiments of the method, the one or more algorithms may be configured to define a clip plane splitting the first image of the target anatomical region into an intersecting mirror plan. Accordingly, the one or more algorithms may mirror the coordinate position and viewing angle in the second image. In some embodiments, the clip plane is associated with the camera at a pre-set offset. In some embodiments, the location of the intersecting mirror plane may be defined by adding a defined buffer offset from the clip plane to the clip plane coordinates in the 3D volume. In some embodiments, the defined buffer offset may be based on at least oneAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0287] PCT APPLICATION
[0288] of known averages from one or more anatomical databases and known sizes of one or more anatomical landmarks.
[0289] The algorithm may use the coordinates of the camera and its viewing direction and mirrors the coordinates based on a mirror plane (blue). The location of the mirror plane may be defined by taking the clip plane coordinates in the 3D scene and adding a defined buffer offset (d2) from the clip plane based on averages collected from anatomical datasets and sizes of anatomical landmarks. The position of the mirrored camera (Cm) source may be created based on the distance of the original camera (C ) to the clip plane (dl), the defined buffer offset (d2) (= location of mirror plane) and adding the same distances to the opposite side of the plane in the same direction (defined by the viewing direction). Thus, the mirrored camera may be placed at the same distance from the mirror plane with the opposed viewing direction to the first, resulting in a second 3D view from the opposite angle with a camera clip function. By zooming in and out of the view, the camera moves the clip plane and the clip on the respective view may be adjusted as part of a natural image inspection workflow.
[0290] The following formula may be used:
[0291] dl = Distance (C, clip plane)
[0292] d2 = Distance (clip plane, mirror plane)
[0293] Mirrored camera position: Cm(x,y,z) = C(x,y,z) + 2*dl + 2*d2 along the viewing direction vector
[0294] Mirrored camera viewing angle: -(viewing vector of C)
[0295] In some embodiments, the position of the mirrored camera source may be calculated based on the distance of the original camera to the clip plane, the defined buffer offset, and by adding one or more same distances to the mirror plane in a same direction as the viewing direction such that the mirrored camera source is placed at a same distance from the mirror plane with an opposed viewing direction to the original camera to thereby provide a second 3D view from an opposite view.
[0296] In some embodiments, the dual view on a landmark improves spatial orientation without creating a new view, either manually or via a multi-step process. In some embodiments, the system may automatically create a second 3D view from the opposite viewing direction. In someAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0297] PCT APPLICATION
[0298] embodiments, the desired 3D view may be oriented in at least one of a particular rotation and a particular zoom.
[0299] As used in any embodiment herein, the term “module” may refer to software, firmware and / or circuitry configured to perform any of the aforementioned operations. Software may be embodied as a software package, code, instructions, instruction sets and / or data recorded on non-transitory computer readable storage medium. Firmware may be embodied as code, instructions, or instruction sets and / or data that are hard-coded (e.g., nonvolatile) in memory devices.
[0300] “Circuitry”, as used in any embodiment herein, may comprise, for example, singly or in any combination, hardwired circuitry, programmable circuitry such as computer processors comprising one or more individual instruction processing cores, state machine circuitry, and / or firmware that stores instructions executed by programmable circuitry. The modules may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, an integrated circuit (IC), system on-chip (SoC), desktop computers, laptop computers, tablet computers, servers, smartphones, etc.
[0301] Any of the operations described herein may be implemented in a system that includes one or more storage mediums having stored thereon, individually or in combination, instructions that when executed by one or more processors perform the methods. Here, the processor may include, for example, a server CPU, a mobile device CPU, and / or other programmable circuitry.
[0302] Also, it is intended that operations described herein may be distributed across a plurality of physical devices, such as processing structures at more than one different physical location. The storage medium may include any type of tangible medium, for example, any type of disk including hard disks, floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic and static RAMs, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), flash memories, Solid State Disks (SSDs), magnetic or optical cards, or any type of media suitable for storing electronic instructions.
[0303] Other embodiments may be implemented as software modules executed by a programmable control device. The storage medium may be non-transitory.
[0304] As described herein, various embodiments may be implemented using hardware elements, software elements, or any combination thereof. Examples of hardware elements mayAttorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0305] PCT APPLICATION
[0306] include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth.
[0307] Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0308] The term "non -transitory" is to be understood to remove only propagating transitory signals per se from the claim scope and does not relinquish rights to all standard computer-readable media that are not only propagating transitory signals per se. Stated another way, the meaning of the term "non-transitory computer-readable medium" and "non-transitory computer-readable storage medium" should be construed to exclude only those types of transitory computer-readable media which were found in In Re Nuijten to fall outside the scope of patentable subject matter under 35 U.S.C. § 101.
[0309] The terms and expressions which have been employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described (or portions thereof), and it is recognized that various modifications are possible within the scope of the claims. Accordingly, the claims are intended to cover all such equivalents.
[0310] Incorporation by Reference
[0311] References and citations to other documents, such as patents, patent applications, patent publications, journals, books, papers, web contents, have been made throughout this disclosure. All such documents are hereby incorporated herein by reference in their entirety for all purposes.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110
[0312] PCT APPLICATION
[0313] Equivalents
[0314] Various modifications of the invention and many further embodiments thereof, in addition to those shown and described herein, will become apparent to those skilled in the art from the full contents of this document, including references to the scientific and patent literature cited herein. The subject matter herein contains important information, exemplification and guidance that can be adapted to the practice of this invention in its various embodiments and equivalents thereof.
Claims
Attorney Docket No.: ONEP-015 / 01WO 36223 / 110PCT APPLICATIONClaims1. A system for generating a simulated view from three-dimensional (3D) ultrasound data, the system comprising:a console configured to be operably associated with an ultrasound imaging device and exchange data therewith, the console comprising a hardware processor coupled to non-transitory, computer-readable memory containing instructions executable by the processor to cause the console to:receive data associated with one or more anatomical regions of interest, the data comprising at least one of catheter-based ultrasound imaging data, catheter position data, and catheter orientation data;process and combine the received data;generate, based on said processing and combining of data, simulated two- dimensional (2D) projection image data; andreconstruct, based on the simulated 2D projection image data, a simulated image of the one or more anatomical regions of interest, wherein the simulated image comprises an estimated catheter position within the simulated image.
2. The system of claim 1, wherein the simulated 2D projection image data comprises simulated fluoroscopy data, wherein a simulated fluoroscopic image of the one or more anatomical regions of interest is generated.
3. The system of claim 1, wherein the simulated image corresponds to a standardized orientation of a view of the anatomical region of interest.
4. The system of claim 3, wherein the standardized orientation comprises at least one of an anterior-posterior (AP) view, a posterior-anterior (PA) view, a left anterior oblique (LAO) view, a right anterior oblique (RAO) view, a left lateral (LL) view, and a right lateral (RR) view.
5. The system of claim 1, wherein the catheter-based ultrasound imaging data is received from an intracardiac echocardiogram (ICE) two-dimensional (2D) and / or 3D catheter.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110PCT APPLICATION6. The system of claim 5, wherein the one or more anatomical regions of interest comprise at least one of cardiac anatomy and vascular anatomy.
7. The system of claim 6, wherein the received data further comprises at least one of intracardiac echocardiography (ICE) data, data associated with one or more anatomical landmarks, data associated with an accessory pathway, cardiac pulse phase data, and a dynamic of a structure of the catheter.
8. The system of claim 6, wherein the system calculates a catheter position and a catheter orientation within the one or more anatomical regions of interest in relation to at least one of a cardiac anatomy, vascular anatomy, and an anatomical landmark.
9. The system of claim 1, wherein the catheter position data and catheter orientation data comprise 3D tracking data.
10. The system of claim 9, wherein the 3D tracking data comprises at least one of electromagnetic (EM)-based tracking data, impedance tracking data, image-based tracking data, and fiber optic shape sensing.
11. The system of claim 1, wherein the simulated image comprises a projection of 3D ultrasound data of the one or more anatomical regions of interest onto a 2D projection plane with the catheter overlay ed onto the 2D projection such that the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
12. The system of claim 1, wherein the simulated image comprises at least one of a sectional reconstruction and a multiplanar reconstruction of the anatomical region of interest onto one or more 2D projection planes visualized in conjunction with the catheter position such that the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
13. The system of claim 1, wherein the simulation comprises at least one of a raytracing-based simulation, a physics-based simulation, and a machine learning-based simulation.
14. The system of claim 1, wherein the simulated image comprises an interactive digital model of the one or more anatomical regions of interest, wherein the interactive digital modelAttorney Docket No.: ONEP-015 / 01WO 36223 / 110PCT APPLICATIONcomprises a common fluoroscopic view generated beginning from a catheter access site to one or more targets within a heart to provide an interactive visualization of patient-specific structure and / or morphology of intra-cardiac structures.
15. The system of claim 1, wherein the catheter-based ultrasound imaging data is received from one or more of a catheter-based ultrasound comprising an endovascular and / or intravascular ultrasound configured for in-body applications and a combination of multiple acquisitions comprising a panoramic scan.
16. The system of claim 15, wherein the ultrasound imaging data is received from a plurality of cylindrical fields-of-view acquired via a rotating transducer array and / or a cylindrical folded transducer matrix.
17. A method for generating a simulated fluoroscopic view from three-dimensional (3D) ultrasound data, the system comprising:providing a console configured to be operably associated with an ultrasound imaging device and exchange data therewith;receiving, via the console, data associated with one or more anatomical regions of interest, the data comprising at least one of catheter-based ultrasound imaging data, catheter position data, and catheter orientation data;processing and combining, via the console, the received data;generating, based on said processing and combining of data, simulated two-dimensional (2D) projection image data; andreconstructing, based on the 2D projection image data, a simulated image of the one or more anatomical regions of interest, wherein the simulated image comprises an estimated catheter position within the simulated image.
18. The method of claim 17, wherein the simulated 2D projection image data comprises simulated fluoroscopy data, wherein a simulated fluoroscopic image of the one or more anatomical regions of interest is generated.
19. The method of claim 17, wherein the simulated image corresponds to a standardized orientation of a view of the anatomical region of interest.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110PCT APPLICATION20. The method of claim 18, wherein the standardized orientations comprise at least one of an anterior-posterior (AP) view, a posterior-anterior (PA) view, a left anterior oblique (LAO) view, a right anterior oblique (RAO) view, a left lateral (LL) view, and a right lateral (RR) view.
21. The method of claim 18, wherein the catheter-based ultrasound imaging data is received from an intracardiac echocardiogram (ICE) two-dimensional (2D) and / or 3D catheter.
22. The method of claim 21, wherein the one or more anatomical regions of interest comprise at least one of cardiac anatomy and vascular anatomy.
23. The method of claim 22, wherein the received data further comprises at least one of intracardiac echocardiography (ICE) data, data associated with one or more anatomical landmarks, data associated with an accessory pathway, cardiac pulse phase data, and a dynamic of a structure of the catheter.
24. The method of claim 22, wherein the system calculates a catheter position and a catheter orientation within the one or more anatomical regions of interest in relation to at least one of a cardiac anatomy, vascular anatomy, and an anatomical landmark.
25. The method of claim 17, wherein the catheter position data and catheter orientation data comprise 3D tracking data.
26. The method of claim 25, wherein the 3D tracking data comprises at least one of electromagnetic (EM)-based tracking data, impedance tracking data, image-based tracking data, and fiber optic shape sensing.
27. The method of claim 17, wherein the simulated image comprises a projection of 3D ultrasound data of the one or more anatomical regions of interest onto a 2D projection plane with the catheter overlay ed onto the 2D projection such that the 2D projection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
28. The method of claim 17, wherein the simulated image comprises at least one of a sectional reconstruction and a multiplanar reconstruction of the anatomical region of interest onto one or more 2D projection planes visualized in conjunction with the catheter position such that the 2DAttorney Docket No.: ONEP-015 / 01WO 36223 / 110PCT APPLICATIONprojection optimizes a view of the orientation of the catheter’s position within the anatomical region of interest.
29. The method of claim 17, wherein the simulation comprises at least one of a raytracing-based simulation, a physics-based simulation, and a machine learning-based simulation.
30. Thes method of claim 17, wherein the simulated image comprises an interactive digital model of the one or more anatomical regions of interest, wherein the interactive digital model comprises a common fluoroscopic view generated beginning from a catheter access site to one or more targets within a heart to provide an interactive visualization of patient-specific structure and / or morphology of intra-cardiac structures.
31. The method of claim 17, wherein the catheter-based ultrasound imaging data is received from a catheter-based ultrasound comprising an endovascular and / or intravascular ultrasound configured for in-body applications.
32. The method of claim 31, wherein the ultrasound imaging data is received from a plurality of cylindrical fields-of-view acquired via a rotating transducer array and / or a cylindrical folded transducer matrix.
33. A system for providing a plurality of view angles of an anatomical region of interest, the system comprising:a console configured to be operably associated with an ultrasound imaging device and exchange data therewith, the console comprising a hardware processor coupled to non-transitory, computer-readable memory containing instructions executable by the processor to cause the console to:receive and analyze three-dimensional (3D) ultrasound image data obtained from an ultrasound imaging device for a target anatomical region, the 3D ultrasound image data being associated with a 3D volume of the target anatomical region oriented in a selected view angle;generate a first image of the target anatomical region at the selected view angle; calculate, via one or more algorithms, data associated with a second image of the target anatomical region; andAttorney Docket No.: ONEP-015 / 01WO 36223 / 110PCT APPLICATIONgenerate the second image of the target region comprising a second view angle different from the selected view angle.
34. The system of claim 33, wherein the second view angle is a defined angle of interest, wherein the angle of interest comprises one or more of an opposite of the selected view angle and an orthogonal view.
35. The system of claim 33, wherein the 3D ultrasound image data comprises coordinate position data of an original camera and a viewing direction at the target anatomical region.
36. The system of claim 35, wherein the algorithm is configured to define a clip plane splitting the first image of the target anatomical region into an intersecting mirror plane, wherein the algorithm mirrors the coordinate position and viewing angle in the second image.
37. The system of claim 36, wherein a location of the intersecting mirror plane is defined by adding a defined buffer offset from the clip plane to the clip plane coordinates in the 3D volume.
38. The system of claim 36, wherein the defined buffer offset is based on at least one of known averages from one or more anatomical databases and known sizes of one or more anatomical landmarks.
39. The system of claim 36, wherein a position of a mirrored camera source is calculated based on the distance of the original camera to the clip plane, the defined buffer offset, and by adding one or more same distances to the mirror plane in a same direction as the viewing direction such that the mirrored camera source is placed at a same distance from the mirror plane with an opposed viewing direction to the original camera to thereby provide a second 3D view from an opposite view.
40. The system of claim 33, wherein a dual view on a landmark improves spatial orientation without creating a new view manually and / or via a multi-step process.
41. The system of claim 33, wherein the system automatically creates a second 3D view from the opposite viewing direction.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110PCT APPLICATION42. The system of claim 33, wherein the desired 3D view is oriented in at least one of a particular rotation and a particular zoom.
43. A method for providing a plurality of view angles of an anatomical region of interest, the method comprising:providing a console configured to be operably associated with an ultrasound imaging device and exchange data therewith;receiving and analyzing, via the console, three-dimensional (3D) ultrasound image data obtained from an ultrasound imaging device for a target anatomical region, the 3D ultrasound image data being associated with a 3D volume of the target anatomical region oriented in a selected view angle;generating, via the console, a first image of the target anatomical region at the selected view angle;calculating, via one or more algorithms, data associated with a second image of the target anatomical region; andgenerating, via the console, the second image of the target region comprising a second view angle different from the selected view angle.
44. The method of claim 43, wherein the second view angle is a defined angle of interest, wherein the angle of interest comprises one or more of an opposite of the selected view angle and an orthogonal view.
45. The method of claim 43, wherein the 3D ultrasound image data comprises coordinate position data of an original camera and a viewing direction at the target anatomical region.
46. The method of claim 45, wherein the algorithm is configured to define a clip plane splitting the first image of the target anatomical region into an intersecting mirror plan, wherein the algorithm mirrors the coordinate position and viewing angle in the second image.
47. The method of claim 46, wherein a location of the intersecting mirror plane is defined by adding a defined buffer offset from the clip plane to the clip plane coordinates in the 3D volume.Attorney Docket No.: ONEP-015 / 01WO 36223 / 110PCT APPLICATION48. The method of claim 46, wherein the defined buffer offset is based on at least one of known averages from one or more anatomical databases and known sizes of one or more anatomical landmarks.
49. The method of claim 46, wherein a position of a mirrored camera source is calculated based on the distance of the original camera to the clip plane, the defined buffer offset, and by adding one or more same distances to the mirror plane in a same direction as the viewing direction such that the mirrored camera source is placed at a same distance from the mirror plane with an opposed viewing direction to the original camera to thereby provide a second 3D view from an opposite view.
50. The method of claim 43, wherein a dual view on a landmark improves spatial orientation without creating a new view manually or via a multi-step process.
51. The method of claim 43, wherein the method automatically creates a second 3D view from the opposite viewing direction.
52. The method of claim 43, wherein the desired 3D view is oriented in at least one of a particular rotation and a particular zoom.
