Interactive surgical collaboration system and methods

Abstract

Described in this specification is an online collaboration tool that extends the traditional audio / video functionality in current offerings by integrating a variety of medical data sources, such as DICOM files, 3D reconstructions and live fluoroscopy streams into a virtual environment. Source information is routed to user-preferred interface destinations, such as imaging workstations, augmented reality headsets and browsers (laptops, tablets, smart phones).

Description

Docket No.: 121165-0032-004-W01INTERACTIVE SURGICAL COLLABORATION SYSTEM AND METHODSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 63 / 760,739, filed on February 20, 2025, entitled INTERACTIVE SURGICAL COLLABORATION SYSTEM AND METHODS, which is entirely incorporated herein by reference.BACKGROUND

[0002] Spatial computing is revolutionizing surgical training and practice providing immersive, interactive environments. Key advancements include enhanced learning experiences, standardized training, and objective assessments. Some platforms offer realistic simulations for various procedures. Integration with technologies like augmented reality (AR) and haptic feedback further enriches training. Despite challenges like cost and technological limitations, AR’s potential for global collaboration and personalized learning paths is promising. This technology is set to transform surgical education and practice, making it safer and more efficient.SUMMARY

[0003] Described in this specification is an online collaboration tool that extends the traditional audio / video functionality in current offerings by integrating a variety of medical data sources, such as DICOM files, 3D reconstructions and live fluoroscopy streams into a virtual environment. Source information is routed to user-preferred interface destinations, such as imaging workstations, augmented reality headsets and browsers (laptops, tablets, smart phones). The technologies described integrate online collaboration technology with innovative surgical systems and methods that integrate and utilize 2D and 3D images, e.g., registration of 2D and 3D images. In particular the technologies leverage correspondence and alignment of (live) intraoperative 2D images (e.g., as taken by fluoroscopy or one or more other intraoperative imaging modalities) with preoperatively acquired 3D images and visualization formats that allow 3D registration to improve surgical and interventional cognition, procedural visibility, general guidance and navigation for minimally invasive and open procedures.

[0004] In an aspect, a surgical collaboration system for interactive sharing of medical image data includes a server module, an interface module, at least one cloud module, and first and second client modules in data communication with each other. The server module is configured to receive, from a Picture Archiving and Communication System (PACS), a set of DICOM images. The server module is configured to generate, from the set of DICOM images, a 3D model of an anatomic structure. The server module is configured to receive, from a fluoroscopy device comprising a C-arm and one or more detectors, fluoroscopy image data. The server module is configured to perform imageregistration operations to align the 3D model and the fluoroscopy image data. The server module is configured to generate combined image data.

[0005] The at least one cloud module is configured to receive, from the server module, the combined image data and store the combined image data for access by the first and / or second client module and / or receive, from the server module, live fluoroscopy image data and transmit the live fluoroscopy image data to the first and / or second client modules;

[0006] The interface module is configured to receive one or more inputs from a user. The inputs cause the interface module to request image data and corresponding meta data from a local DICOM image store. The inputs cause the interface module to transmit image selection and segmentation parameter data to the server module. The inputs cause the interface module to cause the server module to generate the combined image data. The inputs cause the interface module to cause the server module to upload the combined image data to the at least one cloud module.

[0007] The first and second client modules are configured to display the combined image data to a user. The first and second client modules are configured to receive, from the user of the first client module, a display input. The first and second client modules are configured to transmit the display input to the at least one cloud module. The first and second client modules are configured cause the at least one cloud module to transmit updated image data to the second client module.

[0008] In an aspect, a method for interactive sharing of medical image data includes the following steps performed by one or more modules of a surgical collaboration system. The method includes performing, by a server module of a surgical collaboration system, the following steps. The method includes receiving, from a Picture Archiving and Communication System (PACS), a set of DICOM images. The method includes generating, from the set of DICOM images, a 3D model of an anatomic structure. The method includes receiving, from a fluoroscopy device comprising a C-arm and one or more detectors, fluoroscopy image data. The method includes performing image registration operations to align the 3D model and the fluoroscopy image data. The method includes generating combined image data.

[0009] The method includes performing, by at least one cloud module of the surgical collaboration system, the following steps. The method includes receiving, from the server module, the combined image data and storing the combined image data for access by the first and / or second client module. The method includes receiving, from the server module, live fluoroscopy image data and transmitting the live fluoroscopy image data to the first and / or second client modules.

[0010] The method includes performing, by an interface module of the surgical collaboration system, the following steps. The method includes receiving one or more inputs from a user. The inputs cause the interface module to request image data and corresponding meta data from a local DICOM image store. The inputs cause the interface module to transmit image selection and segmentation parameter data to the server module. The inputs cause the interface module to cause the server module togenerate the combined image data. The inputs cause the interface module to cause the server module to upload the combined image data to the at least one cloud module.

[0011] The method includes performing, by first and second client modules of the surgical collaboration system, the following steps. The method includes displaying the combined image data to a user. The method includes receiving, from the user of the first client module, a display input. The method includes transmitting the display input to the at least one cloud module. The method includes causing the at least one cloud module to transmit updated image data to the second client module.INCORPORATION BY REFERENCE

[0012] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent publications and patents or patent applications incorporated by reference contradict the disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such contradictory material.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] A better understanding of the features and advantages of the present technologies will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also “Figure” and “FIG.” herein).

[0014] FIG. 1 is a diagram of an example implementation of a surgical collaboration system as described in this specification.

[0015] FIG.2A shows an image of cranium preoperatively acquired using CTA imaging. FIG.2B shows an image of resulting vessel 3D model generated using the technologies described in this specification.

[0016] FIG.3 is a screenshot illustrating the accurate spatial registration of live X-ray projections (grey) to virtual projections (red) in both frontal and lateral views.

[0017] FIG.4 is a diagram illustrating example information flows during preoperative data preparation.

[0018] FIG.5 is a diagram illustrating example information flows within Aquila during neuroradiological intervention.

[0019] FIG.6 is a diagram illustrating an example guidewire tracking algorithm.

[0020] FIG. 7A is a diagram of an example implementation of a surgical collaboration system as described in this specification illustrating sharing of live medical imaging, e.g. fluoroscopy, in high-resolution / lossless format; FIG. 7B is a diagram of an example implementation of a surgicalcollaboration system as described in this specification illustrating (lower quality) video sharing for the purpose of a collaborative session; FIG. 7C is a diagram of an example implementation of a surgical collaboration system as described in this specification illustrating the messaging between various system components in order to maintain a synchronized state.

[0021] FIG.8 is a diagram illustrating an example overall network topology of an implementation of the surgical collaboration system as described in this specification.

[0022] FIG.9 is a diagram illustrating an example software stack diagram for Ariadne.

[0023] FIG. 10 shows an example prospective patient registration with phantom model (live fluoroscopic input on angiosuite display in the background).

[0024] FIG. 11 is a screenshot illustrating an example step of logging into Ariadne software.

[0025] FIG. 12 is a screenshot illustrating an example step of searching for patient study and downloading from PACS to Local Store.

[0026] FIG. 13 is a screenshot illustrating an example step of retrieving images from Local Store.

[0027] FIG. 14 is a screenshot illustrating an example step of viewing DICOM images with series and frame navigation and brightness / contrast adjustment.

[0028] FIG. 15 is a screenshot illustrating an example step of selecting an image series for segmentation.

[0029] FIG. 16 is a screenshot illustrating an example step of specifying segmentation parameters and segmenting model.

[0030] FIG. 17 is a screenshot illustrating an example step of viewing and uploading segmentation to Azure.

[0031] FIG. 18 is a screenshot illustrating an example step of manually adjustment of fluoroscope parameters.

[0032] FIG. 19 is a screenshot illustrating an example patient registration step.

[0033] FIG.20 is a screenshot illustrating an example registration refinement step.

[0034] FIG.21 is a screenshot illustrating an example step in guidewire tracking.

[0035] FIG.22 is a screenshot illustrating an example step in guidewire 3D reconstruction.

[0036] FIG.23 is a screenshot illustrating an example step of preoperatively selecting viewpoints.

[0037] FIG.24 is a screenshot illustrating an example step of intraoperative ly recalling viewpoints.

[0038] FIG.25 is a diagram of an example implementation of a surgical collaboration system as described in this specification.

[0039] FIG.26 is a diagram of an example implementation of a surgical collaboration system as described in this specification.

[0040] FIG.27 is a block diagram of an example system for performing a method of rendering a 3D visualization with co-visualization of a 2D / 3D registered 2D fluoroscopy image.

[0041] FIG. 28 is a block diagram of an example system, for performing a method of rendering a 3D visualization with co-visualization of two 2D / 3D registered 2D fluoroscopy images, e.g., obtained from two different viewing angles.

[0042] FIG. 29 is a diagram of an example computer system that may used with the technologies described in this specification.

[0043] FIG. 30 is a schematic diagram of an example visualization screen that may be provided on a display screen to provide real-time navigation visualization as described in this specification.

[0044] FIG. 31A is a photograph of a phantom of cerebral vasculature including ‘blood supply’ tubing. FIG. 31B is a photograph of the phantom including circulatory pump and reservoir, a guide wire, and a syringe containing contrast fluid.

[0045] FIG. 32A is an axial slice through original DICOM (CT) data set. FIG. 32B shows the results of an example automated vessel segmentation. Cross-hairs indicate location of middle cerebral artery (MCA) branching.

[0046] FIG. 33A shows a rendering of an example vasculature based on manual segmentation in ITK-SNAP with thresholded region-growing. FIG. 33B shows a rendering of an example vasculature based on the output of first stages of an automated process as described in this specification.

[0047] FIG. 34A shows a rendering of an example raw mesh generated using the technologies described in this specification. FIG. 34B shows a rendering of an example output of the combined merge-trim-decimate-smooth operations.

[0048] FIG. 35 is a photograph showing an example configuration of a head phantom disposed on a table, with frontal / lateral arms of a fluoroscopy device.

[0049] FIG. 36 is a rendering illustrating different configurations of the C-Arm, L-Arc and patient table kinematic chains according to the technologies described in this specification.

[0050] FIG. 37 is a set of renderings of a set of frontal projections for xlO RAOLAO angles and xlO CRACAU angles generated using the technologies described in this specification.

[0051] FIG. 38 is a set of renderings of a set of lateral projections for xlO RAOLAO angles and xlO CRACAU angles generated using the technologies described in this specification

[0052] FIG. 39 is a map showing degree of pixel overwriting for 51x51 = 2,601 angle combinations.

[0053] FIG. 40 shows example lateral and frontal projections of fiducial markers and vasculature (fiducials rendered in green on surface of 3D model). The corresponding 3D image is displayed between two 2D fluoroscopic images.

[0054] FIG. 41 shows example lateral and frontal projections of fiducial markers and vasculature of an example phantom.

[0055] FIG. 42A is a rendering of an initial overlay and FIG. 42B is a rendering of a final overlay in a frontal 2D / 3D alignment of images of an example phantom.

[0056] FIG. 43A is a rendering of an initial overlay and FIG. 43B is a rendering of a final overlay in a lateral 2D / 3D alignment of images of an example phantom.

[0057] FIG. 44 is a graph showing optimization progress over the first three annealing iterations (exponential annealing schedule) of an optimization algorithm as described in this specification applied to imaging of an example phantom.

[0058] FIG. 45A is a rendering of an initial overlay and FIG. 45B is a rendering of a final overlay in a frontal 2D / 3D alignment of images of an example human anatomy.

[0059] FIG. 46A is a rendering of an initial overlay and FIG. 46B is a rendering of a final overlay in a lateral 2D / 3D alignment of images of an example human anatomy.

[0060] FIG. 47 is a graph showing optimization progress over the first three annealing iterations (exponential annealing schedule) of an optimization algorithm as described in this specification applied to imaging of an example human anatomy.

[0061] FIG. 48 is a rendering of virtual rays emanating from one 2D view and ‘scattering’ into the other 2D view.

[0062] FIG. 49 is a rendering of a physically impossible C-arm position with virtually-rendered projection.

[0063] FIG. 50 is a rendering illustrating triangulation of an example guidewire tip position in the 3D space of an example vasculature.

[0064] FIG. 51 is a graph showing results of multiple triangulation tests (minimum ray distance) using the technologies described in this specification.

[0065] FIG. 52A is the frontal view 2D fluoroscopy image corresponding to the rendering in FIG.2A. FIG. 52B is the lateral view 2D fluoroscopy image corresponding to the rendering in FIG. 2A.

[0066] FIG. 53 is a rendering of the control point lattice and resulting geometric deformations (main changes circled and faces rendered in green) in an example vascular model.

[0067] FIG. 54 is a graph showing optimization progress during deformable registration (overlap measured relative to rigid registration as a starting point) in an example registration process as described in this specification.DESCRIPTION

[0068] Described in this specification is an online collaboration tool (XRAI Connect) that extends the traditional audio / video functionality in current offerings by integrating a variety of medical data sources. A schematic overview of the XRAI Connect platform is shown in FIG. 1.

[0069] The XRAI Connect system includes a software platform, Aquila. Aquila is a bespoke software-based medical device with its primary area of applicability being the field of interventional neuroradiology (INR), but its uses can extend to other areas of surgery, e.g., orthopedic, cardiovascular, neurovascular, biliary etc. The application generates three-dimensional (3D) geometric models from preoperative imaging and is capable of aligning them to live 2D fluoroscopic imaging during interventions, e.g., to provide enhanced navigational cues and a native 3D understanding of the spatial relationships between instrumentation and patient anatomy. Thistechnology can help achieve greater levels of safety during procedures, shorten procedure durations and time spent under anesthetic, leading to reduced radiation exposure for both the patient and interventional team, ultimately improving outcomes while reducing the risk of complications. FIG.2A illustrates a typical example of a 3D Computed Tomography Angiogram (CTA) acquired preoperatively that can be used with the technologies described in this specification. FIG. 2B shows a model of the complex network of blood vessels embedded within the brain. Such a model can be used with the technologies described in this specification.

[0070] The opportunity to improve visualization and understanding of imaging during interventional radiology procedures is of particular importance, since current practice necessitates viewing of multiple images (e.g., bi-planar fluoroscopy) in relative positions and orientations quite different from those in effect during physical acquisition. Accordingly, the introduction and navigation of wires, catheters, balloons, stents and other devices within networks of tortuous blood vessels is very challenging. Within the sub-specialty of INR, the device scope extends to mechanical thrombectomy for the treatment of stroke, the deployment of stents and coils to prevent aneurysm rupture, and injection of liquid embolics for the embolization of arteriovenous malformations and tumors. FIG. 3 illustrates an accurate spatial registration (alignment) of the 3D vessel model, built using Aquila, to the two X-ray projections embodied within its virtual model of the fluoroscope.

[0071] Preprocedural planning, and, in particular, the rapid and optimal selection of C-arm angles and positions, is an important factor influencing the outcomes of interventions and postprocedural patient quality of life. In the case of acute ischemic stroke, every minute and second counts while affected regions of the brain are starved of oxygen, potentially leading to irreversible deficits. In the case of cerebral aneurysm coiling, certain cases are deemed to be impossible, or at least unsafe to perform, on account of the orientation of the target anatomy, and the physical constraints imposed by the C-arm geometry, patient body and scanner table. Early identification of such cases must take place to ensure that the patient receives the most appropriate treatment in the first instance, ensuring when applicable that radiographic projections are kept to a minimum, making sure the patient dose is as low as possible.

[0072] Aquila provides a means for both capturing desired fluoroscopic views in both frontal and lateral directions, and recalling them at a later time intraoperatively, taking into account the position of the patient relative to the scanner. Possessing an accurate model of the scanner’s kinematic chain and the ability to perform virtual X-ray projections, the interventionalist is afforded the opportunity to plan, rehearse and review aspects of procedures in advance, that until now have not been possible.

[0073] The restrictions imposed during the COVID-19 pandemic led to the rapid take-up and adoption of software applications facilitating remote audio-visual communication. It is therefore now taken as read that appropriate software medical devices will embody such communication natively and allow interventionalists to involve colleagues who are located remotely, or medical students for educational purposes, sharing with them relevant data feeds available to the team onsite. Remotecommunication can be established with other participants using Aquila. Spatial manipulation of data is mirrored across devices, irrespective of their physical location, through messaging across the local and wide area networks.

[0074] The intended site of application is a suitably equipped angiosuite or operating theater. Aquila comprises software only, and therefore there are no issues concerning contact with tissue, and the device can be described as non-invasive. Use of the device is contra-indicated when no preoperative or intraoperative imaging is available, or if such imaging is of a very poor quality. In terms of frequency and duration of use, the device can be deployed per patient and can typically be used once prior to and once during their intervention. It can run for the duration of the intervention, but typically visualizations may be referred to only at certain key stages.

[0075] With regards to the mode of action, the device comprises software running in a client / server architecture, so that medical images and models derived from such images can be viewed both within a stereoscopic head-mounted augmented reality (AR) display device, and also on traditional tablet and workstation form factors, where native and web-based client applications are supported. These clients act as the user interface for spatial manipulation of data and overall device control. All client implementations support connectivity between multiple devices, such that remote users can participate in audio-visual communication sessions, interact with each other, and appreciate a shared visualization experience. The server has a real-time image capture capability, such that live medical imaging (e.g. fluoroscopy) can be processed, and subsequently viewed on client instances.Operation

[0076] Described in this specification are technologies including systems and methods for obtaining medical images, generating computational models, and sharing images and models between users of various electronic devices. Example implementation of the Aquila pipeline and its components and / or ancillary systems is described below, e.g., in Demonstrative Example 1.

[0077] The device receives medical image data by way of, e.g., two separate modalities. The first of these occurs during the preoperative phase and typically comprises 3D scan data (e.g., CTA of the head and neck). To import such data, the device implements a DICOM node, such that image instances can be ‘pushed’ from the source imaging system, typically the PACS archive within the host medical institution. The device parses incoming image metadata such that it can display image sequences to the user, and also reconstruct the full 3D volume with the correct geometry.

[0078] In order to achieve accurate registration of patient 3D imaging to intraoperative fluoroscopy, a geometric model of the relevant anatomical structures (e.g., blood vessels) is required, such that it can be used to generate virtual projections for pixel-wise comparison. To that end, an example automated segmentation pipeline can be used, including the following stages:a) Query imaging series from DICOM node and sort slices along principal axisb) Apply optional 3D maskc) Apply voxel threshold(s) - lower bound and optional upper boundd) Generate mesh (comprising vertices and triangular faces)e) Merge vertices within specified distance tolerancef) Trim meshg) Perform decimation (i.e. reduction in number of faces)h) Apply smoothingi) Export mesh in OBJ format

[0079] Mesh generation is accomplished using an implementation of, e.g., the Marching Cubes algorithm. The algorithm proceeds through each voxel in the image, considering the eight comers of each such cube, determining the polygon(s) needed to represent the part of the isosurface that may pass through this cube. The algorithm typically results in there being a number of duplicated vertices, and these are merged using a default tolerance. The trim operation considers each disjoint section of the overall generated mesh and removes any such section should it lie entirely within a sphere of the specified radius. This operation is designed to remove any residual geometric ‘noise’ from the resulting mesh (e.g., high intensity imaging artefacts).

[0080] Thereafter, the pipeline continues with the decimation step. This is necessary to ensure that subsequent renderings of the geometry (e.g., during 2D / 3D registration) can occur in an efficient manner. The approach is to reduce the number of mesh faces considerably, while preserving the salient anatomical features. The pipeline employs an implementation of the Quadric Edge Collapse Method, which uses iterative contractions of vertex pairs to simplify models, maintaining surface error approximations using quadric matrices.

[0081] The discrete nature of the underlying volumetric data often results in a non-smooth appearance of the resulting models, despite the best efforts of the marching cubes algorithm.Therefore, as a final step in the pipeline, the mesh is optionally subjected to a number of iterations of the HC (Humphrey’s Classes) Laplacian algorithm, which avoids the well-known problems of deformation and shrinkage caused by other smoothing methods.

[0082] The rotational degrees of freedom afforded by the C-arm mechanical structures are not those which one would ideally want to use to select optimal viewing angles for anatomical targets, since the concatenation of separate rotations is order-dependent, and the result of one rotation may depends on others in the kinematic chain. The software facilitates the selection of optimal views through the considerably more intuitive ‘rolling ball’ rotation interface, where the user makes a sequence of rotational adjustment to the target itself in an independent manner. In effect, the viewpoint remains fixed. The desired transformations are persisted preoperatively as a direct relationship between the models being viewed and the viewport of the user, independent of C-arm geometry, such that when required they can be recalled at a later time once the spatial relationship between the patient and the C-arm coordinate frames has been determined.

[0083] The second imaging modality through which data is consumed is live video, i.e. the frontal and lateral fluoroscopic projections captured in real time during procedures. Rather than digitize the two views separately as independent feeds, the server application, by way of a high-resolution PCI capture card, receives the entire image displayed on the fluoroscope monitor, such that the required regions can be cropped as necessary. Moreover, this method also makes OCR processing possible such that the approximate geometric state of the scanner can be determined, to act as the starting point for subsequent registration optimizations.

[0084] The purpose of 2D / 3D registration, i.e. the alignment of intraoperative 2D fluoroscopy and preoperatively acquired 3D imaging, is to facilitate guidance and navigation during procedures. Given an accurate virtual model of the fluoroscopic device, and with it an ability to produce virtual X-ray projections, when such projections are made to be coincident with the live projections through the setting of the model parameters, a registration can be said to have been achieved. Assuming the registration is accurate, this means that the spatial relationship between the patient’s 3D image coordinate frame and each of the X-ray emitters and detectors is known and, for example, can be used to triangulate locations in 3D space given correspondences in 2D, and perform other geometric calculations, e.g. full 3D reconstructions.

[0085] There are additional degrees of freedom resulting from that fact that the area of patient anatomical interest (e.g., the head) may not be in the same position and orientation when comparing the preoperative and intraoperative settings. For instance, the head may be facing straight up during the initial CT scan, but tilted to one side during later fluoroscopic acquisitions. Therefore a 3 -DOF translation and 3-DOF rotation (yaw-pitch-roll) are appended to the table kinematic chain. These are quantities which must be determined as part of the so-called “patient registration” process.

[0086] The process amounts to finding the values of the degrees of freedom such that real and virtual X-ray projections are aligned accurately in a pixel-wise fashion. A two-stage process is proposed, where first the additional degrees of patient freedom and the longitudinal L-Arc gantry position are determined approximately, using a point-based registration method. Secondly, the initial registration is refined using an image-based comparison metric. The advantage of this method is that, while the patient’s head remains in a static position and orientation, the first stage need only be performed once.

[0087] Subsequently, the registration proceeds as a series of optimization updates. Certain parameters are known and can be inferred from the video feed which represents the real-time output of the fluoroscopy device (e.g., the arm angles, source-to-image distance and effective flat detector size), but only to a limited degree of accuracy. To effect point-based correspondences, clinical-grade CT-compatible fiducial markers are attached to the patient at well-appreciated anatomical locations. In the prospective human setting, for practical reasons the markers are attached only prior to the procedure. However, their geometric locations within the 3D scan coordinate frame are determined in advance. The patient registration proceeds through projecting the fiducial markers onto the respective virtual detectors and comparing their location with the actual fluoroscopic projections. The requiredvalues of the 7 degrees of freedom are determined using Levenberg-Marquardt nonlinear least-squares optimization over the X and Y displacements for each projection.

[0088] Once patient 2D / 3D registration has been achieved, and the relationship between the C-arm and patient anatomy, i.e. the original 3D scan coordinate frame, is known, previously selected optimal viewing angles can be recalled when required. This process amounts to the software iteratively changing the C-arm angles within its model of the kinematic chain until the desired view, expressed as the direct relationship between the scan frame (in which the anatomical models exist) and the viewport, is achieved. Furthermore, the software attempts to position the projection of the isocenter of the scanner to the exact centers of the respective X-ray projections. The required degrees of rotational and translational freedom are again optimized using, e.g., the Levenberg-Marquardt nonlinear leastsquares method.

[0089] Registration refinement proceeds through the repeated invocation of image-based updates, for example in response to the movement of one or more of the imaging arms. Here, the entire model of the anatomy (e.g., vasculature, e.g., of the brain, and / or the biliary system) is projected onto the respective imaging planes, such that the degree of projection overlap can be counted in a pixel-wise manner. An optimization technique, e.g., a technique that combines Simulated Annealing and the downhill simplex method of Nelder and Mead, is employed, since the dimensionality of the problem increases, and partial derivatives of the comparison metric are not available. With an accurate registration and known correspondences in both frontal and lateral 2D images, triangulation can be used to determine the equivalent point in 3D space.

[0090] Triangulation is a useful tool for providing navigational cues natively in 3D, however it presupposes that correspondences are readily available in the frontal and lateral views simultaneously. Such features could be identified manually by the operator or member of the interventional team during a procedure, but this would introduce an undesirable delay, which would be of particular concern in certain time-critical scenarios. To this end, an automatic guidewire tracking capability is introduced, such that the tip of the wire, which is otherwise featureless, can be located in both frontal and lateral images in real time. The guidewire is chosen in the first instance since it typically leads the way as the operator navigates further and further into the patient’s vasculature. Other instruments, including hollow catheters for example, are designed to follow the existing path.

[0091] Two example methods are described, which differ in the way the raw input images are pre-processed in advance of running the actual tracking algorithm. The first, referred to as the “Reference Image” method relies on the capture of a specific reference image from which subsequent images containing the target guidewire are subtracted, revealing the wire alone. This method produces reliable results but is not ideal on account of it needing to maintain state. In contrast, the second method, named here as “Vesselness”, is stateless and convolution-based, relying on eigenvalue analysis of the Hessian matrix in multiple Gaussian scales. This method has the distinct advantagethat it does not require another image to act as a reference and can therefore be applied at any point in time without the inconvenience of having to first acquire images in which no wires are present.

[0092] While knowledge of guidewire tip locations in 3D is useful as a navigational cue, it is possible to extend the triangulation method such that, under reasonable assumptions, the entire visible length of the guidewire can be reconstructed in 3D for subsequent visualization. Since guidewires are typically featureless, only the tip can be used as a guaranteed correspondence. In general, distances measured along the frontal and lateral guidewire projections do not have a straightforward relationship.

[0093] Shape in 3D can be recovered by fitting centripetal Catmull-Rom splines to the respective projected images of the guidewire, and then discretizing the splines to produce sets of points (default of 1000 points in each set) equally spaced with respect to the ID spline parameterization. The correspondences between such points are not known. Rather, a map is produced where all rays back-projected from the frontal points are compared against all rays back-projected from the lateral points.

[0094] While to a reasonable approximation the set of points in 3D that could give rise to the respective projections becomes known, such a set does not necessarily result in a unique path from guidewire tip to the proximal end of the wire. Therefore, a disambiguation step may be required. Making the assumption that no closed loops are observed in the frontal and lateral projections, the ray comparison process in general will result in a tree-like structure within the resulting map. This structure represents the fact that there is no single unique linear shape that gives rise to the respective projections. The disambiguation step proceeds by employing Djikstra’s shortest path algorithm to the weighted graph derived from the map (where edges are weighted with the passing distances of rays), starting at the tip in the top-left comer, and terminating on either the right or bottom edges, which represent the most proximal points visible in the projections. The resulting path defines the unique set of correspondences that make up the length of the guidewire.

[0095] The technologies described in this specification extend beyond 3D model generation and 2D / 3D registration and provide capabilities to share data and interact with multiple users - a collaboration tool for the medical community in the same vein as Zoom is a collaboration tool primarily for the business community. In order to facilitate such interaction, e.g., tele-mentoring, and overall use of the device in collaborative fashion, the software supports remote audiovisual communication via third-party WebRTC APIs using an SFU (Selective Forwarding Unit) topology. Furthermore, messages are exchanged over a parallel data channel such that application state is synchronized when multiple users are active. For example, when a new patient data set is selected, or when a user spatially manipulates a model within the scene, these precise changes are replicated across all other currently connected clients. In an implementation, changes in spatial orientation of a model (e.g., a 2D / 3D model) by one user is broadcast to other participants resulting in all models moving in sync. The system can include one or more software tools that manage data traffic between users, e.g., to prevent multiple users from attempting to manipulate an image at the same time.

[0096] XRAI Connect can operate as a stand-alone product to facilitate general collaboration, but also can be an integral part of other XRAI suite components supporting interactions across the patient journey. This technology can support multidisciplinary team (MDT) meetings for surgical planning and / or, e.g., interventional radiology (IR) procedures during intervention. Moreover, the technology can be used in clinical teaching / training situations.

[0097] In addition to connecting users via audio and video channels in a manner that is familiar to many users (e.g., Zoom, Skype), XRAI Connect’s uniqueness is its ability to integrate data sources into the collaboration space through the use of data channels. Sources can include a variety of information types, such as DICOM fdes and 3D models, as well as streams originating from medical devices, e.g., fluoroscopy, ultrasound, and / or heart monitors, as well as combinations thereof, e.g., registered 2D / 3D images and models. Source information can be routed to destinations that consume that information, such as AR headsets and / or browsers on desktop and tablet computers. Computed information derived from one or more sources, such the position of a guide-wire head determined from a fluoroscopy stream, can be broadcast to users, e.g., in order to update the positioning in a 3D model. This technology provides visibility into real-time tracking to both local and remote users. A user with an augmented reality headset can act as both a source and consumer of information. In an example implementation, if the forward-facing camera on a headset of a consumer is activated, what this consumer sees can be broadcast to other users (i.e., the consumer generates data that serves as input into the data stream consumed by other users). Thus, a remote person can see what the person with the headset sees. In another implementation, a user can manipulate a (shared) image, e.g., zoom in on an area of interest or change a viewing angle, and the system will update the new view on the screens (e.g., headset or tablet) of each user. Such a concept can be applied to multiple scenarios, e.g., a trauma surgeon guiding an EMT in the field or a remote consultant providing guidance to an IR in the cathlab.Example Workflow

[0098] FIG. 4 illustrates information flows during typical usage for the preparation of preoperatively acquired imaging. The typical sequence of operations flows from the top of the figure to the bottom. Horizontally, the columns represent the subcomponents of the Aquila device / pipeline, which communicate with each other. The “Orthanc” component is the DICOM node implementation (a local DICOM image store), which receives incoming data. The adjacent “Ariadne” component (an interface module) accommodates image selection and preliminary 2D visualization and allows the user to coordinate data preparation. The “FluoroHub” component (a server module) provides an interface which Ariadne invokes to perform image segmentation. Once generated, processed patient data is uploaded to “Azure” cloud storage (a cloud module).

[0099] During typical example usage, the following sequence of events normally occur:1. In the first instance, a CT operator acquires the images with a CT scanner and the images are pushed to the PACS system.2. The PACS operator pushes the study from the PACS system to Orthanc.3. A clinician (Ariadne user) starts Ariadne and requests metadata from Orthanc.4. Orthanc send metadata to Ariadne .5. A clinician using Ariadne requests a study from Orthanc.6. A study is downloaded from Orthanc to Ariadne.7. The study is automatically saved to IndexDB locally.8. The clinician using Ariadne may request a dataset from IndexDB.9. Ariadne will load a requested dataset from IndexDB.10. The user will initiate the segmentation process that is performed by FluoroHub.11. FluoroHub will send a request to Orthanc to retrieve the relevant imaging data.12. Orthanc will send the imaging data back to FluoroHub.13. FluoroHub will send the segmented model (mesh) to Ariadne.14. The user will request that the mesh and patient configuration should be uploaded to Azure cloud storage.15. FluoroHub will send the mesh and configuration (i.e. patient) to Azure storage.16. The Aquila client user will request a patient to be downloaded to the headset / tablet device.17. The mesh is downloaded from Azure storage to the Aquila client instance.

[0100] FIG. 5 illustrates information flows during typical interventional usage, following the data preparation stage (either as quickly as possible in the case of stroke, or at some later time). The “CT Fluoro” column represents the source fluoroscope from which live X-ray images are received. The two rightmost columns represent the available client applications, running typically on tablet and headset hardware form factors, respectively. As before, the sequence of events runs from the top of the figure to the bottom. The sequence is split into a preoperative phase, where the user chooses optimal C-arm viewing angles without the need for live imaging from the fluoroscope, and the intraoperative phase, where the software relies upon such a feed.

[0101] During typical usage, the following sequence of events will normally occur1. In the first instance, patient configuration and model files are downloaded from Azure cloud storage.2. The clinician uses the tablet instance of the client application (e.g., a tablet client module) to select desired C-arm views.3. The clinician identifies 3D anatomical landmarks in the preoperatively acquired 3D image data in anticipation of patient registration.4. The 3D landmarks and selected C-arm views are uploaded to Azure cloud storage.The patient configuration (now including 3D landmarks and selected C-arm views) is downloaded onto the tablet client - this may be a different device to that used in the earlier steps.Patient configuration is downloaded to FluoroHub, the server application which implements the registration and tracking functionality invoked from one or more clients.FluoroHub captures the incoming video stream (comprising live X-ray images) from the source imaging device.The live frontal and lateral projections are cropped out and streamed to connected clients (e.g., headset client module) for immediate visualization.The user identifies landmarks in the respective 2D images corresponding to those chosen earlier in step (3).The user initiates patient registration.The 2D correspondences are sent to FluoroHub for processing.The patient registration optimization is performed by FluoroHub, resulting in an approximate alignment of preoperative and intraoperative imaging. The position and orientation of the patient on the fluoroscope table are thus determined.The resulting registration (values of the fluoroscope kinematic chain parameters) are distributed to connected clients.The clinician wishes to visualize the patient models and initial registration natively in 3D, and therefore launches Aquila on headset hardware. Patient configuration is downloaded to the device.The registration performed in step (12) is sent to the new client instance.The live frontal and lateral projections are streamed to the new client instance.The user visualizes the data in the headset.The user selects a predetermined desired viewing perspective using the tablet interface. The request is dispatched to FluoroHub over the local network.FluoroHub calculates the required fluoroscope settings (i.e. C-arm angles and table position) in light of the earlier patient registration.The results are distributed to both tablet and headset client instances.The user initiates further registration (i.e. angle) refinement.The request is dispatched to FluoroHub over the local network.FluoroHub performs optimization with an image-based metric to determine refined C-arm angles and source-to-image distance for each detector.These results are distributed to both tablet and headset client instances.The user initiates guidewire tracking.The request is dispatched to FluoroHub over the local network.FluoroHub tracks the extent of the guide wire in both frontal and lateral images.29. The 2D tip coordinates in the respective views are streamed to connected clients, such that they can perform 3D tip triangulation.30. The user initiates reconstruction of the full 3D guidewire extent.31. The request is dispatched to FluoroHub over the local network.32. FluoroHub performs the reconstruction and disambiguates and non-unique parts of the solution.33. The reconstruction results (sequence of points in 3D space) is sent to all connected clients for 3D visualization.

[0102] In some implementations, the processing undertaken by FluoroHub employs standard algorithms (e.g. the marching cubes method), or standard optimization techniques (e.g. the Levenberg -Marquardt nonlinear least-squares method, the downhill simplex method of Nelder and Mead, and Djikstra’s shortest path algorithm). In an example implementation, the method used for guidewire tracking is bespoke to Aquila. It has two distinct phases: a) image preprocessing, and b) tracing of the guidewire from its start position within each frame to the location of the tip. Since the guidewire is featureless, the tip is the only unique element common to both frontal and lateral images.

[0103] In some implementations, the following preprocessing methods can be used. The “Reference Image” method subtracts a chosen reference frame (typically devoid of the guidewire), from the current frame, thereby revealing the guidewire with a high degree of contrast. The “Vesselness” (or “Frangi”) method works by enhancing vessel-like structures. After pre-processing, the starting point for the guidewire is determined at the image edges. Thereafter, a search is performed in a circular manner around the current point to determine in which direction the algorithm should take a small step. This process is repeated until the tip of the guidewire is reached. Any disjoint segments are connected to form a continuous representation of the wire. FIG. 6 shows the algorithmic flow.

[0104] In some implementations, the technologies described in this specification implement the image processing and registration techniques described in detail in Demonstrative Example 2. In an example implementation, the technologies described in Demonstrative Example 2 are implemented by one or more components of the XRAI Connect platform and / or the Aquila pipeline, e.g., on FluoroHub.Example Device Implementations

[0105] As described above, three major components of the system described in this specification are the Fluorohub, Ariadne, and Aquila components (See, e.g., FIG. 1).

[0106] Fluorohub is a server module that provides real-time video capture from a DVI video source (e.g., up to 4K resolution). This module can broadcast video, commands, and / or actions to connected clients (e.g., Aquila clients as described below). This module further provides, e.g., local storage for on-premises data storage. Flurohub performs, e.g., C-arm angle optimization, registration in thecontext of neuro-intervention, Al-based segmentation and model generation, instrument 3D reconstruction, instrument tip triangulation, and / or Al-based instrument tracking. Flurohub can be implemented on a Windows, Linux, Unix, or other such platform.

[0107] Ariadne is an interface module that is implemented as a web / browser-based tool and supports XRAI Connect (audiovisual conferencing, with shared data visualization and annotation). This module provides a front-end for a data preparation pipeline for various use-cases, e.g., neurointervention (e.g. automated segmentation and 3D model configuration for registration). Ariadne provides authentication and connects to LiveSwitch / Agora WebRTC. The module can query imaging data from local Orthanc and allows selection of studies / series. Via the Fluorohub API, Ariadne can perform automatic segmentation and permits setting of fiducial positions. The module can create new patient instances and uploads to Azure storage. Moreover, Ariadne can provide remote sharing of cursor positions.

[0108] The Aquila software can be implemented on various hardware platforms and can provide different functionalities depending on said platforms.

[0109] Aquila can be implemented on the Microsoft HoloLens platform (AquilaHL), on the Magic Leap (AquilaML) or on the Apple Vision Pro (AquilaAVP) app. Aquila can also be implemented as an app on the Microsoft Windows platform or similar (iOS, Linux, Unix) (AquilaApp). In all implementations, the Aquila software supports XRAI Connect and provides rendering collections of 3D models and a gesture / speech user interface. The software supports remote collaboration: receiving live-streamed input (e.g., from fluoroscopy) and, in some implementation, providing specific support for neuro-intervention procedures, including virtual fluoroscopy, registration, interaction with simulation results, and visualization of tracking output. In some implementations, e.g., in AquilaAVP, the Aquila software provides additional support for cinematic volumetric rendering and direct DICOM import, plus connectivity to 3rd party data back-ends.

[0110] In some implementations, AquilaHL, AquilaML, AquilaAVP, and / or AquilaApp modules are configured to perform one or more functions including executing a kinematic chain model of generic C-arm geometry, transmitting actions or commands over local network (User Datagram Protocol -UDP) and / or WebRTC, performing instrument tip triangulation, and / or performing animation of C-arm geometry updates.

[0111] In some implementations, AquilaHL, AquilaML, and / or AquilaApp modules are configured to perform one or more functions including rendering the 3D models and incoming live video feeds, providing support for scene graph hierarchy, and / or providing support for persisted scene states. The AquilaApp can also be configured to provide spatial anchoring (co-registration).

[0112] In some implementations, the AquilaAVP module is configured to perform one or more functions including visualizing tri-planar greyscale images, performing cinematic volumetric rendering of the 3D model, providing gesture control for spatial manipulation, and / or live-streaming fluoroscopy video.

[0113] In some implementations, the system includes an Aquila Capture module, implemented as an app on the Microsoft Windows platform or similar (iOS, Linux, Unix), which injects additional video feeds into collaborative sessions (built-in and USB-based cameras). This Aquila Capture module connects to LiveSwitch / Agora WebRTC, provides authentication capabilities, and / or queries connected cameras and initiates live streaming.

[0114] FIGS. 7A-7C illustrate example data flows between system components. FIG. 7A illustrates an example process of sharing of live medical imaging, e.g. fluoroscopy, in high-resolution / lossless format. This type of information sharing is distinct from the type of video one might share during a Zoom / MS Teams call. Data can be streamed from a local server or from the WebRTC (e.g., Agora).FIG. 7B illustrates an example process of lower quality video sharing for the purpose of a collaborative session. The video stream can include the Apple Vision Pro Persona or video from a front-facing camera. This sharing is performed through the WebRTC (e.g., Agora). FIG. 7C illustrates an example process of messaging between various system components in order to maintain a synchronized state. Data can be streamed in broadcast mode through a local server or through the WebRTC (e.g., Agora).

[0115] Similar to FIG.l discussed above, FIG. 8 illustrates an example network topology that an example installation of Aquila can use. Since remote communication is supported and the device utilizes cloud storage, the diagram is split into two halves, representing the LAN (shaded) within the host institution and the WAN outside the institution firewall. The network diagram shows how live fluoroscopy is streamed to Aquila client instances over a dedicated local Wi-Fi network. The “Aquila Capture” component is a separate send-only client that can be used to inject additional video streams (e.g., from a fixed camera) into remote communication sessions. With regards to such sessions, the LiveSwitch component represents an example third-party service and APIs used to initiate WebRTC connectivity. In some implementations, the WebRTC module is the Agora cloud module described herein. The “Identity (AD)” component, i.e. Active Directory, represents the cloud service used to authenticate users.

[0116] While the Aquila client / capture applications, and also FluoroHub (during intraoperative use), are intended to be run on specific platforms natively, Ariadne is implemented as a web-based application to maximize platform compatibility and facilitate widespread deployment beyond the angiosuite. It is deployed using “Docker”, a set of platform -as-a-service (PaaS) products that use OS-level virtualization to deliver software in packages known as containers. The resulting software stack is therefore more involved than the components deployed local to the angiosuite and is further described here. FIG. 9 illustrates an example relationship between the software components comprising Ariadne, and their location within the overall software stack.

[0117] Typically, the software is installed and executed in three distinct locations: The first, FluoroHub (1.0), is the server application that typically runs on server hardware local to the angiosuite. The DICOM node Orthanc (1.1) also runs on this server. The second, Ariadne (2.0), isused in the preoperative setting to prepare patient data for the intraoperative phase of procedures and is accessed via a web browser. This application can be run on the server running FluoroHub or can be run on a separate machine. The third, the Aquila client software (3.0), is run on an AR headset device (note that the window-based Aquila client is omitted from the stack diagram for brevity, but occupies an equivalent role).

[0118] The following software environments can be used for deployment:

[0119] FluoroHub (1.2) and Orthanc (1.1) are run within a Docker container within a virtualized operating system, Ubuntu (1.3), that is running within an operating system, namely Microsoft Windows or MacOS (1.4) on a physical PC or Mac (1.5) device. Alternatively, they can be run directly within Ubuntu on the PC without Windows or MacOS providing a wrapper operating system.

[0120] Ariadne (2.0) provides the user interface for data preparation in the preoperative setting. The application logic (2.0.1) controls a variety of code modules including the user interface (2.0.2) that makes use of the 2D and 3D graphics engine (2.0.3). The application logic also handles messages originating from user input (2.0.4) from the mouse / trackpad and keyboard, communication with an interface (2.0.5) to Orthanc, communication with an interface (2.0.6) to the FluoroHub server, DICOM loading and parsing (2.0.7), and the provision of an interface (2.0.8) to IndexDB for local storage functionality.

[0121] Ariadne can run within a web browser (2.1), such as Chrome, Safari, Edge, or Firefox that has an engine (2.1.1) to process HTME (2.1.0), CSS (2.1.1) and JavaScript (2.1.2), and manage IndexDB (2.1.3). The web browser runs within an operating system (2.2) (either Windows or MacOS), providing APIs that allow the web browser to use different components of the system including the mouse / trackpad (2.3.0), network interaction (2.3.1), SSD storage (2.3.2), the graphics processing unit (2.3.3) and ultimately the screen (2.3.4). The operating system is run on a PC or Mac computer (2.3).

[0122] The Aquila client headset software (3.0) runs on the bespoke holographic operating system (3.1) specialized for the device (3.2), and communicates with FluoroHub over a local Wi-Fi network (3.1.0). The C-arm (4.0) provides live X-ray images for FluoroHub consumption.

[0123] FIG. 10 shows an example of the Aquila client in action during a ‘prospective’ bench test of patient registration in the angiosuite. While the patient head (also visible on the main fluoroscope monitor in the background) is a phantom model, the test was carried out under live conditions - that is, the video feed was captured contemporaneously from the fluoroscope, as it would be during an actual patient case, rather than being played back into the system as a pre-recorded video. Within the device user interface, it is possible to see that the registration has resulted in good alignment between the real and virtual X-ray projections in both the frontal and lateral views.Demonstrative Example 1

[0124] The following description and screenshots illustrate an example use of the system described in this specification.

[0125] Step 1 (FIG. 11) - Log into Ariadne Software. This screenshot illustrates the situation where the user has already authenticated with Ariadne and can therefore log into the application immediately.

[0126] Step 2 (FIG.12) - Search for patient study and download from PACS to Local Store. The user enters the patient’s name into the Search Criteria panel. Subsequently, the required study is selected and 30 image series (which have been pushed from PACS) are downloaded from the DICOM node to the Local Store. The imaging preview window is on the righthand side of the screen.

[0127] Step 3 (FIG. 13) - Retrieve images from Local Store. The images are retrieved from the Local Store and the frame slider is used to view the sequence of images in the correct spatial ordering. Relevant patient metadata is displayed alongside the image pane.

[0128] Step 4 (FIG. 14) - View DICOM images with series and frame navigation and brightness / contrast adjustment. Frame and series navigation is demonstrated in addition to brightness and contrast adjustment, where the user can set the lower and upper bounds for pixel intensities. In this specific instance, the contrast-enhanced vessels are isolated from the background anatomy.

[0129] Step 5 (FIG. 15) - Select image series for segmentation. The desired image series is selected for segmentation. The user is afforded the opportunity to check the chosen series visually and confirm the key metadata fields (e.g., name and patient ID).

[0130] Step 6 (FIG. 16) - Specify segmentation parameters and segment model. The user is presented with a set of default segmentation parameters (e.g., the number of mesh faces to use as the decimation target), which they may adjust if required. Segmentation is invoked by pressing the “APPLY” button. Progress is displayed in the “FluoroHub Output” popup window. The “SHOW RESULTS” button (not shown) advances the process to the next stage.

[0131] Step 7 (FIG. 17) - View and upload segmentation to Azure. The 3D mesh model generated by the segmentation process is visualized and spatially manipulated using, e.g., a ‘rolling ball’ interface. Once the user has approved the model, the “UPLOAD” button is pressed to initiate creation of a new patient instance and upload to Azure cloud storage. The user is given the opportunity to change the patient identifier.

[0132] Step 8 (FIG. 18) - Manual adjustment of fluoroscope parameters. Moving into the intraoperative phase with the window-based Ariadne client, the user sees live-streamed fluoroscopy blended with virtual projections (red), together with a 3D representation of the scanner, table and patient. The user is able to adjust the kinematic chain of each C-arm manually with slider controls and enable / disable rendering of individual models.

[0133] Step 9 (FIG. 19) - Patient Registration. The user initiates a ‘patient registration’ where previously identified 2D and 3D correspondences are used to determine the unknown parameters ofkinematic chain. Registration is performed on the server (FluoroHub) and once complete, the results are sent back to the client. The transition to the new state (i.e. set of kinematic chain parameters) is effected with animation.

[0134] Step 10 (FIG. 20) - Registration Refinement. The user initiates a registration refinement. This takes place on the server (FluoroHub), which returns its results to the client. A small adjustment is made to the orientations of the C-arms and other parameters. The user inspects the frontal and lateral 2D views in turn and sees that the degree of overlap between real and virtual X-ray projections has improved.

[0135] Step 11 (FIG. 21) - Guidewire Tracking. The user zooms in for closer inspection of the patient anatomy and, having enable guide wire tracking, can see the estimated location of the guidewire tip based on the current 2D / 3D registration and tracking of the guidewire in both frontal and lateral 2D projections. The relatively small distance of closest passing of the two back-projected rays indicates that the triangulation is accurate.

[0136] Step 12 (FIG. 22) - Guidewire 3D Reconstruction. The server (FluoroHub) has reconstructed the full extent of the guidewire, returning it to the client as a sequence of points in 3D space for subsequent visualization. The resulting representation of the guidewire (in green) can be seen to follow the vessel containing the guidewire closely, but not perfectly. The latter is generated from a preoperatively acquired image, and a certain degree of deformation will be expected in the live setting, induced by the physical stiffness of the guidewire when inserted.

[0137] Step 13 (FIG. 23) - Preoperative Selection of Viewpoints. Two independent ‘rolling ball’ interfaces with 3D crosshairs are used to select desired viewpoints and origins for alignment with the fluoroscope isocentre. Incremental movements are translated immediately into changes to the actual degrees of freedom afforded by the fluoroscope kinematic chain. Desired views are named and saved for future recall.

[0138] Step 14 (FIG. 24) - Intraoperative Recall of Viewpoints. Having requested the recall of existing saved viewpoints (i.e. both frontal and lateral views), the user selects the example “Aneurysm” as the anatomical target. The server (FluoroHub) computes the necessary C-arm angles and other kinematic chain parameters in light of the prevailing patient position and orientation and returns the results to the client. The software animates the transition to the new state.

[0139] The technologies described in this specification can have additional or alternative implementations or configurations of one or more components or modules described. For example, applications can web-based or installed on a local server. Moreover, the degree of automation of each process may vary, e.g., depending on the application. An example implementation of a system as described in this specification is shown in FIG. 25.

[0140] The technologies include a solution to support, e.g., neurovascular interventions, in particular mechanical thrombectomy and the treatment of cerebral aneurysms. In some implementations, these procedures are backed by an Al model to support the pre-processing of CTA-sourced data inpreparation of 3D reconstructions. The technology can also run a real-time Al model to track the guidewire progress in bi-planar fluoroscopic data streams. In some implementations, the technologies described here can support other clinical areas, e.g., ERCP procedures. The technologies used include models for (i) MRI (MRCP) image segmentation, (ii) video analysis of fluoroscopic video streams, (iii) video analysis for adverse event prediction. The processes can also include (i) optimizing imaging protocols to enhance 3D reconstruction, (ii) simulating multiplanar acquisition with sequential capture for 3D triangulation. In some implementations, the technologies use ultrasound as a diagnostic imaging modality to broaden the technical scope of XRAI Connect.

[0141] In some implementations, the XRAI Connect system includes the following modules: The XRAI Controller (a controller module) is central to the operation of XRAI Connect and is used for controlling the distribution of information. XRAI Connect can implement web-based “Controller” functionality that can manage what information gets routed to which participant. The XRAI Client can be configured as an app running on desktops or in browsers for performing user functions. The XRAI Client includes User functions running in an augmented reality device. The XRAI Connect provides core functions to manage Account and Session instances. The XRAI Connect can provide file management functions by interacting with DICOM repositories from 3rdparty, cloud-based imaging providers. The XRAI Connect can provide channel management functions by interacting with 3rdparty, cloud-based media provider (Agora, a cloud module). Another example workflow using the technologies described in this specification includes the following steps implemented in system comprised of a subset of the system components described above (FIG. 26):1. A client has obtained an MRI scan of head.2. A radiographer pushes DICOM images from Local PACS (on premises) to DICOM Store (Orthanc)3. An (first) instance of FluoroHub polls DICOM Store to check for new data.4. FluoroHub picks up new data and performs Al-based segmentation of brain and vessels. 5. FluoroHub creates 3D mesh representations and pushes them to the Local Store (of patient data sets).6. Headset client applications are launched and connect, e.g., over WiFi to a separate instance of FluoroHub.7. A second instance of FluoroHub broadcasts state update messages to connected clients to maintain synchronization.8. The client applications poll the second FluoroHub instance and receive notifications for new data sets.9. New data sets are downloaded onto the respective clients.10. The shared, immersive consultation begins - client and radiologist share the same visual experience.1Demonstrative Example 2

[0142] The following description and screenshots illustrate an example use of a system and methods for guiding surgery as described in this specification. The systems and methods can be integrated into a surgical collaboration system for interactive sharing of medical image data as described above, e.g., to provide 2D imaging and 3D model rendering and registration.

[0143] The use of medical imaging to support a medical procedure is widespread. Such medical imaging may be performed externally to the body (such as by X-ray or ultrasound) or internally to the body (such as endoscopy or ultrasound, e.g., intravascular ultrasound). The use of external medical imaging may support a clinician in navigating a tool through a luminal network while avoiding or reducing the need for an incision into a patient.

[0144] Known medical imaging systems include magnetic resonance imaging (MRI) and computed tomography (CT) X-ray imaging, among others. Although such imaging systems are able to produce detailed 3-dimensional (3D) images, they are mainly used in a pre (or inter) operative context rather than in an intra-operative context. There are a number of reasons why 3D MRI and CT imaging are less suited to use in an intra-operative context. For example, 3D imaging systems may be physically obtrusive and hard to reconcile with the locations of surgical staff around a patient. Such 3D imaging systems are also expensive, and it may not be cost-effective to deploy them for the duration of a medical operation (especially if the imaging is only used for a portion of the medical operation). Another consideration, particularly with regard to CT imaging, is to reduce the exposure of both surgical staff and the patient to X-ray radiation.

[0145] It is known that 3D images produced by inter-operative imaging may not be completely accurate with respect to the current state of a patient during a subsequent surgical operation. For example, a patient might experience some physiological change between the acquisition of a 3D image and the time of the surgery. The patient might also have a different pose for a medical operation compared with the pose for the 3D imaging (such as being on one side rather than lying on his / her back). This can lead to some changes in the orientation and shape of internal soft organs, which can make them difficult to image accurately. Furthermore, the presence of a medical tool inserted intra-operatively into a patient can also distort the internal soft organs.

[0146] Accordingly, other forms of imaging have been developed to provide real-time imaging in an intra-operative context. These other forms of imaging are typically quicker and less obtrusive than pre-operative MRI or CT imaging. A common approach for performing such intra-operative imaging is referred to as fluoroscopic imaging, which is another form of X-ray imaging. Fluoroscopic imaging involves obtaining one or more 2-dimensional (2D) images, each such image representing an X-ray projection through an object being imaged. In such applications, a pair of fluoroscopic images may be acquired from two different orientations. In some procedures, a time sequence of fluoroscopic images is obtained, while other implementations utilize just a single set of one or more fluoroscopic images taken at a particular time.

[0147] X-ray images are generally formed by transmited radiation (in contrast to optical images, which are generally formed by reflected light). An X-ray imaging system typically has an X-ray source to provide a collimated beam of X-rays directed at an object of interest. An X-ray image detector is located behind the object, facing back towards the X-ray source.

[0148] The X-ray source can generally be regarded as forming an X-ray shadow of the object on the image detector. However, whereas optical shadows tend to have a sharp (binary) contrast between black, when a solid is located between the X-ray source and the image detector, or white, when no such intervening object is present, X-rays have a much greater power to penetrate through material such as soft tissue in the human body. The intensity of X-rays received by the image detector corresponds to the original (source) X-ray intensity reduced by the cumulative absorption of X-rays along the path from source to detector.

[0149] One way of navigating a medical instrument to a desired location is first to navigate a guidewire to the location, and then advance the tool over the guidewire. The guidewire is generally relatively small in cross-section and easy to direct along a particular path, thereby assisting in navigation to the desired location. Once the tip (distal end) of the guidewire has reached the desired location, one or more instruments for performing a given medical task (imaging, biopsy, drug release, ablation, etc.) can be mechanically coupled to the guidewire to allow such tools to be inserted along the same path as the guidewire to the desired location.

[0150] The guidewire is guided by the assistance of fluoroscopic images, which may be acquired during navigation of the guidewire to help ensure that the guidewire progresses towards and then arrives at the desired location. As noted above, each fluoroscopic image is a projection of an object onto a flat surface. In practice, the projected position (shadow) of the guidewire can be identified fairly easily in fluoroscopic images since the guidewire is usually formed of material (e.g. metal) having high X-ray absorption. A single fluoroscopic image does not allow the full three-dimensional path of the guidewire to be determined (reconstructed). However, the 3D path of the guidewire becomes more accessible if multiple fluoroscopic images are obtained using known, different orientations of the fluoroscopic imaging device. Most commonly only two fluoroscopic images are obtained (to minimize X-ray exposure to the patient) and these two images are processed to perform reconstruction of the 3D path of the guide wire.

[0151] It is common in medical procedures to insert a tool into a luminal network of a patient to perform a medical task or intervention such as removal of a blockage, endoscopy, biopsy, diagnosis, therapeutic action, and so on. In such procedures, the tool is navigated via the luminal network to the desired location at which the medical task is to be performed.

[0152] It is important that the tool is tracked within the body so the clinician can navigate the tool to a desired location. For example, in the case of a branched blood vessel network with many junctions, the clinician must be able select the correct path for the tool at each junction in order to reach theintended destination. The clinician must also be able to tell when the tool has arrived at the desired location (rather than overshooting or under-shooting).

[0153] Minimally invasive surgery (MIS), characterized by smaller sized incisions compared to standard surgery, has allowed for great advancements in patient treatment and outcomes over the last twenty years. However, there are significant visibility challenges for doctors performing these MIS procedures. Unlike open procedures, where doctors can physically view targeted anatomy and have a direct line of sight to their tool sets in relation to that anatomy, doctors performing MIS procedures or therapy lose their direct line of sight once their surgical or interventional tools enter the patient’s body.

[0154] In MIS procedures, the clinician depends on intraoperative imaging to provide sufficient information to position their instruments relative to target anatomy. Although multiple types of imaging modalities may be utilized in a procedure room (iMRI, iCT, iUS, etc.), many procedures routinely use mono-planar and bi-planar fluoroscopy (interoperative X-rays) due to timeliness of image acquisition and lower levels of radiation resulting from this modality compared to others.

[0155] Fluoroscopy images typically are displayed on two adjacent off-field console screens as two-dimensional (2D) objects, which is not optimal for the clinician as they need to move their tools in the patient’s natural 3D anatomy. Clinicians therefore rely on the two live stream 2D images to estimate, in real-time, the 3D space of the patient where the procedure is being applied - requiring the clinician to cognitively associate two different 2D live views (e.g., lateral and frontal projections) simultaneously to visualize the needed 3D perspective of their patient’s anatomy to navigate their tools.

[0156] This type of spatio-temporal visualization is more of an art than a science, and not all doctors are equally proficient with the required image reading skills. Interventional radiologists who perform MIS procedures often have more training in reading images than general surgeons. A doctor’s ability to successfully navigate an MIS procedure today is dependent on this skill; some doctors take longer than others trying to visualize 3D anatomy. This visualization needs to be done for each progression of tool movement during a procedure. This can be cognitively straining for a proceduralist and is a time-consuming part of a procedure, which can extend the time to complete a procedure. Introducing 3D imaging into the intraoperative environment can reduce this physician cognitive strain, especially for lesser experienced doctors, and allow them to concentrate on the more important non-navigation parts of the procedure to achieve their operative goals.

[0157] Although 2D fluoroscopy is standard protocol for many procedures, there are still flaws in this technology that prevent the doctor from getting their preferred image angle in a given procedure for a given patient positioning on a procedure table. The fluoroscope is a device mounted on a so-called C-arm configured to rotate the imaging device around a table on which the patient is positioned. This C-arm has limitations in terms of movement and thus limitations in field-of-view. For example, the C-arm cannot rotate a full 360 degrees due to constraints (e.g., patient table,operating room floor, etc.). A doctor may need a straight-line view of their patient, such as from the bottom of their foot all the way through to the top of their head. Such a view cannot be achieved today due to the location and geometry of the fluoroscope’s C-arm. These restrictions can provide visibility gaps in navigation today but can be corrected by the introduction of new virtual 2D fluoroscopy screens that can complement existing live 2D real views today.

[0158] 3D modeling capabilities have been slow to develop and adapt to surgery settings. Although advances in Artificial Intelligence (Al) and Augmented Reality (AR) have allowed several digital surgery software companies to utilize preoperative CT scans to create sophisticated 3D volumetric anatomical renderings, these technologies are generally not capable of generating their 3D models (also referred to herein as “3D images”) in a fast enough timeframe to meet the surgical latency needs of the intraoperative (e.g., operating room, angio-suite, hybrid-suite) setting. Additionally, most of the technologies that create 3D images typically only display their holographic 3D images from an arbitrary view location and cannot precisely link or correspond their 3D images to a patient’s anatomy with enough specificity to allow a doctor to use 3D imaging for (non-stereotactic) intraoperative navigation, i.e., navigation without the combined use of imaging and physical stereotactic headframe to provide reference points for targeting. As a result, many 3D available imaging tools provide a proceduralist some gross visualization and communication benefits but not true navigation benefits.Summary of Demonstrative Example 2

[0159] The technologies described in this specification provide an improvement in the field of surgery. Open surgery provides great visibility, natural 3D view, and gives surgeon more spatial maneuverability, but large incisions cause pain, lengthy recovery time, higher infection and readmission rates.

[0160] Laparoscopic surgery provides small incision size leading to less pain, less blood loss, shorter recovery time, and lower readmission rates, but has the disadvantages of very poor visibility, highly dependency on 2D image reads, restriction to 2D navigation, and increased cognitive strain on surgeons. Moreover, laparoscopy is associated with longer procedure times, greater surgeon ergonomic strain, radiation exposure, and increases patient anesthesia time.

[0161] Robotic surgery provides 3D visibility, precision control and motion scaling to a small section of MIS procedures. Most robotic surgery platforms, however, do not provide 3D navigation. In some instances, robotic surgery has displaced the surgeon from the operative field (remote console), but robotic surgery is associated with high cost (capital and procedural) with limited evidence of improved outcomes.

[0162] Moreover, robotic surgery is associated with much longer operative and changeover times, as well as high adverse event risk (electrical arcing, external defibrillation risk, thoracic insufflation risk).

[0163] Laparoscopy and Robotics focused on improving access and doctor's “physical” deficits with better tool sets. Digital surgery technology, e.g., as described in this specification, focuses on improving a doctor’s “decision-making” deficits with improved 2D and 3D information.

[0164] Minimally invasive surgery (MIS) practitioners need new tools / new data to reduce severe adverse events. One In 25 patients still experience a serious adverse event from procedures.Moreover, doctors experience information overload during MIS procedures, which can lead to surgical errors. Doctors need Just In Time (JIT) Information delivery at appropriate parts of procedure. New types of intra-operative data can be created with AR, e.g., as described herein.

[0165] The technologies described in this specification provide enhanced 3D visibility, 3D navigation and decision-making capability to Laparoscopic and Endovascular Procedures at a quarter of the costs of Robotics. The technologies provide shorter procedural times, lower radiation risk, expanded field of operative view (can see behind the target), and improved precision sighting. The technologies are associated with lower costs, which allows for expansion into much larger client base then robotics (currently, only used in 3-4% of all procedures) as digital surgery tools can benefit all types of procedure (Open, MIS-laparoscopy and MIS-Robotic, Micro-surgical Microscope).

[0166] The technologies described herein provide innovative surgical systems and methods that integrate and utilize 2D and 3D images. In an aspect, the technologies include a computer-implemented imaging method for use in guiding a medical instrument through patient anatomy. The method includes receiving, by a processor of an imaging system, a set of first images corresponding to the patient anatomy, generating, by the processor, from the set of first images, a three-dimensional (3D) image of the patient anatomy and storing the 3D image in a memory. The method includes receiving one or more live 2D images corresponding to the patient anatomy, retrieving the stored 3D image, and aligning the one or more live 2D images with the 3D image. The method includes providing real-time 3D visualization of the patient anatomy based on the aligned live 2D images and 3D image.

[0167] In an aspect, the technologies described herein include a method of guiding an instrument through patient vasculature. The method includes the steps of: registering one or more components of the instrument, patient, and surroundings of a procedure room, performing the imaging method described above after such registration, and guiding the instrument through patient vasculature based on the 3D image obtained by the imaging method.

[0168] In an aspect, the technologies described herein include a system configured to perform the methods described above. The system includes means for receiving the 3D image, means for receiving the one or more 2D images, means for processing the 3D image and the one or more 2D images to register the 3D image and the one or more 2D images, and means for simultaneously displaying, on the visualization screen, the 2D images, the 3D image, and an indication of a common landmark in the 2D and 3D images.

[0169] In an aspect, the technologies described herein include a system including one or more processors and a memory storing a computer program that, when executed by the one or more processors, causes the performance of the methods described above.

[0170] In an aspect, the technologies described herein include a computer program including computer readable instructions that, when executed by one or more processors, causes the one or more processors to perform the methods described above.Description of Demonstrative Example 2

[0171] This specification sets forth innovative surgical systems and methods that integrate and utilize 2D and 3D images. Specific applications involve registration of 2D and 3D images, and in particular the correspondence and alignment of (live) intraoperative 2D images (e.g., as taken by fluoroscopy or one or more other intraoperative imaging modalities) with preoperatively acquired 3D images and visualization formats that allow 3D registration to improve surgical and interventional cognition, procedural visibility, general guidance and navigation for minimally invasive and open procedures.

[0172] In general, a computer-implemented imaging method or system is provided for use in guiding a medical instrument through a particular patient anatomy. The system or method would include electronically receiving, by a processor, a set of one or more images corresponding to the patient anatomy, then using the processor to operate on the set of first images and generate a three-dimensional (3D) image of the patient anatomy. That 3D image can be stored in a memory. The processor also receives live 2D images of the patient anatomy and can, optionally, produce 2D images from virtual projections of the 3D image. The processor also retrieves the stored 3D image and aligns it with one or more of the live 2D images and provides a real-time 3D visualization of the patient anatomy based on the aligned live 2D images and 3D image.

[0173] In the disclosed techniques, a clinician can have more precise 3D image guidance during a procedure using an initial, static 3D image model that is corresponded to the continually updating (non-static) intra-operative fluoroscopy imaging (or other 2D imaging) in a geometric model of the relevant anatomical structures (e.g., blood vessels) and, optionally, to the surrounding environment (e.g., the patient table, or other features in the procedure room).

[0174] A system for displaying 2D / 3D correspondence and real-time tracking of a surgical or interventional instrument may be provided on a visualization screen, for example, on an Augmented Reality (AR), Mixed Reality (MR) or Extended Reality (XR) head-mounted display (HMD) unit (e.g., Microsoft HoloLens2, Apple Vision Pro or Magic Leap devices) or another digital interface (e.g., desk top computer, laptop, tablet, or mobile phone) where such display can be viewed by one or more parties simultaneously through a dedicated and / or linked network. For example, the output of the 2D and 3D imaging alignment and the resulting tracking of a surgical tool can be jointly displayed simultaneously on a viewing screen (e.g., side-by-side in a screen on a head-mounted display (HMD) or tablet), with or without alignment markings that identify common fiducial landmarks on the 2D and3D images, and can be seen both by the doctor performing a procedure as well one or more remote doctors that may be advisors to the proceduralist, co-surgeons during a procedure or students being taught by the doctor performing the procedure.

[0175] The 2D-3D registration may be aligned and viewed in a single display in which one or more of the intraoperative 2D images and a preoperatively acquired 3D images are presented with indicators of correspondence between common landmarks in the 2D and 3D images, to identify those common landmarks on the respective images. A clinician can perform a procedure while viewing the correlated 2D and 3D images on the viewing screen.

[0176] Although the 3D image is initially derived from preoperative scans and as a result is generally a static image displayed during an intraoperative procedure, the 3D image is updated by information corresponding from streaming live (2D) fluoroscopy. The technologies described in this specification also allow for the 3D image itself to be updated during a procedure should a major anatomical change to the patient occur (e.g., a ruptured aneurysm) and require such change. This update would occur in the angio-suite by taking a new CT acquisition from which the new 3D image could be derived, e.g., using Flat Detector Computed Tomography (FDCT), e.g., Sine Spin FDCT (SFDCT). The new 3D image would need to be created in a timely manner not to disrupt surgical workflow. The 2D / 3D registration process would then be reconstituted from the original 3D volumetric rendering to the new 3D volumetric rendering derived from the CT allowing physicians to make decisions based on the patient’s most current anatomical status.

[0177] Typically, 2D-3D registration infers that the 2D component being referenced as part of the registration process corresponds to one or both of the live console displays. However, this specification also describes a new method of creating virtual 2D views from existing live fluoroscopy where a 3rd, 4th or 5th virtual fluoroscopy display can be created to supplement the live views and where the 3D image is corresponded to either the real live 2D views or to the virtual 2D views being created, or both simultaneously (e.g., one live view and one virtual view). This new form of Virtual 2D / 3D registration, which can also be deployed using rigid and non-rigid deformable methods, can provide line-of-sight information, notwithstanding the physical limitation constraints of the C-arm.

[0178] Techniques described in this specification for AR-based 3D imaging from previously acquired 3D CT scans can include numerical or Al-based approaches to segmentation, voxel threshold determination, pixelating mesh in a dense triangulation, and fine tuning the 3D output with merging, trimming, decimating, and / or smoothing processes to isolate the desired anatomy (e.g., the endovascular circuit), as disclosed herein, with the option to overlay and display other anatomical components (e.g., nerves, lymphatic system, ligaments, bone, etc.) individually, or in combination, or all simultaneously.

[0179] Data can be shared and manipulated by one or more users interactively over a computer network, e.g., a cloud-based system. Such a system can provide remote collaboration in real time, coaching and training. For example, two specialists can perform different tasks on the sameprocedure, e.g., a first surgeon can remove a tumor while a second surgeon performs endovascular arterial ablation to cut blood flow off the tumor. The data (e.g., the images and models described in this specification) can be displayed on a computer screen, e.g., a tablet or desktop screen, or on a headset, e.g., an AR headset. The technologies can be integrated into networked system providing near zero latency 3D image creation, Al-ML based rigid and non-rigid registration Dynamic limitedlatency tool tracking, real-time fluoroscopy, 0-ARM / iUS updates, 5G remote Telemonitoring collaboration. In some implementations, such integrated technologies can provide full decision support platform fusing simulation with navigation, Al-based etiology and pathophysiology of disease, referenced learning using procedural kinematics database, real time predictive obstacle and path modeling, and predictive outcomes modeling based on selected tool sets.

[0180] In some implementations, the techniques disclosed herein can provide the size and clarity of 3D images with an appropriate number of vertices and triangular faces to allow for the 2D / 3D registration to commence. The registration may finish with sub-second processing time and submillimeter registration accuracy meeting the industry’s need for improved precision of registration over other 3D surgical tools including robotic tools.

[0181] This specification also provides methods of tracking surgical or interventional tool sets using 3D models / images registered to the 2D images, as disclosed herein. The specification also relates to systems configured to perform the methods, and computer programs comprising computer readable instructions for performing the various methods.

[0182] The systems and methods also find application in improving the speed and accuracy of registering 3D models / images to a patient’s anatomical structures and allowing medical professionals to analyze and navigate their procedures, including real time updates of tool location, with 3D visibility.

[0183] The approach provides improved viewing of endovascular (e.g., inside blood vessels) and endoluminal anatomical structures to improve intraoperative clinical decision-making during MIS procedures, but can also include other parts of the anatomy.

[0184] A system for performing one or more of the above methods may include a computer comprising a memory in which the relevant computer program is stored. The computer may interface with the viewing device, which can be implemented in 2D (e.g., a tablet screen) or, preferably, in a 3D display (e.g., a head-mounted display). The screen can display the 2D and 3D images simultaneously to facilitate correspondence of anatomical landmarks on them.

[0185] The system and method are not limited to the particular use described in the examples. The systems and methods may be employed in other settings and applications within or outside the medical field.

[0186] In the disclosed implementations, pre-operative, or inter-operative, one or more 3D images are provided simultaneously with one or more 2D images, and the 3D images are augmented by the processing of the intraoperative 2D imaging (2D images acquired during surgery). In this way, thesystem can mitigate inaccuracies in a 3D visualization arising from, for example, changes in the physiology of a patient since the 3D image was obtained or differences between a position of the patient when the 3D image was acquired and their position during the operation, or a change one or more of the patient’s physiologic parameters, e.g., blood pressure, or due to introduction of guidewires, catheters, or other instrumentation.

[0187] A 3D visualization, based on the 2D / 3D registration, can provide real-time information to the surgeon or interventionalist during the procedure (e.g., such as whether a vessel is above, below, behind or in front of another vessel, the depth of one vessel in comparison to another, or the specific angle of a tool within the vessel). The intraoperative 2D images may be obtained using an imaging method such as fluoroscopic imaging, which can produce images more quickly, conveniently, safely, and cost-effectively when compared to intraoperative 3D imaging methods.

[0188] The improved tracking and / or navigation may, in turn, reduce tool insertion inaccuracies and the incidence of adverse events (AEs) or serious adverse events (SAEs) during a procedure. In some implementations, additional simulation technologies can be included, e.g., fluid dynamics and / or solid mechanics models, e.g., to model and / or display blood flow in arteries and wall shear stress.

[0189] The 3D image(s) may be obtained using, for example, non-contrast or contrast CT or CT-Angiogram (CTA) as the base source of anatomical information. The methods may further include segmentation of such base materials to select the correct image slice / view of relevant anatomy for a physician to use during a procedure and from which to develop the 3D image. One or more views could be used to create a series of 3D images that the doctor may choose from in a preoperative setting prior to using that 3D image in the intraoperative setting. Separate 3D models / images of different anatomical parts can be created (e.g., arteries only, arteries and nerves, bone only, bone and ligaments, etc.) independently and combined. For example, a visualization of the arterial network may be overlaid with a visualization of the patient’s nerves all within the HMD or computer-based digital display.

[0190] FIG. 27 is a block diagram of an example system, or apparatus, that may be used to perform a method of rendering a 3D visualization with co-visualization of a 2D / 3D registered 2D fluoroscopy image. FIG. 28 is a block diagram of an example system, or apparatus, that may be used to perform a method of rendering a 3D visualization with co-visualization of two 2D / 3D registered 2D fluoroscopy images, e.g., obtained from two different viewing angles. In the example systems, the display may be one or more displays, e.g., one or more head-mounted displays. First and second 2D imaging devices may be connected to the computer system via wired or wireless connections.

[0191] FIG. 29 illustrates schematically a computer system that may be included in the system of FIG. 27 and FIG. 28. The computer system includes a memory in which are stored computer-readable instructions for performing a method as described in this specification, a processing arrangement of one or more processors configured to execute those instructions, a receiver for receiving one or more of the 2D images, and a video output for providing video data to the display.

[0192] The computer system may also include a network interface, via which the 2D images and, optionally, 3D image may be received. One or more user interfaces for receiving input from the user, optionally, providing output to the user may be provided. Such user input may be received from a keyboard, mouse, touchscreen or voice input device. Output to the user may be provided via a graphical user interface and / or audio device.

[0193] FIG. 30 schematically illustrates an example visualization / display screen that may real-time navigation visualization for a clinician. In some example implementations, the display screen may be a head-mounted display. In this example, the 3D visualization is shown simultaneously with two 2D images. One or more 2D images can be used. The 2D and 3D images can be updated during the procedure, for example, as an instrument is progressed through the anatomy, to guide a clinician during a procedure. In this manner, the clinician may view the display screen with both 2D and 3D images, with each image optionally being updated temporally, to steer or otherwise control the instrument during the procedure.

[0194] As shown in FIG. 30, an indication of a common landmark is provided on the visualization screen, and that landmark is seen in each of the images. The indication may be in any suitable form. For example, a line may be displayed that links the corresponding location (e.g., a given fiducial marker) in the 2D and the 3D images. Alternatively, or additionally, an indication mark may be used to show the corresponding location. In one example, the common landmark may be a feature of the patient’s anatomy (e.g., a given fiducial marker in a patient’s vascular channel) or an artificial fiducial marker. By visualization of the same fiducial marker in the 3D and 2D images, the clinician can more efficiently guide an instrument (e.g., catheter) past the given fiducial marker during the procedure. In another example, the common landmark may be a feature of the instrument itself as being used in the navigation, such as the tip of a catheter. Although FIG. 30 shows a display that indicates only one common landmark, indications of multiple landmarks may be provided by the display, and each landmark would have corresponding points on each of the images, with indicators showing their correspondence between the images. By this approach, the clinician can view the given landmark in both the 2D and 3D images, and thereby obtain a clearer view of the landmark to guide the navigation.

[0195] Further example implementations are provided below. This disclosure is not, however, limited to the specific examples discussed herein. Various modifications to the disclosed methods and systems are intended to be included in this disclosure, such as the use of equivalents to the various components and mathematical operations set out in the examples. The following embodiments (identified as “claims”) are examples but not limiting.Data Collection

[0196] To demonstrate the technologies described in this specification, data sets comprising 3D CT imaging and biplanar fluoroscopic image sequences were acquired using a physical phantomsimulating human anatomy as follows. A Philips Azurion 7 B20 / 15 model biplanar fluoroscopy device was used to acquire both 2D and 3D images of a Mentice ASISTTM ischemic stroke thrombectomy simulator (see FIGS. 31A and 31B). This phantom simulator model is a device made of a polymer and is designed to mimic a human head and contains a vessel network 3D-printed in silicone based on segmented CT scan images of a real patient, to which other pathologies (e.g., cerebral aneurysms) have been added (FIG. 31A). The model has its own circulator system, consisting of a network of tubing connected to a pump and reservoir. The latter contains a fluid that shares some of the physical properties of real blood. Contrast agents and instrumentation (e.g., guidewires) can be introduced into the system as they would be during a patient case. FIG. 31B shows an example circulatory pump and reservoir, with a guidewire and a syringe containing contrast fluid that can be used a model as described here. The gel inside the stiff outer Perspex ‘skull’ of the model, which supports the silicone vessel network, is designed to mimic the radiographic and mechanical properties of human brain tissue.

[0197] In total, five data collection sessions, which also served as opportunities to iteratively test new developments in a simulated clinical setting, were undertaken in accordance with the schedule set out in Table 1:Table 1

[0198] Altogether, the data acquired was deemed sufficient in quantity and quality to develop and / or validate an automated preoperative 3D CT image segmentation process and methods for intraoperative 2D / 3D image registration as described in this specification.

[0199] The technologies for image acquisition described above can be used for the acquisition of patient data to generate patient specific 3D images. In some implementations, a single data set of a single patient is acquired and processed as described herein before the intervention. In some implementations, multiple patient specific 3D images are acquired from the same patient or from multiple patients. Multiple sets can be used to either update a previous 3D image (e.g., from the same patient) or to create an averaged 3D anatomical model.Automatic 3D CT Image Segmentation

[0200] The technologies described in this specification include the establishment of spatial relationships (mapping or “registration”) of 3D images (e.g., obtained from CT) to 2D images (e.g.,obtained from fluoroscopy). A geometric model of the relevant anatomical structures (e.g., blood vessels) can be used to achieve accurate registration of patient 3D imaging to intraoperative fluoroscopy. This model can be used to generate virtual projections for pixel-wise comparison. To that end, an automated segmentation process is used to translate raw CT images to a 3D digital model (the 3D image). The process includes one or more of the following steps:1. Query a series of parallel Digital Imaging and Communications in Medicine (DICOM) images (“slices”) from a CT scanner and sort slices along their principal axis.2. Optionally, apply a 3D mask to select region of interest. A 3D mask can be used to remove imaging artifacts, e.g., undesired imaged objects, e.g., ECG leads or a breathing apparatus.3. Apply voxel grayscale threshold(s) - lower bound and optional upper bound to reduce noise and artifacts.4. Generate 3D mesh comprising vertices and triangular faces.5. Merge vertices within specified distance tolerance6. Trim mesh.7. Perform decimation (i.e., reduction in number of faces).8. Apply smoothing.9. Export mesh in OBJ (object file) geometry format.Exemplary input parameters and overall execution time of the first three steps for an example data set are shown in Table 2.Table 2An illustration of the partial results generated from this data is given in FIGS. 32A and 32B. FIG.32A shows an example axial slice through an original DICOM data set obtained by CT. FIG. 32B shows an example result of automated vessel segmentation. The regions of interest (lumen of blood vessels) are highlighted in color. Cross-hairs show the location of middle cerebral artery (MCA) branching. After this initial segmentation procedure, a comparison can be made between the results of the initial segmentation step and a manual segmentation of the same 3D image using the ITK-SNAP software, using a thresholding and region-growing technique. FIGS. 33A and 33B illustrate an example comparison between manual segmentation in ITK-SNAP using athresholded region-growing algorithm (FIG. 33A) and first stages of the automated process described in this specification (FIG.33B). The results indicate a high degree of similarity between the manual and automatic methods.

[0201] After segmentation (feature extraction) a computational 3D mesh is generated from the segmented images. In some implementations, mesh generation can be accomplished using an implementation of the Marching Cubes algorithm (an algorithm that creates triangle models of constant density surfaces from 3D medical data) [1]. The algorithm proceeds through each voxel in the image, considering the eight comers of each such cube and determining the polygon(s) needed to represent the part of an isosurface that may pass through this cube. The algorithm typically results in there being a number of duplicated vertices. These vertices are merged using a default tolerance, e.g., of 0.05 mm (or between 0.01 and 0.1 mm). After the mesh is generates, the mesh is trimmed. The trim operation considers each disjoint section of the overall generated mesh and removes any such section that lies entirely within a sphere of a specified radius. This operation is designed to remove residual geometric ‘noise’ from the resulting mesh (e.g., high intensity imaging artefacts).

[0202] After trimming, the process continues with the decimation step. This step is necessary to allow subsequent renderings of the geometry (e.g., during 2D / 3D registration) to occur in a (computationally) efficient manner. The goal is to reduce the number of mesh faces, while preserving the salient anatomical features. The process employs an implementation of the Quadric Edge Collapse Method [2], i.e., a surface simplification algorithm that can rapidly produce high quality approximations of polygonal models using iterative contractions of vertex pairs to simplify models, maintaining surface error approximations using quadric matrices.

[0203] The discrete nature of the underlying volumetric data often results in a non-smooth appearance of the resulting models, despite the best efforts of the marching cubes algorithm.Therefore, as a final step in the segmentation process, the mesh is optionally subjected to a number of iterations of the HC (Humphrey’s Classes) Laplacian algorithm where vertices of the smoothed mesh are pushed back towards their previous locations [3], which avoids problems of deformation and shrinkage caused by other smoothing methods. The relevant input parameters and output characteristics of the latter stages of an example process are listed in Table 3.Table 3< FIGS. 34A and 34B show a comparison of the raw output of the marching cubes algorithm (FIG. 34A) against the final processed mesh, which has undergone the merge-trim-decimate-smooth operations (FIG. 34B). This example illustrates how the segmentation process described herein is capable of generating clean meshes with relatively low face counts, capturing the saliant anatomical features, suitable for the generation of virtual X-ray projections. Once the clean mesh is generated, the mesh information is stored in in OBJ format, i.e., the position of each vertex, the UV (cartesian coordinate) position of each texture coordinate vertex, vertex normals, and the faces that make each polygon defined as a list of vertices, and texture vertices are stored in a file and can be manipulated or “virtually imaged.”Rigid 2D / 3D Registration

[0204] 2D / 3D registration is the process that achieves alignment of intraoperative 2D fluoroscopy and preoperatively acquired 3D imaging, e.g., the 3D CT based model described above. The alignment of 3D imaging and (live) 2D fluoroscopy facilitates real-time guidance and navigation during surgical procedures. FIG. 35 illustrates an example biplanar fluoroscopy device (e.g., Philips Azurion 7 B20 / 15) that can be used during the model development and validation processes (as well as for surgical interventions). In an example implementation, e.g., for model development and validation purposes, this fluoroscopy device can be used to image the head phantom (e.g., a head phantom as described above for 3D image generation), since this phantom model can be imaged with X-rays as many times as necessary. The fluoroscopy device is used to produce actual 2D images of the vasculature. During model development and validation, the vasculature of the phantom is imaged to generate 2D fluoroscopy images of the phantom. During surgery, the fluoroscopy device is used to image the patient to obtain images of the vasculature in real time. The goal of the initial validation scans, however, is to help produce a virtual model of a fluoroscopic device. The parameters of the fluoroscopy device (e.g., size of C-arm, emitter and detector parameters) are used to create virtual X-ray projections of the 3D model of the cranium and vasculature, e.g., using ray tracing or a similar technique, tracing lines from the virtual emitter through the 3D image to the virtual detector. The virtual X-ray projections of the vasculature from such virtual imaging can mimic actual 2D fluoroscopy images. When such virtual projections are made to be coincident with the actual X-ray projections / images (e.g., from the phantom or from live 2D fluoroscopy) through the setting of the model parameters, a successful “registration” has been achieved. After accurate registration, the spatial relationship between the phantom’s or the patient’s 3D image coordinate frame of reference (e.g., in an X-Y-Z, or U-V coordinate system) and the coordinate frames of reference of each of the actual X-ray sources / emitters and detectors of the fluoroscopy device is known and can be used, e.g., to triangulate locations in 3D space given correspondences in 2D, and to perform other geometric calculations.

[0205] To build a virtual model of a fluoroscopic device, a representation of the kinematic chains comprising the fluoroscopy scanner can be created using a mathematical model, such as the Denavit-Hartenberg (DH) convention [4] . In mechanical engineering, the Denavit-Hartenberg parameters (also called DH parameters) are the four parameters associated with a particular convention for attaching reference frames to the links of a spatial kinematic chain, or robot manipulator. Each link in the chain is parameterized by two angles and two translations along coordinate axes, giving rise to a 4x4 transformation matrix (see Equation 1), which represents the link or joint. The concatenation of such matrices then gives a means for determining the coordinates of spatial locations in any of the coordinate frames comprising the kinematic chain.

[0206] Prior to (virtual) scanning with the fluoroscope, the fluoroscope parameters are adjusted, e.g., manually. FIG. 36 illustrates a virtual model of a fluoroscopic device, including both frontal and lateral arms, the table on which the patient lies, the patient (or phantom) head, and the respective virtual X-ray projections from the virtual sources to the virtual flat detector plates. The terms “RAOLAO” refers to the Right Anterior Oblique (RAO)-Left Anterior Oblique (LAO) angle, i.e., right-to-left (relative to the patient) angle, and the term “CAUCRA” refers to the cranial-caudal angle. Left and right images show different configurations of the C-Arm, L-Arc and patient table kinematic chains. The image projection parameters can be verified using a calibration phantom of known geometry.

[0207] In standard procedures, the C-Arm angles are chosen either in a trial-and-error manner at the start of a procedure, or using existing software built into the scanner that requires a CT “spin” at the beginning of the procedure (e.g., an FDCT scan, which exposes the patient to radiation). This process can be time-consuming and exposes the patient to ionizing radiation. Using the technologies described in this specification, C-arm angles can be optimized computationally, as described below.

[0208] For optimal visualization of the vessels of interest, the target anatomy needs to be “opened up,” that is, be visible without any occlusion from other vessels, etc., in the way of the X-ray beams of the fluoroscope. Because rendering virtual projections is computationally cheap, the computer can be used to render thousands of angle combinations, such that the optimal views can be selected automatically, based on a metric measuring the degree of vessel overlap. FIG. 37 shows a set of virtual frontal projections for 10 RAOLAO angles and 10 CAUCRA angles. FIG. 38 shows a set of virtual lateral projections for 10 RAOLAO angles and 10 CAUCRA angles. From the virtual projections, the degree of overlap of vessels in the line of sight (and thus the degree of visualobstruction) in the vessel network can be determined. In an example implementation, the number of times pixels in the image of the vasculature are overwritten (due to overlap of vessels in an image) is counted. FIG. 39 shows an example map showing degree of pixel overwriting for 51x51 = 2,601 angles illustrating potentially good viewing regions with low overlap (blue), versus those where the amount of overlap has been shown to be greater, leading to poor viewing (brown). Such maps can be used to guide the operator during procedure setup, or further still to choose the setup of the C-Arm automatically. For example, the C-Arm angle results with lowest overlap can be used to configure the C-Arm for intervention. When a 2D / 3D registration has been performed, the projections can be generated from preoperative imaging, therefore eliminating the need for a “spin” immediately prior to the procedure.

[0209] Tables 4, 5, and 6 show example DH parameters for the frontal C-Arm, the lateral L-Arc and patient table, respectively, where R is the length of the common normal (assuming a revolute joint, this is the radius about previous z-axis); a is the angle about common normal, from old z-axis to new z-axis D is the offset along previous z to the common normal; and 0 is the angle about previous z from old x to new x (parallel to common normal). The parameters in bold represent the degrees of freedom of the device. The term “SID” refers to the source-to-image distance. The quantity “LPOS” is the longitudinal translation of the lateral L-Arc along the ceiling rail from which it is suspended. The three degrees of freedom in Table 6 correspond to the posterior-anterior, inferior-superior, and rightleft translations of the patient table, in the coordinate frame of the patient.Table 4Table 5Table 6

[0210] Additional degrees of freedom can result from the fact that the area of patient anatomy interest (e.g., the head) may not be in the same position and orientation when comparing the preoperative and intraoperative settings. For instance, the head may be straight during the initial CT scan, but tilted to one side during later fluoroscopic acquisitions. Therefore, an additional 3 -DOF (degree of freedom) translation and / or 3 -DOF rotation (yaw-pitch-roll) can be appended to the kinematic chain table. These quantities are determined as part of the registration process.

[0211] The registration process locates the values of the degrees of freedom such that real and virtual X-ray projections (i.e., real and virtual 2D fluoroscopy images) are aligned accurately in a pixel-wise fashion. A two-stage process can be applied: Firstly, the additional degrees of patient freedom and the longitudinal L-Arc position are determined approximately, using a point-based registration method. Secondly, the initial registration is refined using an image-based comparison metric. The advantage of this method is that, while the patient’s head remains in a static position and orientation, the first stage need only be performed once. Subsequently, in the second stage, the registration proceeds as a series of optimization updates, i.e., based on comparing updated (real-time) 2D patient data with image data used for initial registration or intermediate image data (e.g., previous updates). Certain parameters are known and can be inferred from the video feed, which represents the real-time output of the fluoroscopy device (e.g., the arm angles, source-to-image distance and effective flat detector size), but only to a limited degree of accuracy. Table 7 lists the degrees of freedom that areoptimized as part of the initial so-called “patient registration”, and the accuracies of the inferred parameters (Parameters (in bold) are optimized during patient registration).Table 7

[0212] During the first stage, to effect point-based correspondences (between 2D and 3D images / models), clinical-grade CT-compatible fiducial markers (e.g., one or more radiopaque solid or liquid markers) are attached (e.g., glued) to a patient or phantom model at predetermined anatomical locations. In the simulated (phantom) settings, these markers are permanently attached and therefore appear in 3D imaging. In the human / clinical setting, for practical reasons, the markers may be attached only prior to the procedure. However, their geometric locations within the 3D scan coordinate frame will be determined in advance. The patient registration proceeds through projecting the fiducial markers onto the respective virtual detectors and comparing their location with the actual fluoroscopic projections. FIG. 40 illustrates the registration process, showing lateral and frontal (virtual) projections of fiducial markers and vasculature. The solid arrows point to example fiducials (rendered in green) on the surface of the 3D image. The dashed arrows point to corresponding fiducials on two corresponding virtual 2D fluoroscopy images (lateral and frontal). The 7 degrees of freedom can be determined using Levenberg-Marquardt nonlinear least-squares optimization [5, 6] over the X and Y displacements for each projection (this method employs a maximum neighborhood method, which constitutes an optimum interpolation between the Taylor series method and thegradient method, the interpolation being based upon the maximum neighborhood in which the truncated Taylor series gives an adequate representation of the nonlinear model). In the example illustrated in FIG. 40, the additional overlay of a virtual projection of the vasculature (e.g., shown in red on a screen) shows the initial alignment of the virtual 2D projections with the vasculature of the 3D image.

[0213] The virtual projections shown in FIG. 40 are then compared with actual 2D fluoroscopy images, and registration / alignment in optimized. Table 8 shows example numerical results of the optimization in terms of Euclidean pixel error and the equivalent physical distance. Furthermore, Table 9 gives mean errors over all correspondences. These results are generated in the “retrospective” patient setting, where video of the fluoroscopic feed was captured and then used for analysis offline at a later date. Sub-millimeter mean errors are observed.Table 8Table 9

[0214] As illustrated in FIG. 10, the example registration process described above is repeated in a so-called “prospective” patient registration setting, where a live feed is taken from the fluoroscopy device (live fluoroscopic input shown in the background screen in the image). This (phantom) example represents the actual clinical scenario. The equivalent mean error results are shown inTables 10 and 11. Table 10 shows frontal and lateral fiducial projection errors following prospective patient registration. Table 11 shows residual and mean errors following prospective patient registration. Sub-millimeter mean errors are observed. Since the optimization is point-based, the execution time is virtually not noticeable by a human observer (sub-second).Table 10Table 11

[0215] Equivalent results are generated for a retrospective human data set (“NYUPT-1”), where although no fiducial markers are present on the 3D image, 2D / 3D correspondences are identified in the images manually. FIG. 41 illustrates lateral and frontal projections of fiducial markers and vasculature in the 2D fluoroscopy images. Table 12 shows frontal and lateral fiducial projection errors following NYUPT-1 patient registration. Table 13 shows residual and mean errors following NYUPT-1 patient registration. Again, sub-millimeter mean errors are observed.Table 12Table 13

[0216] During the second stage, registration proceeds through the repeated invocation of imagebased updates, for example, in response to the movement of one or more of the imaging arms. Here, the 3D model or collection of 3D models of the anatomy (e.g., the vasculature of the brain, e.g., one side of the brain imaged using contrast agents) is projected onto the respective imaging planes (flat detector panes), such that the degree of projection overlap can be counted in a pixel-wise manner. That is, the number of pixels that represent, e.g., vasculature in both the actual and the virtual image are counted - the more accurate the registration, the more pixels “overlap.” This pixel overlap is independent of vessel overlap (one vessel being in the line of sight of another). The optimization technique described here aims to adjust registration by finding the greatest number of overlapping pixels. An optimization technique that combines Simulated Annealing (a method for solving unconstrained and bound-constrained optimization problems based on a model of the physical process of heating a material and then slowly lowering the temperature to decrease defects, thus minimizing the system energy) [7] and the downhill simplex method of Nelder and Mead [8] is employed [9] (a method for the minimization of a function of n variables, which depends on the comparison of function values at the (n + 1) vertices of a general simplex, followed by the replacement of the vertex with the highest value by another point), because the dimensionality of the problem increases, and partial derivatives of the comparison metric are not available. Table 14 lists the parameters optimized during registration refinement (bold) and initial simulated annealing deltas. Table 15 lists the algorithm settings and output characteristics from registration refinement.Table 14Table 15

[0217] FIGS. 42A and 42B show initial (FIG. 42A) and final (FIG. 42B) overlays for phantom frontal 2D / 3D alignment. FIGS. 43A and 43B show initial (FIG. 43A) and final (FIG. 43B) overlays for phantom lateral 2D / 3D alignment. The figures illustrate the improvements when comparing the results of patient registration (left, (FIGS. 42A and 42B) and refinement (right, FIGS. 43A and 43B).In an example color display, green represents the actual fluoroscopic image, blue represents the virtual projection, and red accounts for pixels that overlap. These are counted and used as the error metric for each evaluation during the optimization process. FIG. 44 illustrates how the optimization algorithm proceeds over the first three annealing iterations. During the first annealing iteration, the “temperature” is highest, i.e., the algorithm allows for a greater probability that a solution with a lower degree of overlap is accepted as an interim solution, which minimizes the likelihood that the algorithm gets “stuck” in a local maximum. The percentage overlap reaches a plateau at a lower temperature (mostly solutions with a higher degree of overlap are accepted as an interim solution), indicating maximum percentage pixel overlap and thus the most accurate (optimal) registration.

[0218] The phantom-based process described above is repeated to generate corresponding results for the retrospective human data set (NYUPT-1). Table 16 shows Parameters (in bold) optimized during NYUPT-1 registration refinement. Table 17 shows the algorithm settings and output from NYUPT-1 registration refinement.Table 16Table 17

[0219] FIGS. 45 A and 45B shows initial (FIG. 45A) and final (FIG. 45B) overlays for NYUPT-1 frontal 2D / 3D alignment. FIGS. 46A and 46B shows initial (FIG. 46A) and final (FIG. 46B) overlays for NYUPT-1 lateral 2D / 3D alignment. FIG. 47 illustrates how the optimization algorithm proceeds for NYUPT-1 over the first three annealing iterations. The optimization proceeds in a manner similar to that of the phantom model resulting in visual improvement to the overlays.

[0220] Successful 2D / 3D registration can be used to address a number of problems inherent to 2D fluoroscopy. For example, an interventionalist may know the location of a certain feature of interest in one of the fluoroscopic views, e.g., frontal or lateral, but may not be able to determine thecorresponding point location in the other view, depending on the geometric relationship between the C-Arms and anatomy that may be occluding the target in the other view. With a 2D / 3D registration having been performed, the geometry of the patient and projections are known (at least approximately), and therefore it is relatively straightforward to back-project a ray from the point of interest in one view to its X-ray source. Such a ray will intersect with the 3D image between the X-ray source and X-ray detector. Another ray can then be (virtually) projected forwards from the other X-ray source through the point of intersection and onto the other X-ray detector. Thus, points on one of the 2D image “screens” can immediately be ‘scattered’ to their corresponding position on the other 2D screen. FIG. 48 illustrates example virtual rays emanating from one view and ‘scattering’ into the other.

[0221] Another problem that can be addressed using the technologies described in this specification is that certain procedures require X-ray views of the patient’s anatomy that are difficult or even impossible to achieve physically due to limitations in positioning the C-Arm (e.g., the C-Arm cannot be in the same location as the patient’s body, as illustrated in FIG. 49 showing a physically impossible C-arm position with a virtually-rendered projection). The location and orientation of a cerebral aneurysm may be such that without this purely caudal -cranial view, the condition may not be amenable to interventional neuroradiology and may need to be treated via another, potentially riskier, method. The technologies described in this specification can provide additional virtual live fluoroscopic views that alleviate these physical constraints. Analogous to the techniques described above, it is straightforward to create virtual (2D) X-ray projections of the preoperatively acquired 3D vessel model from a desired “impossible” position and orientation. A challenge addressed here may be to ensure that this model accurately represents the current state of the patient during the procedure, where vessels may be subject to deformation. A solution is to extend the rigid registration process so that it operates in real time and accounts for vessel and tissue deformation, e.g., using the techniques for deformable registration described below or using other models, e.g., a real-time finite element simulation, to generate plausible updates to the original static model. These deformable registrations or models can be combined with instrument tracking, e.g., using the triangulation methods described below, and the necessary re-rendering in the virtual views.

[0222] Another problem that can be addressed using the technologies described in this specification is for certain interventional procedures (e.g., catheterization) where it is of considerable benefit to know the manner in which the tip of an instrument will oppose a certain target anatomical feature. For example, it may be beneficial to determine or estimate the behavior of the tip of a microcatheter with respect to the opening and neck of a cerebral aneurysm, once the instrument has been inserted and is close to the target. Challenging insertions and / or progressions of instruments can be mitigated by the pre-bending the instrument tips, but at present this pre-bending is not done in a systematic fashion. Any deviation from the optimal positioning, insertions, and / or progressions can compromise the success of the procedure. Using the technologies described in this specification the location andorientation of devices may be predicted accurately once they are inserted into the vasculature. For example, the imaging techniques described in this specification can be combined with a virtual mechanical model, e.g., a finite-element mechanical model of the vasculature. In an example implementation, a finite element or finite volume model of vascular solid mechanics or fluid / solid mechanics can be generated from the 3D image and known mechanical properties of the tissue and, optionally, from flow data. The positioning of a tip of a medical instrument can be simulated based on a mechanical model of the instrument and simulated path of the instrument tip through the vasculature of the virtual mechanical model.

[0223] With an accurate geometric model of a patient’s vasculature and a model based on the physical (e.g., mechanical) properties and behaviors of the instrument itself (e.g., a spring system), the insertion of instruments can be modelled in real time, such that the location and orientation of the tip can be estimated as it approaches a target. The solution of the model can incorporate vessel deformation, as the energy stored in the instrument during insertion has a non-trivial effect on the vessel walls, which is incorporated into the model. Automated insertions can be performed across a set of initial conditions, such that a probability distribution of potential endpoints can be generated. For example, the system can report the most likely terminal orientations, and can predict ways of preforming the instruments so that said probability distributions are unimodal distributions with thin tails.Triangulation

[0224] Visual confirmation of registration accuracy can be followed by a quantitative analysis. Such analysis can be performed in terms of triangulation accuracy in both the phantom and patient data settings. Triangulation accuracy depends directly on registration accuracy. In an example phantom setting, a guidewire was inserted into the vessels and left in a static position in order to generate correspondences between 2D and 3D images at the tip. FIG. 50 illustrates the back-projections from the frontal and lateral 2D images. The point of near crossing is shown in the insert. In practice, the rays do not intersect exactly, and so the triangulated point is taken to be the midpoint of the points of closest passing. The same was repeated for ten additional guidewire tip locations as the guidewire progresses through the phantom. The technologies can therefore be used to track a medical device or instrument, e.g., a guidewire and / or guidewire tip through a patient vasculature. The results and mean error are illustrated in FIG. 51. The graph shows results of multiple (10) triangulation tests in terms of minimum ray distance. The results are represented numerically in Table 18.Table 18

[0225] Analysis of minimum ray distances can be accompanied by an analysis of triangulation error with respect to a known 3D correspondence. To this end, results were generated for the NYUPT-1 data set, where such correspondences were identified manually. FIGS. 2A and 2B show the 3D anatomical location of an example set of crosshairs within the posterior circulation in a 3D rendering (FIG. 2B) and an example corresponding CT image slice (FIG. 2A). The crosshairs are located on an inferior part of vessel within posterior circulation. The respective frontal and lateral 2D fluoroscopy correspondences are shown in FIG. 52A (frontal view) and FIG. 52B (lateral view).

[0226] Table 19 shows the results of the 3D triangulation experiment numerically. As in the case of the phantom triangulation analysis, where the mean error is sub-millimeter, the 3D correspondence test is similarly accurate. In comparison to the heavy computation burden of registration, the coordinate geometry calculations required for triangulation are less computationally intensive, resulting in almost instantaneous delivery of results. The triangulation calculations can keep up with incoming video frames should features within them be tracked automatically.Table 19Deformable 2D / 3D Registration

[0227] In the clinical context, the utility of rigid registration (where transformations of space and objects within it are composed of globally applied translations, rotations, and scalings) can depend on the assumption that no deformation has occurred when comparing the preoperative image acquisition (e.g., 3D CT) versus intraoperative acquisition (e.g., biplanar 2D fluoroscopy). Such assumption might not always be appropriate, as deformation might occur as a result of a patient being in a different position, or from a change in blood pressure, or, more noticeably, from the introduction of guidewires, catheters and other instrumentation intraoperatively. For triangulation to provide the most useful information to the clinician, for example, any preoperative imaging may need to be modified (deformed) so that it represents the state of the patient during their procedure in a contemporaneous manner. To this end, a deformable module can be implemented, e.g., based on a finite-element model or a spline model.

[0228] In an example implementation, a volumetric free-form deformation (FFD) technique is introduced, based on 3D cubic B-splines

[0010] , Equation 2 defines the local deformation field in terms of the tensor product of ID cubic basis functions given in equations 3, 4, 5 and 6.

[0229] The B-Splines are supported with respect to the three coordinate axes by a lattice <|) of control points. The user determines the lattice spacing along each of the axes (X, Y, Z). With knowledge of the size of the models(s), this spacing, in turn, determines the number of control points equally spaced along each axis (and therefore in total). As one or more of the control points are moved, the space surrounding them deforms in a mathematically smooth and continuous manner. Therefore, the geometric (3D) models of anatomy used to generate virtual projections can be placed within such a deformation field, and the resulting deformed anatomy can be registered to the contemporaneous 2D imaging using the same image-based metric and optimization process as described above (e.g., simulated annealing can be used to update the positions of the control points, which, in turn, affect how the models are deformed in an attempt to obtain a better registration). The coordinates of the control points become the degrees of freedom over which the search for optimal alignment takes place. Table 20 lists the input parameters and output in respect of the example NYUPT-1 data set.Table 20

[0230] FIG. 53 illustrates the control point lattice (starting positions of points that have moved are shown in red on a color display and indicated by white arrows in the insert). The original undeformed mesh is rendered in grey, whereas the deformed regions of the mesh are highlighted and rendered in green. The deformation results in an increase in the number of overlapping pixels when real and virtual projections are compared, which indicates an improved alignment of 2D and updated 3D images. This is illustrated in FIG. 54, showing the optimization progress during deformable registration (overlap measured relative to rigid registration as a starting point). An initial marked increase in relative percentage pixel overlap during the first annealing iteration is followed by a series of smaller improvements.

[0231] It should be understood that the results and images shown are results of example implementations of the technologies described in this specification. Variations and modifications will occur to those of skill in the art after reviewing this disclosure. The disclosed features may be implemented, in any combination and subcombination (including multiple dependent combinations and subcombinations), with one or more other features described herein. The various features described or illustrated above, including any components thereof, may be combined or integrated in other systems. Moreover, certain features may be omitted or not implemented.

[0232] In general, embodiments of the subject matter, including the systems and their modules, and methods, and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, a data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter affecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus.

[0233] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpretedlanguages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0234] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0235] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices.

[0236] Examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and could be made without departing from the scope of the information disclosed herein. All references cited herein are incorporated by reference in their entirety and made part of this application.References[1] Lorensen, W., Cline, H. (1987) Marching cubes: A high resolution 3D surface construction algorithm. ACM Computer Graphics, 21(4), pp. 163-169.[2] Garland, M., Heckbert, P. (1997) Surface simplification using quadric error metrics. Proceedings of the 24th annual conference on computer graphics and interactive techniques, pp. 209-216.[3] Vollmer, J., Mencl, R., Muller, H. (1999) Improved Laplacian smoothing of noisy surface meshes. Eurographics 18(3), pp. 131-138.[4] Denavit, J., Hartenberg, R. (1955) A kinematic notation for lower-pair mechanisms based on matrices. Journal of Applied Mechanics, 22(2), pp. 215-221.[5] Levenberg, K. (1944) A method for the solution of certain non-linear problems in least squares. Quarterly Journal of Applied Mathematics, 2(2), pp. 164-168.[6] Marquardt, D. (1963) An algorithm for least-squares estimation of nonlinear parameters. SIAM journal on Applied Mathematics, 11(2), pp. 431-441.[7] Kirkpatrick, S., Gelatt Jr, C., Vecchi, M. (1983) Optimization by simulated annealing. Science 220(4598), pp. 671-680.[8] Nelder, J., Mead, R. (1965) A simplex method for function minimization. Computer Journal, 7(4), pp. 308-313.[9] Press, W., Teukolsky, S., Vetterling, W., Flannery, B. (2007) Numerical recipes - the art of scientific computing (third edition), Cambridge University Press.

[0010] Rueckert, D., Sonoda, L., Hayes, C., Hill, D., Leach, M., Hawkes, D. (1999) Nonrigid registration using free-form deformations: application to breast MR images. IEEE Transactions on Medical Imaging, 18(8), pp. 712-721.Itemized Implementations

[0237] Item Al. A surgical collaboration system for interactive sharing of medical image data comprising: a server module, an interface module, at least one cloud module, and first and second client modules in data communication with each other, the server module configured to: receive, from a Picture Archiving and Communication System (PACS), a set of DICOM images; generate, from the set of DICOM images, a 3D model of an anatomic structure; receive, from a fluoroscopy device comprising a C-arm and one or more detectors, fluoroscopy image data; perform image registration operations to align the 3D model and the fluoroscopy image data; and generate combined image data; the at least one cloud module configured to: receive, from the server module, the combined image data and store the combined image data for access by the first and / or second client module; and / or receive, from the server module, live fluoroscopy image data and transmit the live fluoroscopy image data to the first and / or second client modules; the interface module configured to: receive one or more inputs from a user, the inputs causing the interface module to (i) request image data and corresponding meta data from a local DICOM image store, (ii) transmit image selection and segmentation parameter data to the server module, (iii) cause the server module to generate the combined image data, and / or (iv) cause the server module to upload the combined image data to the at least one cloud module; and the first and second client modules configured to: display the combined image data to a user; receive, from the user of the first client module, a display input; transmit the display input to the at least one cloud module; and / or cause the at least one cloud module to transmit updated image data to the second client module.

[0238] Item A2. The system of item Al, wherein the at least one cloud module comprises: a first cloud module configured to: receive, from the server module, the combined image data and store the combined image data for access by the first and / or second client module; and a second cloud moduleconfigured to: receive, from the server module, live fluoroscopy image data and transmit the live fluoroscopy image data to the first and / or second client modules.

[0239] Item A3. The system of item A2, wherein the interface module is configured to cause the server module to upload the combined image data to the first cloud module; and the first and second client modules are configured to: transmit the display input to the first cloud module; and / or cause the first cloud module to transmit updated image data to the second client module.

[0240] Item A4. The system as in any one of items A2-A3, wherein the first client module is configured to transmit selected 3D landmarks and selected C-arm views to the first cloud module and receive patient configuration data from the first cloud module.

[0241] Item A5. The system as in any one of items A1-A4, wherein the first client module is a tablet module.

[0242] Item A6. The system as in any one of items A2-A5, wherein the server module is configured to receive the configuration data and transmit incoming video stream (comprising live X-ray images) to the first client module via the second cloud module.

[0243] Item A7. The system as in any one of items A1-A6, wherein the first client module is configured to transmit 2D landmarks corresponding to the 3D landmarks to the server module and transmit, to the server module, a command to initialize registration.

[0244] Item A8. The system of item A7, wherein the server module is configured to transmit resulting registration data to the first or second client modules.

[0245] Item A9. The system of item A8, wherein the registration data comprises values of the fluoroscopy device’s kinematic chain parameters.

[0246] Item A 10. The system as in any one of items A1-A9, wherein the server module is configured to transmit parameters for live frontal and lateral projections of an anatomical structure to the second client module.

[0247] Item All. The system as in any one of items Al -A 10, wherein the second client module is a headset module.

[0248] Item A 12. The system as in any one of items Al-All, wherein the first client module is configured to transmit predetermined desired viewing perspectives to the server module.

[0249] Item A13. The system as in any one of items Al -A 12, wherein the server module is configured to calculate required fluoroscopy device settings based on the patient registration and transmit the results to the first or second client modules.

[0250] Item A 14. The system of item A 13, wherein the fluoroscopy device settings comprise C-arm angles or patient table position.

[0251] Item A15. The system as in any one of items Al -A 14, wherein the server module is configured to perform one or more optimization steps using source-to-image distance for each detector of the fluoroscopy device and an image-based metric to determine refined C-arm angles, and transmit the results to the first or second client modules.

[0252] Item A 16. The system as in any one of items Al -A 15, wherein the server module is configured to track a guidewire in frontal and lateral images.

[0253] Item A17. The system of item A16, wherein the server module is configured to perform 3D tip triangulation and reconstruction of a full 3D guidewire extent.

[0254] Item Al 8. The system of item A 17, wherein the server module is configured to transmit reconstruction results to the first or second client modules.

[0255] Item A19. The system as in any one of items A1-A18, wherein the server module comprises a local data storage device.

[0256] Item A20. The system as in any one of items Al -A 19, wherein the server module comprises the local DICOM image store.

[0257] Item A21. The system as in any one of items A2-A20, wherein generating the 3D model comprises one or more of the steps of: performing image segmentation to generate a set of segmented images; generating, from the segmented images, a computational mesh; generating, from the mesh, a patient data set; and uploading the patient data set to the first cloud module or the local storage device.

[0258] Item A22. The system as in any one of items A1-A21, wherein the system is in electronic communication with a scanner device.

[0259] Item A23. The system of item K 1, wherein the scanner device is a CT scanner or an MRI scanner.

[0260] Item A24. The system as in any one of items A7-A23, wherein the server module is configured to perform image registration between the 3D model and the fluoroscopy image data using fiducial markers comprising the 2D landmarks and 3D landmarks.

[0261] Item A25. The system of item A24, wherein the server module is configured to perform a set of image registration refinement steps.

[0262] Item A26. The system as in any one of items A1-A25, wherein the server module is configured to generate and / or execute a kinematic chain model of generic C-arm geometry and / or perform C-arm optimization of the fluoroscopy device.

[0263] Item A27. The system of item A26, wherein the C-arm optimization comprises performing registration proceeds as a series of optimization updates.

[0264] Item A28. The system of item A27, wherein the optimization process comprises generating a virtual model of the fluoroscopy device to produce virtual X-ray projections.

[0265] Item A29. The system of item A28, wherein the optimization process comprises aligning the virtual X-ray projections with live fluoroscopy projections generated by the fluoroscopy device.

[0266] Item A30. The system as in any one of items A27-A29, wherein the optimization process comprises appending a 3-DOF translation and / or a 3-DOF rotation (yaw-pitch-roll) to the kinematic chain model.

[0267] Item A31. The system as in any one of items A27-A30, wherein the optimization process comprises iteratively changing the C-arm angles until the desired view, expressed as the direct relationship between the scan frame (in which the anatomical models exist) and the viewport, is achieved.

[0268] Item A32. The system as in any one of items A1-A31, wherein the system is configured for use with a surgical instrument.

[0269] Item A33. The system of item A32, wherein the surgical instrument is a catheter or a guidewire, each having a tip at their distal end.

[0270] Item A34. The system as in any one of items A32-A33, wherein the server module is configured to perform one or more of the following: track at least a part of the surgical instrument; perform computational 3D reconstruction of at least a part of the instrument to generate an instrument model; combine the instrument model and the 3D model; and perform an instrument tip triangulation operation.

[0271] Item A35. The system of item A34, where in the part of the surgical instrument is the tip.

[0272] Item A36. The system as in any one of items Al -A35, wherein the first and / or second client modules are configured to perform one or more of the following: execute a kinematic chain model of generic C-arm geometry; transmit actions or commands over local network (User Datagram Protocol - UDP) and / or WebRTC; perform instrument tip triangulation; and perform animation of C-arm geometry updates.

[0273] Item A37. The system as in any one of items Al -A36, wherein the first and / or second client modules are configured to perform one or more of the following: render the 3D models and incoming live video feeds; provide support for scene graph hierarchy; provide support for persisted scene states; and provide spatial anchoring (co-registration).

[0274] Item A38. The system as in any one of items Al -A37, wherein the first and / or second client modules are configured to perform one or more of the following: visualize triplanar greyscale images; perform cinematic volumetric rendering of the 3D model; provide gesture control for spatial manipulation; and live-stream fluoroscopy video.

[0275] Item A39. The system as in any one of items A1-A38, wherein the first and / or second client modules are configured to perform one or more of the following: perform an authentication operation; connect to the at least one cloud module; and query connected cameras and initiate live streaming or inject additional video feeds.

[0276] Item A40. The system as in any one of items Al -A39, comprising a controller module configured to direct and prioritize data exchange between the first and second client modules, the server module, and / or the at least one cloud module.

[0277] Item A41. The system as in any one of items A1-A40, comprising a connection module configured to control data exchange with one or more remote DICOM image servers and / or one or more cloud-based media providers.

[0278] Item A42. The system as in any one of items A1-A41, comprising one or more local client modules.

[0279] Item A43. The system as in any one of items A1-A42, wherein the server module is configured to broadcast video to the one or more local client modules; and broadcast actions or commands to the local client modules to achieve synchronization.

[0280] Item A44. The system as in any one of items A2-A43, wherein the first cloud module comprises a cloud image provider.

[0281] Item B 1. A surgical collaboration system comprising: a host module and a client module in data communication with each other, the host module configured to: receive, from a scanner device comprising a PACS, a set of DICOM images of an anatomic structure; perform a checking operation to check for new data; perform image segmentation; and generate the 3D model of the anatomic structure; and the client module configured to: connect to the host module; receive, from the host module, update messages to maintain synchronization; receive notifications of new patient data sets; download the new patient data sets; and display combined 3D model and patient data to a user.

[0282] Item B2. The system of item Bl, wherein the client module is connected to the host module via WiFi.

[0283] Item B3. The system as in any one of items B1-B2, wherein the scanner device is a CT scanner or an MRI scanner.

[0284] Item B4. The system as in any one of items B1-B3, wherein the patient data is or comprises fluoroscopy data.

[0285] Item Cl. A method for interactive sharing of medical image data comprising: performing, by a server module of a surgical collaboration system, the steps of: receiving, from a Picture Archiving and Communication System (PACS), a set of DICOM images; generating, from the set of DICOM images, a 3D model of an anatomic structure; receiving, from a fluoroscopy device comprising a C-arm and one or more detectors, fluoroscopy image data; performing image registration operations to align the 3D model and the fluoroscopy image data; and generating combined image data; performing, by at least one cloud module of the surgical collaboration system, the steps of: receiving, from the server module, the combined image data and storing the combined image data for access by the first and / or second client module; and / or receiving, from the server module, live fluoroscopy image data and transmitting the live fluoroscopy image data to the first and / or second client modules; performing, by an interface module of the surgical collaboration system, the steps of: receiving one or more inputs from a user, the inputs causing the interface module to (i) request image data and corresponding meta data from a local DICOM image store, (ii) transmit image selection and segmentation parameter data to the server module, (iii) cause the server module to generate the combined image data, and / or (iv) cause the server module to upload the combined image data to the at least one cloud module; and performing, by first and second client modules of the surgical collaboration system, the steps of: displaying the combined image data to a user; receiving, fromthe user of the first client module, a display input; transmitting the display input to the at least one cloud module; and / or causing the at least one cloud module to transmit updated image data to the second client module.

[0286] Item C2. The method of item Cl, wherein the at least one cloud module comprises a first cloud module and a second cloud module, the first cloud module receiving, from the server module, the combined image data and storing the combined image data for access by the first and / or second client module; and the second cloud module: receiving, from the server module, live fluoroscopy image data and transmitting the live fluoroscopy image data to the first and / or second client modules.

[0287] Item C3. The method of item C2, comprising causing, by the interface module, the server module to upload the combined image data to the first cloud module; and causing the first and second client modules to: transmit the display input to the first cloud module; and / or cause the first cloud module to transmit updated image data to the second client module.

[0288] Item C4. The method as in any one of items C2-C3, comprising transmitting, by the first client module, selected 3D landmarks and selected C-arm views to the first cloud module and receiving, by the first client module, patient configuration data from the first cloud module.

[0289] Item C5. The method as in any one of items C1-C4, wherein the first client module is a tablet module.

[0290] Item C6. The method as in any one of items C2-C5, comprising receiving, by the server module, the configuration data and transmitting, by the server module, incoming video stream (comprising live X-ray images) to the first client module via the second cloud module.

[0291] Item C7. The method as in any one of items C1-C6, comprising transmitting, by the first client module, 2D landmarks corresponding to the 3D landmarks to the server module and transmitting, by the first client module, to the server module, a command to initialize registration.

[0292] Item C8. The method of item C7, comprising transmitting, by the server module, resulting registration data to the first or second client modules.

[0293] Item C9. The method of item C8, wherein the registration data comprises values of the fluoroscopy device’s kinematic chain parameters.

[0294] Item CIO. The method as in any one of items C1-C9, comprising, transmitting, by the server module, parameters for live frontal and lateral projections of an anatomical structure to the second client module.

[0295] Item C 11. The method as in any one of items C 1 -C 10, wherein the second client module is a headset module.

[0296] Item C12. The method as in any one of items Cl-Cl 1, comprising transmitting, by the first client module, predetermined desired viewing perspectives to the server module.

[0297] Item C13. The method as in any one of items Cl -Cl 2, comprising calculating, by the server module, required fluoroscopy device settings based on the patient registration and transmitting, by the server module, the results to the first or second client modules.

[0298] Item C14. The method of item Cl 3, wherein the fluoroscopy device settings comprise C-arm angles or patient table position.

[0299] Item C15. The method as in any one of items Cl -Cl 4, comprising performing, by the server module, one or more optimization steps using source-to-image distance for each detector of the fluoroscopy device and an image-based metric to determine refined C-arm angles, and transmitting, by the server module, the results to the first or second client modules.

[0300] Item Cl 6. The method as in any one of items Cl -Cl 5, comprising tracking, by the server module, a guidewire in frontal and lateral images.

[0301] Item C17. The method of item C16, comprising performing, by the server module, 3D tip triangulation and reconstruction of a full 3D guidewire extent.

[0302] Item Cl 8. The method of item Cl 7, comprising transmitting, by the server module, reconstruction results to the first or second client modules.

[0303] Item C19. The method as in any one of items C1-C18, wherein the server module comprises a local data storage device.

[0304] Item C20. The method as in any one of items Cl -Cl 9, wherein the server module comprises the local DICOM image store.

[0305] Item C21. The method as in any one of items C2-C20, wherein generating the 3D model comprises one or more of the steps of: performing image segmentation to generate a set of segmented images; generating, from the segmented images, a computational mesh; generating, from the mesh, a patient data set; and uploading the patient data set to the first cloud module or the local storage device.

[0306] Item C22. The method as in any one of items C1-C21, comprising electronic communication by the surgical collaboration system with a scanner device.

[0307] Item C23. The method of item C22, wherein the scanner device is a CT scanner or an MRI scanner.

[0308] Item C24. The method as in any one of items C7-C23, comprising performing, by the server module, image registration between the 3D model and the fluoroscopy image data using fiducial markers comprising the 2D landmarks and 3D landmarks.

[0309] Item C25. The method of item C24, comprising performing, by the server module, a set of image registration refinement steps.

[0310] Item C26. The method as in any one of items C1-C25, comprising generating and / or executing, by the server module, a kinematic chain model of generic C-arm geometry and / or performing, by the server module, C-arm optimization of the fluoroscopy device.

[0311] Item C27. The method of item C26, wherein the C-arm optimization comprises performing registration proceeds as a series of optimization updates.

[0312] Item C28. The method of item C27, wherein the optimization process comprises generating a virtual model of the fluoroscopy device to produce virtual X-ray projections.

[0313] Item C29. The method of item C28, wherein the optimization process comprises aligning the virtual X-ray projections with live fluoroscopy projections generated by the fluoroscopy device.

[0314] Item C30. The method as in any one of items C27-C29, wherein the optimization process comprises appending a 3-DOF translation and / or a 3-DOF rotation (yaw-pitch-roll) to the kinematic chain model.

[0315] Item C31. The method as in any one of items C27-C30, wherein the optimization process comprises iteratively changing the C-arm angles until the desired view, expressed as the direct relationship between the scan frame (in which the anatomical models exist) and the viewport, is achieved.

[0316] Item C32. The method as in any one of items C1-C31, comprising performing the steps in a before or during a surgical procedure using a surgical instrument.

[0317] Item C33. The method of item C32, wherein the surgical procedure comprises using a catheter or a guidewire, each having a tip at their distal end.

[0318] Item C34. The method as in any one of items C32-C33, comprising performing, by the server module, one or more of the steps of: tracking at least a part of the surgical instrument; performing computational 3D reconstruction of at least a part of the instrument to generate an instrument model; combining the instrument model and the 3D model; and performing an instrument tip triangulation operation.

[0319] Item C35. The method of item C34, where in the part of the surgical instrument is the tip.

[0320] Item C36. The method as in any one of items C1-C35, comprising performing, by the first and / or second client modules, one or more of the steps of: executing a kinematic chain model of generic C-arm geometry; transmitting actions or commands over local network (User Datagram Protocol - UDP) and / or WebRTC; performing instrument tip triangulation; and performing animation of C-arm geometry updates.

[0321] Item C37. The method as in any one of items C1-C36, comprising performing, by the first and / or second client modules, one or more of the steps of: rendering the 3D models and incoming live video feeds; providing support for scene graph hierarchy; providing support for persisted scene states; and providing spatial anchoring (co-registration).

[0322] Item C38. The method as in any one of items C1-C37, comprising performing, by the first and / or second client modules, one or more of the steps of: visualizing triplanar greyscale images; performing cinematic volumetric rendering of the 3D model; providing gesture control for spatial manipulation; and live-streaming fluoroscopy video.

[0323] Item C39. The method as in any one of items C1-C38, comprising performing, by the first and / or second client modules, one or more of the steps of: performing an authentication operation; connecting to the at least one cloud module; and querying connected cameras and initiate live streaming or inject additional video feeds.

[0324] Item C40. The method as in any one of items C1-C39, comprising directing and prioritizing, by a controller module of the surgical collaboration system, data exchange between the first and second client modules, the server module, and / or the at least one cloud module.

[0325] Item C41. The method as in any one of items C1-C40, comprising controlling, by a connection module of the surgical collaboration system, data exchange with one or more remote DICOM image servers and / or one or more cloud-based media providers.

[0326] Item C42. The method as in any one of items C1-C41, comprising broadcasting, by the server module, video to the one or more local client modules; and broadcasting, by the server module, actions or commands to the local client modules to achieve synchronization.

[0327] Item C43. The method as in any one of items C2-C42, wherein the first cloud module comprises a cloud image provider.

[0328] Item DI. A method comprising: performing, by a host module, the steps of: receiving, from a scanner device comprising a PACS, a set of DICOM images of an anatomic structure; performing a checking operation to check for new data; performing image segmentation; and generating the 3D model of the anatomic structure; and performing, by a client module, the steps of: connecting to the host module; receiving, from the host module, update messages to maintain synchronization; receiving notifications of new patient data sets; downloading the new patient data sets; and displaying combined 3D model and patient data to a user.

[0329] Item D2. The method of item DI, wherein the client module is connected to the host module via WiFi.

[0330] Item D3. The method as in any one of items D1-D2, wherein the scanner device is a CT scanner or an MRI scanner.

[0331] Item D4. The method as in any one of items D1-D3, wherein the patient data is or comprises fluoroscopy data.

[0332] Item EA1. The method as in any one of items C1-C43 or D1-D4, comprising performing, by one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module, the steps of receiving, a set of first images corresponding to the patient anatomy; generating, from the set of first images, a three-dimensional (3D) image of patient anatomy; storing the 3D image in a memory; receiving one or more live 2D images corresponding to the patient anatomy; retrieving the stored 3D image; aligning the one or more live 2D images with the 3D image; and providing a real-time 3D visualization of the patient anatomy based on the aligned live 2D images and 3D image.

[0333] Item EA2. The method of item EA1, wherein the 3D image is derived from a computer model based on first images of anatomy of multiple patients or multiple scans of the same patient.

[0334] Item EA3. The method as in any one of items EA1-EA2, wherein the 3D image is derived from a computer model based on first images from a phantom model.

[0335] Item EA4. The method as in any one of items EA1-EA3, wherein the first images include Digital Imaging and Communications in Medicine (DICOM) images (“slices”) from computed tomography (CT) imaging.

[0336] Item EA5. The method as in any one of items EA1-EA4, wherein generating the 3D image comprises performing an image segmentation process of the first images, the segmentation process comprising: sorting the first images along their principal axis and storing the sorted images as a set of voxels; optionally, apply a 3D mask to select region of interest in each one of the first images; applying, to each one of the first images, one or more voxel threshold(s); generating, from the set of voxels, a virtual 3D mesh comprising vertices and triangular faces; merging vertices within specified distance tolerance; trimming the mesh; performing a decimation step; applying a smoothing algorithm; and exporting the mesh in OBJ geometry format.

[0337] Item EA6. The method as in any one of items EA1-EA5, wherein the aligning of the 2D images with the 3D images includes a registration step that aligns the 3D image with the one or more 2D images.

[0338] Item EA7. The method of item EA6, wherein the 2D images are obtained from a biplanar fluoroscopy device.

[0339] Item EA8. The method of item EA7, wherein imaging angles of a C-arm of the fluoroscopy device are selected based on a set of virtual projections of the 3D image.

[0340] Item EA9. The method of item EA8, comprising selecting the imaging angle with a least amount of occlusion of imaged features.

[0341] Item EA10. The method of item EA9, comprising counting, for each virtual projection, the number of times pixels are overwritten due to overlap of vessels in a 3D image.

[0342] Item EA11. The method as in any one of items EA1-EA10, wherein the aligning comprises generating a virtual model of a fluoroscopy device.

[0343] Item EA12. The method of item EA11, wherein the virtual model is a mathematical model based on the Denavit-Hartenberg (DH) convention.

[0344] Item EA13. The method of item EA12, wherein the virtual model comprises DH parameters for a frontal C-Arm of a fluoroscopy device, a lateral L-Arc of a fluoroscopy device, a patient table, or a combination thereof.

[0345] Item EA14. The method of item EA13, wherein the virtual model comprises a 3-DOF (degree of freedom) translation or a 3-DOF rotation (yaw-pitch-roll), or both.

[0346] Item EA15. The method as in any one of items EA1-EA14, comprising attaching physical CT-compatible fiducial markers to patient anatomy.

[0347] Item EA16. The method of item EA15, comprising projecting imaged fiducial markers on the 3D image onto virtual detectors in the virtual model, thereby generating one or more virtual 2D fluoroscopy images.

[0348] Item EA17. The method of item EA16, comprising comparing the imaged fiducial markers in the one or more virtual 2D fluoroscopy images with locations of fiducial markers in the live 2D images.

[0349] Item EA18. The method of item EA17, comprising repeating the comparing step and determining a degree of projection overlap.

[0350] Item EA19. The method of item EA18, comprising optimizing alignment using an imagebased comparison metric, wherein the one or more virtual 2D fluoroscopy images are projected onto a set of live 2D images and projection overlap between the one or more virtual 2D fluoroscopy images and one image of the set of live 2D images is determined by counting overlapping pixels and wherein a set of registration parameters is adjusted such that the amount of overlap between the one or more virtual 2D fluoroscopy images and the one image of the set of live 2D images is maximized.

[0351] Item EA20. The method of item EA19, comprising optimizing alignment using a simulated annealing algorithm.

[0352] Item EA21. The method of item EA20, wherein the percentage overlap reaches a plateau in the annealing algorithm indicating maximum pixel overlap.

[0353] Item EA22. The method as in any one of items EA20-EA21, wherein optimizing alignment comprises using a downhill simplex method.

[0354] Item EA23. The method as in any one of items EA7-EA22, comprising projecting a feature from a first view of the biplanar fluoroscopy device to a second, simulated, view of the biplanar fluoroscopy device, the second view having a viewing angle different from the viewing angle of the first view.

[0355] Item EA24. The method as in any one of items EA7-EA23, comprising generating a simulated fluoroscopy view that simulates a view outside the range of a physical fluoroscopy device.

[0356] Item EA25. The method as in any one of items EA1-EA24, comprising generating, from the 3D image, a virtual model based on the mechanical properties of imaged tissue and modelling insertion of an instrument into the tissue.

[0357] Item EA26. The method as in any one of items EA7-EA25, comprising determining registration accuracy using a triangulation scheme.

[0358] Item EA27. The method as in any one of items EA7-EA26, wherein the registration step comprises a modification step of updating the 3D image to account for transformations of space and objects comprising translation, rotation, scaling, or a combination thereof.

[0359] Item EA28. The method of item EA27, wherein the modification comprises applying a volumetric free-form deformation (FFD) technique based on cubic B-splines.

[0360] Item EA29. The method of item EA28, wherein the B-splines are supported by a lattice of control points that are moveable based on updated images.

[0361] Item EA30. The method of item EA29, wherein the updated images are 2D fluoroscopy images.

[0362] Item EA31. The method of item EA30, comprising optimizing alignment between the updated images and corresponding virtual projections of the 3D image using a simulated annealing algorithm, and moving the control points based on the optimized alignment information.

[0363] Item EA32. The method as in any one of items EA29-EA31, wherein the updated images are CT images.

[0364] Item EA33. The method of any one of items EA1-EA32, wherein the 3D visualization includes information indicating the location of an object within the patient.

[0365] Item EA34. The method of item EA33, wherein: the object is a surgical tool; and the information indicating the location of the object is provided in real time.

[0366] Item EA35. The method of any one of items EA1-EA34, comprising simultaneously displaying the one or more 2D images and the 3D image on a visualization screen, and displaying on the visualization screen an indication of a common landmark in the 3D image and in the one or more 2D images, the common landmark representing a single anatomic feature of the patient.

[0367] Item EA36. The method of item EA35, wherein the 3D image is displayed on the visualization screen in between the 2D fluoroscopy images.

[0368] Item EA37. The method of any one of items EA1-EA36, comprising a step of updating an initial 3D image originating from preoperative CT scans with another 3D image derived from an intraoperative CT-spin scan should the patient experience an emergent condition that changes anatomy during a procedure (e.g., ruptured aneurysm).

[0369] Item EA38. The method of item EA37, comprising a step of re-registering the 2D fluoroscopy to the newly updated 3D image.

[0370] Item EA39. The method of any one of items EA35-EA38, wherein the common landmark is a specific portion of the patient’s vasculature, and the instrument is moved past that specific portion based on visualization of the simultaneous display of the 3D and one or more 2D images.

[0371] Item EA40. The method of any of items EA1-EA40, wherein the steps are performed on the server module, the interface module, or a combination thereof.

[0372] Item EA41. A system configured to perform the method of any of items EA1-EA40, the system comprising: means for receiving the 3D image; means for receiving the one or more 2D images; means for processing the 3D image and the one or more 2D images to register the 3D image and the one or more 2D images; and means for simultaneously displaying, on the visualization screen, the 2D images, the 3D image, and an indication of a common landmark in the 2D and 3D images.

[0373] Item EA42. A system comprising: one or more processors; and a memory storing a computer program that, when executed by the one or more processors, causes the performance of the method of any of items EA 1 -EA41.

[0374] Item EB 1. The system as in any one of items A 1 -A44 or B 1 -B4, wherein one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to receive a set of first images corresponding to the patient anatomy; generatefrom the set of first images, a three-dimensional (3D) image of the patient anatomy; store the 3D image in a memory; receive one or more live 2D images corresponding to the patient anatomy; retrieve the stored 3D image; align the one or more live 2D images with the 3D image; and cause to display a real-time 3D visualization of the patient anatomy based on the aligned live 2D images and 3D image.

[0375] Item EB2. The system of item EB1, wherein the 3D image is derived from a computer model based on first images of anatomy of multiple patients or multiple scans of the same patient.

[0376] Item EB3. The system as in any one of items EB1E-B2, wherein the 3D image is derived from a computer model based on first images from a phantom model.

[0377] Item EB4. The system as in any one of items EB1-EB3, wherein the first images include Digital Imaging and Communications in Medicine (DICOM) images (“slices”) from computed tomography (CT) imaging.

[0378] Item EB5. The system as in any one of items EB1-EB4, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to generate the 3D image by performing an image segmentation process of the first images, the segmentation process comprising: sorting the first images along their principal axis and storing the sorted images as a set of voxels; optionally, apply a 3D mask to select region of interest in each one of the first images; applying, to each one of the first images, one or more voxel threshold(s); generating, from the set of voxels, a virtual 3D mesh comprising vertices and triangular faces; merging vertices within specified distance tolerance; trimming the mesh; performing a decimation step; applying a smoothing algorithm; and exporting the mesh in OBJ geometry format.

[0379] Item EB6. The system as in any one of items EB1-EB5, wherein the aligning of the 2D images with the 3D images includes a registration step that aligns the 3D image with the one or more 2D images.

[0380] Item EB7. The system of item EB6, wherein the 2D images are obtained from a biplanar fluoroscopy device.

[0381] Item EB8. The system of item EB7, wherein imaging angles of a C-arm of the fluoroscopy device are selected based on a set of virtual projections of the 3D image.

[0382] Item EB9. The system of item EB8, wherein the imaging angle with a least amount of occlusion of imaged features is selected.

[0383] Item EB 10. The system of item EB9, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to to count, for each virtual projection, the number of times pixels are overwritten due to overlap of vessels in a 3D image.

[0384] Item EB 11. The system as in any one of items EB 1-EB 10, wherein the aligning comprises generating a virtual model of a fluoroscopy device.

[0385] Item EB 12. The system of item EB 11, wherein the virtual model is a mathematical model based on the Denavit-Hartenberg (DH) convention.

[0386] Item EB13. The system of item EB12, wherein the virtual model comprises DH parameters for a frontal C-Arm of a fluoroscopy device, a lateral L-Arc of a fluoroscopy device, a patient table, or a combination thereof.

[0387] Item EB14. The system of item EB13, wherein the virtual model comprises a 3-DOF (degree of freedom) translation or a 3-DOF rotation (yaw-pitch-roll), or both.

[0388] Item EB15. The system as in any one of items EB1-EB14, comprising physical CT-compatible fiducial markers attached to patient anatomy.

[0389] Item EB 16. The system of item EB 15, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to to project imaged fiducial markers on the 3D image onto virtual detectors in the virtual model, thereby generating one or more virtual 2D fluoroscopy images.

[0390] Item EB 17. The system of item EB 16, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to to compare the imaged fiducial markers in the one or more virtual 2D fluoroscopy images with locations of fiducial markers in the live 2D images.

[0391] Item EB 18. The system of item EB 17, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to to repeat the comparing step and determining a degree of projection overlap.

[0392] Item EB19. The system of item EB18, wherein the one ormore of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to optimize alignment using an image-based comparison metric, wherein the one or more virtual 2D fluoroscopy images are projected onto a set of live 2D images and projection overlap between the one or more virtual 2D fluoroscopy images and one image of the set of live 2D images is determined by counting overlapping pixels and wherein a set of registration parameters is adjusted such that the amount of overlap between the one or more virtual 2D fluoroscopy images and the one image of the set of live 2D images is maximized.

[0393] Item EB20. The system of item EB 19, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to to optimize alignment using a simulated annealing algorithm.

[0394] Item EB21. The system of item EB20, wherein the percentage overlap reaches a plateau in the annealing algorithm indicating maximum pixel overlap.

[0395] Item EB22. The system as in any one of items EB20-EB21, wherein optimizing alignment comprises using a downhill simplex method.

[0396] Item EB23. The system as in any one of items EB7-EB22, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloudmodule are configured to project a feature from a first view of the biplanar fluoroscopy device to a second, simulated, view of the biplanar fluoroscopy device, the second view having a viewing angle different from the viewing angle of the first view.

[0397] Item EB24. The system as in any one of items EB7-EB23, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to to generate a simulated fluoroscopy view that simulates a view outside the range of a physical fluoroscopy device.

[0398] Item EB25. The system as in any one of items EB1-EB24, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to to generate, from the 3D image, a virtual model based on the mechanical properties of imaged tissue and model insertion of an instrument into the tissue.

[0399] Item EB26. The system as in any one of items EB7-EB25, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to to determine registration accuracy using a triangulation scheme.

[0400] Item EB27. The system as in any one of items EB7-EB26, wherein the registration step comprises a modification step of updating the 3D image to account for transformations of space and objects comprising translation, rotation, scaling, or a combination thereof.

[0401] Item EB28. The system of item EB27, wherein the modification comprises applying a volumetric free-form deformation (FFD) technique based on cubic B-splines.

[0402] Item EB29. The system of item EB28, wherein the B-splines are supported by a lattice of control points that are moveable based on updated images.

[0403] Item EB30. The system of item EB29, wherein the updated images are 2D fluoroscopy images.

[0404] Item EB31. The system of item EB30, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to optimize alignment between the updated images and corresponding virtual projections of the 3D image using a simulated annealing algorithm, and moving the control points based on the optimized alignment information.

[0405] Item EB32. The system as in any one of items EB29-EB31, wherein the updated images are CT images.

[0406] Item EB33. The system of any one of items EB1-EB32, wherein the 3D visualization includes information indicating the location of an object within the patient.

[0407] Item EB34. The system of item EB33, wherein: the object is a surgical tool; and the information indicating the location of the object is provided in real time.

[0408] Item EB35. The system of any one of items EB1-EB34, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to simultaneously display the one or more 2D images and the 3D image on avisualization screen, and display on the visualization screen an indication of a common landmark in the 3D image and in the one or more 2D images, the common landmark representing a single anatomic feature of the patient.

[0409] Item EB36. The system of item EB35, wherein the 3D image is displayed on the visualization screen in between the 2D fluoroscopy images.

[0410] Item EB37. The system of any one of items EB1-EB36, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to update an initial 3D image originating from preoperative CT scans with another 3D image derived from an intraoperative CT-spin scan should the patient experience an emergent condition that changes anatomy during a procedure (e.g., ruptured aneurysm).

[0411] Item EB38. The system of item EB37, wherein the one or more of the server module, the at least one cloud module, the interface module, and / or the first and second cloud module are configured to re-register the 2D fluoroscopy to the newly updated 3D image.What is claimed is:

Claims

CLAIMS1. A surgical collaboration system for interactive sharing of medical image data comprising:a server module, an interface module, at least one cloud module, and first and second client modules in data communication with each other,the server module configured to:receive, from a Picture Archiving and Communication System (PACS), a set of DICOM images;generate, from the set of DICOM images, a 3D model of an anatomic structure;receive, from a fluoroscopy device comprising a C-arm and one or more detectors, fluoroscopy image data;perform image registration operations to align the 3D model and the fluoroscopy image data; andgenerate combined image data;the at least one cloud module configured to:receive, from the server module, the combined image data and store the combined image data for access by the first and / or second client module; and / orreceive, from the server module, live fluoroscopy image data and transmit the live fluoroscopy image data to the first and / or second client modules; the interface module configured to:receive one or more inputs from a user, the inputs causing the interface module to (i) request image data and corresponding meta data from a local DICOM image store, (ii) transmit image selection and segmentation parameter data to the server module, (iii) cause the server module to generate the combined image data, and / or (iv) cause the server module to upload the combined image data to the at least one cloud module; andthe first and second client modules configured to:display the combined image data to a user;receive, from the user of the first client module, a display input; transmit the display input to the at least one cloud module; and / or cause the at least one cloud module to transmit updated image data to the second client module.

2. The system of claim 1, wherein the at least one cloud module comprises:a first cloud module configured to:receive, from the server module, the combined image data and store the combined image data for access by the first and / or second client module; and a second cloud module configured to:receive, from the server module, live fluoroscopy image data and transmit the live fluoroscopy image data to the first and / or second client modules.

3. The system of claim 2, whereinthe interface module is configured to cause the server module to upload the combined image data to the first cloud module; andthe first and second client modules are configured to:transmit the display input to the first cloud module; and / orcause the first cloud module to transmit updated image data to the second client module.

4. The system as in any one of claims 2-3, wherein the first client module is configured to transmit selected 3D landmarks and selected C-arm views to the first cloud module and receive patient configuration data from the first cloud module.

5. The system as in any one of claims 1-4, wherein the first client module is a tablet module.

6. The system as in any one of claims 2-5, wherein the server module is configured to receive the configuration data and transmit incoming video stream (comprising live X-ray images) to the first client module via the second cloud module.

7. The system as in any one of claims 1-6, wherein the first client module is configured to transmit 2D landmarks corresponding to the 3D landmarks to the server module and transmit, to the server module, a command to initialize registration.

8. The system of claim 7, wherein the server module is configured to transmit resulting registration data to the first or second client modules.

9. The system of claim 8, wherein the registration data comprises values of the fluoroscopy devices’s kinematic chain parameters.

10. The system as in any one of claims 1-9, wherein the server module is configured to transmit parameters for live frontal and lateral projections of an anatomical structure to the second client module.

11. The system as in any one of claims 1-10, wherein the second client module is a headset module.

12. The system as in any one of claims 1-11, wherein the first client module is configured to transmit predetermined desired viewing perspectives to the server module.

13. The system as in any one of claims 1-12, wherein the server module is configured to calculate required fluoroscopy device settings based on the patient registration and transmit the results to the first or second client modules.

14. The system of claim 13, wherein the fluoroscopy device settings comprise C-arm angles or patient table position.

15. The system as in any one of claims 1-14, wherein the server module is configured to perform one or more optimization steps using source-to-image distance for each detector of the fluoroscopy device and an image-based metric to determine refined C-arm angles, and transmit the results to the first or second client modules.

16. The system as in any one of claims 1-15, wherein the server module is configured to track a guidewire in frontal and lateral images.

17. The system of claim 16, wherein the server module is configured to perform 3D tip triangulation and reconstruction of a full 3D guidewire extent.

18. The system of claim 17, wherein the server module is configured to transmit reconstruction results to the first or second client modules.

19. The system as in any one of claims 1-18, wherein the server module comprises a local data storage device.

20. The system as in any one of claims 1-19, wherein the server module comprises the local DICOM image store.

21. The system as in any one of claims 2-20, wherein generating the 3D model comprises one or more of the steps of:performing image segmentation to generate a set of segmented images; generating, from the segmented images, a computational mesh;generating, from the mesh, a patient data set; anduploading the patient data set to the first cloud module or the local storage device.

22. The system as in any one of claims 1-21, wherein the system is in electronic communication with a scanner device.

23. The system of claim 22, wherein the scanner device is a CT scanner or an MRI scanner.

24. The system as in any one of claims 7-23, wherein the server module is configured to perform image registration between the 3D model and the fluoroscopy image data using fiducial markers comprising the 2D landmarks and 3D landmarks.

25. The system of claim 24, wherein the server module is configured to perform a set of image registration refinement steps.

26. The system as in any one of claims 1-25, wherein the server module is configured to generate and / or execute a kinematic chain model of generic C-arm geometry and / or perform C-arm optimization of the fluoroscopy device.

27. The system of claim 26, wherein the C-arm optimization comprises performing registration proceeds as a series of optimization updates.

28. The system of claim 27, wherein the optimization process comprises generating a virtual model of the fluoroscopy device to produce virtual X-ray projections.

29. The system of claim 28, wherein the optimization process comprises aligning the virtual X-ray projections with live fluoroscopy projections generated by the fluoroscopy device.

30. The system as in any one of claims 27-29, wherein the optimization process comprises appending a 3-DOF translation and / or a 3-DOF rotation (yaw-pitch-roll) to the kinematic chain model.

31. The system as in any one of claims 27-30, wherein the optimization process comprises iteratively changing the C-arm angles until the desired view, expressed as the direct relationship between the scan frame (in which the anatomical models exist) and the viewport, is achieved.

32. The system as in any one of claims 1-31, wherein the system is configured for use with a surgical instrument.

33. The system of claim 32, wherein the surgical instrument is a catheter or a guidewire, each having a tip at their distal end.

34. The system as in any one of claims 32-33, wherein the server module is configured to perform one or more of the following:track at least a part of the surgical instrument;perform computational 3D reconstruction of at least a part of the instrument to generate an instrument model;combine the instrument model and the 3D model; andperform an instrument tip triangulation operation.

35. The system of claim 34, where in the part of the surgical instrument is the tip.

36. The system as in any one of claims 1-35, wherein the first and / or second client modules are configured to perform one or more of the following:execute a kinematic chain model of generic C-arm geometry;transmit actions or commands over local network (User Datagram Protocol - UDP) and / or WebRTC;perform instrument tip triangulation; andperform animation of C-arm geometry updates.

37. The system as in any one of claims 1-36, wherein the first and / or second client modules are configured to perform one or more of the following:render the 3D models and incoming live video feeds;provide support for scene graph hierarchy;provide support for persisted scene states; andprovide spatial anchoring (co-regi strati on).

38. The system as in any one of claims 1-37, wherein the first and / or second client modules are configured to perform one or more of the following:visualize triplanar greyscale images;perform cinematic volumetric rendering of the 3D model;provide gesture control for spatial manipulation; andlive-stream fluoroscopy video.

39. The system as in any one of claims 1-38, wherein the first and / or second client modules are configured to perform one or more of the following:perform an authentication operation;connect to the at least one cloud module; andquery connected cameras and initiate live streaming or inject additional video feeds.

40. The system as in any one of claims 1-39, comprising a controller module configured to direct and prioritize data exchange between the first and second client modules, the server module, and / or the at least one cloud module.

41. The system as in any one of claims 1-40, comprising a connection module configured to control data exchange with one or more remote DICOM image servers and / or one or more cloud-based media providers.

42. The system as in any one of claims 1-41, comprising one or more local client modules.

43. The system as in any one of claims 1-42, wherein the server module is configured tobroadcast video to the one or more local client modules; andbroadcast actions or commands to the local client modules to achieve synchronization.

44. The system as in any one of claims 2-43, wherein the first cloud module comprises a cloud image provider.

45. A surgical collaboration system comprising:a host module and a client module in data communication with each other, the host module configured to:receive, from a scanner device comprising a PACS, a set of DICOM images of an anatomic structure;perform a checking operation to check for new data;perform image segmentation; andgenerate the 3D model of the anatomic structure; andthe client module configured to:connect to the host module;receive, from the host module, update messages to maintain synchronization; receive notifications of new patient data sets;download the new patient data sets; anddisplay combined 3D model and patient data to a user.

46. The system of claim 45, wherein the client module is connected to the host module via WiFi.

47. The system as in any one of claims 45-46, wherein the scanner device is a CT scanner or an MRI scanner.

48. The system as in any one of claims 45-47, wherein the patient data is or comprises fluoroscopy data.

49. A method for interactive sharing of medical image data comprising:performing, by a server module of a surgical collaboration system, the steps of:receiving, from a Picture Archiving and Communication System (PACS), a set of DICOM images;generating, from the set of DICOM images, a 3D model of an anatomic structure;receiving, from a fluoroscopy device comprising a C-arm and one or more detectors, fluoroscopy image data;performing image registration operations to align the 3D model and the fluoroscopy image data; andgenerating combined image data;performing, by at least one cloud module of the surgical collaboration system, the steps of:receiving, from the server module, the combined image data and storing the combined image data for access by the first and / or second client module; and / orreceiving, from the server module, live fluoroscopy image data and transmitting the live fluoroscopy image data to the first and / or second client modules;performing, by an interface module of the surgical collaboration system, the steps of:receiving one or more inputs from a user, the inputs causing the interface module to (i) request image data and corresponding meta data from a local DICOM image store, (ii) transmit image selection and segmentation parameter data to the server module, (iii) cause the server module to generate the combined image data, and / or (iv) cause the server module to upload the combined image data to the at least one cloud module; andperforming, by first and second client modules of the surgical collaboration system, the steps of:displaying the combined image data to a user;receiving, from the user of the first client module, a display input; transmitting the display input to the at least one cloud module; and / or causing the at least one cloud module to transmit updated image data to the second client module.

50. The method of claim 49, wherein the at least one cloud module comprises a first cloud module and a second cloud module,the first cloud modulereceiving, from the server module, the combined image data and storing the combined image data for access by the first and / or second client module; and the second cloud module:receiving, from the server module, live fluoroscopy image data and transmitting the live fluoroscopy image data to the first and / or second client modules.

51. The method of claim 50, comprisingcausing, by the interface module, the server module to upload the combined image data to the first cloud module; andcausing the first and second client modules to:transmit the display input to the first cloud module; and / orcause the first cloud module to transmit updated image data to the second client module.

52. The method as in any one of claims 50-51, comprising transmitting, by the first client module, selected 3D landmarks and selected C-arm views to the first cloud module and receiving, by the first client module, patient configuration data from the first cloud module.

53. The method as in any one of claims 49-52, wherein the first client module is a tablet module.

54. The method as in any one of claims 50-53, comprising receiving, by the server module, the configuration data and transmitting, by the server module, incoming video stream (comprising live X-ray images) to the first client module via the second cloud module.

55. The method as in any one of claims 49-54, comprising transmitting, by the first client module, 2D landmarks corresponding to the 3D landmarks to the server module and transmitting, by the first client module, to the server module, a command to initialize registration.

56. The method of claim 55, comprising transmitting, by the server module, resulting registration data to the first or second client modules.

57. The method of claim 56, wherein the registration data comprises values of the fluoroscopy device’s kinematic chain parameters.

58. The method as in any one of claims 49-57, comprising, transmitting, by the server module, parameters for live frontal and lateral projections of an anatomical structure to the second client module.

59. The method as in any one of claims 49-58, wherein the second client module is a headset module.

60. The method as in any one of claims 49-59, comprising transmitting, by the first client module, predetermined desired viewing perspectives to the server module.

61. The method as in any one of claims 49-60, comprising calculating, by the server module, required fluoroscopy device settings based on the patient registration and transmitting, by the server module, the results to the first or second client modules.

62. The method of claim 61, wherein the fluoroscopy device settings comprise C-arm angles or patient table position.

63. The method as in any one of claims 49-62, comprising performing, by the server module, one or more optimization steps using source-to-image distance for each detector of the fluoroscopy device and an image-based metric to determine refined C-arm angles, and transmitting, by the server module, the results to the first or second client modules.

64. The method as in any one of claims 49-63, comprising tracking, by the server module, a guidewire in frontal and lateral images.

65. The method of claim 64, comprising performing, by the server module, 3D tip triangulation and reconstruction of a full 3D guidewire extent.

66. The method of claim 65, comprising transmitting, by the server module, reconstruction results to the first or second client modules.

67. The method as in any one of claims 49-66, wherein the server module comprises a local data storage device.

68. The method as in any one of claims 49-67, wherein the server module comprises the local DICOM image store.

69. The method as in any one of claims 50-68, wherein generating the 3D model comprises one or more of the steps of:performing image segmentation to generate a set of segmented images; generating, from the segmented images, a computational mesh;generating, from the mesh, a patient data set; anduploading the patient data set to the first cloud module or the local storage device.

70. The method as in any one of claims 49-69, comprising electronic communication by the surgical collaboration system with a scanner device.

71. The method of claim 70, wherein the scanner device is a CT scanner or an MRI scanner.

72. The method as in any one of claims 55-71, comprising performing, by the server module, image registration between the 3D model and the fluoroscopy image data using fiducial markers comprising the 2D landmarks and 3D landmarks.

73. The method of claim 72, comprising performing, by the server module, a set of image registration refinement steps.

74. The method as in any one of claims 49-73, comprising generating and / or executing, by the server module, a kinematic chain model of generic C-arm geometry and / or performing, by the server module, C-arm optimization of the fluoroscopy device.

75. The method of claim 74, wherein the C-arm optimization comprises performing registration proceeds as a series of optimization updates.

76. The method of claim 75, wherein the optimization process comprises generating a virtual model of the fluoroscopy device to produce virtual X-ray projections.

77. The method of claim 76, wherein the optimization process comprises aligning the virtual X-ray projections with live fluoroscopy projections generated by the fluoroscopy device.

78. The method as in any one of claims 75-78, wherein the optimization process comprises appending a 3-DOF translation and / or a 3-DOF rotation (yaw-pitch-roll) to the kinematic chain model.

79. The method as in any one of claims 75-78, wherein the optimization process comprises iteratively changing the C-arm angles until the desired view, expressed as the direct relationship between the scan frame (in which the anatomical models exist) and the viewport, is achieved.

80. The method as in any one of claims 49-79, comprising performing the steps in a before or during a surgical procedure using a surgical instrument.

81. The method of claim 80, wherein the surgical procedure comprises using a catheter or a guidewire, each having a tip at their distal end.

82. The method as in any one of claims 80-81, comprising performing, by the server module, one or more of the steps of:tracking at least a part of the surgical instrument;performing computational 3D reconstruction of at least a part of the instrument to generate an instrument model;combining the instrument model and the 3D model; andperforming an instrument tip triangulation operation.

83. The method of claim 82, where in the part of the surgical instrument is the tip.

84. The method as in any one of claims 49-83, comprising performing, by the first and / or second client modules, one or more of the steps of:executing a kinematic chain model of generic C-arm geometry;transmitting actions or commands over local network (User Datagram Protocol -UDP) and / or WebRTC;performing instrument tip triangulation; andperforming animation of C-arm geometry updates.

85. The method as in any one of claims 49-84, comprising performing, by the first and / or second client modules, one or more of the steps of:rendering the 3D models and incoming live video feeds;providing support for scene graph hierarchy;providing support for persisted scene states; andproviding spatial anchoring (co-regi strati on).

86. The method as in any one of claims 49-85, comprising performing, by the first and / or second client modules, one or more of the steps of:visualizing triplanar greyscale images;performing cinematic volumetric rendering of the 3D model;providing gesture control for spatial manipulation; andlive-streaming fluoroscopy video.

87. The method as in any one of claims 49-86, comprising performing, by the first and / or second client modules, one or more of the steps of:performing an authentication operation;connecting to the at least one cloud module; andquerying connected cameras and initiate live streaming or inject additional video feeds.

88. The method as in any one of claims 49-87, comprising directing and prioritizing, by a controller module of the surgical collaboration system, data exchange between the first and second client modules, the server module, and / or the at least one cloud module.

89. The method as in any one of claims 49-88, comprising controlling, by a connection module of the surgical collaboration system, data exchange with one or more remote DICOM image servers and / or one or more cloud-based media providers.

90. The method as in any one of claims 48-89, comprisingbroadcasting, by the server module, video to the one or more local client modules; andbroadcasting, by the server module, actions or commands to the local client modules to achieve synchronization.

91. The method as in any one of claims 50-90, wherein the first cloud module comprises a cloud image provider.

92. A method comprising:performing, by a host module, the steps of:receiving, from a scanner device comprising a PACS, a set of DICOM images of an anatomic structure;performing a checking operation to check for new data;performing image segmentation; andgenerating the 3D model of the anatomic structure; andperforming, by a client module, the steps of:connecting to the host module;receiving, from the host module, update messages to maintain synchronization;receiving notifications of new patient data sets;downloading the new patient data sets; anddisplaying combined 3D model and patient data to a user.

93. The method of claim 92, wherein the client module is connected to the host module via WiFi.

94. The method as in any one of claims 92-93, wherein the scanner device is a CT scanner or an MRI scanner.

95. The method as in any one of claims 92-94, wherein the patient data is or comprises fluoroscopy data.

Images