Anatomical visualization system using intraoperative imaging
The surgical imaging and guidance system addresses the challenges of precise intraoperative visualization and tool tracking by integrating RGB/depth imaging and external monitoring data, providing real-time anatomical mapping and navigation enhancements.
Patent Information
- Application Number
- PCT/IB2026/054701
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-12
- Filing Date
- 2026-05-12
- Publication Date
- 2026-11-19
AI Technical Summary
Existing surgical navigation techniques struggle with precise intraoperative visualization, anatomical mapping, and dynamic tool tracking, particularly in complex medical procedures, lacking real-time integration of preoperative and intraoperative imaging data and external monitoring systems.
A surgical imaging and guidance system that integrates intraoperative RGB/depth imaging, provides dynamic anatomical models, and tracks surgical tools, enhancing visualization and navigation by registering preoperative and intraoperative data, and fusing external monitoring data in real-time.
Enables precise anatomical mapping, dynamic tool tracking, and improved surgical planning and safety by overlaying intraoperative data on preoperative images, integrating external monitoring data for enhanced spatial awareness and efficiency.
Smart Images

Figure IB2026054701_19112026_PF_FP_ABST
Abstract
Description
[0001] ANATOMICAL VISUALIZATION SYSTEM USING INTRAOPERATIVE IMAGING
[0002] CROSS-REFERENCES TO RELATED APPLICATIONS The present application claims priority from U.S. Provisional Patent Application No.
[0003] 63 / 804,145 to Racheli, filed May 12, 2025, entitled "Real-Time Anatomical Visualization and Tracking System Using Intraoperative 3D Imaging," which is incorporated herein by reference.
[0004] The present application is related to PCT Application No. PCT / IB2023 / 061590 filed November 16, 2023 (published as WO 2024 / 105607), to Elimelech, entitled "Apparatus and Methods for Performing a Medical Procedure," which is incorporated herein by reference.
[0005] TECHNICAL FIELD
[0006] The present disclosure relates to methods and apparatus for use in medical procedures, and particularly surgical imaging and navigation apparatus and methods.
[0007] BACKGROUND
[0008] Surgical navigation techniques are used in several different types of medical procedures, such as neurosurgery, spinal surgery, orthopedic surgery, and pulmonary procedures. Such techniques allow physicians to observe the current location of a surgical instrument with respect to preoperative imaging data. Typically, the preoperative imaging data includes CT and / or MRI images. In some cases, pre-planning of the surgery is performed with reference to the preoperative imaging data and the navigation allows the physician to observe the current location of the instrument with respect to a pre-planned trajectory and / or with respect to a pre-planned target, such as a lesion, a tumor, etc.
[0009] In order to perform the surgical navigation, the patient’s anatomy is coregistered to the preoperative imaging data, such that corresponding points in the patient’s anatomy and the preoperative imaging data are aligned with each other within a common coordinate system. By virtue of the coregistration of the patient’s anatomy to the preoperative imaging data, the location of the surgical instrument with respect to the preoperative imaging data and / or with respect to the preoperative planning is thereby derived.SUMMARY
[0010] In accordance with some applications of the present disclosure, a surgical imaging and guidance system is provided that provides intraoperative image frames, which may be two-dimensional (“2D”) image frames and / or three-dimensional (“3D”) image frames. For some applications, the surgical imaging and guidance system integrates intraoperative RGB imaging with depth sensing to generate three-dimensional RGB-depth image frames (also referred to hereinafter as a three-dimensional RGB-depth point cloud, or as a “3D RGB-D point cloud”) that provide a dynamic anatomical model in real time. Alternatively or additionally, the surgical imaging and guidance system integrates grayscale imaging with depth sensing to generate three-dimensional grayscale image frames. The system enhances intraoperative visualization, enables precise anatomical mapping, supports intelligent object segmentation, and provides dynamic tool tracking during surgical procedures. For some applications, certain features are applied to intraoperative two-dimensional (“2D”) image frames, as described in further detail hereinbelow.
[0011] Designed for surgical environments, the system typically enables precise visualization of anatomical structures, dynamic tracking of surgical tools, and intelligent scene interpretation. The system typically delivers enhanced spatial awareness, supports surgical planning and navigation, and improves safety and efficiency during complex procedures. For some applications, the features described herein are applied to two-dimensional intraoperative imaging data and / or three-dimensional intraoperative imaging data.
[0012] For some applications, registration between pre-operative imaging data and intraoperative image frames is performed by matching between an intraoperative image frame of an anatomical structure of the patient (e.g., a rigid anatomical structure, such as the patient’s face) and a corresponding mesh that is extracted from the same anatomical structure from the pre-operative imaging data. Typically, the intraoperative image frame of the anatomical structure of the patient is a 3D intraoperative image frame of the anatomical structure of the patient and the corresponding mesh that is extracted from the same anatomical structure from the pre-operative imaging data is a 3D mesh that is extracted from the same anatomical structure from the pre-operative imaging data.
[0013] For some applications, the system automatically segments structures within intraoperative 3D image frames and / or within intraoperative 2D image frames. In some embodiments, the system is configured to segment anatomical structures, at least partiallybased on temporal behaviors of the structures. For some applications, only specific structures or classes of structure are segmented. For example, only cortical parts or only anatomical structures may be segmented. For some applications, healthy tissue and tumors are segmented.
[0014] For some applications, the system detects a tool and identifies the tool tip within an intraoperative image frame using geometric cues. Typically, the orientation of the tool is determined by comparing the observed tool profile with a known mechanical model, thereby providing a precise estimation of the pose of the tool in 6 degrees-of-freedom (“DOF”). The mechanical model can be generated either from reference data or by scanning the tool using an imaging device of the system.
[0015] For some applications, the computer processor tracks identified anatomical and non-anatomical elements across sequential intraoperative 3D or 2D image frames. For some applications, the tracked element is assigned a color in order to allow the user to easily follow the tracked element in successive image frames. For some applications, the structure(s) is colored in accordance with a color scheme, as described hereinbelow. For some such applications, motion trajectories of given structures are calculated even in the presence of occlusion (i.e., if the view of the structure is obscured), thereby providing continuity in the tracking of relevant structures. Typically, a level of confidence (e.g., a confidence score) is assigned to the tracking of each element between frames, and the computer processor accepts or ignores the tracking based on the level of confidence of the tracking. For some applications, in order to perform the tracking, a light beam is projected over the surgical field of view from an external light source (e.g., a laser light source) and the computer processor tracks the light beam. For some applications, the user provides an input designating an element for tracking and optionally annotating the element. Subsequently, the designated (and optionally colored and / or annotated) element is tracked. For some applications, a video stream is stabilized around the tracked (and optionally colored and / or annotated) element, such that the tracked element appears stationary.
[0016] For some applications, subsequent to the registration between the pre-operative imaging data and the intraoperative image frames, the current position of structures that have been segmented and tracked within the intraoperative image frames are overlaid on a 2D multi-planar-reconstruction (hereinafter “MPR”) slice (e.g., an axial, sagittal, and / or coronal slice) that is generated from preoperative imaging data. Alternatively, or additionally, the current position of structures that have been segmented and tracked within the intraoperative imageframes are overlaid on a 3D mesh that is generated from the preoperative imaging data (referred to herein as the “preoperative 3D mesh”).
[0017] For some applications, one or more of the intraoperative image frames are segmented (e.g., as described hereinabove), and only specific structures or classes of structure are overlaid on the 2D multi-planar-reconstruction slice and / or the 3D mesh that is based on preoperative imaging. For example, only cortical parts or only anatomical structures may be included in the overlay. For some applications, the MPR slice upon which the structure(s) is overlaid is selected based on the position of the structure within the intraoperative image frame. For some applications, the structure(s) is colored in accordance with a color scheme, as described hereinbelow.
[0018] For some applications, elements in the intraoperative 3D image frames or in intraoperative 2D image frames are colored. For some such applications, the elements are colored automatically by the computer processor, for example, based on real-time input from external monitoring systems (e.g., neuromonitoring, thermal sensing), and / or based on segmentation of specific structures within the intraoperative image frame. For example, a structure may be segmented based on parameters of the structure that are detected by external monitoring systems (e.g., neuromonitoring, thermal sensing). Alternatively or additionally, the elements are colored manually with user-defined color settings to emphasize or track specific structures.
[0019] There is therefore provided, in accordance with some applications of the present disclosure, apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus including a computer processor configured to:
[0020] receive preoperative imaging data of the portion of the subject's body;
[0021] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0022] register between the preoperative imaging data and the intraoperative imaging data; receive data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; and fuse the data from the one or more external physiological monitoring systems with the preoperative imaging data via the registration between the preoperative imaging data and the intraoperative imaging data.In some applications, the computer processor is further configured to drive a display device to display a visualization of the preoperative imaging data overlaid with the data from the one or more external physiological monitoring systems based on the fused data.
[0023] In some applications, the computer processor is further configured to drive the display device to display the visualization in real time during the surgical procedure.
[0024] In some applications, the computer processor is further configured to drive the display device to display the visualization on demand in response to a user input.
[0025] In some applications, the one or more external physiological monitoring systems include at least one monitoring system selected from the group consisting of: a neuromonitoring system, a temperature sensing system, a tissue impedance measurement system, and an ultrasound imaging system.
[0026] In some applications, the computer processor is further configured to color elements within the intraoperative imaging data based on the received data from the one or more external physiological monitoring systems.
[0027] In some applications, the computer processor is further configured to segment at least one intraoperative image frame of the intraoperative imaging data, and to color a segmented structure based on a parameter of the segmented structure detected by the one or more external physiological monitoring systems.
[0028] In some applications, the computer processor is further configured to receive patient monitoring signals including one or more monitoring signals selected from the list consisting of: heart rate, blood pressure, and oxygen saturation, and to display the patient monitoring signals in association with the intraoperative imaging data.
[0029] In some applications, the computer processor is further configured to spatially reference the data from the one or more external physiological monitoring systems to the intraoperative imaging data by tracking a sensor of the one or more external physiological monitoring systems using the intraoperative imaging device.
[0030] In some applications, the computer processor is further configured to spatially reference the data from the one or more external physiological monitoring systems to the intraoperative imaging data by means of a user input indicating a spatial correspondence between the data and a location within the intraoperative imaging data.In some applications, the computer processor is further configured to spatially reference the data from the one or more external physiological monitoring systems to the intraoperative imaging data by identifying corresponding landmarks in the data and in the intraoperative imaging data.
[0031] In some applications, the computer processor is further configured to generate a two-dimensional multi-planar-reconstruction slice from the preoperative imaging data, and to modify a display of the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data.
[0032] In some applications, the computer processor is configured to modify the display of the two-dimensional multi-planar-reconstruction slice by deforming the preoperative imaging data based on the intraoperative imaging data to match a current position and shape of the portion of the subject's body, and to generate the two-dimensional multi -planar-reconstruction slice from the deformed preoperative imaging data.
[0033] In some applications, the computer processor is further configured to drive a display device to display the two-dimensional multi-planar-reconstruction slice and a view of the intraoperative imaging data in a side-by-side arrangement.
[0034] In some applications, the computer processor is further configured to drive the display device to display a marker line on the two-dimensional multi-planar-reconstruction slice and a corresponding marker line on the view of the intraoperative imaging data, the marker line indicating a spatial correspondence between the two-dimensional multi-planar-reconstruction slice and the intraoperative imaging data.
[0035] In some applications, the computer processor is configured to update a position of the marker line on the view of the intraoperative imaging data in response to a user adjustment of the two-dimensional multi-planar-reconstruction slice.
[0036] In some applications, the marker line on the view of the intraoperative imaging data indicates a plane from which the two-dimensional multi-planar-reconstruction slice is derived.
[0037] In some applications, the computer processor is configured to segment at least one intraoperative image frame of the intraoperative imaging data, and to modify the two-dimensional multi-planar-reconstruction slice based on at least a portion of the segmented intraoperative image frame.In some applications, the computer processor is configured to modify the two-dimensional multi-planar-reconstruction slice based on a specific class of structure from the segmented intraoperative image frame, the specific class of structure including one or more structures selected from the group consisting of anatomical structures, cortical structures, tissue, vessels, and tumors.
[0038] In some applications, the computer processor is configured to select the two-dimensional multi-planar-reconstruction slice based on a position of a structure within the intraoperative image frame.
[0039] In some applications, the intraoperative imaging data includes sequential intraoperative image frames, and the computer processor is configured to segment the at least one intraoperative image frame at least partially based on temporal behaviors of structures across the sequential intraoperative image frames.
[0040] In some applications, the computer processor is configured to segment the at least one intraoperative image frame using an artificial intelligence model trained on annotated surgical datasets.
[0041] In some applications, the computer processor is further configured to modify a three-dimensional mesh that is generated from the preoperative imaging data based on at least a portion of the intraoperative imaging data.
[0042] In some applications, the computer processor is configured to register between the preoperative imaging data and the intraoperative imaging data by matching between an image frame of an anatomical structure of the subject as provided by the intraoperative imaging data and a corresponding three-dimensional mesh that is extracted from the same anatomical structure from the preoperative imaging data.
[0043] In some applications, the computer processor is further configured to:
[0044] detect a tool within the intraoperative imaging data;
[0045] identify a tool tip of the tool using geometric cues; and
[0046] determine an orientation of the tool by comparing an observed tool profile with a known mechanical model to estimate a pose of the tool in six degrees of freedom.
[0047] In some applications, the computer processor is further configured to overlay the tool tip and an orientation of the tool on the preoperative imaging data.
[0048] In some applications, the computer processor is further configured to:receive a user input defining one or more spatial zones within the intraoperative imaging data;
[0049] monitor proximity of the tool to the one or more spatial zones; and
[0050] generate an alert when the detected tool enters or exits the one or more spatial zones. In some applications, the computer processor is further configured to:
[0051] receive a user input designating an element within the intraoperative imaging data for tracking; and
[0052] track the designated element across sequential intraoperative image frames.
[0053] In some applications, the computer processor is further configured to receive a user annotation associated with the designated element, and to display the annotation on the preoperative imaging data at a position corresponding to a tracked position of the designated element.
[0054] In some applications, the computer processor is further configured to integrate intraoperative information derived from the intraoperative imaging data into the preoperative imaging data to generate updated preoperative imaging data, and to export the updated preoperative imaging data.
[0055] In some applications, the intraoperative imaging device includes a set of imaging devices configured to generate three-dimensional red-green-blue-plus-depth imaging data, the set of imaging devices including one or more red-green-blue cameras and a depth sensor, and the computer processor is configured to receive the intraoperative imaging data of the portion of the subject's body from the one or more red-green-blue cameras and the depth sensor.
[0056] In some applications, the intraoperative imaging device further includes one or more near-infrared cameras, and the computer processor is configured to use data from both the one or more red-green-blue cameras and the one or more near-infrared cameras to generate the intraoperative imaging data.
[0057] In some applications, the one or more red-green-blue cameras and the one or more nearinfrared cameras are pre-calibrated with respect to each other such that pixels in the nearinfrared cameras are co-registered with corresponding pixels in the red-green-blue cameras, and the computer processor is configured to use data from the pre-calibrated one or more red-green-blue cameras and one or more near-infrared cameras to generate the intraoperative imaging data.
[0058] In some applications, the computer processor is further configured to:receive two-dimensional imaging data of the portion of the subject's body from a two-dimensional imaging device; and
[0059] modify the two-dimensional imaging data based on the intraoperative imaging data to produce three-dimensional imaging data.
[0060] In some applications, the two-dimensional imaging device includes a surgical microscope, and the computer processor is configured to receive two-dimensional imaging data from the surgical microscope.
[0061] In some applications, the computer processor is further configured to model depth information in regions of the two-dimensional imaging data where depth data from the intraoperative imaging device is not available.
[0062] In some applications, the computer processor is further configured to generate an augmented reality visualization by compositing fused data with the two-dimensional imaging data to visualize an intersection between a physical surgical field and preoperative planning data.
[0063] In some applications, the computer processor is further configured to:
[0064] detect a light beam projected over a surgical field of view from an external light source; and
[0065] track a position of the light beam across sequential intraoperative image frames.
[0066] In some applications, the intraoperative imaging device includes a hyperspectral camera, and the computer processor is configured to use spectral imaging data from the hyperspectral camera to register between the preoperative imaging data and the intraoperative imaging data.
[0067] In some applications, the computer processor is further configured to:
[0068] assign a color to a tracked element within the intraoperative imaging data; and display the tracked element with the assigned color across sequential intraoperative image frames.
[0069] In some applications, the computer processor is further configured to identify an element within the intraoperative imaging data, and to track the identified element by stabilizing a video stream around the identified element such that the identified element appears stationary within the video stream.In some applications, the computer processor is further configured to detect intraoperative changes to the portion of the subject's body based on the intraoperative imaging data, and to apply an inverse deformation such that the detected intraoperative changes are mapped back to an original state of the preoperative imaging data, thereby generating a virtual image that displays the intraoperative changes in a format corresponding to the original preoperative imaging data.
[0070] In some applications, the computer processor is further configured to, subsequent to fusing the data from the one or more external physiological monitoring systems with the preoperative imaging data, update the location of the fused data based on changes in the intraoperative imaging data as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
[0071] In some applications, the computer processor is further configured to update the intraoperative imaging data based on detected changes in locations of one or more data points of the fused data as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
[0072] In some applications, the computer processor is further configured to update the preoperative imaging data based on detected changes in locations of one or more data points of the fused data as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
[0073] In some applications, the intraoperative imaging data includes three-dimensional intraoperative imaging data.
[0074] In some applications, the intraoperative imaging data includes two-dimensional intraoperative imaging data.
[0075] There is further provided, in accordance with some applications of the present disclosure, apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a display device and an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus including a computer processor configured to:
[0076] receive preoperative imaging data of the portion of the subject's body;receive the intraoperative imaging data of the portion of the subject's body from the intraoperative imaging device during the surgical procedure;
[0077] register between the preoperative imaging data and the intraoperative imaging data; generate a two-dimensional multi-planar-reconstruction slice from the preoperative imaging data;
[0078] modify the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data; and
[0079] drive the display device to display the modified two-dimensional multi-planar-reconstruction slice.
[0080] In some applications, the computer processor is configured to modify the display of the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data by deforming the preoperative imaging data based on the intraoperative imaging data to match a current position and shape of the portion of the subject's body, and the computer processor is further configured to generate the two-dimensional multi-planar-reconstruction slice by generating the two-dimensional multi-planar-reconstruction slice from the deformed preoperative imaging data.
[0081] In some applications, the computer processor is further configured to drive the display device to display the two-dimensional multi-planar-reconstruction slice and a view of the intraoperative imaging data in a side-by-side arrangement.
[0082] In some applications, the computer processor is further configured to drive the display device to display a marker line on the two-dimensional multi-planar-reconstruction slice and a corresponding marker line on the view of the intraoperative imaging data, the marker line indicating a spatial correspondence between the two-dimensional multi-planar-reconstruction slice and the intraoperative imaging data.
[0083] In some applications, the computer processor is configured to update a position of the marker line on the view of the intraoperative imaging data in response to a user adjustment of the two-dimensional multi-planar-reconstruction slice.
[0084] In some applications, the marker line on the view of the intraoperative imaging data indicates a plane from which the two-dimensional multi-planar-reconstruction slice is derived.
[0085] In some applications, the computer processor is configured to register between the preoperative imaging data and the intraoperative imaging data by matching between an image frame of an anatomical structure of the subject as provided by the intraoperative imaging dataand a corresponding three-dimensional mesh that is extracted from the same anatomical structure from the preoperative imaging data.
[0086] In some applications, the computer processor is configured to segment at least one intraoperative image frame of the intraoperative imaging data, and to modify the two-dimensional multi-planar-reconstruction slice based on at least a portion of the segmented intraoperative image frame.
[0087] In some applications, the computer processor is configured to modify the two-dimensional multi-planar-reconstruction slice based on a specific class of structure from the segmented intraoperative image frame, the specific class of structure including one or more structures selected from the group consisting of: anatomical structures, cortical structures, tissue, vessels, and tumors.
[0088] In some applications, the computer processor is configured to select the two-dimensional multi-planar-reconstruction slice based on a position of a structure within the intraoperative image frame.
[0089] In some applications, the intraoperative imaging data includes sequential intraoperative image frames, and the computer processor is configured to segment the at least one intraoperative image frame at least partially based on temporal behaviors of structures across the sequential intraoperative image frames.
[0090] In some applications, the computer processor is configured to segment the at least one intraoperative image frame using an artificial intelligence model trained on annotated surgical datasets.
[0091] In some applications, the computer processor is further configured to modify a three-dimensional mesh that is generated from the preoperative imaging data based on at least a portion of the intraoperative imaging data.
[0092] In some applications, the computer processor is further configured to detect a tool within the intraoperative imaging data, identify a tool tip of the tool using geometric cues, determine an orientation of the tool by comparing an observed tool profile with a known mechanical model to estimate a pose of the tool in six degrees of freedom, and modify the two-dimensional multi-planar-reconstruction slice to display the tool tip and an orientation of the tool.In some applications, the computer processor is further configured to receive a user input designating an element within the intraoperative imaging data for tracking, track the designated element across sequential intraoperative image frames, receive a user annotation associated with the designated element, and display the annotation on the two-dimensional multi-planar-reconstruction slice at a position corresponding to a tracked position of the designated element.
[0093] In some applications, the computer processor is further configured to receive data from one or more external physiological monitoring systems, and to overlay the two-dimensional multi-planar-reconstruction slice with the received data from the one or more external physiological monitoring systems.
[0094] In some applications, the intraoperative imaging data includes three-dimensional intraoperative imaging data.
[0095] In some applications, the intraoperative imaging data includes two-dimensional intraoperative imaging data.
[0096] There is further provided, in accordance with some applications of the present disclosure, apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus including a computer processor configured to:
[0097] receive preoperative imaging data of the portion of the subject's body;
[0098] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0099] register between the preoperative imaging data and the intraoperative imaging data; integrate intraoperative information derived from the intraoperative imaging data into the preoperative imaging data to generate updated preoperative imaging data; and
[0100] export the updated preoperative imaging data.
[0101] In some applications, the intraoperative information includes a resection area identified from the intraoperative imaging data.
[0102] In some applications, the intraoperative information includes anatomical shift data derived from a comparison between the intraoperative imaging data and the preoperative imaging data.In some applications, the intraoperative information includes surgeon annotations received via a user input during the surgical procedure.
[0103] In some applications, the intraoperative information includes data received from one or more external physiological monitoring systems during the surgical procedure.
[0104] In some applications, the data received from the one or more external physiological monitoring systems includes one or more data selected from the list consisting of: neuromonitoring data, temperature sensing data, tissue impedance measurement data, and ultrasound imaging data.
[0105] In some applications, the computer processor is further configured to deform the preoperative imaging data based on the intraoperative imaging data to match a current position and shape of an anatomical structure prior to integrating the intraoperative information.
[0106] In some applications, the computer processor is further configured to segment at least one intraoperative image frame of the intraoperative imaging data, and to integrate only a segmented portion of the intraoperative imaging data into the preoperative imaging data.
[0107] In some applications, the updated preoperative imaging data is exported in a format compatible with a medical imaging system for use in subsequent clinical procedures or analysis.
[0108] In some applications, the intraoperative imaging data includes three-dimensional intraoperative imaging data.
[0109] In some applications, the intraoperative imaging data includes two-dimensional intraoperative imaging data.
[0110] There is further provided, in accordance with some applications of the present disclosure, apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a three-dimensional intraoperative imaging device configured to generate three-dimensional intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus including a computer processor configured to:
[0111] receive the three-dimensional intraoperative imaging data from the three-dimensional intraoperative imaging device during the surgical procedure;
[0112] receive two-dimensional imaging data from a two-dimensional imaging device during the surgical procedure;determine a correspondence between the three-dimensional intraoperative imaging data and the two-dimensional imaging data; and
[0113] modify the two-dimensional imaging data based on the three-dimensional intraoperative imaging data by projecting depth information from the three-dimensional intraoperative imaging data onto the two-dimensional imaging data to produce three-dimensional imaging data from the two-dimensional imaging device.
[0114] In some applications, the two-dimensional imaging device includes a surgical microscope, and the computer processor is configured to receive two-dimensional imaging data from the surgical microscope.
[0115] In some applications, the two-dimensional imaging device has a higher spatial resolution than the three-dimensional intraoperative imaging device.
[0116] In some applications, a field of view of the two-dimensional imaging device includes a region not within a field of view of the three-dimensional intraoperative imaging device, and the computer processor is further configured to model depth information in the region based on the three-dimensional intraoperative imaging data.
[0117] In some applications, the computer processor is further configured to model depth information in regions of the two-dimensional imaging data where depth data from the three-dimensional intraoperative imaging device is not available.
[0118] In some applications, the computer processor is further configured to calibrate between the three-dimensional intraoperative imaging device and the two-dimensional imaging device such that pixels in the two-dimensional imaging device are co-registered with corresponding data points in the three-dimensional intraoperative imaging data.
[0119] In some applications, the computer processor is further configured to segment the three-dimensional intraoperative imaging data to identify anatomical structures, and to project the identified anatomical structures onto the two-dimensional imaging data.
[0120] In some applications, the computer processor is further configured to detect a tool within the three-dimensional intraoperative imaging data, and to display a position and orientation of the tool on the two-dimensional imaging data based on the correspondence.
[0121] In some applications, the computer processor is further configured to receive preoperative imaging data of the portion of the subject's body, to register between thepreoperative imaging data and the three-dimensional intraoperative imaging data, and to overlay preoperative imaging data onto the two-dimensional imaging data.
[0122] In some applications, the three-dimensional intraoperative imaging device includes one or more red-green-blue cameras and a depth sensor, and the two-dimensional imaging device is separate from the three-dimensional intraoperative imaging device, and the computer processor is configured to receive the three-dimensional intraoperative imaging data from the one or more red-green-blue cameras and a depth sensor.
[0123] There is further provided, in accordance with some applications of the present disclosure, apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus including a computer processor configured to:
[0124] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data including sequential intraoperative image frames; and
[0125] segment at least one intraoperative image frame of the intraoperative imaging data into structures at least partially based on temporal behaviors of the structures within the sequential intraoperative image frames.
[0126] In some applications, the computer processor is further configured to segment the at least one intraoperative image frame additionally based on shape and geometry of the structures.
[0127] In some applications, the computer processor is further configured to segment the at least one intraoperative image frame additionally based on spatial relationships and proximity between the structures.
[0128] In some applications, the computer processor is further configured to segment the at least one intraoperative image frame additionally based on consistency of appearance of the structures over time.
[0129] In some applications, the segmentation distinguishes between anatomical structures and non-anatomical elements, the anatomical structures including at least one of tissue and vessels, and the non-anatomical elements including at least one of tools, clips, absorbing materials, and hands.In some applications, the segmentation is performed using an artificial intelligence model trained on annotated surgical datasets.
[0130] In some applications, the segmentation is performed using one or more methods selected from the group consisting of: feature extraction, contour detection, clustering, and motion tracking.
[0131] In some applications, the computer processor is further configured to track the segmented structures across the sequential intraoperative image frames, and to calculate motion trajectories of the segmented structures in the presence of occlusion.
[0132] In some applications, the computer processor is further configured to assign a level of confidence to the tracking of each segmented structure between frames, and to accept or ignore the tracking based on the level of confidence.
[0133] In some applications, the computer processor is further configured to detect areas of blood pooling within the intraoperative imaging data based on the segmentation.
[0134] In some applications, the computer processor is further configured to assign a color to each segmented structure based on a class of the segmented structure, and to display the segmented structures with the assigned colors across the sequential intraoperative image frames.
[0135] In some applications, the intraoperative imaging data includes three-dimensional intraoperative imaging data.
[0136] In some applications, the intraoperative imaging data includes two-dimensional intraoperative imaging data.
[0137] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0138] receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0139] registering between the preoperative imaging data and the intraoperative imaging data;receiving data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; and fusing the data from the one or more external physiological monitoring systems with the preoperative imaging data via the registration between the preoperative imaging data and the intraoperative imaging data.
[0140] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a display device and an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0141] receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data of the portion of the subject's body from the intraoperative imaging device during the surgical procedure;
[0142] registering between the preoperative imaging data and the intraoperative imaging data; generating a two-dimensional multi-planar-reconstruction slice from the preoperative imaging data;
[0143] modifying the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data; and
[0144] driving the display device to display the modified two-dimensional multi-planar-reconstruction slice.
[0145] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0146] receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0147] registering between the preoperative imaging data and the intraoperative imaging data; integrating intraoperative information derived from the intraoperative imaging data into the preoperative imaging data to generate updated preoperative imaging data; and exporting the updated preoperative imaging data.There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a three-dimensional intraoperative imaging device configured to generate three-dimensional intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0148] receiving the three-dimensional intraoperative imaging data from the three-dimensional intraoperative imaging device during the surgical procedure;
[0149] receiving two-dimensional imaging data from a two-dimensional imaging device during the surgical procedure;
[0150] determining a correspondence between the three-dimensional intraoperative imaging data and the two-dimensional imaging data; and
[0151] modifying the two-dimensional imaging data based on the three-dimensional intraoperative imaging data by projecting depth information from the three-dimensional intraoperative imaging data onto the two-dimensional imaging data to produce three-dimensional imaging data from the two-dimensional imaging device.
[0152] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0153] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data including sequential intraoperative image frames; and
[0154] segmenting at least one intraoperative image frame of the intraoperative imaging data into structures at least partially based on temporal behaviors of the structures within the sequential intraoperative image frames.
[0155] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0156] registering between the preoperative imaging data and the intraoperative imaging data; receiving data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; and fusing the data from the one or more external physiological monitoring systems with the preoperative imaging data via the registration between the preoperative imaging data and the intraoperative imaging data.
[0157] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a display device and an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:
[0158] receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data of the portion of the subject's body from the intraoperative imaging device during the surgical procedure;
[0159] registering between the preoperative imaging data and the intraoperative imaging data; generating a two-dimensional multi-planar-reconstruction slice from the preoperative imaging data;
[0160] modifying the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data; and
[0161] driving the display device to display the modified two-dimensional multi-planar-reconstruction slice.
[0162] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0163] registering between the preoperative imaging data and the intraoperative imaging data; integrating intraoperative information derived from the intraoperative imaging data into the preoperative imaging data to generate updated preoperative imaging data; and exporting the updated preoperative imaging data.
[0164] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a three-dimensional intraoperative imaging device configured to generate three-dimensional intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:
[0165] receiving the three-dimensional intraoperative imaging data from the three-dimensional intraoperative imaging device during the surgical procedure;
[0166] receiving two-dimensional imaging data from a two-dimensional imaging device during the surgical procedure;
[0167] determining a correspondence between the three-dimensional intraoperative imaging data and the two-dimensional imaging data; and
[0168] modifying the two-dimensional imaging data based on the three-dimensional intraoperative imaging data by projecting depth information from the three-dimensional intraoperative imaging data onto the two-dimensional imaging data to produce three-dimensional imaging data from the two-dimensional imaging device.
[0169] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subj ect, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data including sequential intraoperative image frames; and
[0170] segmenting at least one intraoperative image frame of the intraoperative imaging data into structures at least partially based on temporal behaviors of the structures within the sequential intraoperative image frames.
[0171] There is further provided, in accordance with some applications of the present disclosure, an apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a three-dimensional intraoperative imaging device configured to generate three-dimensional intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:
[0172] receive the three-dimensional intraoperative imaging data from the three-dimensional intraoperative imaging device during the surgical procedure;
[0173] detect a tool within the three-dimensional intraoperative imaging data;
[0174] identify a tool tip of the tool using geometric cues; and
[0175] determine an orientation of the tool by comparing an observed tool profile with a known mechanical model to estimate a pose of the tool in six degrees of freedom.
[0176] There is further provided, in accordance with some applications of the present disclosure, an apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:
[0177] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames;
[0178] track at least one structure across the sequential intraoperative image frames; calculate a motion trajectory of the at least one tracked structure; and
[0179] predict the location of the at least one tracked structure in an intraoperative image frame of the sequential intraoperative image frames in which the at least one tracked structure is occluded, based on the calculated trajectory.There is further provided, in accordance with some applications of the present disclosure, an apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:
[0180] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames;
[0181] track at least one structure across the sequential intraoperative image frames; assign a level of confidence to the tracking of the at least one structure between frames based on the tracking; and
[0182] accept or ignore the tracking in at least one intraoperative image frame of the sequential intraoperative image frames based on the level of confidence.
[0183] There is further provided, in accordance with some applications of the present disclosure, an apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:
[0184] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0185] receive an input defining one or more spatial zones within the intraoperative imaging data;
[0186] track an element across sequential intraoperative image frames of the intraoperative imaging data; and
[0187] generate an alert when the tracked element enters or exits the one or more spatial zones. There is further provided, in accordance with some applications of the present disclosure, an apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:
[0188] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames;segment structures within the intraoperative image frames of the intraoperative imaging data;
[0189] assign a color to each segmented structure; and
[0190] display the segmented structures with the assigned colors across the sequential intraoperative image frames.
[0191] There is further provided, in accordance with some applications of the present disclosure, an apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:
[0192] receive preoperative imaging data of the portion of the subject's body;
[0193] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0194] register between the preoperative imaging data and the intraoperative imaging data; detect intraoperative changes to the portion of the subject's body based on the intraoperative imaging data; and
[0195] apply an inverse deformation to the preoperative imaging data such that the intraoperative changes are mapped back to an original undeformed state of the preoperative imaging data.
[0196] There is further provided, in accordance with some applications of the present disclosure, an apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:
[0197] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0198] receive data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; fuse the data from the one or more external physiological monitoring systems with the intraoperative imaging data at a corresponding anatomical location; and
[0199] track the fused data at the corresponding anatomical location across sequential intraoperative image frames as the portion of the subject's body undergoes anatomical shift during the surgical procedure.There is further provided, in accordance with some applications of the present disclosure, an apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:
[0200] receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0201] receive data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; fuse the data from the one or more external physiological monitoring systems with the intraoperative imaging data at a corresponding anatomical location; and
[0202] update the intraoperative imaging data based on detected changes in locations of one or more data points of the fused data as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
[0203] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a three-dimensional intraoperative imaging device configured to generate three-dimensional intraoperative image data of the portion of the subject's body during the surgical procedure, the method including:
[0204] receiving the three-dimensional intraoperative image data from the three-dimensional intraoperative imaging device during the surgical procedure;
[0205] detecting a tool within the three-dimensional intraoperative image data; identifying a tool tip of the tool using geometric cues; and
[0206] determining an orientation of the tool by comparing an observed tool profile with a known mechanical model to estimate a pose of the tool in six degrees of freedom.
[0207] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames;
[0208] tracking at least one structure across the sequential intraoperative image frames; calculating a motion trajectory of the at least one tracked structure; and predicting the location of the at least one tracked structure in an intraoperative image frame of the sequential intraoperative image frames in which the at least one tracked structure is occluded, based on the calculated trajectory.
[0209] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0210] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames;
[0211] tracking at least one structure across the sequential intraoperative image frames; assigning a level of confidence to the tracking of the at least one structure between frames based on the tracking; and
[0212] accepting or ignoring the tracking in at least one intraoperative image frame of the sequential intraoperative image frames based on the level of confidence.
[0213] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0214] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0215] receiving an input defining one or more spatial zones within the intraoperative imaging data;
[0216] tracking an element across sequential intraoperative image frames of the intraoperative imaging data; andgenerating an alert when the tracked element enters or exits the one or more spatial zones.
[0217] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0218] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames;
[0219] segmenting structures within the intraoperative image frames of the intraoperative imaging data;
[0220] assigning a color to each segmented structure; and
[0221] displaying the segmented structures with the assigned colors across the sequential intraoperative image frames.
[0222] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0223] receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0224] registering between the preoperative imaging data and the intraoperative imaging data; detecting intraoperative changes to the portion of the subject's body based on the intraoperative imaging data; and
[0225] applying an inverse deformation to the preoperative imaging data such that the intraoperative changes are mapped back to an original undeformed state of the preoperative imaging data.
[0226] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generateintraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0227] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0228] receiving data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; fusing the data from the one or more external physiological monitoring systems with the intraoperative imaging data at a corresponding anatomical location; and
[0229] tracking the fused data at the corresponding anatomical location across sequential intraoperative image frames as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
[0230] There is further provided, in accordance with some applications of the present disclosure, a method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method including:
[0231] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0232] receiving data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; fusing the data from the one or more external physiological monitoring systems with the intraoperative imaging data at a corresponding anatomical location; and
[0233] updating the intraoperative imaging data based on detected changes in locations of one or more data points of the fused data as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
[0234] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a three-dimensional intraoperative imaging device configured to generate three-dimensional intraoperative image data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:receiving the three-dimensional intraoperative image data from the three-dimensional intraoperative imaging device during the surgical procedure;
[0235] detecting a tool within the three-dimensional intraoperative image data; identifying a tool tip of the tool using geometric cues; and
[0236] determining an orientation of the tool by comparing an observed tool profile with a known mechanical model to estimate a pose of the tool in six degrees of freedom.
[0237] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subj ect, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:
[0238] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames;
[0239] tracking at least one structure across the sequential intraoperative image frames; calculating a motion trajectory of the at least one tracked structure; and predicting the location of the at least one tracked structure in an intraoperative image frame of the sequential intraoperative image frames in which the at least one tracked structure is occluded, based on the calculated trajectory.
[0240] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subj ect, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:
[0241] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames;
[0242] tracking at least one structure across the sequential intraoperative image frames;assigning a level of confidence to the tracking of the at least one structure between frames based on the tracking; and
[0243] accepting or ignoring the tracking in at least one intraoperative image frame of the sequential intraoperative image frames based on the level of confidence.
[0244] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subj ect, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:
[0245] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0246] receiving an input defining one or more spatial zones within the intraoperative imaging data;
[0247] tracking an element across sequential intraoperative image frames of the intraoperative imaging data; and
[0248] generating an alert when the tracked element enters or exits the one or more spatial zones.
[0249] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subj ect, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:
[0250] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames;
[0251] segmenting structures within the intraoperative image frames of the intraoperative imaging data;
[0252] assigning a color to each segmented structure; anddisplaying the segmented structures with the assigned colors across the sequential intraoperative image frames.
[0253] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subj ect, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:
[0254] receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0255] registering between the preoperative imaging data and the intraoperative imaging data; detecting intraoperative changes to the portion of the subject's body based on the intraoperative imaging data; and
[0256] applying an inverse deformation to the preoperative imaging data such that the intraoperative changes are mapped back to an original undeformed state of the preoperative imaging data.
[0257] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subj ect, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:
[0258] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0259] receiving data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; fusing the data from the one or more external physiological monitoring systems with the intraoperative imaging data at a corresponding anatomical location; andtracking the fused data at the corresponding anatomical location across sequential intraoperative image frames as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
[0260] There is further provided, in accordance with some applications of the present disclosure, a computer software product for use during a surgical procedure that is performed on a portion of a body of a subj ect, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product including a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:
[0261] receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;
[0262] receiving data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; fusing the data from the one or more external physiological monitoring systems with the intraoperative imaging data at a corresponding anatomical location; and
[0263] updating the intraoperative imaging data based on detected changes in locations of one or more data points of the fused data as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
[0264] The present disclosure will be more fully understood from the following detailed description of applications thereof, taken together with the drawings, in which:
[0265] BRIEF DESCRIPTION OF THE DRAWINGS
[0266] Fig. 1 A is a schematic illustration of a physician performing brain surgery on a patient using surgical navigation, in accordance with some applications of the present disclosure;
[0267] Fig. IB is a schematic illustration of a physician performing brain surgery on a patient using surgical navigation, in accordance with some applications of the present disclosure;
[0268] Fig. 2 is a schematic illustration of a segmented intraoperative image frame demonstrating segmentation into different classes, in accordance with some applications of the present disclosure;
[0269] Fig. 3 is a schematic illustration of intraoperative imaging data overlaid on preoperative images, in accordance with some applications of the present disclosure;Fig. 4 is a schematic illustration of an anatomical landmark marked on a brain surface and displayed on various views, in accordance with some applications of the present disclosure;
[0270] Fig. 5 is a schematic illustration of a side-by-side arrangement of three-dimensional intraoperative imaging data and a two-dimensional multi -planar-reconstruction slice, in accordance with some applications of the present disclosure; and
[0271] Fig. 6 is a schematic illustration of a side-by-side arrangement showing the effect of brain deformation during surgery, in accordance with some applications of the present disclosure.
[0272] DETAILED DESCRIPTION
[0273] Reference is now made to Fig. 1 A, which is a schematic illustration of a physician 20 performing brain surgery on a patient 22 using surgical navigation, in accordance with some applications of the present disclosure. Reference is also made to Fig. IB is a schematic illustration of physician 20 performing brain surgery on patient 22 using surgical navigation, in accordance with some additional applications of the present disclosure. Although Figs. 1 A and IB show the apparatus and methods being described herein being used during brain surgery, the scope of the present disclosure includes using the apparatus and methods described herein in the context of other surgical procedures, mutatis mutandis.
[0274] For some applications, prior to the procedure being performed, preoperative imaging data are acquired. For some applications, a 3D imaging modality is used to acquire the preoperative imaging data. For example, 3D CT, MRI, PET, PET-CT, radiographical, ultrasound images, and / or other types of images are acquired. Alternatively, or additionally, a 2D imaging modality is used to acquire the preoperative imaging data. For example, x-ray, ultrasound, MRI, and / or other types of images are acquired. In some cases, additional preoperative data is utilized, for example non-patient-specific data, e.g., an anatomical atlas or other data that reflect known anatomical structures or parameters. For some applications, preoperative planning is performed with respect to the preoperative imaging data. For example, the trajectory of a surgical instrument through the subject’s anatomy is pre-planned using the preoperative imaging data. Alternatively, or additionally, a target tissue, such as a lesion or a tumor may be located within the preoperative imaging data. For some applications, the preoperative planning includes planning the delivery and / or the deployment of an implant, for example, the implantation of an electrode in the brain, and / or the implantation of a cage (or other implant) in the spine.For some applications, the operating room includes an intraoperative imaging device 24, such as a digital camera (and typically, a stereoscopic high-resolution digital camera). For some applications, the intraoperative imaging device is configured to acquire intraoperative two-dimensional image frames and / or intraoperative three-dimensional image frames. For some applications, the intraoperative imaging device is configured to acquire intraoperative two-dimensional image frames and certain features are applied to intraoperative two-dimensional image frames, as described in further detail hereinbelow. For some applications, the intraoperative imaging device includes one or more red-green-blue (“RGB”) cameras. For some applications, the intraoperative imaging device includes a three-dimensional intraoperative imaging device such as a set of imaging devices that are configured to generate real-time red-green-blue-plus-depth (“RGB-D”) data. Alternatively, or additionally, the surgical imaging and guidance system integrates grayscale imaging with depth sensing to generate three-dimensional grayscale image frames. For example, the three-dimensional intraoperative imaging device includes a pair of RGB cameras arranged to provide stereoscopic vision. For some applications, the three-dimensional intraoperative imaging device includes one or more RGB cameras and a depth sensor, for example, a time-of-flight depth sensor. For some applications, the three-dimensional intraoperative imaging device is configured to generate three-dimensional RGB-depth image frames (also referred to hereinafter as a three-dimensional RGB-depth point cloud, or as a “3D RGB-D point cloud”) that provides a realtime anatomical model based on data acquired by the RGB cameras and / or the depth sensor.
[0275] For some applications, the operating room includes a monitoring system 30 configured to track the position and orientation of objects within the surgical field. As shown in Figs. 1A and IB, the monitoring system 30 has a field of view that encompasses the surgical site on the patient 22. For some applications, the monitoring system 30 comprises an optical tracking system that detects markers or features on surgical instruments, the patient, and / or other objects within the surgical field. For some applications, the monitoring system 30 is used in conjunction with the intraoperative imaging device 24 to provide enhanced tracking capabilities.
[0276] For some applications, the system includes a display device 34 configured to present visual information to the physician 20 during the surgical procedure. As shown in Fig. IB, for some applications, the display device 34 is mounted on a wall or other surface within the operating room and shows an image of the surgical site, such as an image of the brain with a surgical instrument. For some applications, the display device 34 presents real-timevisualization of the surgical procedure, including the intraoperative imaging data, the two-dimensional multi-planar-reconstruction slice, and / or overlays of preoperative planning data. For some applications, the display device 34 is positioned such that the physician 20 can view the display device while performing the surgical procedure. Alternatively, or additionally, as shown in Fig. 1A, the physician 20 wears an augmented reality display 32. For some applications, the display device comprises the augmented reality display 32, which presents visual information directly within the physician's field of view.
[0277] For some applications, the display device 34 comprises a monitor positioned on a mobile cart or mounted on an articulating arm, allowing the physician 20 to adjust the position and orientation of the display device 34 for optimal viewing during the surgical procedure. For some applications, the display device 34 comprises a touchscreen display that allows the physician 20 or other surgical team members to interact with the displayed information, such as adjusting the displayed slice, zooming, or annotating structures. For some applications, the display device 34 comprises multiple monitors arranged to display different views simultaneously, such as the intraoperative imaging data on one monitor and the two-dimensional multi-planar-reconstruction slice on another monitor. For some applications, the display device 34 is integrated into a surgical navigation system console that includes additional controls and interfaces for managing the surgical procedure.
[0278] For some applications, the intraoperative imaging device includes one or more nearinfrared (“NIR”) cameras, e.g., a pair of lenses and corresponding sensors arranged to provide stereoscopic vision. For some applications, a combination of RGB and NIR cameras is used, with the two types of camera typically working in parallel such as to increase the accuracy and / or efficiency of one or more of the image processing steps described hereinbelow. For some applications, the cameras are pre-calibrated with respect to each other (e.g., when the cameras are assembled in the manufacturing process), such that each pixel in a given camera is coregistered with a corresponding pixel on the other cameras. For example, light that is detected by the NIR camera (e.g., light generated by a random structure laser light source as described hereinbelow) are automatically registered with pixels within the RGB cameras. For some applications, this increases the accuracy and / or efficiency of one or more of the image processing steps described hereinbelow.
[0279] For some applications, the intraoperative imaging device includes a hyperspectral camera, and imaging data acquired using the hyperspectral camera are used to perform the registration of intraoperative imaging data with preoperative imaging data. For example,spectral imaging data that are indicative of a given tissue type are used to perform the registration. For some applications, data acquired using the hyperspectral camera are used to perform one or more additional steps of the procedure.
[0280] For some applications, raw depth data (for example, acquired using stereoscopic cameras and / or one or more depth sensors) are refined using spatial and temporal noise filtering techniques to improve stability and accuracy, especially in dynamic surgical scenes.
[0281] For some applications, registration between pre-operative imaging data and intraoperative 3D image frames is performed by matching between an intraoperative 3D image frame of an anatomical structure of the patient (e.g., a rigid anatomical structure, such as the patient’s face) and a corresponding 3D mesh that is extracted from the same anatomical structure from the pre-operative imaging data. Typically, the intraoperative image frame of the anatomical structure of the patient is a 3D intraoperative image frame of the anatomical structure of the patient and the corresponding mesh that is extracted from the same anatomical structure from the pre-operative imaging data is a 3D mesh that is extracted from the same anatomical structure from the pre-operative imaging data.
[0282] For some applications, the computer processor is configured to track anatomical structures and update the registration between the patient's anatomy and the preoperative imaging data without the need for external markers. For some applications, the computer processor updates the registration at any moment and / or repeatedly during the procedure by matching between the visible anatomy surface as captured by the intraoperative imaging device and the corresponding surface from the preoperative imaging data. For example, the computer processor matches visible surfaces such as skin or bone to update and / or refine the registration in real time.
[0283] For some applications, the computer processor is configured to refine the registration at any moment and / or repeatedly during the procedure and / or to identify new visible features for tracking when the surface has new visibility. For example, after removing a covering, cutting skin, or shaving, new anatomical features may become visible, and for some applications, the computer processor identifies these newly visible features and uses them to refine the registration and for subsequent tracking. For some applications, this allows anatomy tracking during the procedure without external markers.
[0284] For some applications, the system enables tool-free navigation, in which the computer processor tracks the position and orientation of surgical instruments and / or anatomy within theintraoperative imaging data without requiring fiducial markers to be disposed on the surgical instruments. For some applications, the computer processor detects and tracks the surgical instruments and / or anatomy using the intraoperative imaging device by visually identifying the instruments and / or anatomy within the intraoperative image frames, for example using the geometric cues and / or artificial intelligence models described herein. For some applications, this allows the surgeon to navigate without the need for specialized tracked and / or tracking instruments.
[0285] For some applications, the computer processor refines the registration during the procedure by progressively matching additional surface area between the intraoperative imaging data captured by the intraoperative imaging device and the preoperative imaging data. For some applications, the computer processor incorporates tracking of the camera image such that the registration is continuously maintained and locked to the patient's coordinate system. For some applications, this allows the system to maintain accurate registration even as the intraoperative imaging device moves or as more of the patient's anatomy becomes visible during the procedure.
[0286] For some applications, the registration includes a step of deforming the preoperative imaging data to match the current position and shape of the organ and / or structure of interest, for example using techniques as described in WO 24 / 105607 to Elimelech et al., which is incorporated herein by reference. For example, the preoperative imaging data (e.g., CT and / or MRI imaging data) of an organ may include data relating to the shape of a soft tissue organ which is different than the intraoperative shape of the organ in surgery. In addition, during the procedure, the shape of the organ may undergo changes (e.g., due to natural movement, due to movement of the organ by the surgical instruments, and / or due to the organ being cut, for example, a flap being cut). For some applications, in such cases, the registration includes a step of deforming the preoperative imaging data to match the current position and shape of the organ and / or structure of interest.
[0287] For some applications, the computer processor is configured to deform the preoperative imaging data based on the intraoperative imaging data to match a current position and shape of the portion of the subject's body, and to generate the two-dimensional multi -planar-reconstruction slice from the deformed preoperative imaging data. For some applications, the MPR slice displayed to the surgeon reflects the current deformed state of the anatomy rather than the original preoperative state. For example, if the brain has shifted during surgery due to cerebrospinal fluid drainage, the computer processor deforms the preoperative MRI data tomatch the observed brain position and then generates the MPR slice from this deformed data, thereby providing the surgeon with an accurate representation of the current anatomical configuration.
[0288] For some applications, the computer processor is configured to modify the entire preoperative image volume (e.g., the full three-dimensional MRI or CT dataset) based on the intraoperative imaging data. For some applications, the modification includes deforming and / or modifying the entire preoperative image volume to match the current position and shape of the anatomy as observed in the intraoperative imaging data. For some applications, subsequent to modifying the entire preoperative image volume, any two-dimensional multi-planar-reconstruction slice or other view is generated from the modified volume. For some applications, this ensures that all views derived from the preoperative imaging data reflect the current intraoperative state of the anatomy. For some applications, this approach is combined with other features described herein, such as overlaying segmented intraoperative structures on the views, displaying tool positions on the views, integrating data from external physiological monitoring systems, and / or exporting the modified volume as updated preoperative imaging data.
[0289] For some applications, the computer processor is configured to apply an inverse deformation to the preoperative imaging data such that intraoperative changes are mapped back to an original and / or undeformed state of the preoperative imaging data. For some applications, this generates a virtual post-operative image that displays the intraoperative changes in a format corresponding to the original preoperative imaging data. For example, when a tumor has been resected during the surgical procedure, the computer processor maps the resection area back to the original and / or undeformed preoperative imaging data, such that the resection can be visualized on the original preoperative image slices. For some applications, the computer processor drives the display device to display a current tool location on the original undeformed preoperative image slices by applying the inverse deformation to the tracked tool position. For some applications, this provides the surgeon with a virtual post-operative medical image in a familiar format, similar to an expected MRI or CT image, allowing the surgeon to visualize the surgical progress and outcomes in a format that corresponds to the original preoperative imaging data.
[0290] The scope of the present disclosure includes applying the apparatus and methods described herein to procedures that are performed on any portion of a subject's body, including but not limited to the brain, kidneys, the spleen, the liver, the spine, etc.For some applications, the computer processor is configured to track tissue deformation in real time based on the intraoperative imaging data. For some applications, the computer processor compares sequential intraoperative image frames to detect changes in the position and shape of anatomical structures. For example, during brain surgery, the brain may shift or deform due to cerebrospinal fluid drainage, tissue resection, or pressure applied by surgical instruments. The computer processor tracks these changes and updates the registration between the preoperative imaging data and the intraoperative imaging data accordingly.
[0291] For some applications, the computer processor generates a deformable model of the anatomical structure based on the preoperative imaging data. The deformable model is updated in real time based on the intraoperative imaging data to reflect the current position and shape of the anatomical structure. For some applications, the deformable model accounts for multiple factors that affect tissue deformation, including biomechanical effects, pharmacological effects, gravitational effects, forces applied by surgical tools, and biological responses to tissue resection. For some applications, the computer processor receives input indicating administration of certain medications to the patient, and the deformable model incorporates known deformation effects caused by the medication and / or adjusts the biomechanical model parameters based on the pharmacological effects of the medication. For some applications, the computer processor detects the patient's orientation and / or incorporates gravitational effects on the tissue into the deformable model. For some applications, the deformable model accounts for forces applied by surgical tools as detected from the intraoperative imaging data. For some applications, the deformable model incorporates biological responses to tissue resection, including swelling, bleeding, and / or other tissue responses. For some applications, the computer processor uses finite element modeling or other biomechanical modeling techniques to predict how the tissue will deform in response to surgical manipulation. For some applications, the computer processor uses artificial intelligence techniques in conjunction with the biomechanical model to accelerate the computation of tissue deformation predictions, thereby enabling real-time or near-real-time deformation modeling during the surgical procedure. For some applications, the artificial intelligence techniques include machine learning models that have been trained on biomechanical simulation data to rapidly approximate the results of finite element modeling computations. For some applications, the computer processor uses biomechanical modeling techniques to predict how the tissue will deform in response to surgical manipulation. For some applications, the predicted deformationis then validated and refined based on the observed deformation in the intraoperative image frames.
[0292] For some applications, the computer processor uses surface-based tracking to monitor the position of the exposed tissue surface. The computer processor identifies anatomical landmarks or features on the tissue surface and tracks their movement across sequential intraoperative image frames. For some applications, this surface-based tracking provides information about the overall displacement and deformation of the tissue.
[0293] For some applications, the computer processor uses volumetric tracking to monitor deformation throughout the volume of the tissue, not just at the surface. The volumetric tracking is performed by propagating the observed surface deformation into the subsurface tissue using biomechanical and / or geometric models. For some applications, this allows the computer processor to estimate the position of subsurface structures, such as tumors or blood vessels, based on the observed surface deformation.
[0294] For some applications, the computer processor integrates data from multiple sources to improve the accuracy of tissue tracking and modeling. For example, the computer processor combines data from the intraoperative imaging device with data from intraoperative ultrasound or other imaging modalities to track subsurface structures. Alternatively, or additionally, the computer processor uses data from external physiological monitoring systems, such as neuromonitoring systems, to validate the tissue tracking.
[0295] Referring to Fig. 2, in accordance with some applications of the present disclosure, an intraoperative 3D image frame or 2D image frame is segmented into anatomical (and / or non-anatomical) structures. For example, Fig. 2 shows a segmented intraoperative image frame 40 of the brain segmented into cortex 42, and dura 44, and also shows the segmentation of a tool 46. For some applications, additional surgical objects, such as absorber 48, are also segmented. For some applications, each image frame is analyzed to segment the scene into anatomical structures (e.g., tissue, vessels) and non-anatomical elements (e.g., tools, clips, absorbing materials, hands). For some applications, the segmentation distinguishes between anatomical structures and non-anatomical elements. In addition, detection of areas with other artifacts like blood pooling are typically identified and segmented. For some applications, the computer processor is configured to detect areas of blood pooling within the intraoperative imaging data based on the segmentation. For example, the segmentation algorithm may identify regions with characteristic color, reflectance, and / or temporal behavior indicative of pooled blood, andclassify these regions accordingly. For some applications, the segmentation is performed using traditional feature-based methods. Alternatively, or additionally, artificial intelligence (“Al”) driven models that have been trained on annotated surgical datasets are used to perform the segmentation.
[0296] For some applications, the system automatically segments structures within the intraoperative 3D image frames and / or within intraoperative 2D image frames. In some embodiments, the system is configured to segment anatomical structures, at least partially based on temporal behaviors of the structures. For some applications, only specific structures or classes of structure are segmented. For example, only cortical parts or only anatomical structures may be segmented. For some applications, healthy tissue and tumors are segmented.
[0297] For some applications, the computer processor detects a tool and identifies the tool tip within intraoperative 3D image frames using geometric cues. For some applications, the orientation of the tool is determined by comparing the observed tool profile with a known mechanical model, thereby providing a precise estimation of the pose of the tool in 6 degrees-of-freedom (“DOF”). The mechanical model can be generated either from reference data or by scanning the tool using a 3D imaging device of the system.
[0298] For some applications, the computer processor detects and / or identifies the tool and / or the tool tip using an artificial intelligence model or other trained model that has been trained to visually identify tools and / or tool tips from imaging data. For some applications, the artificial intelligence model is trained on annotated images of surgical tools to recognize tool types, tool tips, and / or tool orientations based on visual appearance. For some applications, the computer processor uses a combination of the artificial intelligence model and the mechanical model to identify the tool and determine its pose.
[0299] For some applications, the geometric cues used for tool tip identification include one or more of shape characteristics, edge profiles, elongated geometry, tapering toward the tip, symmetry properties, and angular features. For some applications, the geometric cues include detection of a transition from a wider shaft region to a narrower tip region. For some applications, the geometric cues include identification of characteristic tool shapes such as pointed tips, rounded tips, or flat tips. For some applications, the computer processor analyzes the three-dimensional point cloud data to extract geometric features that distinguish the tool tip from other portions of the tool and from surrounding structures.For some applications, the computer processor uses the detected tool position and orientation to model tissue deformation. For some applications, the computer processor predicts how the tissue will deform when contacted by the tool at the detected location and in the detected orientation. For some applications, the computer processor updates the deformable model of the anatomical structure based on the detected tool position and orientation, such that the predicted tissue deformation accounts for forces applied by the tool. For some applications, this allows the system to anticipate and display tissue deformation in real time as the surgeon manipulates the tool within the surgical field.
[0300] For some applications, the computer processor segments and classifies anatomical structures detected in the intraoperative 2D and / or 3D image frames based on temporal and spatial behavior. For some applications, objects are segmented and / or classified based on a variety of characteristics, including but not limited to shape and geometry, temporal behavior (e.g., movement patterns across frames), spatial relationships and proximity to other elements, and / or consistency of appearance over time.
[0301] For some applications, segmentation and / or classification of structures (e.g., anatomical structures) is performed using traditional methods (e.g., feature extraction, contour detection, clustering, motion tracking). Alternatively, or additionally, segmentation and / or classification of structures (e.g., anatomical structures) is performed using Al-based methods, in which models are trained to automatically learn and extract relevant features from labeled data (e.g., deep learning models that are trained to identify surgical tools, tissue types, or other scene components).
[0302] For some applications, the computer processor tracks identified anatomical and non-anatomical elements across sequential intraoperative 3D or 2D image frames. For some applications, a tracked element is assigned a color in order to allow the user to easily follow the tracked element in successive image frames. For some applications, the structure(s) is colored in accordance with a color scheme, as described hereinbelow. For some such applications, motion trajectories of given structures are calculated even in the presence of occlusion (i.e., if the view of the structure is obscured), thereby providing continuity in the tracking of relevant structures. For some applications, a level of confidence (e.g., a confidence score) is assigned to the tracking of each element between frames and the computer processor accepts or ignores the tracking based on the level of confidence of the tracking. For some applications, in order to perform the tracking, a light beam is projected over the surgical field of view from an external light source (e.g., a laser light source) and the computer processordetects the light beam and tracks the position of the light beam across sequential intraoperative image frames. For some applications, the light beam is used by the surgeon to point to or annotate specific anatomical structures or regions of interest within the surgical field. The computer processor detects the position of the light beam within the intraoperative imaging data and tracks the position of the light beam across sequential intraoperative image frames. For some applications, the tracked position of the light beam is displayed on the two-dimensional multi-planar-reconstruction slice and / or on the preoperative three-dimensional mesh, thereby allowing the surgeon to correlate the pointed location with the preoperative imaging data. For some applications, the user provides an input designating an element for tracking and optionally annotating the element. For some applications, the computer processor receives a user annotation associated with the designated element, and displays the annotation on the two-dimensional multi-planar-reconstruction slice at a position corresponding to a tracked position of the designated element. For some applications, this allows the surgeon to mark and label specific anatomical features or regions of interest and have those annotations follow the tracked element as it moves during the procedure. Subsequently, the designated (and optionally colored and / or annotated) element is tracked. For some applications, the computer processor is configured to identify an element within the intraoperative imaging data, and to track the identified element by stabilizing a video stream around the identified element such that the identified element appears stationary within the video stream. For some applications, the video stream stabilization is performed by the computer processor adjusting the display of sequential intraoperative image frames such that the tracked element remains at a fixed position within the displayed video stream, while the surrounding structures appear to move relative to the tracked element. For some applications, this stabilization is useful when the surgeon wishes to focus on a particular anatomical structure or region of interest while performing a procedure.
[0303] As described hereinabove, for some such applications, motion trajectories of given structures are calculated even in the presence of occlusion (i.e., if the view of the structure is obscured), thereby providing continuity in the tracking of relevant structures. For some applications, the computer processor predicts the location of a tracked element in a future frame (e.g., the next frame). For example, the computer processor uses Al techniques to predict the location of a tracked element in a future frame. For some applications, the computer processor uses the predicted location of the tracked element, in the event that the tracked element becomes obscured.For some applications, the computer processor is configured to track segmented structures across sequential intraoperative image frames, and to calculate motion trajectories of the segmented structures even in the presence of an occlusion. For some applications, the computer processor assigns a level of confidence to the tracking of each segmented structure between frames, and accepts or ignores the tracking based on the level of confidence. For some applications, this allows the system to maintain reliable tracking of anatomical and non-anatomical structures throughout the surgical procedure, even when portions of the surgical field are temporarily obscured by instruments, hands, or other objects.
[0304] For some applications, the computer processor is configured to identify an element within the intraoperative imaging data. For some applications, the identification is performed automatically by the computer processor, for example, based on segmentation algorithms, detection algorithms, or pattern recognition techniques. Alternatively, or additionally, the identification is performed in response to a user input designating the element for identification. For some applications, the identified element comprises an anatomical structure, a surgical tool, or another object within the surgical field.
[0305] Reference is now made to panel A of Fig. 3, which is a display showing a tool 46 being placed on a brain 50, and to panel B of Fig. 3, which is an image showing intraoperative 3D imaging data overlaid on a two-dimensional multi-planar-reconstruction (“MPR”) slice 52 that is generated from preoperative imaging data, in accordance with some applications of the present disclosure. Reference is also made to panel C of Fig. 3, which is an image showing intraoperative 3D imaging data overlaid on a preoperative three-dimensional mesh 54, in accordance with some applications of the present disclosure. Reference is also made to panel A of Fig. 4, which is a display showing an anatomical landmark marker 56 (e.g., a virtual cross) marked over the model brain 50, representing an anatomical landmark that the surgeon has selected. For some applications, the computer processor 28 is configured to track the anatomical landmark marker 56 within the intraoperative imaging data across sequential intraoperative image frames. For some applications, the tracked anatomical landmark marker 56 is displayed on the two-dimensional multi-planar-reconstruction slice and / or on the preoperative mesh at a position corresponding to the tracked position of the anatomical landmark marker 56. Reference is also made to panels B of Fig. 4, which is an image showing data from a two-dimensional multi-planar-reconstruction slice 52, with the anatomical landmark marker 56 presented at the anatomical position as marked on the model brain 50 in accordance with some applications of the present disclosure, and to panel C of Fig. 4, which isan image showing a preoperative three-dimensional mesh 54, with the anatomical landmark marker 56 presented at the anatomical position as marked on the model brain 50 in accordance with some applications of the present disclosure.
[0306] For some applications, the computer processor provides visual integration of live imaging data with preoperative planning images and real-time surgical feedback.
[0307] For some applications, subsequent to the registration between the pre-operative imaging data and the intraoperative 3D image frames, the current position of structures that have been segmented and tracked within the intraoperative 3D image frames are overlaid on a 2D multi-planar-reconstruction (hereinafter “MPR”) slice (e.g., an axial, sagittal, and / or coronal slice) that is generated from preoperative imaging data. Alternatively, or additionally, the current position of structures that have been segmented and tracked within the intraoperative 3D image frames are overlaid on a 3D mesh that is generated from the preoperative imaging data (referred to herein as the “preoperative 3D mesh”).
[0308] For some applications, the computer processor is configured to modify a display of the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data. For some applications, the modification includes overlaying segmented structures from the intraoperative imaging data onto the MPR slice, adjusting the displayed anatomy to reflect tissue deformation detected in the intraoperative imaging data, and / or highlighting regions of interest based on the intraoperative imaging. For some applications, this modification allows the surgeon to view the preoperative imaging data in a manner that reflects the current intraoperative state of the anatomy.
[0309] For some applications, a virtual 3Dview(i.e., a 3D view that isn’t shown in the acquired images) of an organ or a portion thereof is generated based on the preoperative and / or intraoperative imaging.
[0310] For some applications, one or more of the intraoperative 3D image frames are segmented (e.g., as described hereinabove), and only specific structures or classes of structure are overlaid on the 2D multi-planar-reconstruction slice and / or the 3D mesh that is based on preoperative imaging. For example, only cortical parts or only anatomical structures may be included in the overlay. For some applications, the MPR slice upon which the structure(s) is overlaid is selected based on the position of the structure within the intraoperative 3D image frame. For some applications, the structure(s) is colored in accordance with a color scheme, as described hereinbelow.For some applications, the computer processor is configured to identify and segment anatomical structures in the preoperative imaging data based on data from the intraoperative imaging data. For some applications, the computer processor uses the intraoperative imaging data to guide or refine the identification of structures in the preoperative imaging data that, for example, may have been difficult to identify or segment using the preoperative imaging data alone. For some applications, the computer processor identifies a structure in the intraoperative imaging data, determines the corresponding location in the preoperative imaging data based on the registration between the preoperative imaging data and the intraoperative imaging data, and then searches for and segments the corresponding structure in the preoperative imaging data at or near the determined location. For some applications, characteristics of the structure as observed in the intraoperative imaging data, such as size, shape, color, texture, or spatial relationships with neighboring structures, are used to guide the identification and segmentation of the corresponding structure in the preoperative imaging data. For example, if a blood vessel or other anatomical structure is identified in the intraoperative imaging data and the intraoperative imaging data is registered with the preoperative imaging data, the computer processor may search for corresponding features in the preoperative imaging data. For some applications, the computer processor identifies unique features in the preoperative imaging data that could not be determined without the intraoperative data. For some applications, the computer processor extracts new segmentation from the preoperative imaging data based on the intraoperative findings. For some applications, this method is used to find additional data and structures in the preoperative imaging data that were not previously identified.
[0311] For some applications, the tracked tool tip and / or its 3D orientation are overlaid on MPR slices and / or on a 3D mesh that is based on preoperative imaging.
[0312] For some applications, the computer processor displays the position of tracked anatomical or non-anatomical elements on the two-dimensional multi-planar-reconstruction slice and / or on the preoperative three-dimensional mesh. This display may include a virtual representation of the tracked element (e.g., a virtual tool representation showing the current position and orientation of a surgical instrument) overlaid on the preoperative imaging data. For some applications, the virtual representation allows the surgeon to visualize the current position of tracked elements with respect to the preoperative anatomy, even when the elements are not directly visible in the preoperative imaging data.
[0313] For some applications, the computer processor is configured to display an up-to-date representation of a tumor, growth, lesion, or other region of interest on the intraoperativeimaging data and / or on the preoperative imaging data. For some applications, the representation shows the portion of the tumor, growth, lesion, or other region of interest that intersects the surface of the anatomy (e.g., the brain surface) with a highlighted outline, while the portion of the tumor, growth, lesion, or other region of interest that is below the surface is displayed with a faded, semi-transparent, or otherwise visually distinguished appearance to indicate that it is subsurface. For some applications, for a partially resected tumor, growth, lesion, or other region of interest, the representation is updated to reflect the current state of the tumor, growth, lesion, or other region of interest based on the intraoperative imaging data. For some applications, this visualization is combined with deformation modeling, such that the displayed position and shape of the tumor, growth, lesion, or other region of interest are updated based on tissue deformation tracked during the procedure. For some applications, this visualization is combined with tracking, such that the representation of the tumor, growth, lesion, or other region of interest follows the tracked anatomical structures as they move during the procedure.
[0314] For some applications, the computer processor overlays a region of interest identified in the preoperative imaging data, such as a tumor boundary, lesion outline, or planned resection margin, onto the intraoperative imaging data such that the region of interest is visible to the surgeon during the procedure. For some applications, the overlaid region of interest is continually tracked and updated based on the registration between the preoperative imaging data and the intraoperative imaging data, such that the overlaid region of interest follows the anatomical structures as they move and deform during the procedure. For some applications, this allows regions of interest to be tracked without specialized tools. For some applications, this visualization is combined with deformation modeling, such that the displayed position and shape of the region of interest are updated based on tissue deformation tracked during the procedure.
[0315] For some applications, the intraoperative 3D image frames are rendered over a preoperative 3D mesh, thereby allowing intuitive 3D navigation.
[0316] Reference is now made to Fig. 5, which is a schematic illustration of a side-by-side arrangement of a view of intraoperative imaging data 60 and a two-dimensional multi-planar-reconstruction slice 52, in accordance with some applications of the present disclosure. For some applications, the computer processor is configured to drive the display device 34 to display the two-dimensional multi-planar-reconstruction slice 52 and a view of the intraoperative imaging data 60 in a side-by-side arrangement. For some applications, this side-by-side arrangement allows the user to simultaneously visualize the intraoperative surgical view and the corresponding preoperative imaging data reconstruction.
[0317] For some applications, the computer processor is configured to drive the display device 34 to display a marker line 58 on the two-dimensional multi-planar-reconstruction slice 52 and a corresponding marker line 58 on the view of the intraoperative imaging data 60. The marker line 58 indicates a spatial correspondence between the two-dimensional multi-planar-reconstruction slice 52 and the intraoperative imaging data. For some applications, the marker line 58 on the view of the intraoperative imaging data 60 indicates a plane from which the two-dimensional multi-planar-reconstruction slice 52 is derived. For example, as shown in Fig. 5, the marker line 58 is shown as a contour running circumferentially along the curved surface of the brain in the view of the intraoperative imaging data 60, and a corresponding marker line 58 is displayed on the outer perimeter of the two-dimensional multi-planar-reconstruction slice 52 at a position corresponding to the contour.
[0318] For some applications, the computer processor is configured to update a position of the marker line 58 on the view of the intraoperative imaging data 60 in response to a user adjustment of the two-dimensional multi-planar-reconstruction slice 52. For example, if the user scrolls through different axial, sagittal, or coronal slices, the computer processor drives the display device 34 to update the marker line 58 on the view of the intraoperative imaging data 60 to indicate the plane corresponding to the currently displayed slice.
[0319] For some applications, the computer processor 28 is configured to apply an inverse deformation to the marker line 58 and display the inversely deformed marker lines on the preoperative imaging data at the corresponding position in reference to the preoperative image. For some applications, this helps the surgeon visualize the correspondence between the deformed anatomy, such as a deformed brain, and the original two-dimensional multi-planar-reconstruction slice being viewed.
[0320] Reference is now made to Fig. 6, which is a schematic illustration of a side-by-side arrangement of a view of intraoperative imaging data 60 and a two-dimensional multi-planar-reconstruction slice 52, showing the effect of brain deformation during surgery, in accordance with some applications of the present disclosure. As shown in Fig. 6, a tool 46 is shown pressing on the brain surface, creating a visibly deformed region 62 where the brain tissue is indented by the tool pressure. The marker line 58 is shown as a three-dimensional contour running circumferentially along the brain surface, with the contour visibly deformed to follow the newshape of the brain at the location of the indentation caused by tool 46. The two-dimensional multi-planar-reconstruction slice 52 displays a corresponding deformed region 62 where the brain outline has an inward indentation matching the tool location. The marker line 58 on the two-dimensional multi-planar-reconstruction slice 52 is also visibly deformed to follow the deformed brain contour, indicating that the deformation of the preoperative imaging data is updated based on the intraoperative imaging data to match the current position and shape of the brain.
[0321] For some applications, the computer processor is configured to model and display internal deformation of the brain structure within the two-dimensional multi-planar-reconstruction slice 52 based on the observed surface deformation in the intraoperative imaging data. The internal deformation is propagated from the surface into subsurface tissue using biomechanical and / or geometric models, such that the two-dimensional multi-planar-reconstruction slice 52 reflects not only the surface indentation but also the displacement of internal anatomical structures, such as ventricles, white matter tracts, and subsurface lesions. For some applications, this allows the surgeon to visualize how tool pressure or tissue resection affects structures beneath the visible surface, providing enhanced spatial awareness during the procedure.
[0322] For some applications, the computer processor receives an input indicating spatial zones or regions of interest within the point cloud. For some applications, the system monitors the proximity of tracked tools to these zones and generates an alert, such as a visual alert or an auditory alert, when a tool enters or exits a defined area, thereby improving surgical safety and awareness.
[0323] For some applications, elements in the intraoperative 3D image frames or in intraoperative 2D image frame are colored or are marked with a graphical symbol (e.g., an anatomical landmark marker 56 such as a cross as shown in panel A of Fig. 4). For some such applications, the elements are colored automatically by the computer processor, for example, based on real-time input from external monitoring systems (e.g., neuromonitoring, thermal sensing), and / or based on segmentation of specific structures within the intraoperative 3D image frame. For example, a structure may be segmented based on parameters of the structure that are detected by external monitoring systems (e.g., neuromonitoring, thermal sensing). Alternatively, or additionally, the elements are colored manually with user-defined color settings to emphasize or track specific structures.For some applications, the computer processor is configured to assign a color to each segmented structure based on a class of the segmented structure. For example, different anatomical structures (e.g., cortex, dura, vessels) may be assigned different colors, and different non-anatomical elements (e.g., tools, absorbers, clips) may be assigned other colors. The computer processor displays the segmented structures with the assigned colors across sequential intraoperative image frames, thereby allowing the user to quickly identify and distinguish between different types of structures within the surgical field.
[0324] For some applications, the computer processor described herein functions as part of a module that is configured to interact with external physiological monitoring inputs to enhance intraoperative decision-making. For example, the computer processor receives inputs from one or more external physiological monitoring systems that provide intraoperative neuromonitoring, temperature sensing, tissue impedance measurements, and / or ultrasound imaging. For some applications, these inputs offer functional or safety -related insights during surgery. For some applications, the computer processor is configured to fuse the data from the one or more external physiological monitoring systems with the preoperative imaging data via registration between the preoperative imaging data and the intraoperative imaging data.
[0325] As used herein, the term "external" in "external physiological monitoring systems" refers to monitoring systems that are separate from and / or independent of the intraoperative imaging device 24. For some applications, the external physiological monitoring systems comprise standalone devices that are not integrated into the hardware of the intraoperative imaging device 24. For some applications, the external physiological monitoring systems acquire data through separate sensors, electrodes, probes, or cameras that are distinct from the imaging sensors of the intraoperative imaging device 24. For some applications, the external physiological monitoring systems communicate with the computer processor 28 via wired or wireless connections that are separate from the data connection between the intraoperative imaging device 24 and the computer processor 28.
[0326] As used herein, the term "sensor" in the context of the one or more external physiological monitoring systems refers to any device configured to acquire data from the patient's anatomy, including but not limited to a probe, an electrode, a transducer, or a camera. For some applications, the sensor of the one or more external physiological monitoring systems comprises a camera configured to detect fluorescence emitted by tissue that has been administered a fluorescent agent. For some applications, the fluorescent agent comprises afluorescent dye or marker, such as indocyanine green (ICG), 5 -aminolevulinic acid (5-ALA), or fluorescein, that is administered to the patient prior to or during the surgical procedure.
[0327] For some applications, the computer processor 28 is configured to guide the surgeon to acquire external physiological monitoring data at specific anatomical locations. For example, the computer processor 28 may prompt the surgeon to obtain neuromonitoring measurements at particular regions of the brain to verify motor or sensory activity, for example to verify activity that is assumed based on a general anatomical atlas or other non-patient-specific anatomical data. For some applications, the computer processor identifies anatomical regions of interest based on the registration between the preoperative imaging data and the intraoperative imaging data, and generates guidance indicators directing the surgeon to position a monitoring sensor (e.g., a probe or electrode) at these regions. For some applications, this allows the surgeon to confirm or refine the assumed functional mapping derived from atlas data with patient-specific physiological measurements obtained during the procedure.
[0328] For some applications, the data from the one or more external physiological monitoring systems is spatially referenced to the intraoperative imaging data. For some applications, the fusion of the data from the one or more external physiological monitoring systems with the preoperative imaging data is achieved via the registration between the preoperative imaging data and the intraoperative imaging data, such that the intraoperative imaging data serves as a common reference frame through which both data sets are linked. For some applications, the computer processor is configured to spatially reference the data from the one or more external physiological monitoring systems to the intraoperative imaging data by tracking a sensor of the one or more external physiological monitoring systems using the intraoperative imaging device. For some applications, the computer processor is configured to spatially reference the data from the one or more external physiological monitoring systems to the intraoperative imaging data by means of a user input indicating a spatial correspondence between the data and a location within the intraoperative imaging data. For some applications, the computer processor is configured to spatially reference the data from the one or more external physiological monitoring systems to the intraoperative imaging data by identifying corresponding landmarks in the data and in the intraoperative imaging data.
[0329] As used herein, the term "fuse," when used in the context of fusing data from external physiological monitoring systems with preoperative imaging data, refers to spatially combining the data from the external physiological monitoring systems with the preoperative imaging data within a common coordinate system, such that the data from the external physiologicalmonitoring systems is associated with corresponding spatial locations within the preoperative imaging data. For some applications, the fusion comprises modifying the preoperative imaging data to include the external monitoring data at the corresponding spatial locations. For some applications, the fusion comprises generating a combined dataset that includes both the preoperative imaging data and the spatially referenced external monitoring data. For some applications, the fusion comprises overlaying the external monitoring data on the preoperative imaging data for joint visualization. For some applications, the fusion comprises storing the external monitoring data in association with the corresponding spatial locations within the preoperative imaging data.
[0330] For some applications, external data is integrated with intraoperative 3D data and preoperative imaging to create a unified visualization. For some applications, this provides a combined view, which supports advanced guidance by correlating spatial anatomy with physiological and functional data in real time.
[0331] For some applications, the computer processor is configured to integrate intraoperative information derived from the intraoperative imaging data into the preoperative imaging data to generate updated preoperative imaging data. For some applications, the integration involves modifying the volumetric preoperative imaging data (e.g., MRI or CT data) to incorporate the intraoperative information, such that the updated preoperative imaging data reflects changes that occurred during the surgical procedure. The intraoperative information may include, for example, a resection area identified from the intraoperative imaging data, anatomical shift data derived from a comparison between the intraoperative imaging data and the preoperative imaging data, surgeon annotations received via a user input during the surgical procedure, and / or data received from one or more external physiological monitoring systems during the surgical procedure. For some applications, the computer processor is configured to deform the preoperative imaging data based on the intraoperative imaging data to match a current position and shape of an anatomical structure prior to integrating the intraoperative information. For some applications, the computer processor is configured to segment at least one intraoperative image frame of the intraoperative imaging data, and to integrate only a segmented portion of the intraoperative imaging data into the preoperative imaging data. For some applications, the computer processor is configured to export the updated preoperative imaging data (i.e., to convert the updated preoperative imaging data from a first application format into a different, more universal file format for use in other applications or systems). For some applications, the updated preoperative imaging data is exported in a format compatible with a medical imagingsystem, such as Digital Imaging and Communications in Medicine (“DICOM”) format, for use in subsequent clinical procedures or analysis. For some applications, this allows the surgical team to retain a record of the intraoperative changes and to use the updated imaging data for postoperative planning, follow-up imaging comparisons, or subsequent surgical procedures.
[0332] For some applications, the intraoperative data, including data from one or more external physiological monitoring systems, is merged with the patient's anatomy and tracked according to anatomical shift during the procedure. For some applications, the merged data is overlaid on the preoperative imaging data. For some applications, the merged data is exported as updated preoperative imaging data, for example, as an MRI that includes information collected during the procedure. For some applications, the computer processor is configured to track an ultrasound sensor (e.g., an ultrasound probe or transducer) using the intraoperative imaging device, and to merge ultrasound data acquired by the ultrasound sensor with the patient's anatomy based on the tracked position and orientation of the ultrasound sensor. For some applications, the computer processor converts the ultrasound data into three-dimensional data by correlating the ultrasound images with the spatial position and orientation of the ultrasound sensor as tracked by the intraoperative imaging device.
[0333] For some applications, once data from the one or more external physiological monitoring systems is detected and spatially referenced to the patient's anatomy, the data is associated with the corresponding anatomical location such that the data is continuously associated with this location. In other words, the fused data is “virtually painted” on the anatomy at this location. For some applications, the computer processor continues to update the location of the fused data as the anatomy undergoes shift during the procedure, such that the fused and / or associated data follows the anatomical structures as they move and deform. For some applications, the fused data continues to be updated based on changes in the intraoperative imaging data, maintaining the spatial association between the external monitoring data and the corresponding anatomical locations even as the portion of the subject's body undergoes anatomical shift during the surgical procedure. For some applications, the computer processor detects changes in locations of one or more data points of the fused data as the anatomy undergoes shift during the procedure, and updates the intraoperative imaging data and / or preoperative imaging data to correspond to the detected changes in the one or more data points of the fused data. For some applications, the intraoperative imaging data and / or preoperative imaging data continue to be updated based on changes in the detected changes in locations of one or more data points of the fused data, thereby maintaining the spatialassociation between the anatomical locations and the corresponding external monitoring data even as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
[0334] For some applications, the external physiological monitoring systems include patient monitoring systems that provide patient monitoring signals such as heart rate, blood pressure, and / or oxygen saturation. For some applications, the computer processor is configured to display the patient monitoring signals in association with the intraoperative imaging data. For some applications, the visualization of the intraoperative imaging data modified based on the received data from the one or more external physiological monitoring systems is generated in real time during the surgical procedure. Alternatively, or additionally, the visualization is generated on demand in response to a user input.
[0335] For some applications, the operating room includes a two-dimensional imaging device, such as a surgical microscope, that is separate from a three-dimensional intraoperative imaging device 24. The computer processor is configured to receive two-dimensional imaging data from the two-dimensional imaging device during the surgical procedure, determine a correspondence between the three-dimensional intraoperative imaging data and the two-dimensional imaging data, and modify the two-dimensional imaging data based on the three-dimensional intraoperative imaging data. For some applications, the computer processor projects depth information from the three-dimensional intraoperative imaging data onto the two-dimensional imaging data to produce three-dimensional imaging data from the two-dimensional imaging device. For some applications, the system can effectively convert a two-dimensional imaging device, such as a surgical microscope, into a three-dimensional imaging device by combining the two-dimensional imaging data with depth information from the three-dimensional intraoperative imaging device 24.
[0336] For some applications, the two-dimensional imaging device has a higher spatial resolution than the three-dimensional intraoperative imaging device 24. For some applications, this configuration allows the system to leverage the superior image quality and detail of the two-dimensional imaging device while still benefiting from the depth information provided by the three-dimensional intraoperative imaging device 24, thereby providing enhanced visualization that combines high-resolution imagery with three-dimensional spatial awareness. For some applications, a field of view of the two-dimensional imaging device includes a region not within a field of view of the three-dimensional intraoperative imaging device 24, and the computer processor is configured to model depth information in the region based on the three-dimensional intraoperative imaging data. For some applications, this capability extends the effective coverage of three-dimensional imaging beyond the physical limitations of the three-dimensional intraoperative imaging device 24, enabling comprehensive spatial visualization across a wider surgical field. For some applications, the computer processor is configured to model depth information in regions of the two-dimensional imaging data where depth data from the three-dimensional intraoperative imaging device is not available. For some applications, the computer processor is configured to calibrate between the three-dimensional intraoperative imaging device 24 and the two-dimensional imaging device such that pixels in the two-dimensional imaging device are co-registered with corresponding data points in the three-dimensional intraoperative imaging data. For some applications, this co-regi strati on ensures accurate spatial alignment between the two imaging modalities, facilitating seamless integration of data and improving the precision of depth projection onto the two-dimensional imaging data.
[0337] For some applications, the computer processor is configured to generate an augmented reality visualization by compositing fused data with two-dimensional imaging data to visualize an intersection between a physical surgical field and preoperative planning data. For example, the augmented reality visualization displays planned tumor boundaries overlaid on a partially resected tumor as viewed through the surgical microscope or other two-dimensional imaging device. For some applications, the augmented reality visualization is displayed on augmented reality display 32 worn by physician 20, as shown in Fig. 1 A.
[0338] With reference to computer processor 28, it is noted that although the computer processor is schematically illustrated as being a device within the operating room, the scope of the present disclosure includes any one of the steps described herein being performed by one or more remote computer processors that perform some of the algorithms described herein and that communicate with a local computer processor via a communications network. For some applications, a computer processor is built into the physician’s eyewear and the computer processor performs one or more of the steps described herein. For some applications, a computer processor that is built into the physician’s eyewear communicates with one or more remote computer processors via a communications network, and the remote computer processors perform one or more of the steps described herein. For some applications, a computer processor is built into intraoperative imaging device 24 and the computer processor performs one or more of the steps described herein. For some applications, a computer processor that is built into the intraoperative imaging device communicates with one or moreremote computer processors via a communications network, and the remote computer processors perform one or more of the steps described herein.
[0339] Although some applications of the present disclosure have been described as being related to a procedure that is performed on a patient's brain, the scope of the present disclosure includes applying the apparatus and methods described herein to other portions of a patient's body, mutatis mutandis.
[0340] Applications of the disclosure described herein can take the form of a computer program product accessible from a computer-usable or computer-readable medium (e.g., a non-transitory computer-readable medium) providing program code for use by or in connection with a computer or any instruction execution system, such as computer processor 28. For the purpose of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Typically, the computer-usable or computer readable medium is a non-transitory computer-usable or computer readable medium.
[0341] Examples of a computer-readable medium include a semiconductor or solid-state memory, magnetic tape, a removable computer diskette, a random-access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read / write (CD-R / W) and DVD.
[0342] A data processing system suitable for storing and / or executing program code will include at least one processor (e.g., computer processor 28) coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. The system can read the inventive instructions on the program storage devices and follow these instructions to execute the methodology of the embodiments of the disclosure.
[0343] Network adapters may be coupled to the processor to enable the processor to become coupled to other processors or remote printers or storage devices through intervening privateor public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
[0344] Computer program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the C programming language or similar programming languages.
[0345] It will be understood that blocks of the flowcharts shown in the figures and combinations of blocks in the flowcharts, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general -purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer (e.g., computer processor 28) or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowcharts and / or algorithms described in the present application. These computer program instructions may also be stored in a computer-readable medium (e.g., a non-transitory computer-readable medium) that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function / act specified in the flowchart blocks and algorithms. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowcharts and / or algorithms described in the present application.
[0346] Computer processor 28 is typically a hardware device programmed with computer program instructions to produce a special purpose computer. For example, when programmed to perform the algorithms described with reference to the figures, computer processor 28 typically acts as a special purpose surgical-navigation computer processor. For some applications, the operations described herein that are performed by computer processor 28 transform the physical state of a memory, which is a real physical article, to have a different magnetic polarity, electrical charge, or the like depending on the technology of the memory that is used. For some applications, operations that are described as being performed bycomputer processor 28 are performed by a plurality of computer processors in combination with each other. For example, as described hereinabove, the scope of the present disclosure includes any one of the steps described herein being performed by one or more remote computer processors that perform some of the algorithms described herein and that communicate with a local computer processor via a communications network. For some applications, a computer processor is built into the physician’s eyewear and the computer processor performs one or more of the steps described herein. For some applications, a computer processor that is built into the physician’s eyewear communicates with one or more remote computer processors via a communications network, and the remote computer processors perform one or more of the steps described herein. For some applications, a computer processor is built into intraoperative imaging device 24 and the computer processor performs one or more of the steps described herein. For some applications, a computer processor that is built into the intraoperative imaging device communicates with one or more remote computer processors via a communications network, and the remote computer processors perform one or more of the steps described herein.
[0347] It will be appreciated by persons skilled in the art that the present disclosure is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present disclosure includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof that are not in the prior art, which would occur to persons skilled in the art upon reading the foregoing description.
Claims
CLAIMS1. Apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:receive preoperative imaging data of the portion of the subject's body;receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;register between the preoperative imaging data and the intraoperative imaging data; receive data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; and fuse the data from the one or more external physiological monitoring systems with the preoperative imaging data via the registration between the preoperative imaging data and the intraoperative imaging data.
2. The apparatus according to claim 1, wherein the computer processor is further configured to drive a display device to display a visualization of the preoperative imaging data overlaid with the data from the one or more external physiological monitoring systems based on the fused data.
3. The apparatus according to claim 2, wherein the computer processor is further configured to drive the display device to display the visualization in real time during the surgical procedure.
4. The apparatus according to claim 2 or claim 3, wherein the computer processor is further configured to drive the display device to display the visualization on demand in response to a user input.
5. The apparatus according to any one of claims 1-4, wherein the one or more external physiological monitoring systems comprise at least one monitoring system selected from the group consisting of: a neuromonitoring system, a temperature sensing system, a tissue impedance measurement system, and an ultrasound imaging system.
6. The apparatus according to any one of claims 1-5, wherein the computer processor is further configured to color elements within the intraoperative imaging data based on the received data from the one or more external physiological monitoring systems.
7. The apparatus according to any one of claims 1-6, wherein the computer processor is further configured to segment at least one intraoperative image frame of the intraoperative imaging data, and to color a segmented structure based on a parameter of the segmented structure detected by the one or more external physiological monitoring systems.
8. The apparatus according to any one of claims 1-7, wherein the computer processor is further configured to receive patient monitoring signals comprising one or more monitoring signals selected from the list consisting of: heart rate, blood pressure, and oxygen saturation, and to display the patient monitoring signals in association with the intraoperative imaging data.
9. The apparatus according to any one of claims 1-8, wherein the computer processor is further configured to spatially reference the data from the one or more external physiological monitoring systems to the intraoperative imaging data by tracking a sensor of the one or more external physiological monitoring systems using the intraoperative imaging device.
10. The apparatus according to any one of claims 1-9, wherein the computer processor is further configured to spatially reference the data from the one or more external physiological monitoring systems to the intraoperative imaging data by means of a user input indicating a spatial correspondence between the data and a location within the intraoperative imaging data.
11. The apparatus according to any one of claims 1-10, wherein the computer processor is further configured to spatially reference the data from the one or more external physiological monitoring systems to the intraoperative imaging data by identifying corresponding landmarks in the data and in the intraoperative imaging data.
12. The apparatus according to any one of claims 1-11, wherein the computer processor is further configured to generate a two-dimensional multi-planar-reconstruction slice from the preoperative imaging data, and to modify a display of the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data.
13. The apparatus according to claim 12, wherein the computer processor is configured to modify the display of the two-dimensional multi-planar-reconstruction slice by deforming the preoperative imaging data based on the intraoperative imaging data to match a current position and shape of the portion of the subject's body, and to generate the two-dimensional multi-planar-reconstruction slice from the deformed preoperative imaging data.
14. The apparatus according to any one of claims 12-13, wherein the computer processor is further configured to drive a display device to display the two-dimensional multi-planar-reconstruction slice and a view of the intraoperative imaging data in a side-by-side arrangement.
15. The apparatus according to claim 14, wherein the computer processor is further configured to drive the display device to display a marker line on the two-dimensional multi-planar-reconstruction slice and a corresponding marker line on the view of the intraoperative imaging data, the marker line indicating a spatial correspondence between the two-dimensional multi-planar-reconstruction slice and the intraoperative imaging data.
16. The apparatus according to claim 15, wherein the computer processor is configured to update a position of the marker line on the view of the intraoperative imaging data in response to a user adjustment of the two-dimensional multi-planar-reconstruction slice.
17. The apparatus according to any one of claims 15-16, wherein the marker line on the view of the intraoperative imaging data indicates a plane from which the two-dimensional multi-planar-reconstruction slice is derived.
18. The apparatus according to any one of claims 12-17, wherein the computer processor is configured to segment at least one intraoperative image frame of the intraoperative imaging data, and to modify the two-dimensional multi-planar-reconstruction slice based on at least a portion of the segmented intraoperative image frame.
19. The apparatus according to claim 18, wherein the computer processor is configured to modify the two-dimensional multi-planar-reconstruction slice based on a specific class of structure from the segmented intraoperative image frame, the specific class of structure comprising one or more structures selected from the group consisting of: anatomical structures, cortical structures, tissue, vessels, and tumors.
20. The apparatus according to any one of claims 18-19, wherein the computer processor is configured to select the two-dimensional multi-planar-reconstruction slice based on a position of a structure within the intraoperative image frame.
21. The apparatus according to any one of claims 18-20, wherein the intraoperative imaging data comprises sequential intraoperative image frames, and wherein the computer processor is configured to segment the at least one intraoperative image frame at least partially based on temporal behaviors of structures across the sequential intraoperative image frames.
22. The apparatus according to any one of claims 18-21, wherein the computer processor is configured to segment the at least one intraoperative image frame using an artificial intelligence model trained on annotated surgical datasets.
23. The apparatus according to any one of claims 1-22, wherein the computer processor is further configured to modify a three-dimensional mesh that is generated from the preoperative imaging data based on at least a portion of the intraoperative imaging data.
24. The apparatus according to any one of claims 1-23, wherein the computer processor is configured to register between the preoperative imaging data and the intraoperative imaging data by matching between an image frame of an anatomical structure of the subject as provided by the intraoperative imaging data and a corresponding three-dimensional mesh that is extracted from the same anatomical structure from the preoperative imaging data.
25. The apparatus according to any one of claims 1-24, wherein the computer processor is further configured to:detect a tool within the intraoperative imaging data;identify a tool tip of the tool using geometric cues; anddetermine an orientation of the tool by comparing an observed tool profile with a known mechanical model to estimate a pose of the tool in six degrees of freedom.
26. The apparatus according to claim 25, wherein the computer processor is further configured to overlay the tool tip and an orientation of the tool on the preoperative imaging data.
27. The apparatus according to claim 25 or claim 26, wherein the computer processor is further configured to:receive a user input defining one or more spatial zones within the intraoperative imaging data;monitor proximity of the tool to the one or more spatial zones; andgenerate an alert when the detected tool enters or exits the one or more spatial zones.
28. The apparatus according to any one of claims 1-27, wherein the computer processor is further configured to:receive a user input designating an element within the intraoperative imaging data for tracking; andtrack the designated element across sequential intraoperative image frames.
29. The apparatus according to claim 28, wherein the computer processor is further configured to receive a user annotation associated with the designated element, and to display the annotation on the preoperative imaging data at a position corresponding to a tracked position of the designated element.
30. The apparatus according to any one of claims 1-29, wherein the computer processor is further configured to integrate intraoperative information derived from the intraoperative imaging data into the preoperative imaging data to generate updated preoperative imaging data, and to export the updated preoperative imaging data.
31. The apparatus according to any one of claims 1 -30, wherein the intraoperative imaging device includes a set of imaging devices configured to generate three-dimensional red-greenblue-plus-depth imaging data, the set of imaging devices including one or more red-green-blue cameras and a depth sensor, and the computer processor is configured to receive the intraoperative imaging data of the portion of the subject's body from the one or more red-green-blue cameras and the depth sensor.
32. The apparatus according to claim 31, wherein the intraoperative imaging device further includes one or more near-infrared cameras, and wherein the computer processor is configured to use data from both the one or more red-green-blue cameras and the one or more near-infrared cameras to generate the intraoperative imaging data.
33. The apparatus according to claim 32, wherein the one or more red-green-blue cameras and the one or more near-infrared cameras are pre-calibrated with respect to each other such that pixels in the near-infrared cameras are co-registered with corresponding pixels in the red-green-blue cameras, and wherein the computer processor is configured to use data from the pre-calibrated one or more red-green-blue cameras and one or more near-infrared cameras to generate the intraoperative imaging data.
34. The apparatus according to any one of claims 1-33, wherein the computer processor is further configured to:receive two-dimensional imaging data of the portion of the subject's body from a two-dimensional imaging device; andmodify the two-dimensional imaging data based on the intraoperative imaging data to produce three-dimensional imaging data.
35. The apparatus according to claim 34, wherein the two-dimensional imaging device comprises a surgical microscope, and the computer processor is configured to receive two-dimensional imaging data from the surgical microscope.
36. The apparatus according to claim 34 or claim 35, wherein the computer processor is further configured to model depth information in regions of the two-dimensional imaging data where depth data from the intraoperative imaging device is not available.
37. The apparatus according to any one of claims 34-36, wherein the computer processor is further configured to generate an augmented reality visualization by compositing fused data with the two-dimensional imaging data to visualize an intersection between a physical surgical field and preoperative planning data.
38. The apparatus according to any one of claims 1-37, wherein the computer processor is further configured to:detect a light beam projected over a surgical field of view from an external light source; andtrack a position of the light beam across sequential intraoperative image frames.
39. The apparatus according to any one of claims 1-38, wherein the intraoperative imaging device includes a hyperspectral camera, and wherein the computer processor is configured to use spectral imaging data from the hyperspectral camera to register between the preoperative imaging data and the intraoperative imaging data.
40. The apparatus according to any one of claims 1-39, wherein the computer processor is further configured to:assign a color to a tracked element within the intraoperative imaging data; and display the tracked element with the assigned color across sequential intraoperative image frames.
41. The apparatus according to any one of claims 1-40, wherein the computer processor is further configured to identify an element within the intraoperative imaging data, and to track the identified element by stabilizing a video stream around the identified element such that the identified element appears stationary within the video stream.
42. The apparatus according to any one of claims 1-41, wherein the computer processor is further configured to detect intraoperative changes to the portion of the subject's body based on the intraoperative imaging data, and to apply an inverse deformation such that the detected intraoperative changes are mapped back to an original state of the preoperative imaging data,thereby generating a virtual image that displays the intraoperative changes in a format corresponding to the original preoperative imaging data.
43. The apparatus according to any one of claims 1-42, wherein the computer processor is further configured to, subsequent to fusing the data from the one or more external physiological monitoring systems with the preoperative imaging data, update the location of the fused data based on changes in the intraoperative imaging data as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
44. The apparatus according to any one of claims 1-43, wherein the computer processor is further configured to update the intraoperative imaging data based on detected changes in locations of one or more data points of the fused data as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
45. The apparatus according to any one of claims 1-44, wherein the computer processor is further configured to update the preoperative imaging data based on detected changes in locations of one or more data points of the fused data as the portion of the subject's body undergoes anatomical shift during the surgical procedure.
46. The apparatus according to any one of claims 1-45, wherein the intraoperative imaging data comprises three-dimensional intraoperative imaging data.
47. The apparatus according to any one of claims 1-46, wherein the intraoperative imaging data comprises two-dimensional intraoperative imaging data.
48. Apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a display device and an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:receive preoperative imaging data of the portion of the subject's body;receive the intraoperative imaging data of the portion of the subject's body from the intraoperative imaging device during the surgical procedure;register between the preoperative imaging data and the intraoperative imaging data; generate a two-dimensional multi-planar-reconstruction slice from the preoperative imaging data;modify the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data; anddrive the display device to display the modified two-dimensional multi-planar-reconstruction slice.
49. The apparatus according to claim 48, wherein the computer processor is configured to modify the display of the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data by deforming the preoperative imaging data based on the intraoperative imaging data to match a current position and shape of the portion of the subject's body, and the computer processor is further configured to generate the two-dimensional multi -planar-reconstruction slice by generating the two-dimensional multi-planar-reconstruction slice from the deformed preoperative imaging data.
50. The apparatus according to any one of claims 48-49, wherein the computer processor is further configured to drive the display device to display the two-dimensional multi-planar-reconstruction slice and a view of the intraoperative imaging data in a side-by-side arrangement.
51. The apparatus according to claim 50, wherein the computer processor is further configured to drive the display device to display a marker line on the two-dimensional multi-planar-reconstruction slice and a corresponding marker line on the view of the intraoperative imaging data, the marker line indicating a spatial correspondence between the two-dimensional multi-planar-reconstruction slice and the intraoperative imaging data.
52. The apparatus according to claim 51, wherein the computer processor is configured to update a position of the marker line on the view of the intraoperative imaging data in response to a user adjustment of the two-dimensional multi-planar-reconstruction slice.
53. The apparatus according to any one of claims 51-52, wherein the marker line on the view of the intraoperative imaging data indicates a plane from which the two-dimensional multi-planar-reconstruction slice is derived.
54. The apparatus according to any one of claims 48-53, wherein the computer processor is configured to register between the preoperative imaging data and the intraoperative imaging data by matching between an image frame of an anatomical structure of the subject as provided by the intraoperative imaging data and a corresponding three-dimensional mesh that is extracted from the same anatomical structure from the preoperative imaging data.
55. The apparatus according to any one of claims 48-54, wherein the computer processor is configured to segment at least one intraoperative image frame of the intraoperative imagingdata, and to modify the two-dimensional multi-planar-reconstruction slice based on at least a portion of the segmented intraoperative image frame.
56. The apparatus according to claim 55, wherein the computer processor is configured to modify the two-dimensional multi-planar-reconstruction slice based on a specific class of structure from the segmented intraoperative image frame, the specific class of structure comprising one or more structures selected from the group consisting of: anatomical structures, cortical structures, tissue, vessels, and tumors.
57. The apparatus according to any one of claims 55-56, wherein the computer processor is configured to select the two-dimensional multi-planar-reconstruction slice based on a position of a structure within the intraoperative image frame.
58. The apparatus according to any one of claims 55-57, wherein the intraoperative imaging data comprises sequential intraoperative image frames, and wherein the computer processor is configured to segment the at least one intraoperative image frame at least partially based on temporal behaviors of structures across the sequential intraoperative image frames.
59. The apparatus according to any one of claims 55-57, wherein the computer processor is configured to segment the at least one intraoperative image frame using an artificial intelligence model trained on annotated surgical datasets.
60. The apparatus according to any one of claims 48-59, wherein the computer processor is further configured to modify a three-dimensional mesh that is generated from the preoperative imaging data based on at least a portion of the intraoperative imaging data.
61. The apparatus according to any one of claims 48-60, wherein the computer processor is further configured to detect a tool within the intraoperative imaging data, identify a tool tip of the tool using geometric cues, determine an orientation of the tool by comparing an observed tool profile with a known mechanical model to estimate a pose of the tool in six degrees of freedom, and modify the two-dimensional multi-planar-reconstruction slice to display the tool tip and an orientation of the tool.
62. The apparatus according to any one of claims 48-61, wherein the computer processor is further configured to receive a user input designating an element within the intraoperative imaging data for tracking, track the designated element across sequential intraoperative image frames, receive a user annotation associated with the designated element, and display the annotation on the two-dimensional multi-planar-reconstruction slice at a position corresponding to a tracked position of the designated element.
63. The apparatus according to any one of claims 48-62, wherein the computer processor is further configured to receive data from one or more external physiological monitoring systems, and to overlay the two-dimensional multi-planar-reconstruction slice with the received data from the one or more external physiological monitoring systems.
64. The apparatus according to any one of claims 48-63 , wherein the intraoperative imaging data comprises three-dimensional intraoperative imaging data.
65. The apparatus according to any one of claims 48-64, wherein the intraoperative imaging data comprises two-dimensional intraoperative imaging data.
66. Apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:receive preoperative imaging data of the portion of the subject's body;receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;register between the preoperative imaging data and the intraoperative imaging data; integrate intraoperative information derived from the intraoperative imaging data into the preoperative imaging data to generate updated preoperative imaging data; andexport the updated preoperative imaging data.
67. The apparatus according to claim 66, wherein the intraoperative information comprises a resection area identified from the intraoperative imaging data.
68. The apparatus according to claim 66 or claim 67, wherein the intraoperative information comprises anatomical shift data derived from a comparison between the intraoperative imaging data and the preoperative imaging data.
69. The apparatus according to any one of claims 66-68, wherein the intraoperative information comprises surgeon annotations received via a user input during the surgical procedure.
70. The apparatus according to any one of claims 66-69, wherein the intraoperative information comprises data received from one or more external physiological monitoring systems during the surgical procedure.
71. The apparatus according to claim 70, wherein the data received from the one or more external physiological monitoring systems comprises one or more data selected from the list consisting of: neuromonitoring data, temperature sensing data, tissue impedance measurement data, and ultrasound imaging data.
72. The apparatus according to any one of claims 66-71, wherein the computer processor is further configured to deform the preoperative imaging data based on the intraoperative imaging data to match a current position and shape of an anatomical structure prior to integrating the intraoperative information.
73. The apparatus according to any one of claims 66-72, wherein the computer processor is further configured to segment at least one intraoperative image frame of the intraoperative imaging data, and to integrate only a segmented portion of the intraoperative imaging data into the preoperative imaging data.
74. The apparatus according to any one of claims 66-73, wherein the updated preoperative imaging data is exported in a format compatible with a medical imaging system for use in subsequent clinical procedures or analysis.
75. The apparatus according to any one of claims 66-74, wherein the intraoperative imaging data comprises three-dimensional intraoperative imaging data.
76. The apparatus according to any one of claims 66-75, wherein the intraoperative imaging data comprises two-dimensional intraoperative imaging data.
77. Apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a three-dimensional intraoperative imaging device configured to generate three-dimensional intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:receive the three-dimensional intraoperative imaging data from the three-dimensional intraoperative imaging device during the surgical procedure;receive two-dimensional imaging data from a two-dimensional imaging device during the surgical procedure;determine a correspondence between the three-dimensional intraoperative imaging data and the two-dimensional imaging data; andmodify the two-dimensional imaging data based on the three-dimensional intraoperative imaging data by projecting depth information from the three-dimensionalintraoperative imaging data onto the two-dimensional imaging data to produce three-dimensional imaging data from the two-dimensional imaging device.
78. The apparatus according to claim 77, wherein the two-dimensional imaging device comprises a surgical microscope, and the computer processor is configured to receive two-dimensional imaging data from the surgical microscope.
79. The apparatus according to claim 77 or claim 78, wherein the two-dimensional imaging device has a higher spatial resolution than the three-dimensional intraoperative imaging device.
80. The apparatus according to any one of claims 77-79, wherein a field of view of the two-dimensional imaging device includes a region not within a field of view of the three-dimensional intraoperative imaging device, and wherein the computer processor is further configured to model depth information in the region based on the three-dimensional intraoperative imaging data.
81. The apparatus according to any one of claims 77-80, wherein the computer processor is further configured to model depth information in regions of the two-dimensional imaging data where depth data from the three-dimensional intraoperative imaging device is not available.
82. The apparatus according to any one of claims 77-81, wherein the computer processor is further configured to calibrate between the three-dimensional intraoperative imaging device and the two-dimensional imaging device such that pixels in the two-dimensional imaging device are co-registered with corresponding data points in the three-dimensional intraoperative imaging data.
83. The apparatus according to any one of claims 77-82, wherein the computer processor is further configured to segment the three-dimensional intraoperative imaging data to identify anatomical structures, and to project the identified anatomical structures onto the two-dimensional imaging data.
84. The apparatus according to any one of claims 77-83, wherein the computer processor is further configured to detect a tool within the three-dimensional intraoperative imaging data, and to display a position and orientation of the tool on the two-dimensional imaging data based on the correspondence.
85. The apparatus according to any one of claims 77-84, wherein the computer processor is further configured to receive preoperative imaging data of the portion of the subject's body,to register between the preoperative imaging data and the three-dimensional intraoperative imaging data, and to overlay preoperative imaging data onto the two-dimensional imaging data.
86. The apparatus according to any one of claims 77-85, wherein the three-dimensional intraoperative imaging device comprises one or more red-green-blue cameras and a depth sensor, and wherein the two-dimensional imaging device is separate from the three-dimensional intraoperative imaging device, and wherein the computer processor is configured to receive the three-dimensional intraoperative imaging data from the one or more red-green-blue cameras and a depth sensor.
87. Apparatus for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the apparatus comprising a computer processor configured to:receive the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames; andsegment at least one intraoperative image frame of the intraoperative imaging data into structures at least partially based on temporal behaviors of the structures within the sequential intraoperative image frames.
88. The apparatus according to claim 87, wherein the computer processor is further configured to segment the at least one intraoperative image frame additionally based on shape and geometry of the structures.
89. The apparatus according to claim 87 or claim 88, wherein the computer processor is further configured to segment the at least one intraoperative image frame additionally based on spatial relationships and proximity between the structures.
90. The apparatus according to any one of claims 87-89, wherein the computer processor is further configured to segment the at least one intraoperative image frame additionally based on consistency of appearance of the structures over time.
91. The apparatus according to any one of claims 87-90, wherein the segmentation distinguishes between anatomical structures and non-anatomical elements, the anatomical structures comprising at least one of tissue and vessels, and the non-anatomical elements comprising at least one of tools, clips, absorbing materials, and hands.
92. The apparatus according to any one of claims 87-91, wherein the segmentation is performed using an artificial intelligence model trained on annotated surgical datasets.
93. The apparatus according to any one of claims 87-91, wherein the segmentation is performed using one or more methods selected from the group consisting of: feature extraction, contour detection, clustering, and motion tracking.
94. The apparatus according to any one of claims 87-93, wherein the computer processor is further configured to track the segmented structures across the sequential intraoperative image frames, and to calculate motion trajectories of the segmented structures in the presence of occlusion.
95. The apparatus according to claim 94, wherein the computer processor is further configured to assign a level of confidence to the tracking of each segmented structure between frames, and to accept or ignore the tracking based on the level of confidence.
96. The apparatus according to any one of claims 87-95, wherein the computer processor is further configured to detect areas of blood pooling within the intraoperative imaging data based on the segmentation.
97. The apparatus according to any one of claims 87-96, wherein the computer processor is further configured to assign a color to each segmented structure based on a class of the segmented structure, and to display the segmented structures with the assigned colors across the sequential intraoperative image frames.
98. The apparatus according to any one of claims 87-97, wherein the intraoperative imaging data comprises three-dimensional intraoperative imaging data.
99. The apparatus according to any one of claims 87-98, wherein the intraoperative imaging data comprises two-dimensional intraoperative imaging data.
100. A method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method comprising:receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;registering between the preoperative imaging data and the intraoperative imaging data;receiving data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; and fusing the data from the one or more external physiological monitoring systems with the preoperative imaging data via the registration between the preoperative imaging data and the intraoperative imaging data.
101. A method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a display device and an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method comprising:receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data of the portion of the subject's body from the intraoperative imaging device during the surgical procedure;registering between the preoperative imaging data and the intraoperative imaging data; generating a two-dimensional multi-planar-reconstruction slice from the preoperative imaging data;modifying the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data; anddriving the display device to display the modified two-dimensional multi-planar-reconstruction slice.
102. A method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method comprising:receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;registering between the preoperative imaging data and the intraoperative imaging data; integrating intraoperative information derived from the intraoperative imaging data into the preoperative imaging data to generate updated preoperative imaging data; and exporting the updated preoperative imaging data.
103. A method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a three-dimensional intraoperative imaging device configured togenerate three-dimensional intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method comprising:receiving the three-dimensional intraoperative imaging data from the three-dimensional intraoperative imaging device during the surgical procedure;receiving two-dimensional imaging data from a two-dimensional imaging device during the surgical procedure;determining a correspondence between the three-dimensional intraoperative imaging data and the two-dimensional imaging data; andmodifying the two-dimensional imaging data based on the three-dimensional intraoperative imaging data by projecting depth information from the three-dimensional intraoperative imaging data onto the two-dimensional imaging data to produce three-dimensional imaging data from the two-dimensional imaging device.
104. A method for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the method comprising:receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames; andsegmenting at least one intraoperative image frame of the intraoperative imaging data into structures at least partially based on temporal behaviors of the structures within the sequential intraoperative image frames.
105. A computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product comprising a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;registering between the preoperative imaging data and the intraoperative imaging data;receiving data from one or more external physiological monitoring systems during the surgical procedure, the data being spatially referenced to the intraoperative imaging data; and fusing the data from the one or more external physiological monitoring systems with the preoperative imaging data via the registration between the preoperative imaging data and the intraoperative imaging data.
106. A computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a display device and an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product comprising a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data of the portion of the subject's body from the intraoperative imaging device during the surgical procedure;registering between the preoperative imaging data and the intraoperative imaging data; generating a two-dimensional multi-planar-reconstruction slice from the preoperative imaging data;modifying the two-dimensional multi-planar-reconstruction slice based on at least a portion of the intraoperative imaging data; anddriving the display device to display the modified two-dimensional multi-planar-reconstruction slice.
107. A computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product comprising a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:receiving preoperative imaging data of the portion of the subject's body; receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure;registering between the preoperative imaging data and the intraoperative imaging data;integrating intraoperative information derived from the intraoperative imaging data into the preoperative imaging data to generate updated preoperative imaging data; and exporting the updated preoperative imaging data.
108. A computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with a three-dimensional intraoperative imaging device configured to generate three-dimensional intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product comprising a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:receiving the three-dimensional intraoperative imaging data from the three-dimensional intraoperative imaging device during the surgical procedure;receiving two-dimensional imaging data from a two-dimensional imaging device during the surgical procedure;determining a correspondence between the three-dimensional intraoperative imaging data and the two-dimensional imaging data; andmodifying the two-dimensional imaging data based on the three-dimensional intraoperative imaging data by projecting depth information from the three-dimensional intraoperative imaging data onto the two-dimensional imaging data to produce three-dimensional imaging data from the two-dimensional imaging device.
109. A computer software product for use during a surgical procedure that is performed on a portion of a body of a subject, and for use with an intraoperative imaging device configured to generate intraoperative imaging data of the portion of the subject's body during the surgical procedure, the computer software product comprising a non-transitory computer-readable medium storing instructions that, when executed by a computer processor, cause the computer processor to perform the steps of:receiving the intraoperative imaging data from the intraoperative imaging device during the surgical procedure, the intraoperative imaging data comprising sequential intraoperative image frames; andsegmenting at least one intraoperative image frame of the intraoperative imaging data into structures at least partially based on temporal behaviors of the structures within the sequential intraoperative image frames.
