Method and system for supporting image-guided navigation of a tool
The method and system utilize AI and ML for dynamically adjusting registration and tracking tool movement in image-guided surgery without fiducials, addressing inefficiencies in existing systems by enabling rapid alignment, dynamic tracking, and simple setup, even during occlusions.
Patent Information
- Application Number
- PCT/CA2026/050662
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2026-05-06
- Publication Date
- 2026-11-12
AI Technical Summary
Existing image-guided surgery systems rely heavily on fiducial markers, rigid hardware setups, and complex registration processes, leading to inefficiencies, increased surgical setup times, limited procedural flexibility, and high computational loads.
A method and system for dynamically adjusting registration and tracking the movement of a medical tool using Artificial Intelligence (AI) and Machine Learning (ML) processes, without fiducials or dedicated tools, employing off-the-shelf hardware components and stereo imaging to align patient anatomy with preoperative imaging, adapt to patient movement, and maintain tracking even when anatomy is occluded.
Enables rapid alignment of patient anatomy, dynamic tracking of tool movement, and simple setup, maintaining tracking during occlusions, while reducing computational load and setup time, and enhancing procedural flexibility.
Smart Images

Figure CA2026050662_12112026_PF_FP_ABST
Abstract
Description
[0001] METHOD AND SYSTEM FOR SUPPORTING IMAGE-GUIDED NAVIGATION OF A TOOL
[0002] FIELD
[0003] The present application relates to the field of medical methods and systems and, more particularly, to a method and system for supporting image-guided navigation of a tool such as a medical or surgical tool by providing dynamically adjusted registration and dynamically tracked movement of a tool without employing fiducials or dedicated tools.
[0004] BACKGROUND
[0005] The incorporation of image guidance systems into various medical procedures allows a user to correlate a predetermined portion of a patient's body to images taken pre-operatively or intra-operatively using various imaging modalities such as x- rays, ultrasounds, CT scans, MRI's, etc. The use of image guidance systems also imparts the ability to look through superficial layers of anatomy to visualize deeper targets of interest. Further, image guidance systems provide the guidance needed to access target areas of interest within the patient's anatomy through the use of pre-defined entry points and / or target zones. Often, physicians rely heavily on imaging systems when a target cannot be directly visualized, in order to avoid damage to surrounding anatomical structures and to minimize unnecessary tissue trauma.
[0006] Image guidance systems may also be adapted for tracking a surgical tool and displaying its location with respect to a patient's body in order to facilitate precise navigation during Image-Guided Surgery (IGS). IGS employs various tracking technologies, including optical, electromagnetic, and ultrasonic systems, to create a mapped representation of the patient’s body during a medical or surgical procedure. IGS is commonly applied in neurosurgery, otolaryngology, and orthopedics.
[0007] Despite advancements in IGS, as evidenced in:
[0008] US Pat. Appl. 2024180634A1 “SURGICAL NAVIGATION SYSTEMS AND METHODS INCLUDING MATCHING OF MODEL TO ANATOMY WITHIN BOUNDARIES”;US Pat. Appl. 2024081917A1 “MARKERLESS NAVIGATION USING Al COMPUTER VISION”;
[0009] INT. Pat. Appl. WO2024095134A1 “REGISTRATION AND NAVIGATION IN CRANIAL PROCEDURES”;
[0010] US Pat. Appl. 2023363831A1 “MARKERLESS NAVIGATION SYSTEM”;
[0011] US Pat. Appl. 2023122724A1 “METHOD FOR OPERATING A MEDICAL NAVIGATION FOR AN IMAGE-GUIDED SURGICAL PROCEDURE”; INT. Pat. Appl. WO2023250131A1 “SYSTEMS FOR REAL-TIME NONINVASIVE SURGICAL NAVIGATION”;
[0012] US Pat. Appl. 2022079685A1 “SURGICAL NAVIGATION SYSTEM, MEDICAL IMAGING SYSTEM WITH SURGICAL NAVIGATION FUNCTION, AND REGISTRATION METHOD OF MEDICAL IMAGES FOR SURGICAL NAVIGATION”;
[0013] US Pat. 12,059,222B2 “ROBOTIC SURGICAL METHODS AND APPARATUSES”;
[0014] US Pat. ll,712,304B2 “SYSTEMS AND METHODS FOR PERFORMING INTRAOPERATIVE SURFACE-BASED REGISTRATION”; and, US Pat. 11,419,685B2 “APPARATUS AND METHOD FOR REGISTERING FACIAL LANDMARKS FOR SURGICAL NAVIGATION SYSTEM”, state-of-the-art systems and methods often depend on fiducial markers, rigid hardware setups, and / or complex registration processes, consequently introducing inefficiencies, increasing surgical setup times, requiring external tracking tools that may limit procedural flexibility, and / or requiring high computational loads.
[0015] There is a need for a method and system for supporting image-guided navigation of a medical tool by providing dynamically adjusted registration and dynamically tracked movement of a medical tool without employing fiducials or dedicated tools. There is also a need for a method and system that enables rapid alignment of a patient’s anatomy with preoperative imaging. There is further a need for a method and system that dynamically adapts to the patient’s movement. There is yet further a need for a method and system that maintains tracking even when the patient’s anatomy is occluded. There is yet further a need for a method and system that is simple to set up and employs off-the-shelf hardware components.This background information is provided for making information believed by the applicant to be of possible relevance to the present application. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the subject matter presented herein.
[0016] SUMMARY
[0017] An aspect of the present application is to provide a method and system for supporting image-guided navigation of a tool by providing dynamically adjusted registration and dynamically tracked movement of a tool without employing fiducials or specific tracking supporting tools. The method and system:
[0018] • enable rapid alignment of a patient’s anatomy with preoperative imaging;
[0019] • dynamically adapt to the patient’s movement; and,
[0020] • are simple to set up and employ off-the-shelf hardware components.
[0021] Another aspect of the present application is to provide a method and system that maintain tracking even when the patient’s anatomy is occluded.
[0022] Various steps of the method for supporting image-guided navigation of a tool include use of Artificial Intelligence (Al) technology, and in particular Machine Learning (ML) processes.
[0023] In accordance with one aspect, the present application provides a method for dynamically adjusting a registration of a patient’s 3D anatomical model to features extracted from a predetermined body surface part of the patient, for example during a medical or surgical procedure. The method comprises the steps of:
[0024] using at least a processor with program instructions:
[0025] a) receiving cross-sectional imaging data of the patient;
[0026] b) determining the patient’s 3D anatomical model in dependence upon the cross-sectional imaging data of the patient;
[0027] c) determining a 3D model of the predetermined body surface part of the patient in dependence upon the cross-sectional imaging data of the patient;
[0028] using at least a camera configured for stereo imaging:
[0029] d) sensing paired images of the predetermined body surface part of the patient;
[0030] using the at least a processor with program instructions:e) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;
[0031] f) aligning the depth map with the 3D model of the predetermined body surface part of the patient and registering the patient’s 3D anatomical model to the depth map;
[0032] using a display:
[0033] g) displaying image data indicative of the patient’s 3D anatomical model registered to the depth map; and,
[0034] successively repeating steps d) to g) for dynamically adjusting the registration.
[0035] In one embodiment the depth map is generated using a pre-trained convolutional neural network process for analyzing at least a disparity between the paired images and a region of interest for the procedure is extracted from the sensed predetermined body surface part of the patient using a face detection or anatomical landmark detection process.
[0036] In accordance with the aspect, the present application further provides a method for dynamically tracking movement of a tool with respect to a patient’s 3D anatomical model. The method comprises the steps of:
[0037] using at least a camera configured for stereo imaging:
[0038] a) sensing paired images of a predetermined body surface part of the patient with the tool; using at least a processor with program instructions:
[0039] b) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;
[0040] c) identifying the tool within the paired images using a machine learning process;
[0041] d) determining position and orientation of the tool with respect to the depth map;
[0042] e) aligning the depth map with a 3D model of the predetermined body surface part of the patient and registering the patient’s 3D anatomical model to the depth map;
[0043] using a display:
[0044] f) displaying image data indicative of the patient’s 3D anatomical model overlay ed with the tool; and,
[0045] successively repeating steps a) to f) for dynamically tracking the movement of the tool.
[0046] The tracking method may further comprise the steps of:
[0047] using the at least a camera configured for stereo imaging:
[0048] g) sensing paired images of a user’s hand;using the at least a processor with program instructions:
[0049] h) identifying landmarks of the user’s hand using a machine learning process;
[0050] i) determining a position of at least a predetermined tracking point of the user’s hand in dependence upon the landmarks; and,
[0051] j) executing steps b) to f) in dependence upon the position of the at least a predetermined tracking point.
[0052] The tracking method may yet further comprise the steps of:
[0053] using the at least a processor with program instructions:
[0054] k) identifying hand gestures of the user’s hand using the machine learning process; and, l) executing steps b) to f) in dependence upon the identified hand gestures.
[0055] The tracking method may yet further comprise the steps of:
[0056] affixing to the patient’s body at least one of at least an inertia measurement sensor and at least a magnetometer;
[0057] using the at least a processor with program instructions:
[0058] receiving motion sensor data from the at least one of at least an inertia measurement sensor and at least a magnetometer;
[0059] processing the motion sensor data for determining patient movement in dependence thereupon;
[0060] providing an alert message when the patient movement is above a predetermined threshold, pausing repeating steps b) to f) when the patient movement is above a predetermined first threshold; and,
[0061] continuing repeating steps b) to f) when the patient movement is below a predetermined second threshold.
[0062] The tracking method may yet further comprise the steps of:
[0063] re-registering the patient’s 3D anatomical model based on the motion sensor data; and, intermittently re-registering the patient’s 3D anatomical model by performing steps a) to f).
[0064] In accordance with the aspect, the present application further provides a system for dynamically tracking movement of a tool with respect to a patient’s 3D anatomical model, for example during a medical or surgical procedure. The system comprises:
[0065] at least a camera configured for stereo imaging;at least a processor with program instructions, the at least a processor when executing the program instructions performing:
[0066] a) receiving data indicative of the paired images of a predetermined body surface part of the patient with the tool;
[0067] b) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;
[0068] c) identifying the tool within the paired images using a machine learning process; d) determining position and orientation of the tool with respect to the depth map; e) aligning the depth map with the patient’s 3D anatomical model;
[0069] f) generating image data indicative of the patient’s 3D anatomical model overlay ed with the tool; and,
[0070] g) successively repeating steps a) to f) for dynamically tracking the movement of the tool; and,
[0071] a display adapted for displaying the image data indicative of the patient’s 3D anatomical model overlayed with the tool.
[0072] In accordance with the aspect, the present application further provides a computer program product comprising a computer readable memory storing computer executable instructions thereon that when executed by a computer cause the computer to perform the method for dynamically adjusting a registration of a patient’s 3D anatomical model to features extracted from a predetermined body surface part of the patient, for example during a medical or surgical procedure.
[0073] In accordance with the aspect, the present application further provides a computer program product comprising a computer readable memory storing computer executable instructions thereon that when executed by a computer cause the computer to perform the method for dynamically tracking movement of a tool with respect to a patient’s 3D anatomical model.
[0074] BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Drawings of embodiments are provided herewith and briefly described as follows:
[0076] Figure 1 is a schematic diagram of a system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein;Figure 2 is a flow diagram of a method for providing dynamically adjusted registration described herein;
[0077] Figure 3 is a schematic diagram of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein with the display showing anatomical landmark detection;
[0078] Figure 4 is a flow diagram of a method for providing dynamically tracked movement of a tool described herein;
[0079] Figure 5 is a schematic diagram of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during tool detection;
[0080] Figure 6 is a flow diagram of another embodiment of the method for providing dynamically tracked movement of a tool described herein;
[0081] Figure 7 is a schematic diagram of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during hand landmark detection;
[0082] Figure 8 is a schematic diagram of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein;
[0083] Figure 9 is a schematic diagram of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during hand gesture detection; Figure 10 is a schematic diagram of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during hand landmark detection used for distance measurement;
[0084] Figure 11 is a schematic diagram of another embodiment of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein;
[0085] Figure 12 is a flow diagram of yet another embodiment of the method for providing dynamically tracked movement of a tool described herein;
[0086] Figure 13 is a schematic diagram of the other embodiment of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during physics-based tracking;
[0087] Figure 14 is a flow diagram of yet another embodiment of the method for providing dynamically tracked movement of a tool described herein;
[0088] Figure 15 is a schematic diagram of the other embodiment of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein comprising a reference magnet;Figure 16 is a schematic diagram of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein employed during orthopedic surgery; and,
[0089] Figure 17 is a schematic diagram of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein employed during spinal surgery.
[0090] DETAILED DESCRIPTION
[0091] Disclosed herein is a method and system for supporting image-guided navigation of a tool such as a medical or surgical tool by providing dynamically adjusted registration and dynamically tracked movement of the tool without employing fiducials or dedicated tools. The movement of a tool with respect to a patient’s 3D anatomical model is dynamically tracked, for example during a medical or surgical procedure, using a system comprising:
[0092] • at least a camera configured for stereo imaging;
[0093] • at least a processor with program instructions; and,
[0094] • a display.
[0095] At first, cross-sectional imaging data of a desired portion of the patient (e.g. the head of the patient for neuro-surgery) taken prior to a procedure, using various imaging modalities such as x- rays, ultrasounds, CT scans, MRI's, etc., are processed to create the patient’s 3D anatomical model of the desired portion of the patient, as well as a 3D model of the corresponding body surface part of the patient.
[0096] The movement of the tool may then be dynamically tracked following the steps of:
[0097] • sensing paired images of the body surface part of the patient with the tool;
[0098] • generating a depth map of the sensed body surface part of the patient based on the paired images;
[0099] • identifying the tool within the paired images;
[0100] • determining position and orientation of the tool with respect to the depth map;
[0101] • aligning the depth map with a 3D model of the predetermined body surface part of the patient and registering the patient’s 3D anatomical model to the depth map;
[0102] • displaying image data indicative of the patient’s 3D anatomical model overlayed with the tool; and,successively repeating the above steps for dynamically tracking the movement of the tool.
[0103] The depth map may be generated using a machine learning process such as, for example, a pre-trained convolutional neural network process, for analyzing at least a disparity between the paired images. Furthermore, machine learning processes may be employed for identifying the tool and extracting a region of interest from the sensed predetermined body surface part of the patient.
[0104] Machine learning processes may further be employed for identifying landmarks of the user’s hand and determining a position of at least a predetermined tracking point of the user’s hand in dependence upon the landmarks, as well as for identifying hand gestures of the user’s hand.
[0105] The system may further comprise an inertia measurement sensor and / or a magnetometer affixed to the patient’s body to maintain tracking even when the patient’s body surface part is occluded by enabling re-registration of the patient’s 3D anatomical model based on the inertia measurement sensor and / or a magnetometer data.
[0106] Figure 1 illustrates a system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein;
[0107] Figure 2 illustrates a method for providing dynamically adjusted registration described herein;
[0108] Figure 3 illustrates the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein with the display showing anatomical landmark detection;
[0109] Figure 4 illustrates a method for providing dynamically tracked movement of a tool described herein;
[0110] Figure 5 illustrates the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during tool detection;
[0111] Figure 6 illustrates another embodiment of the method for providing dynamically tracked movement of a tool described herein;
[0112] Figure 7 illustrates the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during hand landmark detection;Figure 8 illustrates the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein;
[0113] Figure 9 illustrates the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during hand gesture detection; Figure 10 illustrates the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during hand landmark detection used for distance measurement;
[0114] Figure 11 illustrates another embodiment of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein;
[0115] Figure 12 illustrates yet another embodiment of the method for providing dynamically tracked movement of a tool described herein;
[0116] Figure 13 illustrates the other embodiment of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein during physicsbased tracking;
[0117] Figure 14 illustrates yet another embodiment of the method for providing dynamically tracked movement of a tool described herein;
[0118] Figure 15 illustrates the other embodiment of the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein comprising a reference magnet;
[0119] Figure 16 illustrates the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein employed during orthopedic surgery; and,
[0120] Figure 17 illustrates the system for providing dynamically adjusted registration and dynamically tracked movement of a tool described herein employed during spinal surgery.
[0121] The system 100 for supporting image-guided navigation of a tool has been designed such that it: is simple to set up; employs off-the-shelf hardware components; and enables use of mobile computers such as, for example, laptops and tablets. Figure 1 illustrates a first embodiment of the system 100 placed in a theater for performing neurosurgery. Camera system 10 comprises two or more cameras 11 configured for stereo imaging. The camera system 10 is connected to computer 15, such as, for example, a laptop or a tablet having a display 16, via a wired connection 12 such as, for example, high speed USB or a wireless connection 13 such as, for example, high speed Bluetooth®.The camera system 10 is positioned a predetermined distance D from the anatomical region of interest (body surface part) 14A of the patient 14, for example, approximately 18-24 inches from the face 14A, such that it is sufficiently close for capturing details of the region of interest with sufficiently high resolution while also being a sufficient distance away for enabling free movement of the user.
[0122] In an example implementation an off-the-shelf Acer Nitro Laptop has been used with:
[0123] • Intel 13th gen i5 processor;
[0124] • nVidia RTX 4050 mobile GPU at 6GB of dedicated VRAM;
[0125] • 16GB DDR5 system RAM (not shared with GPU); and,
[0126] • 512GB SSD.
[0127] Various off-the-shelf high-resolution camera systems 10 may be employed such as, for example, (1080p, 2K, or 4K) cameras equipped with either colored CMOS / CCD sensors or alternative monochrome sensors (capable of operating in the visible or infrared spectra) with the sensors being capable to capture raw image data at “real-time” frame rates of 15 to 60 fps. Some example off-the-shelf camera systems are:
[0128] • Intel RealSense RGB-D camera system with depth map output;
[0129] • Luxonis Oak D; and,
[0130] • StereoLab ZED mini.
[0131] It is noted that multiple camera systems 10, as well as structured light and / or LiDAR technologies may be employed for imaging the region of interest.
[0132] These camera systems 10 provide onboard signal processing such as:
[0133] • noise reduction and filtering for minimizing sensor noise and enhancing image quality before further processing; and,
[0134] • optionally, image compression for reducing the size of the image data while maintaining critical details required for depth mapping, thereby reducing latency and the load on subsequent processing stages.
[0135] The processed image frames may then be transmitted through a dedicated onboard input / output (I / O) subsystem from the camera system 10 to the computer 15. Once the image frames are received by the computer 15, they may be immediately stored in memory for further processing. The onboard I / O systems ensure that the data transfer occurs seamlessly,keeping pace with the high frame rate of the image capture. The stored image frames are then processed using a combination of CPU and GPU resources. The GPU, in particular, accelerates the computation required for depth map generation through parallel processing to ensure “real-time” performance at frame rates of 15 to 60 fps.
[0136] Alternatively, the camera system 10 is provided with sufficient processing capacity (CPU and memory) to enable onboard depth map generation. In this embodiment, instead of transmitting full-resolution image frames to the computer 15, only the computed depth map is sent via the I / O subsystem. This approach reduces the bandwidth requirement and offloads significant processing tasks from the computer 15, potentially enhancing system responsiveness and reducing latency in “real-time” applications at frame rates of 15 to 60 fps.
[0137] The system 100 may be provided as a complete suite of hardware components such as computer 15 and one or more camera systems 10 (depending on user preferences) with the program instructions for performing the various embodiments of the method for supporting image-guided navigation of a tool (depending on user preferences). The program instructions may be already installed and stored in non-volatile memory of the hardware components or be provided for installation by the user, for example, as a download from a server computer via the Internet or stored on a non-volatile storage medium such as, for example, flash memory. Alternatively, only the program instructions may be provided for installation by the user, enabling the user to purchase off-the-shelf hardware components (that meet the system requirements for executing the program instructions) and install the program instructions thereon.
[0138] Before depth extraction can be performed, a state-of-the-art calibration process is conducted comprising two steps:
[0139] • first, each individual camera is initially calibrated to correct lens distortion and dewarp the image, ensuring that subsequent measurements are accurate; and,
[0140] • second, once individual camera calibrations are complete, a stereo calibration is performed to determine the intrinsic (focal length, principal point) and extrinsic (rotation and translation vectors) parameters. These calibration parameters are critical for triangulation, where disparity between corresponding image points is used to compute depth.The method for supporting image-guided navigation of a tool comprises two main components:
[0141] 1. a method for dynamically adjusting a registration of a patient’s 3D anatomical model to features extracted from a predetermined body surface part of the patient; and, 2. method for dynamically tracking movement of the tool with respect to a patient’s 3D anatomical model,
[0142] with the first component being fully integrated in the second as will be described hereinbelow.
[0143] The steps of the method for dynamically adjusting a registration of a patient’s 3D anatomical model to features extracted from a predetermined body surface part of the patient are outlined in Figure 2.
[0144] The registration process begins by initially, for example, pre-operatively or intra-operatively, acquiring high-resolution cross-sectional imaging data such as, for example, CT or MRI scans, of the desired portion of the patient 14, which are stored within state-of-the-art Picture Archiving and Communication (PAC) systems. The PAC system automatically searches PAC repositories to locate and download the required imaging series in DICOM format, ensuring that all pertinent metadata (e.g., slice thickness, orientation, and acquisition parameters) is preserved.
[0145] Once the DICOM data is loaded - step a) - onto the computer 15, which may be equipped with a high-performance processing unit including CPU, GPU, and sufficient RAM for parallel processing, the next step involves determining the patient’s 3D anatomical model in dependence upon the cross-sectional imaging data of the patient - step b). for example, this is accomplished using state-of-the-art thresholding techniques, where specific Hounsfield unit ranges (for CT) or signal intensity ranges (for MRI) are selected to distinguish anatomical structures from surrounding tissues. The threshold parameters can be dynamically adjusted to account for variations in imaging protocols and patient-specific characteristics.
[0146] In step c) a 3D model of the corresponding body surface part of the patient is determined in dependence upon the cross-sectional imaging data of the patient by applying a state-of-the-art process such as, for example, the marching cubes process, to segmented volumetric data. Thisprocess iterates through the voxel grid to construct a triangulated surface mesh that accurately represents the patient’s body surface part. Given the critical nature of medical procedures, the mesh generation process may be accelerated using GPU parallelization, allowing for high-speed computation of high-resolution imaging data.
[0147] The 3D model of the corresponding body surface part of the patient may be post-processed using state-of-the-art techniques comprising:
[0148] • denoising by utilizing techniques such as median fdtering or anisotropic diffusion to suppress image noise without blurring significant anatomical edges;
[0149] • smoothing by utilizing mesh smoothing techniques (e.g., Laplacian smoothing) to remove irregularities while maintaining the integrity of anatomical features; and, • removing stray vertices by applying outlier detection to identify and eliminate vertices that do not conform to the expected anatomical contours, ensuring a cleaner, more accurate surface.
[0150] The resulting refined 3D model of the corresponding body surface part of the patient forms the foundational reference for the subsequent dynamically adjusted registration and tracking processes.
[0151] It is noted that the above processing steps may be performed using another computer such as a personal computer or workstation to generate the patient’s 3D anatomical model and the 3D model of the corresponding body surface part of the patient, which are then transferred to the computer 15 for the subsequent dynamically adjusted registration and tracking processes.
[0152] The registration of a patient’s 3D anatomical model to features extracted from a predetermined body surface part of the patient may then be dynamically adjusted following the steps of:
[0153] using camera system 10 configured for stereo imaging:
[0154] d) sensing paired images of the predetermined body surface part 14A of the patient 14;
[0155] using the computer 15 with program instructions:
[0156] e) generating a depth map of the sensed predetermined body surface part 14A of the patient 14 based on the paired images;
[0157] f) aligning the depth map with the 3D model of the predetermined body surface partof the patient and registering the patient’s 3D anatomical model to the depth map; using display 16:
[0158] g) displaying image data indicative of the patient’s 3D anatomical model registered to the depth map.
[0159] Steps d) to g) are then successively repeated for dynamically adjusting the registration.
[0160] In step e) the depth map may be generated using a pre-trained state-of-the-art machine learning process such as, for example, a Convolutional Neural Network (CNN) process, for analyzing the disparity between the paired images captured by the stereo camera system 10. The CNN is pre-trained using a large dataset of stereo image pairs, alongside corresponding calibration data and high-fidelity depth maps to determine depth more accurately by effectively reducing noise and compensating for variations in lighting and texture. Once trained, the CNN is capable of processing the paired images and generating a high-density, noise-reduced depth map in “real-time” applications at frame rates of 15 to 60 fps with each pixel of the paired images being associated with a precise depth measurement to enable resolving fine anatomical details.
[0161] To further enhance the accuracy of the depth mapping a region of interest is extracted from the entire image scene by applying a state-of-the-art landmark detection process such as, for example, a face landmark detection process in cranial surgery procedures to identify the patient’s face 17, as illustrated in Figure 3. Detected bounding boxes are then expanded into 3D volumes, ensuring that only the depth points corresponding to the area of clinical interest are retained. The region of interest extraction not only improves the signal-to-noise ratio by eliminating irrelevant background data but also facilitates subsequent computation by eliminating the processing of unnecessary depth points.
[0162] Optionally, additional techniques such as, for example, structured light techniques may be employed. Using structured light, grids, lines, or dots may be projected onto the body surface part 14A of the patient 14 with the deformation of this pattern being captured by the camera system 10 from which depth can be inferred when processing the image data. Employment of structured light can particularly be useful in situations where the patient’s body surface part is relatively flat, such as a patient’s back during a spinal surgery procedure.The next step f) involves aligning the depth map with the 3D model of the predetermined body surface part of the patient derived from the DICOM data. This registration process is key to ensuring that the sensed “live” body surface data correctly overlays on the patient’s 3D anatomical model, derived from the DICOM data as described hereinabove, for accurate surgical navigation.
[0163] At first, an initial estimate of a transformation matrix is determined for approximately aligning the depth map with the 3D model of the predetermined body surface part of the patient. The initial estimate of the transformation matrix is determined by extracting anatomical features from both the 3D model of the predetermined body surface part and the “live” depth map. These features may include prominent edges, curvature descriptors, or key landmark points which are identified using a state-of-the-art machine learning process. Initial correspondences between the 3D model data set and the “live” depth map data set may be determined using a state-of-the-art Random Sample Consensus (RANSAC) process. The RANSAC process filters out outliers by iteratively selecting subsets of points, computing tentative transformations, and evaluating consensus among the remaining points. The RANSAC process provides a rough estimate of the transformation matrix (rotation, translation, and scaling) required to approximately align the “live” depth map with the 3D model.
[0164] Following the initial alignment the transformation matrix is refined using, for example, a state-of-the-art Iterative Closest Point (ICP) process. The ICP process minimizes the distance between corresponding points by iteratively updating the transformation until the change in error falls below a predetermined threshold. For example, error metrics such as Mean Squared Error (MSE) or point-to-plane distances are determined at each iteration step.
[0165] Convergence is determined once the improvement between successive iterations is below a predetermined threshold, ensuring both precision and stability.
[0166] Optionally, the refinement is only performed on localized regions of the depth map, particularly focusing on areas with high anatomical complexity or curvature.
[0167] Further optionally, regions of the depth map with higher clinical significance (e.g., areas near critical anatomical structures) may be assigned higher weights during the alignment process, ensuring that the registration remains most accurate where it is most needed.By successively repeating steps d) to g) the registration is continuously updated in “realtime”, for example at frame rates of 15 to 60 fps. Any changes in the patient’s position or minor deformations in the anatomy are immediately detected and fed back into the registration process, allowing for dynamic adjustments of the registration.
[0168] Optionally, the steps d) to g) are performed at predetermined time intervals (e.g., every 1, 3, or 5 seconds), thus balancing computational load with the need for “real-time” responsiveness. The intervals may be dynamically adjusted by the user.
[0169] Further optionally, consecutively sensed images may be compared and a similarity index determined. When the similarity index is below a predetermined threshold steps e) and g) are performed, i.e. a significant drop in similarity, indicating movement or change in the scene, triggers an immediate re-registration.
[0170] Further optionally, motion may be detected using bounding boxes or regions of interest. For example, in cranial surgeries, a face detection process isolates the region of interest. Any detected shift in the position or size of the face bounding box beyond a preset threshold serves as a trigger to update registration.
[0171] Alternatively, in step e) the depth map may be generated using a face mesh and warping process, which may be implemented using, for example, an open-source framework such as Media Pipe to morph a canonical face mesh onto the face detected in each image frame. In this process, each vertex of the canonical mesh is triangulated, yielding a set of 200-300 registration points. The advantage of this method lies in its speed and lower hardware resource requirements, albeit with a potential trade-off in the accuracy of the registration compared to the depth mapping described hereinabove.
[0172] Further alternatively, single image frames are used to detect motion by mapping a canonical face mesh to each frame. When motion is detected between consecutive image frames, the system 100 initiates a full depth mapping and re-registration process by performing steps d) to g). This dual approach leverages the speed of the face mesh process for motion detection while ensuring high accuracy through subsequent full-scale depth mapping.
[0173] Optionally, as a fallback the system 100 may retain the last successful registration in caseswhere data quality temporarily degrades (e.g., due to occlusion or lighting variations), and attempt to acquire new, better-quality image data.
[0174] Further optionally, multiple optical sensors and parallel processing pathways may be implemented to verify data integrity and to maintain registration continuity, even if one sensor or processing module encounters technical problems.
[0175] Referring to Figure 4, the movement of a tool with respect to a patient’s 3D anatomical model, for example during a medical procedure may be dynamically tracked following the steps of:
[0176] using camera system 10 configured for stereo imaging:
[0177] a) sensing paired images of a predetermined body surface part of the patient with the tool;
[0178] using computer 15 with program instructions:
[0179] b) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;
[0180] c) identifying the tool within the paired images using a machine learning process; d) determining position and orientation of the tool with respect to the depth map; e) aligning the depth map with a 3D model of the predetermined body surface part of the patient and registering the patient’s 3D anatomical model to the depth map; using display 16:
[0181] f) displaying image data indicative of the patient’s 3D anatomical model overlay ed with the tool.
[0182] Steps a) to f) are then successively repeated during use of the tool for dynamically tracking the movement of the tool.
[0183] Use of stereo-depth camera systems 10 in surgical navigation enables employment of state-of-the-art machine learning processes for precise and adaptive tracking of tools. A variety of machine learning processes such as, for example, CNN processes, may be employed depending on the specific needs of the procedure. The system 100 is designed to be highly flexible, allowing for training on standard tool datasets or even grouping and fine-tuning models based on specialized surgical domains such as neurosurgery, Ear Nose and Throat (ENT) surgery, or spinal procedures. This modular approach enables that each tool can be selected for use in the training process, with the option to incorporate sparse dataset learning.Furthermore, users can simply capture a series of images from various angles of a selected tool, and the system 100 can then recognize and integrate this tool into the tracking framework without requiring extensive datasets.
[0184] In the detection phase, the system 100 can operate on one or both images captured by the stereo camera system 10. The machine learning process processes these images to identify -step c) - the tool within the scene. By training on both large-scale datasets and using techniques that work well with sparse data, the system 100 supports rapid integration of newly selected tools. Therefore, even with limited sample images, the machine learning process can learn distinctive features that enable reliable detection. Moreover, the system 100 can categorize tools based on their intended surgical application, ensuring that recognition and tracking are optimized for the specific operational context (e.g., neurosurgical instruments might be handled differently than ENT tools).
[0185] Once the tool is detected, the localization process - step d) - begins by generating a corresponding 3D bounding volume. This bounding volume significantly reduces the processing load by narrowing the search area to a region of interest where the tool is likely located. From this confined space, the system 100 then determines the tool’s pose - its position and orientation - by applying a state-of-the-art pose estimation process that analyzes the geometry and depth information. Alternatively, an internal vector representation of the tool may be constructed, such as a skeletal model or segmented line structures. This approach is particularly useful for flexible or non-rigid tools. For example, when tracking the surgeon’s finger, the system 100 can extract a dynamic “skeleton” model from the stereo images and fit this model to the depth map, ensuring accurate localization in three-dimensional space.
[0186] Given the critical nature of intra-operative navigation, processing efficiency is paramount. The steps a) to f) are performed at “real-time” frame rates of, for example, 15 to 60 fps to maintain a seamless “real-time” connection between the physical tool and its representation in the digital imaging space. The extracted location data is further integrated using state-of-the-art projection algorithms that maximize visualization flexibility. This enables adjustment and rendering of various views, such as overlaying the tool’s trajectory or highlighting regions of interest within the 3D anatomical model, ensuring that the user has continuous, intuitive feedback during the procedure. The overall design prioritizes low latency, ensuring that any movements of the tool - whether rigid or flexible - are captured and updated withminimal delay, thus enhancing the precision and safety of medical or surgical procedures.
[0187] Figure 5 illustrates in an example depiction a tracked tool 24 such as, for example, a pointer, or a drill. The determined pose information is processed by the computer 15 and displayed in window 25, providing real-time visualization of both the tool and the patient’s anatomy.
[0188] Alternatively, a hybrid tracking strategy is employed in step c) that begins with a rapid feature detection process to narrow the search space for the tool. In this method, the process first identifies the general region where the tool is likely to be located.
[0189] Optionally, the tool is provided with a plurality of “anchor points” in the form of retro-reflective markers or high-contrast patterns such as, for example, checkerboard designs, at different locations. In steps c) and d) the markers are used as reference landmarks for determining the position and the orientation of the tool. When used in combination with localized depth mapping, the “anchor points” significantly enhance the precision of the tool’s spatial localization and orientation, even when the overall imaging data is complex or cluttered.
[0190] In another embodiment, the user’s finger serves as both a pointing device and an interactive tool for system control by performing the following steps illustrated in Figure 6:
[0191] g) sensing paired images of a user’s hand;
[0192] h) identifying landmarks of the user’s hand using a machine learning process;
[0193] i) determining a position of at least a predetermined tracking point of the user’s hand in dependence upon the landmarks; and,
[0194] j) executing steps b) to f) in dependence upon the position of the at least a predetermined tracking point.
[0195] Steps g) to j) are executed concurrently with steps b) to f). Steps h) and i) may be executed using computer 15 or processing means of the camera system 10.
[0196] A machine learning process such as, for example, MediaPipe, may be employed for extracting the finger’s central line and key joint positions from the stereo images. By triangulating the key joint positions from the stereo images, the 3D coordinates of the finger joints can be accurately determined. Once the central line is extracted, the system 100 may extend the same by a predetermined distance (e.g., 5mm) to account for the natural tip offset.This adjustment ensures that the actual pointing position is accurately represented in the imaging space. The extracted centreline may define a sampling window - extending a few millimeters on either side - to gather additional depth points. This localized depth mapping refines the finger’s position, further enhancing the accuracy of tracking and ensuring that the displayed image remains correctly aligned with the user’s intended target.
[0197] Figure 7 illustrates in an example depiction of a tracked index finger 18 (with annotation showing central line 18A and key joints 18B) used for pointing. The trained machine learning process extracts key features (central line 18A and key joints 18B) from the imaged finger 18, enabling determination of the 3D position and orientation (pose) of the finger 18. Once all six degrees of freedom are defined, the determined pose is integrated with the data of the depth map and / or the patient’s 3D anatomical model 19 to facilitate registration and “real-time” tracking for surgical navigation.
[0198] Figure 8 illustrates in an example depiction a tracked index finger 18 used for pointing. A predetermined virtual extension 20 is applied to define the tip of the pointer. This virtual tip dynamically updates cross-sectional medical images (e.g., CT scans) on display 16, with the crosshair position 22 corresponding to the pointer tip. A 3D rendering of the patient’s body surface part 14A includes a visual representation of the pointing vector or tip direction as an arrow 23. This allows visualization of deeper anatomical structures on medical imaging that is otherwise obscured by surface anatomy.
[0199] Furthermore, the system 100 may be adapted to recognize hand gestures and to associate the same with respective control commands for providing interactive control of various functionalities of the system 100 by performing the following steps illustrated in Figure 6:
[0200] k) identifying hand gestures of the user’s hand using the machine learning process; and,
[0201] l) executing steps b) to f) in dependence upon the identified hand gestures.
[0202] Steps k) and 1) are executed concurrently with steps b) to j). Step k) may be executed using computer 15 or processing means of the camera system 10.
[0203] Since the finger and hand positions can be determined using the machine learning process, the system 100 can be trained to recognize a set of common gestures such as, for example, thumbs up, thumbs down, and the victory sign. Each gesture can be associated with arespective control command for providing interactive control of various functionalities such as, for example:
[0204] • Thumbs Up for initiating live updated navigation;
[0205] • Thumbs Down for pausing navigation to display a static image for detailed analysis or intra-operative discussion; and,
[0206] • Victory Sign for switching between different visualization modes (e.g., classic view versus trajectory view).
[0207] These gesture-based commands provide a touch-free method for the user to interact with the system 100 during the procedure.
[0208] Figure 9 illustrates in an example depiction a gesture-based command (Thumbs Up). When a recognized gesture exceeds a confidence threshold, the corresponding system behavior is triggered. In this example, a Thumbs Up gesture 35 by the user is detected and used to initiate live navigation mode using the user’s hand. System feedback is provided to the user 34, and navigation of the anatomical region of interest 14A proceeds as described in Figure 8.
[0209] The system 100 may further be adapted to recognize different fingers and to determine distances between specific landmarks such as, for example, the gap between the tip of the thumb and the tip of the index finger.
[0210] This distance measurement functionality is particularly useful in clinical settings for:
[0211] • providing a reference scale for determining the dimensions of tumors or other anatomical anomalies;
[0212] • assisting in the planning of incisions by measuring lengths directly on the patient's body surface; and,
[0213] • enabling various other required metric evaluations that could assist in a medical or surgical procedure.
[0214] By incorporating these measurement tools, the system 100 not only enhances navigational accuracy but also provides additional diagnostic and planning capabilities directly in the surgical field.
[0215] Figure 10 illustrates in an example depiction hand landmark detection used for distance measurement. The machine learning process extracts anatomical landmarks and estimates the3D positions of the fingertips and joints 42. In this example, the system 100 isolates the tips of the index finger and thumb, determines their 3D positions using stereo or depth-sensing capabilities of the camera system 10, and calculates the spatial distance between them 43. This measurement is projected onto the navigation display 44, where it can be overlaid onto multiplanar reconstructions 45 and visualized as a line segment or vector within a 3D rendering. When the user adjusts the fingers to span a target structure, for example, for estimating the size of a tumor projected onto the patient’s scalp, the measured distance can be recorded. This recording can be triggered through standard input methods (e.g., touchscreen, keyboard, mouse), or via gesture-based input as described hereinabove.
[0216] In another embodiment, the system 100 employs a combination of an InfraRed (IR) depth camera system and a single image frame visual spectrum camera system such as, for example, a Red Green Blue (RGB) camera system, to balance performance and computational load. Here, the RGB camera system performs initial feature extraction, identifying the region corresponding to the tool. This approach minimizes the computational complexity typically associated with dual-camera feature detection. The IR depth camera system, operating as an active stereo system with a structured light source, concurrently generates a high-resolution depth map. This depth map is then used to refine the initially detected region from the RGB camera system, resulting in an enhanced and precise localization of the tool. This camera system combination offers a robust yet resourceefficient solution for “real-time” surgical navigation.
[0217] This process for localizing the tool comprises the following steps:
[0218] using the single image frame RGB camera sensing a single RGB image of a predetermined body surface part of the patient with the tool;
[0219] identifying a region containing the tool based on the single RGB image;
[0220] using the IR camera configured for stereo imaging sensing paired IR images of the predetermined body surface part of the patient with the tool;
[0221] generating a depth map of the sensed predetermined body surface part of the patient with the tool based on the paired IR images; and,
[0222] determining within the identified region position and orientation of the tool with respect to the depth map.In yet another embodiment, a small high-contrast marker or pattern may be disposed on the surface of a rigid tool featuring a shaft-like structure. The machine learning process is used to first coarsely localize the tool. Once initial coarse localization is achieved, the high contrast markers can be used to more accurately determine the tool’s pose. The system 100 may additionally leverage the surrounding depth map to perform a refined pose estimation. This two-step process of:
[0223] • starting with a quick, coarse estimation; and,
[0224] • followed by detailed refinement using high contrast feature of the marker, allows for high accuracy in determining the tool’s 6-DOF pose. Moreover, by requiring only a small, dedicated marker, the approach supports the development of relatively small, ergonomically optimized tools that maintain high tracking accuracy without compromising design or functionality.
[0225] In yet another embodiment the system 100 incorporates a motion detection module 26 affixed to the patient’s body, for example, the patient’s head, as illustrated in Figure 11, that enables continuously monitoring of the patient’s anatomical movement to maintain optimal alignment between the medical image and the patient’s body. The motion detection module 26 may include visually trackable elements such as retroreflective markers, high-contrast patterns (e.g., checkerboards or QR codes), enabling the camera system 10 to track the reference object even if a portion of the patient’s body surface becomes obscured by drapes or moves out of view. The motion detection module 26 may also include motion sensors such as, for example, Inertial Measurement Units (IMUs) and magnetometers, which are battery operated and connected to the computer 15 via wireless communication such as, for example, high speed Bluetooth®.
[0226] The IMU sense both linear and angular accelerations and provide sensor data in “real time” to the computer 15, thus enabling the system 100 to accurately determine the extent and direction of the patient’s motion. Precise motion measurements are needed for accurately determining when re-registration is necessary.
[0227] The magnetometer measures the motion detection module’s 26 orientation with respect to the earth’s magnetic field. The magnetometer aids in establishing an absolute spatial frame of reference, which, when combined with the IMU data, enhances the overall accuracy of thetracking system 100.
[0228] The system 100 continuously monitors for patient movement. When significant movement 28 is detected, an alert 29 is issued to inform the user that tracking accuracy may be compromised, as illustrated in Figure 11. This immediate feedback allows the user to make informed decisions during the procedure.
[0229] Upon detecting substantial movement, the system 100 may automatically pause the tracking. This pause allows the patient’s body to stabilize, reducing the risk of propagating errors in the navigational data.
[0230] Re-registration is triggered only when motion stabilization criteria are met, once the system 100 detects that the patient’s movement has ceased for a predetermined period, it automatically recalibrates the navigation setup to re-establish accurate alignment. A visual notification system communicates to the user that navigation has been paused and that recalibration is in progress, ensuring procedural transparency and safety.
[0231] The motion detection may be incorporated into the tracking process by performing the following steps as illustrated in Figure 12:
[0232] receiving motion sensor data from the at least one of at least an inertia measurement sensor and at least a magnetometer;
[0233] processing the motion sensor data for determining patient movement in dependence thereupon;
[0234] providing an alert message when the patient movement is above a predetermined threshold;
[0235] pausing repeating steps b) to f) when the patient movement is above a predetermined first threshold: and,
[0236] continuing repeating steps b) to f) when the patient movement is below a predetermined second threshold.
[0237] During a surgical procedure, anatomical landmarks such as, for example, portions of the patient’s face, may occasionally become obstructed due to surgical drapes or other occlusions. To address this, the system 100 may rely on the motion detection module 26, which is equipped with both optical markers and motion sensors.The motion data collected from the motion detection module 26 is analyzed using a comprehensive physics model that accounts for the biomechanical constraints of the patient’s anatomy. For example, the head’s movement is naturally constrained by its connection to the neck and torso. Known biomechanical kinematics of the cervical spine and neck allow the system to limit the degrees of freedom in the head’s motion model, thereby enabling more accurate position updates by cross-referencing the IMU data with the biomechanical constraints to accurately determine the new head position.
[0238] Before surgical draping, an initial calibration step may be performed to generate a baseline model of the patient’s anatomical kinematics. For example, during this calibration, the patient’s head is guided through a series of controlled motions such as, for example, rotation, flexion, extension, and lateral bending. Assuming minimal linear translation, the full range of the head’s movement is characterized using the IMU sensor readings. This baseline model of the patient’s anatomical kinematics allows for accurate tracking even when subsequent visual inputs are partially obscured.
[0239] To mitigate the impact of sensor drift or model inaccuracies, the system 100 periodically prompts the exposure of the reference device to the optical sensor. In situations where direct optical tracking is not possible, the system 100 transparently notifies the user that “physicsbased tracking” is active. As illustrated in Figure 13, the computer 15 displays a message 30 to notify the user that physics-based tracking is active. A visual representation of the biomechanical model 31 may also be displayed, accompanied by a warning indicating potential reduction in accuracy and recommending re-registration using visual input when feasible. This ensures continuous navigation while maintaining the highest possible level of tracking accuracy.
[0240] The re-registration based on motion sensor data may be incorporated into the tracking process by performing the following steps as illustrated in Figure 14:
[0241] receiving motion sensor data from the at least one of at least an inertia measurement sensor and at least a magnetometer;
[0242] processing the motion sensor data for determining patient movement in dependence thereupon;
[0243] re-registering the patient’s 3D anatomical model based on the motion sensor data; and,intermittently re-registering the patient’s 3D anatomical model by performing steps a) to f).
[0244] In yet another embodiment of the system 100, low-profde magnetic reference 31 is provided, as illustrated in Figure 15. This assembly is designed using a single large magnet or numerous small arrays of high-strength permanent magnets (e.g., NdFeB) that generate a stable and predictable magnetic field gradient 33 across the region of interest such as, for example, the patient’s head. The key advantage of this design is that the magnetic reference remains stationary relative to the patient’s body, providing a constant and well-characterized magnetic field that can be measured with high precision by a magnetometer 32.
[0245] The magnetic field gradient 33 generated by this dedicated reference serves as an external coordinate anchor, significantly improving the magnetic localization accuracy when combined with data from an IMU. State-of-the-art sensor fusion processes combine magnetometer readings with accelerometer and gyroscope data, reducing drift and compensating for any transient disturbances. Moreover, since the magnetic field 33 from the reference 31 is deliberately engineered to be strong and uniform over the target region, it provides a superior baseline compared to the Earth’s magnetic field. It may also be used to compensate for sensor drift that can occur with IMU based systems, especially if movement is slow with very low linear or angular acceleration.
[0246] Further calibration is achieved by integrating an optical tracking system. The optical system captures different head poses and correlates them with the measured magnetic field readings, thereby establishing a mapping between the patient’s head position and the magnetic field gradient 34. This dual-calibration approach using both the engineered magnetic reference and optical calibration enables the system 100 to dynamically adjust and correct the sensor fusion process, ensuring that the navigation and tracking remain accurate even if the patient’s head or the patient’s body undergoes subtle movements or occlusions during the surgical procedure.
[0247] By combining these elements, the system 100 not only improves the spatial localization of the patient’s body but also ensures that image-guided navigation remains robust and accurate throughout the medical or surgical procedure.Figure 16 illustrates an example application of the system 100 in orthopedic surgery, specifically for joint replacement procedures. The anatomy of interest 14B is the knee joint. The camera system 10 captures the target anatomy, and a pre-trained machine learning process detects the knee, defines a 3D bounding volume, and extracts the 3D point cloud specific to the knee joint 36. This point cloud is then registered to pre-operative CT scan imaging data, enabling integration of surgical plans. The femur and tibia are registered independently 37 to account for intra-operative joint movement. Marker-less tracking is also supported using features such as the user’s finger 18 as a pointing tool.
[0248] Figure 17 illustrates an example application of the system 100 in a spinal surgery. The camera system 10, is mounted directly onto the patient's back and connected to the computer 15. Registration is achieved using a C-arm 38, which captures part of the camera system 10 and co-registered integrated markers 39 that are both optically visible and radio-opaque. These markers are overlaid on the patient’s anatomy, enabling alignment between optical tracking and X-ray imaging. This registration allows the user to perform navigation based on 2D fluoroscopic images using any compatible, pre-trained tool 40. Optionally, 2D-to-3D registration processes may be employed to align the patient’s anatomy 14C with a preoperative CT scan for enhanced navigation capabilities 41.
[0249] Provided herein are various embodiments, including the following:
[0250] Embodiment 1 : A method for dynamically adjusting a registration of a patient’s 3D anatomical model to features extracted from a predetermined body surface part of the patient comprising:
[0251] using at least a processor with program instructions:
[0252] a) receiving cross-sectional imaging data of the patient;
[0253] b) determining the patient’s 3D anatomical model in dependence upon the cross-sectional imaging data of the patient;
[0254] c) determining a 3D model of the predetermined body surface part of the patient in dependence upon the cross-sectional imaging data of the patient;
[0255] using at least a camera configured for stereo imaging:
[0256] d) sensing paired images of the predetermined body surface part of the patient;
[0257] using the at least a processor with program instructions:e) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;
[0258] f) aligning the depth map with the 3D model of the predetermined body surface part of the patient and registering the patient’s 3D anatomical model to the depth map;
[0259] using a display:
[0260] g) displaying image data indicative of the patient’s 3D anatomical model registered to the depth map; and,
[0261] successively repeating steps d) to g) for dynamically adjusting the registration.
[0262] Embodiment 2: The method of embodiment 1, wherein in step e) the depth map is generated using a pre-trained convolutional neural network process for analyzing at least a disparity between the paired images.
[0263] Embodiment 3 : The method of embodiment 1 or 2, wherein in step e) a region of interest is extracted from the sensed predetermined body surface part of the patient using a face detection or anatomical landmark detection process.
[0264] Embodiment 4:The method of embodiment 1, 2 or 3, wherein in step f) comprises determining an initial estimate of a transformation matrix for approximately aligning the depth map with the 3D model of the predetermined body surface part of the patient.
[0265] Embodiment 5 : The method of embodiment 4, wherein the initial estimate of the transformation matrix is determined using a random sample consensus process.
[0266] Embodiment 6: The method of embodiment 4 or 5, comprising refining the transformation matrix using an iterative closest point process.
[0267] Embodiment 7: The method of embodiment 6, comprising determining a mean squared error or point-to-plane distances at each iteration and determining convergence when a change in the error or distances is below a predetermined threshold.
[0268] Embodiment 8: The method of embodiment 5, 6 or 7, wherein the transformation matrix is refined in selected areas of the depth map.Embodiment 9: The method of any one of embodiments 1 to 8, wherein the alignment of selected areas of the depth map is determined with higher accuracy than non-selected areas.
[0269] Embodiment 10: The method of any one of embodiments 1 to 9, wherein the steps d) to g) are performed at a speed of at least 15 fps.
[0270] Embodiment 11: The method of any one of embodiments 1 to 10, wherein the steps d) to g) are performed at predetermined time intervals and wherein a user is enabled to change the time intervals.
[0271] Embodiment 12: The method of any one of embodiments 1 to 11, comprising: comparing consecutively sensed images and calculating a similarity index; and, performing steps e) and g) when the similarity index is below a predetermined threshold.
[0272] Embodiment 13: The method of any one of embodiments 1 to 12, wherein a region of interest is selected and wherein steps e) and g) are performed when a shift in the position of the selected region of interest or a change in size of the selected region of interest is above a predetermined threshold.
[0273] Embodiment 14: The method of any one of embodiments 1 to 13, wherein the depth map is determined using a surface mesh and warping method.
[0274] Embodiment 15: The method of any one of embodiments 1 to 14, comprising: mapping a canonical surface mesh to consecutively sensed images for determining motion; and,
[0275] performing steps d) to g) when the motion is above a predetermined threshold.
[0276] Embodiment 16: The method of any one of embodiments 1 to 15, wherein step e) is performed using a processor of the at least a camera.
[0277] Embodiment 17: A method for dynamically tracking movement of a tool with respect to a patient’s 3D anatomical model comprising:
[0278] using at least a camera configured for stereo imaging:
[0279] a) sensing paired images of a predetermined body surface part of the patient with the tool;using at least a processor with program instructions:
[0280] b) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;
[0281] c) identifying the tool within the paired images using a machine learning process;
[0282] d) determining position and orientation of the tool with respect to the depth map;
[0283] e) aligning the depth map with a 3D model of the predetermined body surface part of the patient and registering the patient’s 3D anatomical model to the depth map;
[0284] using a display:
[0285] f) displaying image data indicative of the patient’s 3D anatomical model overlay ed with the tool; and,
[0286] successively repeating steps a) to f) for dynamically tracking the movement of the tool.
[0287] Embodiment 18: The method of embodiment 17, comprising training the machine learning process based on data indicative of images of at least a tool.
[0288] Embodiment 19: The method of embodiment 17 or 18, comprising:
[0289] capturing a plurality of images from various angles of the tool; and,
[0290] training the machine learning process based on data indicative of the plurality of images.
[0291] Embodiment 20: The method of embodiment 17, 18 or 19, wherein the machine learning process is a convolutional neural networks process.
[0292] Embodiment 21: The method of any one of embodiments 17 to 20, wherein step d) comprises determining a 3D bounding volume corresponding to a region of interest where the tool is likely located.
[0293] Embodiment 22: The method of any one of embodiments 17 to 21, wherein in step d) the position and orientation of the tool are determined using a pose estimation process for analyzing geometry and depth information of the paired images.
[0294] Embodiment 23: The method of any one of embodiments 17 to 21, wherein in step d) the position and orientation of the tool are determined using internal vector representation of the tool.Embodiment 24: The method of any one of embodiments 17 to 23, wherein in step f) the image data are indicative of the patient’s 3D anatomical model overlay ed with a trajectory of the tool.
[0295] Embodiment 25: The method of any one of embodiments 17 to 24, wherein steps a) to f) are performed at a speed of at least 15 fps.
[0296] Embodiment 26: The method of any one of embodiments 17 to 25, wherein the tool comprises a plurality of markers disposed thereon at different locations and wherein in steps c) and d) the markers are used as reference landmarks for determining the position and the orientation of the tool.
[0297] Embodiment 27: The method of any one of embodiments 17 to 26, comprising:
[0298] using the at least a camera configured for stereo imaging:
[0299] g) sensing paired images of a user’s hand;
[0300] using the at least a processor with program instructions:
[0301] h) identifying landmarks of the user’s hand using a machine learning process;
[0302] i) determining a position of at least a predetermined tracking point of the user’s hand in dependence upon the landmarks; and,
[0303] j) executing steps b) to f) in dependence upon the position of the at least a predetermined tracking point.
[0304] Embodiment 28: The method of embodiment 27, wherein steps g) to j) are executed concurrently with steps b) to f).
[0305] Embodiment 29: The method of embodiment 27 or 28, wherein step i) comprises: determining a center line of at least a finger; and,
[0306] determining a pointing position in dependence upon the center line.
[0307] Embodiment 30: The method of embodiment 29, comprising extending the center line by a predetermined distance.
[0308] Embodiment 31: The method of embodiment 29 or 30, comprising localized depth mapping in proximity to the center line.Embodiment 32: The method of any one of embodiments 27 to 31, comprising:
[0309] using the at least a processor with program instructions:
[0310] k) identifying hand gestures of the user’s hand using the machine learning process; and, l) executing steps b) to f) in dependence upon the identified hand gestures.
[0311] Embodiment 33: The method of embodiment 32, wherein steps k) and 1) are executed concurrently with steps b) to j).
[0312] Embodiment 34: The method of any one of embodiments 27 to 33, determining at least a distance between at least two predetermined landmarks of the user’s hand.
[0313] Embodiment 35: The method of any one of embodiments 17 to 34, comprising:
[0314] using a single image frame RGB camera sensing a single RGB image of a predetermined body surface part of the patient with the tool; and,
[0315] identifying a region containing the tool based on the single RGB image.
[0316] Embodiment 36: The method of embodiment 35, comprising:
[0317] using an IR camera configured for stereo imaging sensing paired IR images of the predetermined body surface part of the patient with the tool;
[0318] generating a depth map of the sensed predetermined body surface part of the patient with the tool based on the paired IR images; and,
[0319] determining within the identified region position and orientation of the tool with respect to the depth map.
[0320] Embodiment 37: The method of any one of embodiments 17 to 36, wherein the tool is rigid and has a shaft-like structure and comprises a marker with a high-contrast feature, and wherein step d) comprises refining the position and orientation of the tool with respect to the depth map based on image data of the high-contrast feature sensed in step a).
[0321] Embodiment 38: The method of any one of embodiments 1 to 16, comprising: affixing to the patient’s body at least one of at least an inertia measurement sensor and at least a magnetometer;
[0322] using the at least a processor with program instructions:
[0323] receiving motion sensor data from the at least one of at least an inertia measurement sensorand at least a magnetometer; and,
[0324] processing the motion sensor data for determining patient movement in dependence thereupon.
[0325] Embodiment 39: The method of embodiment 38, comprising providing an alert message when the patient movement is above a predetermined threshold.
[0326] Embodiment 40: The method of embodiment 38 or 39, comprising pausing repeating steps b) to f) when the patient movement is above a predetermined first threshold.
[0327] Embodiment 41: The method of any one of embodiments 38 to 40, comprising continuing repeating steps b) to f) when the patient movement is below a predetermined second threshold.
[0328] Embodiment 42: The method of any one of embodiments 38 to 41, wherein the patient’s 3D anatomical model is re-registered based on the motion sensor data.
[0329] Embodiment 43: The method of any one of embodiments 38 to 42, wherein the motion sensor data are processed based on biomechanical kinematics of the patient’s anatomy.
[0330] Embodiment 44: The method of any one of embodiments 38 to 43, comprising:
[0331] sensing a plurality of predetermined patient movements and generating motion sensor data in dependence thereupon; and,
[0332] determining a model of the patient’s biomechanical kinematics based on the predetermined patient movements and the motion sensor data.
[0333] Embodiment 45: The method of embodiment 42, wherein the patient’s 3D anatomical model is intermittently re-registered by performing steps a) to f).
[0334] Embodiment 46: The method of any one of embodiments 38 to 45, comprising providing a reference magnet in proximity to the predetermined body surface part of the patient.
[0335] Embodiment 47: A system for dynamically tracking movement of a tool with respect to a patient’s 3D anatomical model comprising:at least a camera configured for stereo imaging;
[0336] at least a processor with program instructions, the at least a processor when executing the program instructions performing:
[0337] a) receiving data indicative of the paired images of a predetermined body surface part of the patient with the tool;
[0338] b) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;
[0339] c) identifying the tool within the paired images using a machine learning process;
[0340] d) determining position and orientation of the tool with respect to the depth map;
[0341] e) aligning the depth map with the patient’s 3D anatomical model;
[0342] f) generating image data indicative of the patient’s 3D anatomical model overlay ed with the tool; and,
[0343] g) successively repeating steps a) to f) for dynamically tracking the movement of the tool; and,
[0344] a display adapted for displaying the image data indicative of the patient’s 3D anatomical model overlayed with the tool.
[0345] Embodiment 48: The system of embodiment 47, comprising a central computer having a CPU and a GPU with the CPU and the GPU being adapted for parallel processing.
[0346] Embodiment 49: The system of embodiment 48, wherein the central computer is adapted for being connected to the at least a camera via an I / O subsystem.
[0347] Embodiment 50: The system of embodiment 48 or 49, wherein the central computer is a handheld device comprising a graphical user interface.
[0348] Embodiment 51: The system of any one of embodiments 47 to 50, wherein the at least a processor is adapted for performing the steps a) to f) at a speed of at least 15 fps.
[0349] Embodiment 52: The system of any one of embodiments 47 to 51, wherein the at least a camera is a high-resolution camera comprising one of:
[0350] a colour sensor;
[0351] a monochrome sensor capable of operating in the visible spectrum; and,
[0352] a monochrome sensor capable of operating in the infrared spectrum.Embodiment 53: The system of any one of embodiments 47 to 52, wherein the at least a camera is a high-resolution camera comprising a monochrome sensor capable of operating in the infrared spectrum and wherein the system further comprises a single frame high-resolution camera comprising a colour sensor.
[0353] Embodiment 54: The system of any one of embodiments 47 to 53, wherein the at least a camera comprises a processor capable of performing the steps a) and b).
[0354] Embodiment 55: The system of any one of embodiments 47 to 54, comprising at least one of at least an inertia measurement sensor and at least a magnetometer adapted for being affixed to the patient’s body and
[0355] wherein
[0356] the at least a processor with program instructions is adapted for performing:
[0357] receiving motion sensor data from the at least one of at least an inertia measurement sensor and at least a magnetometer; and,
[0358] processing the motion sensor data for determining patient movement in dependence thereupon;
[0359] when executing the program instructions.
[0360] Embodiment 56: The system of any one of embodiments 47 to 55, wherein the at least a processor is adapted for being connected to at least a PAC repository for receiving imaging data in DICOM format and wherein the at least a processor performs:
[0361] receiving cross-sectional imaging data of the patient;
[0362] determining the patient’s 3D anatomical model in dependence upon the cross-sectional imaging data of the patient; and,
[0363] determining a 3D model of the predetermined body surface part of the patient in dependence upon the cross-sectional imaging data of the patient;
[0364] when executing the program instructions.
[0365] Embodiment 57: A computer program product comprising a computer readable memory storing computer executable instructions thereon that when executed by a computer cause the computer to perform the method steps of any one of embodiments 1 to 16.Embodiment 58: A computer program product comprising a computer readable memory storing computer executable instructions thereon that when executed by a computer cause the computer to perform the method steps of any one of embodiments 17 to 45.
[0366] Embodiment 59: A method according to any one of embodiments 1 to 45, wherein the tool is a medical tool.
[0367] Embodiment 60: A method according to any one of embodiments 1 to 45, wherein the tool is a surgical tool.
[0368] Embodiment 61: A system according to any one of embodiments 47 to 56, wherein the tool is a medical tool.
[0369] Embodiment 62: A system according to any one of embodiments 47 to 56, wherein the tool is a surgical tool.
[0370] Embodiment 63: A method according to any one of embodiments 1 to 45, wherein the method steps are carried out in the context of a medical procedure.
[0371] Embodiment 64: A method according to any one of embodiments 1 to 45, wherein the method steps are carried out in the context of a surgical procedure.
[0372] Embodiment 65: A system according to any one of embodiments 47 to 56, wherein the system is for use in the context of a medical procedure.
[0373] Embodiment 66: A system according to any one of embodiments 47 to 56, wherein the system is for use in the context of a surgical procedure.
[0374] The above disclosure and figures are intended to be illustrative and not exhaustive. The description will suggest many variations and alternatives to one of ordinary skill in the art. Those familiar with the art may recognize other equivalents to the specific embodiments described herein within, without departing from the scope thereof.
Claims
What is claimed is:
1. A method for dynamically adjusting a registration of a patient’s 3D anatomical model to features extracted from a predetermined body surface part of the patient comprising: using at least a processor with program instructions:a) receiving cross-sectional imaging data of the patient;b) determining the patient’s 3D anatomical model in dependence upon the cross- sectional imaging data of the patient;c) determining a 3D model of the predetermined body surface part of the patient in dependence upon the cross-sectional imaging data of the patient;using at least a camera configured for stereo imaging:d) sensing paired images of the predetermined body surface part of the patient; using the at least a processor with program instructions:e) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;f) aligning the depth map with the 3D model of the predetermined body surface part of the patient and registering the patient’s 3D anatomical model to the depth map; using a display:g) displaying image data indicative of the patient’s 3D anatomical model registered to the depth map; and,successively repeating steps d) to g) for dynamically adjusting the registration.
2. The method of claim 1, wherein in step e) the depth map is generated using a pretrained convolutional neural network process for analyzing at least a disparity between the paired images.
3. The method of claim 1, wherein in step e) a region of interest is extracted from the sensed predetermined body surface part of the patient using a face detection or anatomical landmark detection process.
4. The method of claim 1, wherein in step f) comprises determining an initial estimate of a transformation matrix for approximately aligning the depth map with the 3D model of the predetermined body surface part of the patient.
5. The method of claim 4, wherein the initial estimate of the transformation matrix is determined using a random sample consensus process.
6. The method of claim 4, comprising refining the transformation matrix using an iterative closest point process.
7. The method of claim 6, comprising determining a mean squared error or point-to-plane distances at each iteration and determining convergence when a change in the error or distances is below a predetermined threshold.
8. The method of claim 5, wherein the transformation matrix is refined in selected areas of the depth map.
9. The method of claim 1, wherein the alignment of selected areas of the depth map is determined with higher accuracy than non-selected areas.
10. The method of claim 1, wherein the steps d) to g) are performed at a speed of at least 15 fps.
11. The method of claim 1, wherein the steps d) to g) are performed at predetermined time intervals and wherein a user is enabled to change the predetermined time intervals.
12. The method of claim 1, comprising:comparing consecutively sensed images and calculating a similarity index; and, performing steps e) and g) when the similarity index is below a predetermined threshold.
13. The method of claim 1, wherein a region of interest is selected and wherein steps e) and g) are performed when a shift in the position of the selected region of interest or a change in size of the selected region of interest is above a predetermined threshold.
14. The method of claim 1, wherein the depth map is determined using a surface mesh and warping method.
15. The method of claim 1, comprising:mapping a canonical surface mesh to consecutively sensed images for determining motion; and,performing steps d) to g) when the motion is above a predetermined threshold.
16. The method of claim 1, wherein step e) is performed using a processor of the at least a camera.
17. A method for dynamically tracking movement of a tool with respect to a patient’s 3D anatomical model comprising:using at least a camera configured for stereo imaging:a) sensing paired images of a predetermined body surface part of the patient with the tool;using at least a processor with program instructions:b) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;c) identifying the tool within the paired images using a machine learning process; d) determining position and orientation of the tool with respect to the depth map; e) aligning the depth map with a 3D model of the predetermined body surface part of the patient and registering the patient’s 3D anatomical model to the depth map; using a display:f) displaying image data indicative of the patient’s 3D anatomical model overlay ed with the tool; and,successively repeating steps a) to f) for dynamically tracking the movement of the tool.
18. The method of claim 17, comprising training the machine learning process based on data indicative of images of at least a tool.
19. The method of claim 18, comprising:capturing a plurality of images from various angles of the tool; and,training the machine learning process based on data indicative of the plurality of images.
20. The method of claim 17, wherein the machine learning process is a convolutional neural networks process.
21. The method of claim 17, wherein step d) comprises determining a 3D bounding volume corresponding to a region of interest where the tool is likely located.
22. The method of claim 21, wherein in step d) the position and orientation of the tool are determined using a pose estimation process for analyzing geometry and depth information of the paired images.
23. The method of claim 21, wherein in step d) the position and orientation of the tool are determined using internal vector representation of the tool.
24. The method of claim 17, wherein in step f) the image data are indicative of the patient’s 3D anatomical model overlay ed with a trajectory of the tool.
25. The method of claim 17, wherein steps a) to f) are performed at a speed of at least 15 fps.
26. The method of claim 17, wherein the tool comprises a plurality of markers disposed thereon at different locations and wherein in steps c) and d) the markers are used as reference landmarks for determining the position and the orientation of the tool.
27. The method of claim 17, comprising:using the at least a camera configured for stereo imaging:g) sensing paired images of a user’s hand;using the at least a processor with program instructions:h) identifying landmarks of the user’s hand using a machine learning process;i) determining a position of at least a predetermined tracking point of the user’s hand in dependence upon the landmarks; and,j) executing steps b) to f) in dependence upon the position of the at least a predetermined tracking point.
28. The method of claim 27, wherein steps g) to j) are executed concurrently with steps b) to f).
29. The method of claim 27, wherein step i) comprises:determining a center line of at least a finger; and,determining a pointing position in dependence upon the center line.
30. The method of claim 29, comprising extending the center line by a predetermined distance.
31. The method of claim 29, comprising localized depth mapping in proximity to the center line.
32. The method of claim 27, comprising:using the at least a processor with program instructions:k) identifying hand gestures of the user’s hand using the machine learning process; and,l) executing steps b) to f) in dependence upon the identified hand gestures.
33. The method of claim 32, wherein steps k) and 1) are executed concurrently with steps b) toj).
34. The method of claim 27, determining at least a distance between at least two predetermined landmarks of the user’s hand.
35. The method of claim 17, comprising:using a single image frame RGB camera sensing a single RGB image of a predetermined body surface part of the patient with the tool; and,identifying a region containing the tool based on the single RGB image.
36. The method of claim 35, comprising:using an IR camera configured for stereo imaging sensing paired IR images of the predetermined body surface part of the patient with the tool;generating a depth map of the sensed predetermined body surface part of the patient with the tool based on the paired IR images; and,determining within the identified region position and orientation of the tool with respect to the depth map.
37. The method of claim 17, wherein the tool is rigid and has a shaft-like structure and comprises a marker with a high-contrast feature, and wherein step d) comprises refining the position and orientation of the tool with respect to the depth map based on image data of the high-contrast feature sensed in step a).
38. The method of claim 1, comprising:affixing to the patient’s body at least one of at least an inertia measurement sensor and at least a magnetometer;using the at least a processor with program instructions:receiving motion sensor data from the at least one of at least an inertia measurement sensor and at least a magnetometer; and,processing the motion sensor data for determining patient movement in dependence thereupon.
39. The method of claim 38, comprising providing an alert message when the patient movement is above a predetermined threshold.
40. The method of claim 38, comprising pausing repeating steps b) to f) when the patient movement is above a predetermined first threshold.
41. The method of claim 40, comprising continuing repeating steps b) to f) when the patient movement is below a predetermined second threshold.
42. The method of claim 38, wherein the patient’s 3D anatomical model is re-registered based on the motion sensor data.
43. The method of claim 42, wherein the motion sensor data are processed based on biomechanical kinematics of the patient’s anatomy.
44. The method of claim 43, comprising:sensing a plurality of predetermined patient movements and generating motion sensor data in dependence thereupon; and,determining a model of the patient’s biomechanical kinematics based on the predetermined patient movements and the motion sensor data.
45. The method of claim 42, wherein the patient’s 3D anatomical model is intermittently re-registered by performing steps a) to f).
46. The method of claim 38, comprising providing a reference magnet in proximity to the predetermined body surface part of the patient.
47. A system for dynamically tracking movement of a tool with respect to a patient’s 3D anatomical model comprising:at least a camera configured for stereo imaging;at least a processor with program instructions, the at least a processor when executing the program instructions performing:a) receiving data indicative of the paired images of a predetermined body surface part of the patient with the tool;b) generating a depth map of the sensed predetermined body surface part of the patient based on the paired images;c) identifying the tool within the paired images using a machine learning process; d) determining position and orientation of the tool with respect to the depth map; e) aligning the depth map with the patient’s 3D anatomical model;f) generating image data indicative of the patient’s 3D anatomical model overlay ed with the tool; and,g) successively repeating steps a) to f) for dynamically tracking the movement of the tool; and,a display adapted for displaying the image data indicative of the patient’s 3D anatomical model overlayed with the tool.
48. The system of claim 47, comprising a central computer having a CPU and a GPU with the CPU and the GPU being adapted for parallel processing.
49. The system of claim 48, wherein the central computer is adapted for being connected to the at least a camera via an I / O subsystem.
50. The system of claim 49, wherein the central computer is a handheld device comprising a graphical user interface.
51. The system of claim 47, wherein the at least a processor is adapted for performing the steps a) to f) at a speed of at least 15 fps.
52. The system of claim 47, wherein the at least a camera is a high-resolution camera comprising one of:a colour sensor;a monochrome sensor capable of operating in the visible spectrum; and,a monochrome sensor capable of operating in the infrared spectrum.
53. The system of claim 47, wherein the at least a camera is a high-resolution camera comprising a monochrome sensor capable of operating in the infrared spectrum and wherein the system further comprises a single frame high-resolution camera comprising a colour sensor.
54. The system of claim 47, wherein the at least a camera comprises a processor capable of performing the steps a) and b).
55. The system of claim 47, comprising at least one of at least an inertia measurement sensor and at least a magnetometer adapted for being affixed to the patient’s body and whereinthe at least a processor with program instructions is adapted for performing:receiving motion sensor data from the at least one of at least an inertia measurement sensor and at least a magnetometer; and,processing the motion sensor data for determining patient movement in dependence thereupon;when executing the program instructions.
56. The system of claim 47, wherein the at least a processor is adapted for being connected to at least a PAC repository for receiving imaging data in DICOM format and wherein the at least a processor performs:receiving cross-sectional imaging data of the patient;determining the patient’s 3D anatomical model in dependence upon the cross- sectional imaging data of the patient; and,determining a 3D model of the predetermined body surface part of the patient independence upon the cross-sectional imaging data of the patient;when executing the program instructions.
57. A computer program product comprising a computer readable memory storing computer executable instructions thereon that when executed by a computer cause the computer to perform the method steps of any one of claims 1 to 16.
58. A computer program product comprising a computer readable memory storing computer executable instructions thereon that when executed by a computer cause the computer to perform the method steps of any one of claims 17 to 45.
