A computer-implemented method for surgical assistance

Abstract

The present disclosure generally relates to a computer-implemented method for surgical assistance, specifically adapted to determine and visualize a spatial relationship between a surgical tool (302, 310) and a marker unit (200) physically related to an internal body portion (306) of a patient The present disclosure also relates to a corresponding computer system (100) and computer program product.

Description

[0001] A COMPUTER-IMPLEMENTED METHOD FOR SURGICAL ASSISTANCE

[0002] TECHNICAL FIELD

[0003] The present disclosure generally relates to a computer-implemented method for surgical assistance, specifically adapted to determine and visualize a spatial relationship between a surgical tool and a marker unit physically related to an internal body portion of a patient. The present disclosure also relates to a corresponding computer system and computer program product.

[0004] BACKGROUND

[0005] Minimally invasive surgery (MIS) has emerged as a preferred technique due to its advantages in reducing patient recovery time, minimizing surgical trauma, and lowering the risk of post-surgery complications and infection. Among the various technologies enabling MIS, laparoscopic surgery stands out, offering the ability to perform complex procedures through small incisions with the assistance of a camera and specialized instruments. Despite its advantages, laparoscopic surgery presents unique challenges, primarily due to the surgeon’s reliance on two-dimensional video feeds to navigate the three-dimensional internal anatomy of the patient. Such a limitation may result in difficulties of spatial orientation, identification of specific anatomical structures, accurate assessment of the position and orientation of surgical tools, and the precise manipulation of surgical tools.

[0006] To address these challenges, augmented reality (AR) technologies have been introduced to the surgical field, aiming to enhance the surgeon’s visual field by overlaying critical virtual information onto the real-world image. AR in surgery promises to merge the physical and digital worlds, providing real-time, three-dimensional navigational aids directly within the surgeon’s field of view. However, the practical application of AR in surgical procedures demands high accuracy in the alignment and tracking of both anatomical structures and surgical tools, which remains a significant technical hurdle.

[0007] One exemplary system attempting to solve the challenges associated with AR in surgery is presented in US9646423B1, which introduces an AR system for MIS that leverages pre-operative image data. The system presented in US9646423B1 seeks to enhance surgical precision by merging pre-operative images with intra-operative views captured by an endoscope, aiming to provide real-time augmented reality overlays during surgical procedures. The solution as presented in US9646423B1 represents a great step towardsintegrating AR technology into MIS, facilitating a more informed surgical process by allowing surgeons to visualize structures that are not immediately apparent during surgery.

[0008] However, the solution proposed in US9646423B1, necessitates significant computational resources for achieving the desired merging effect, and may encounter reliability issues in dynamically changing surgical environments. The reliance on the processing and registration of complex image data sets may also introduce latency and will as such not always offer the real-time accuracy required for certain surgical interventions.

[0009] Furthermore, the method as presented in US9646423B1 relies on blood vessel-based feature detection for tracking the position and orientation of the endoscope, which may encounter difficulties in consistently identifying reliable anatomical landmarks. Accordingly, there is an apparent risk that such a dependency could affect the ability of the system to accurately correlate between pre-operative and intra-operative images, thus impacting the overall reliability of the augmented reality overlays during critical surgical tasks.

[0010] With the above in mind, there is an apparent desire to provide further enhancements for surgical assistance, specifically focusing on user-friendly AR-assisted navigation in surgical settings and for enhancing the capabilities of surgeons to perform complex operations with greater reliability and accuracy, without the limitations observed in currently available prior art solutions, such as exemplified by US9646423B1.

[0011] Further attention is drawn to US20180049622A1, presenting a mixed reality surgical navigation system, which relies on stereoscopic tracking using multiple cameras and / or sensor fusion involving e.g. IMU and SLAM to derive spatial data such as depth and orientation through triangulation and mapping. Such as solution, similar to the solution presented in US9646423B1, results in significant computational resources for achieving the desired merging effect.

[0012] SUMMARY

[0013] According to an aspect of the present disclosure, the above is at least partly alleviated by a computer-implemented method for surgical assistance using a computer system comprising a processing unit, wherein the method comprises the steps of receiving, at the processing unit, image data illustrating a first surgical tool and an internal body portion of a patient, wherein the internal body portion is physically related to a marker unit, the marker unit comprising optically detectable features, identifying, using the processing unit and based on the image data, one or more geometrical features associated with the first surgical tool, determining, using the processing unit, a spatial relationship between the first surgical tooland the marker unit based on the identified geometrical features of the first surgical tool, the optically detectable features of the marker unit, and a predefined size of at least a portion of the marker unit, forming, using the processing unit, a graphical visualization representing the spatial relationship between the first surgical tool and the marker unit, the graphical visualization being based on the identified features of the first surgical tool and the determined spatial relationship, and displaying, at a display device arranged in communication with the computer system, the graphical visualization.

[0014] By means of the present disclosure, it is made possible to enhance the accuracy and efficiency of minimally invasive surgeries (MIS), particularly by addressing the challenge of accurately determining and visualizing the position and orientation of surgical tools in relation to an internal body portion. Such an enhancement directly improves spatial orientation and the precise manipulation of surgical tools, both of which are obvious challenges in surgical practice.

[0015] The present disclosure provides a method in which image data illustrating at least one surgical tool and an internal body portion of a patient is received and processed. The internal body portion is physically related to a marker unit, which provides a reference for tracking. Based on the image data, features associated with the first surgical tool are identified, allowing the system to determine its spatial relationship with the marker unit. A graphical visualization is then generated, representing the relative position and orientation of the tool in real time. The visualization may be presented separately from the image data or as an augmented reality (AR) overlay integrated into the live surgical feed.

[0016] Such an approach significantly improves surgical guidance by offering enhanced visualization and spatial awareness, particularly in procedures where depth perception is limited. Specifically, by providing an intuitive, real-time interface that aligns with the surgeon’s workflow, the present disclosure streamlines tool navigation, reduces the risk of unintended tissue interaction, and ultimately contributes to improved surgical precision and patient safety.

[0017] As used herein, the term “internal body portion” refers to any anatomical structure within a patient’s body, including soft tissue organs (e.g., liver, kidney, pancreas), vascular structures, and skeletal components, depending on the surgical context.

[0018] In line with the present disclosure, the technique of forming a graphical visualization of the first surgical tool relative to the marker unit may overcome several technical hurdles present in currently available surgical navigation systems. Specifically, by using a marker unit with optically detectable features, the system provides a robust andconsistent reference point, ensuring that the computed spatial relationship remains highly accurate even in dynamic surgical environments.

[0019] The term “graphical visualization” as used within the context of the present disclosure refers to a digitally generated representation that illustrates the spatial relationship between the first surgical tool and the marker unit. Such a visualization is formed based on features identified in the image data and the computed spatial relationship between the first surgical tool and the marker unit. The graphical visualization may be presented either separately from the image data or as an augmented reality (AR) overlay integrated within the image feed. The AR approach ensures that real-time spatial cues remain aligned with the captured image, allowing seamless navigation while preserving the operator’s natural field of view, in turn minimizing workflow disruptions by embedding essential spatial information directly into the live surgical feed.

[0020] The term “marker unit” as used within the context of the present disclosure refers to a device or marker designed to be physically related to an internal body portion of a patient, facilitating localization and tracking within the surgical field. As stated above, the marker unit comprises optically detectable features, enabling identification within the image data. These optically detectable features may include, for example, geometrical figures, visual markers, or machine-readable codes that allow for accurate detection using image processing techniques. Optionally, the marker unit may also comprise radiopaque features, making it visible under imaging modalities such as CT or X-ray imaging. Such a dual functionality allows the marker unit to be used across multiple imaging platforms, enhancing the ability of the computer system to correlate the physical position of the internal body portion with its digital representation. Specifically, the inclusion of radiopaque features allows for additional alignment capabilities when preoperative imaging modalities are used, improving integration with preoperative surgical planning data.

[0021] The term “surgical tool” as used within the context of the present disclosure refers broadly to any instrument, device, or implement used in relation to a surgical procedure. The expression should be interpreted inclusively, covering both manual and powered instruments used for diagnosis, intervention, or assistance during surgery. Examples of surgical tools may include, but are not limited to, laparoscopic instruments, robot-assisted surgical tools, electrosurgical devices, surgical forceps, scalpels, needle holders, and catheters. Additionally, imaging-related tools such as handheld ultrasonic scanning probes or endoscopic ultrasound devices may also be considered surgical tools within the meaning ofthe present disclosure, particularly when used for intraoperative guidance or tissue characterization.

[0022] Furthermore, the term “physically related to” is to comprise any spatial arrangement where the marker unit shares a coordinate reference with the internal body portion. In preferred embodiments, such relationship defines the marker unit as fixedly attached, adhered, or mechanically coupled to the internal body portion such that they move in unison. The fixed correlation between the marker unit and the internal body portion ensures that the spatial transformation between the marker unit and the anatomy remains valid throughout the procedure, regardless of the specific method of attachment employed.

[0023] It should be appreciated that while the embodiments described herein are primarily discussed in the context of real-time surgical assistance, the present disclosure is not limited thereto. The computer-implemented method as disclosed is equally applicable to the processing of pre-recorded image data, for instance, during post-operative review, surgical training simulations, or the analysis of surgical workflows. In such non-surgical scenarios, the step of receiving image data refers to retrieving data from a storage medium rather than receiving a live feed from an image capturing device. Consequently, the method steps described herein are capable of being executed physically separately from the actual surgical intervention and do not require a simultaneous surgical step to be performed.

[0024] Furthermore, it is to be understood that the determined spatial relationship between the first surgical tool and the marker unit may serve as a basis for at least partially autonomous robotic control. Since the present disclosure provides a precise, real-time definition of the position of the anatomy (by means of the marker unit), the position of the tool and the relative geometry therebetween, the computer system may be configured to generate control signals for a robotic surgical system. In such a possible embodiment, the graphical visualization data or the underlying spatial coordinates are fed into a robot control unit to enable a closed-loop control mechanism. Consequently, the solution as defined in line with the present disclosure effectively bridges the gap between perception and action, allowing a robotic agent to navigate or perform actions relative to the internal body portion with a high degree of spatial autonomy.

[0025] Additionally, such an implementation is generally advantageous for establishing e.g. safety constraints in an autonomous or semi-autonomous workflow. By utilizing the fixed geometric correlation provided by the marker unit, the processing unit may for example define precise operational boundaries, such as cutting trajectories or no-go zones, relative to the internal body portion. The solution as is defined by means of the presentdisclosure may then automatically restrict the movement of the first surgical tool or actively guide the tool along a predefined path based on the continuously updated spatial relationship. Thus, the disclosed method not only aids human visualization but also provides the requisite high-fidelity spatial feedback loop necessary for advancing towards autonomous surgical intervention.

[0026] In some possible embodiment of the present disclosure the solution as is defined by means of the present disclosure may also be configured to allow for fully autonomous surgical tasks. By leveraging the continuous, real-time spatial data derived from the marker unit and the surgical tool as discussed above, the processing unit may actively plan and execute specific surgical actions without direct human intervention (or with minimal intervention). For example, the processing unit may be configured to calculate a trajectory for a surgical maneuver, such as a suture, an incision, or a resection, and autonomously drive the actuation of the surgical tool along said trajectory. As such, by updating the path of the tool in response to any detected movement of the marker unit, and consequently the internal body portion, the computer system may be arranged to ensure that the autonomous action remains accurate and synchronized with the overall anatomy of the patient.

[0027] Furthermore, within the scope of the present disclosure, it may be possible to include a second surgical tool, and possibly even further surgical tools, allowing for multiple instruments to be tracked and visualized concurrently. The system may accommodate a combination of different surgical tools, each of which may be identified using techniques specific to the tool type. For example, rigid instruments may be identified based on predefined geometric patterns, while flexible instruments, such as catheters, may be tracked using shape-sensing technologies. The identification of surgical tools, including potential tracking and feature extraction methods, is elaborated further below.

[0028] In an embodiment, the graphical visualization is presented separately from the image data. By displaying the visualization independently from the image data, the system enables an alternative perspective on the spatial relationship between the surgical tool and the marker unit. Such an implementation may be particularly beneficial in providing the operator with an additional side view representation, allowing for enhanced depth perception that may not be readily available in a standard two-dimensional endoscopic or laparoscopic feed.

[0029] A “side view visualization” may depict the first surgical tool and the marker unit within a coordinate system, visually indicating their relative positioning without obstruction from surrounding anatomical structures. Such a perspective allows the operator to better anticipate and control tool movements, especially in procedures where precision iscritical. Furthermore, presenting the visualization separately provides flexibility in how the information is displayed. For example, it may be shown on a secondary screen, a headmounted display, or integrated into a multi-view surgical navigation interface (preferably also illustrating the received image data).

[0030] Possible advantages of presenting the graphical visualization separately may for example include that it provides a dedicated spatial reference, free from the complexities of real-time video feeds, lighting variations, and image noise. Such an approach may be particularly useful in cases where tool positioning relative to a fixed anatomical structure (e.g., a specific landmark or incision site) needs to be continuously monitored without being obscured by dynamic changes in the surgical scene.

[0031] In another embodiment, the method according to the present disclosure further comprises augmenting, using the processing unit, the image data with the graphical visualization. In this embodiment, the graphical visualization is overlaid directly onto the received image data, integrating real-time visual guidance within the same field of view as the live surgical scene.

[0032] One potential implementation of such an augmentation is the display of a virtual distance indicator between a specific point on the first surgical tool (e.g., its tip) and a corresponding reference point on the marker unit. Such an indicator may be presented as a colored line, a segmented path, or numerical distance markers, visually connecting the tool and the marker to convey depth information. Such an enhanced visualization may help the operator estimate the proximity of the surgical tool to the internal body portion, particularly in minimally invasive procedures where depth perception is inherently limited due to the use of two-dimensional imaging.

[0033] Additionally, augmentation may include dynamic feedback mechanisms, such as changing the color or opacity of the visualization based on proximity. For instance, as the tip of the surgical tool approaches a predefined distance from the marker unit, the visualized line may shift from e.g. green (safe) to yellow (caution) to red (critical proximity). Such a form of real-time visual guidance may assist the surgeon in maintaining precise tool positioning, reducing the likelihood of unintentional contact with delicate anatomical structures.

[0034] By integrating the graphical visualization into the actual surgical image feed, this embodiment provides immediate, context-aware spatial awareness without requiring the operator to shift focus between multiple screens or external reference views, possibly furtherenhancing procedural efficiency by streamlining the cognitive load of depth estimation and tool navigation.

[0035] Additionally, it may in some embodiments be desirable to arrange the scheme to comprise the step of applying, using the processing unit, a predefined augmented reality (AR) technique adapted to utilize the correlation between the marker unit’s position and the computer system’s coordinates to accurately overlay the graphical visualization onto the image data. Such an implementation ensures that the graphical visualization, representing the spatial relationship between the first surgical tool and the marker unit, is precisely aligned with the actual position and orientation of the surgical tool as viewed in the image data.

[0036] Incorporating such a predefined AR technique into the process provides substantial technical advantages. First, it significantly improves the ability to interpret the spatial relationship between the surgical tool and the marker unit within the surgeon’s direct field of view, thereby enhancing precision and reducing the likelihood of errors. The accurate overlay of spatial indicators, distance markers, or trajectory guides onto the real-world image allows operators to navigate more confidently during procedures, particularly when performing complex maneuvers or when visibility is limited.

[0037] Furthermore, the technique’s dependency on the marker unit’s position for overlay accuracy simplifies the alignment process, making the AR assistance more robust and reliable even in dynamic surgical environments where the patient’s positioning or tool orientation might change. Such a solution may in turn contribute to a more efficient surgical workflow and can potentially reduce operation times by providing intuitive, real-time depth perception.

[0038] Still further, in some embodiments of the present disclosure, it may be desirable to additionally include a step of calculating, using the processing unit, a three-dimensional distance map between the first surgical tool and the marker unit to facilitate AR alignment. Such a calculation serves to enhance the accuracy of augmented visual cues, such as depth indicators or projected alignment guides, ensuring that the overlaid graphical visualization accurately reflects the real-world positioning of the surgical tool.

[0039] The inclusion of a three-dimensional distance map calculation offers substantial technical advantages, mainly by elevating the precision of depth representation and spatial feedback within the surgical field. Such a solution ensures that virtual elements such as distance lines, warning zones, or tool trajectory projections are precisely superimposed over the relevant reference points in the image data. Additionally, a moreaccurate AR alignment reduces the likelihood of misinterpretation by ensuring that virtual indicators dynamically adjust in response to real-time changes in tool positioning.

[0040] By integrating such techniques, the present disclosure may offer an effective and intuitive method for visualizing tool positioning during surgery, thereby improving accuracy, reducing the cognitive load on the surgeon, and enhancing the overall safety of minimally invasive procedures.

[0041] Preferably, the graphical visualization further comprises a three-dimensional representation of the internal body portion, the representation being formed based on the spatial relationship between the first surgical tool and the marker unit. By incorporating such a three-dimensional model of the internal body portion into the graphical visualization, the system provides an enhanced spatial reference that allows for improved depth perception and localization of surgical tools during a procedure.

[0042] The three-dimensional representation may serve multiple purposes. Firstly, it provides the operator with a contextual anatomical reference that aids in positioning and orienting the surgical tool relative to critical structures. Secondly, when combined with the spatial relationship data obtained from the marker unit, it enables real-time visualization of tool movements in relation to the internal body portion, helping to improve precision and reduce unintended interactions with surrounding tissues.

[0043] A significant advantage of incorporating a three-dimensional representation into the graphical visualization is the ability to offer multiple perspectives on the surgical site. For example, a side view, cross-sectional view, or overlay visualization may be provided to enhance the operator’s understanding of tool positioning within the internal anatomy. Additionally, depth-enhanced visual cues, such as semi-transparent rendering of deeper structures or adjustable opacity based on depth, can be applied to further refine the surgeon’s spatial awareness.

[0044] Furthermore, the system may dynamically update the three-dimensional representation in real-time based on changes in the detected spatial relationship between the surgical tool and the marker unit. Such an implementation ensures that the displayed visualization remains accurate as the procedure progresses, providing continuous feedback to the operator.

[0045] In some embodiments, it may be desirable to allow the data acquired from at least one imaging modality to be selected from the group consisting of cone beam computed tomography (CBCT), ultrasound (US), computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single-photon emission computedtomography (SPECT), and X-ray imaging (XRAY). Other similar technologies are of course possible and within the scope of the present disclosure.

[0046] The possibility of using different types of technologies allows for selecting the most suitable imaging technique based on the specific surgical needs, the anatomical area of interest, and the distinct characteristics of the tissue or structure being examined. For instance, MRI offers soft tissue contrast, ideal for visualizing organs and neural structures, whereas CT and CBCT provide high-resolution images of bone and dense tissues.

[0047] The technical advantages of incorporating such a diverse range of imaging modalities are manifold. Firstly, such an approach enhances the precision of the three-dimensional model creation, directly influencing the accuracy of subsequent surgical planning and guidance. It further allows for tailoring the imaging process to the specific requirements of each surgery, optimizing visualization of the target area, and thereby facilitating more precise interventions. Still further, by providing alternatives that can circumvent limitations associated with a particular imaging technology the scheme according to the present disclosure enhances patient safety and improves the overall quality of surgical outcomes.

[0048] In accordance with the present disclosure, the optically detectable features and radiopaque markers are preferably provided in a geometrical pattern that does not exhibit rotational symmetry, which allows for the rotational position of the marker unit to be determined. Additionally, the geometrical patterns of the optically detectable features and the radiopaque markers have a fixed, predetermined correlation. Such a configuration pennits an accurate identification of the marker unit’s position from all directions, using both medical imaging data acquired by the imaging modality and optical data captured by the image capturing device, since they use the same transformation between medical imaging modality three-dimensional model data and the suggested product’s coordinate system.

[0049] The use of radiopaque markers, in some embodiments defined as cylinders made of e.g. metal, may further enhance acquiring of the three-dimensional model of the internal body portion of the patient. One example of an exemplary metal is gold, where gold’s high visibility under radiographic imaging techniques ensures that these markers stand out in the imaging data.

[0050] In addition to the above, the scheme according to the present disclosure encompasses the use of e.g. an image capturing device for generating the image data, wherein the image capturing device for example may be implemented by means of a laparoscopic or endoscopic camera. A laparoscopic camera is typically inserted into the patient’s bodythrough minimally invasive incisions and / or ports, whereas an endoscopic camera, such as one used in bronchoscopy, may instead be introduced through a natural orifice. The process is enhanced by utilizing the marker unit to ensure the alignment of the camera’s captured images with the pre-acquired three-dimensional model of the internal body portion. Thus, the integration of the camera within the surgical procedure is seamlessly achieved, enhancing the visualization of the internal anatomy. It should be stressed that other forms of image capturing devices may be used within the scope of the present disclosure, such as for example a bronchoscopic camera. Similar to a laparoscopic camera, a bronchoscopic camera is designed for internal examination but is specifically arranged for accessing and visualizing the airways and lungs.

[0051] In an embodiment, the spatial relationship between the first surgical tool and the marker unit is determined using a predefined size of at least a portion of the marker unit. By utilizing the known physical dimensions of the marker unit, the system can accurately estimate distances and spatial relationships within the image data. Since the marker unit serves as a fixed reference point, the computer system can analyze the captured image data and compare the detected size of the marker unit with its predefined real-world dimensions. Such a solution enables precise depth estimation, distance calculation, and perspective correction in relation to the surgical tool.

[0052] One advantage of such an approach is that it eliminates the need for additional depth sensors or stereo imaging to estimate distances. Instead, the system can leverage monocular camera input and use the marker unit as a scale reference, ensuring reliable and consistent spatial alignment even in single-camera surgical setups.

[0053] Furthermore, size-based scaling techniques may be applied to correct for perspective distortions caused by variations in camera angle or distance. If the marker unit appears smaller or larger in the image data than expected, the system can automatically adjust for these variations to maintain accurate spatial positioning.

[0054] Preferably, the graphical visualization further comprises a representation of the spatial relationship between the first surgical tool and the marker unit in a predefined plane relative to the marker unit. By introducing such a predefined plane, the system ensures a structured and consistent frame of reference for visualizing the tool’s positioning, whereby the predefined plane may be aligned with a relevant anatomical structure, such as an organ surface or a surgical workspace, allowing the operator to assess tool placement relative to the internal body portion with improved clarity.The predefined plane may be dynamically computed based on a fixed transformation relative to the marker unit, ensuring consistent spatial alignment regardless of camera positioning.

[0055] Furthermore, it may in some embodiments be desirable for the graphical visualization to include distance indicators visually representing the separation between the first surgical tool and predefined features of the internal body portion associated with the marker unit. By integrating distance indicators within the visualization, the system provides real-time spatial feedback, enabling the operator to precisely monitor the proximity of the surgical tool to critical anatomical landmarks. These indicators may be dynamically updated to reflect changes in positioning, ensuring that the visualization remains accurate as the tool moves within the surgical workspace.

[0056] The distance indicators may be computed using either direct Euclidean measurements or depth-projection techniques. Euclidean calculations determine the linear spatial separation between predefined reference points (e.g., the tool tip and the marker unit). In contrast, depth-projection techniques dynamically map the tool’s relative position onto an adaptive reference plane. By continuously updating the depth-projection framework, the system may be adapted to ensure accuracy even when the image capturing device shifts perspective, maintaining stable real-time depth cues. Furthermore, the distance indicators may be presented in various formats, such as numerical readouts, segmented visual guides, or color-coded depth lines, where closer proximity triggers dynamic color transitions to visually communicate critical thresholds.

[0057] The inclusion of distance indicators offers several technical advantages. For example, by visually representing separation distances it is made possible to improve depth perception, addressing a fundamental challenge in minimally invasive procedures where conventional two-dimensional imaging provides limited depth information, possibly allowing the operator to make more informed adjustments to the tool’s trajectory. In some implementations, distance indicators may be further enhanced through visual cues, such as color changes or varying line thickness, to intuitively convey proximity information. For example, as the surgical tool moves closer to a predefined structure, the indicator may transition from green to yellow and ultimately to red, signaling increased proximity.

[0058] By providing structured visualization of spatial relationships and integrating real-time distance indicators, the present disclosure significantly enhances the accuracy and efficiency of surgical tool navigation. The combination of a predefined plane and dynamicdistance indicators ensures that the operator receives immediate and interpretable spatial feedback, ultimately contributing to improved precision and safer surgical outcomes.

[0059] In an embodiment, the method may further comprise the step of applying, using the processing unit, a predefined noise reduction algorithm to the image data to enhance the accuracy of the spatial relationship determination and the graphical visualization. In medical imaging and surgical visualization, noise can arise from various sources, including sensor limitations, lighting conditions, or artifacts introduced by the imaging modality. For example, by applying a noise reduction algorithm to the image data, the system improves the accuracy of feature extraction, particularly in identifying the optically detectable features of the marker unit and the surgical tool. Such an enhancement ensures that the determination of spatial relationships remains robust, even when image quality is suboptimal.

[0060] Noise reduction may be implemented using a range of computational techniques, such as Gaussian filtering, adaptive thresholding, or machine learning-based denoising methods. By refining the image data before spatial processing, the system minimizes erroneous detections and improves the reliability of the graphical visualization, possibly leading to a more precise overlay of spatial relationships, reducing the risk of misinterpretation by the operator. Additionally, real-time noise suppression may enhance the responsiveness of the system, ensuring that updates to the graphical visualization remain smooth and free from visual artifacts.

[0061] Furthermore, the method may also comprise the step of updating, using the processing unit, the graphical visualization in real-time based on changes in the position or orientation of the first surgical tool relative to the marker unit. The dynamic nature of surgical procedures requires that the graphical visualization continuously reflects the latest positioning of the surgical tool. By implementing real-time updates, the system ensures that the visualization remains accurate and responsive to movements made by the operator. As the first surgical tool changes position or orientation relative to the marker unit, the processing unit recalculates the spatial relationship and modifies the visualization accordingly.

[0062] Such real-time updating may in some situations be essential for maintaining situational awareness in minimally invasive surgery (MIS), where direct depth perception is limited. Without continuous updates, there is a risk that the graphical visualization becomes misaligned with the actual tool position, reducing its effectiveness as a navigational aid. To achieve high-frequency real-time updates, the system may utilize low-latency tracking algorithms, ensuring that even rapid tool movements are accurately reflected without perceptible delay.Still further, the scheme according to the present disclosure may also comprise the step of generating, using the processing unit, a virtual surgical plan that overlays navigational cues on the at least one display device, based on the augmented visualization of the internal body portion, for guiding surgical instruments during an associated surgical procedure. The cues may for example be dynamically aligned with the augmented visualization of the patient’s internal anatomy, providing a real-time guide for the precise manipulation and placement of surgical instruments. The process leverages augmented reality to transform abstract surgical planning into an intuitive, visually guided experience, directly overlaying crucial navigational information onto the operator’s view of the surgical field.

[0063] The inclusion of a virtual surgical plan presents numerous technical advantages, and foremost, it substantially reduces the cognitive load on operators by providing an intuitive, visual guide that aligns with their natural visual processing capabilities. Such an inclusion may not only accelerate the decision-making process during surgeries but also enhances precision by providing clear, contextual cues for instrument navigation. As a result, the risk of navigational errors may possibly be reduced, and the likelihood of achieving surgical objectives with greater accuracy and efficiency is increased.

[0064] Additionally, the graphical visualization may in some embodiments further comprise a depth-adjusted representation of the internal body portion, the depth being determined based on the spatial relationship between the first surgical tool, the marker unit, and the image data. Accurately perceiving depth information is one of the primary challenges in minimally invasive surgery, where traditional two-dimensional imaging provides limited cues regarding the relative positioning of structures. By incorporating a depth-adjusted representation of the internal body portion, the present disclosure enhances the operator’s ability to assess the tool’s interaction with surrounding anatomy.

[0065] The depth-adjusted representation may be achieved by modulating the opacity, shading, or color intensity of anatomical structures within the graphical visualization, ensuring that deeper structures appear more transparent or visually distinct from those closer to the surgical tool. Such an approach improves depth perception within the visualization, helping the operator make more precise movements and avoid unintended contact with delicate tissues.

[0066] Additionally, the depth representation may dynamically update as the spatial relationship between the surgical tool and the marker unit changes. By continuously adjusting the visualization based on real-time tool movements, the system provides a highly responsiveand intuitive method for visualizing three-dimensional surgical environments, ultimately contributing to improved accuracy and procedural safety.

[0067] Preferably, the method further comprises the step of providing, using the processing unit, a user interface configured to allow an operator to adjust the graphical visualization by modifying at least one parameter selected from the group consisting of scale, transparency, and orientation. By integrating an interactive user interface (UI), the present disclosure enhances the flexibility and adaptability of the graphical visualization to accommodate different surgical preferences and procedural requirements. Through the UI, the operator may adjust the scale of the visualization to focus on specific regions of interest, modify transparency levels to emphasize or de-emphasize particular structures, or rotate and reposition the visualization to achieve an optimal viewing angle.

[0068] Providing an adjustable graphical visualization is particularly beneficial in minimally invasive surgery (MIS), where the available field of view is often constrained, and the ability to manipulate the visualization ensures that the most relevant spatial information remains prominent. The UI may be implemented on a touchscreen monitor, a physical control interface, or even a voice-command system, allowing for seamless intraoperative adjustments without disrupting the surgical workflow.

[0069] In some embodiments, the one or more features associated with the first surgical tool are identified using feature extraction techniques applied to the image data. Feature extraction techniques play a critical role in ensuring accurate and robust identification of the surgical tool within the image data. These techniques may for example involve processing the image to detect and isolate defining characteristics of the tool, such as edges, contours, texture patterns, or reflection properties. By analyzing these features, the processing unit may reliably distinguish the tool from surrounding anatomical structures and surgical instruments.

[0070] The selection of feature extraction techniques may depend on the specific imaging conditions and the tool’s visual characteristics. In high-contrast environments, edge detection algorithms such as Canny or Sobel filtering may be applied to highlight the tool’s contours. In cases where tools exhibit complex or variable geometries, machine learningbased object detection models, such as convolutional neural networks (CNNs), may be employed to enhance feature recognition and reduce false detections. Furthermore, robustness to occlusions can be improved through multi-angle detection methods, where the system analyzes successive frames to reconstruct partially obscured tool features, ensuring uninterrupted tracking during dynamic surgical movements.Feature extraction techniques facilitate real-time tool tracking by continuously identifying and updating the tool’s spatial position within the image data. Depending on imaging conditions, different methods may be applied for optimal detection. In high-contrast environments, edge detection techniques such as Canny filtering effectively highlight tool contours. For complex tool geometries, deep-leaming approaches, including convolutional neural networks (CNNs), enhance recognition accuracy while minimizing false detections. To further improve robustness in dynamic surgical environments, multi-frame tracking algorithms may be employed to reconstruct partially obscured tool features and maintain consistent tracking despite occlusions or sudden tool movements.

[0071] Additionally, the one or more features associated with the first surgical tool may in some embodiments be identified based on a physical identifier arranged on the surgical tool. Incorporating a physical identifier on the surgical tool provides an additional layer of reliability and specificity in tool recognition. A physical identifier may take the form of a QR code, a barcode, a radio-frequency identification (RFID) tag, or a passive optical pattern embedded onto the surface of the surgical tool. Such an approach ensures that even if lighting conditions, reflections, or occlusions affect standard feature extraction techniques, the identifier remains detectable.

[0072] The advantage of using a physical identifier is that it enables unique tool recognition, distinguishing between different types of instruments in the surgical field. For instance, in robot-assisted surgery or multi-tool procedures, the system may differentiate between a scalpel, forceps, or a cautery device, ensuring that the correct spatial visualization is applied to each instrument. Additionally, physical identifiers facilitate automatic calibration, where the system can adjust the graphical visualization based on the specific tool detected, optimizing overlays for the instrument in use.

[0073] Still further, the one or more features associated with the first surgical tool may in some embodiments be identified based on a geometric pattern, shape, or color associated with the surgical tool. By utilizing predefined geometric patterns, unique tool shapes, or specific color markers, the system enhances its ability to accurately track the surgical tool. Tools often have characteristic lengths, angles, or curved surfaces, which can be detected and matched against stored reference models, allowing the processing unit to recognize and differentiate them.

[0074] Color-based detection may further enhance tool identification, particularly in scenarios where certain instruments are color-coded for different surgical functions. For example, laparoscopic instruments designed for suturing, dissection, or coagulation mayfeature distinct colored components that assist the system in categorizing and tracking them effectively.

[0075] Preferably, the identification of the one or more features associated with the first surgical tool comprises detecting, using the processing unit, a machine-readable code associated with the first surgical tool, the code being optically detectable within the image data, or extracting, using the processing unit, a contour or boundary of the surgical tool from the image data. Machine-readable codes, such as QR codes or Data Matrix codes, provide an efficient and highly reliable tool identification mechanism. These codes may be directly printed or engraved onto the instrument and allow for rapid and unambiguous identification when scanned by the image capturing device. By utilizing optical recognition algorithms, the system can extract and interpret the encoded information, ensuring that the correct tool is detected in the image data.

[0076] Alternatively, the processing unit may extract the contour or boundary of the surgical tool from the image data to determine its position and orientation, possibly relying on edge detection, shape analysis, and contour-tracking algorithms to precisely identify the tool’s profile within the surgical field. The extracted boundary data can then be used to calculate the tool’s spatial position relative to the marker unit, ensuring that the graphical visualization remains accurately aligned throughout the procedure. By integrating multiple identification techniques, including feature extraction, physical identifiers, shape analysis, and machine-readable codes, the present disclosure ensures a robust, reliable, and versatile approach to tool tracking in surgical environments.

[0077] It may in some embodiments be desirable to additionally (or optionally) implement a user interface on the display device for allowing an operator to interactively adjust parameters of the overlayed visualization to tailor the augmented reality view.

[0078] Operators, such as surgeons, may as such be provided with the option to modify aspects such as the scale, orientation, and possibly the transparency of the virtual overlay directly from the interface. The inclusion of such a functionality may also grant operators more control over the visualization tools at their disposal, making the AR system not just a passive display tool but an interactive aid that can be fine-tuned in real-time during surgeries.

[0079] It may of course be possible, and within the scope of the present disclosure, to provide a separate user device, such as a tablet or a laptop, for presenting the user interface and / or a portion of the user interface. Such an implementation may allow for further improvements in the interaction between the present system and users of the system.The implementation of an interactive user interface may allow for operators to modify the AR overlay in ways that might be most conducive to the task at hand, enhancing the practical utility of the AR system as a surgical guide. For instance, adjusting the opacity or contrast of the overlay could help in highlighting specific anatomical features or surgical landmarks, thus aiding in precision tasks such as the removal of a tumor or the navigation around critical structures. Moreover, the operator may add additional virtual objects to the set of visuals, both segmentations of patient tissues, e.g., blood vessels, bile ducts, and nearby organs, but also prerendered geometries of known sizes, e.g., rulers, cubes, and spheres. The operator may even draw virtual lines on specific planes correlated to the marker unit using a laparoscopic tool that is instantly added to the set of virtual objects on display, to further enhance the practical utility of the AR system. Furthermore, the ability to interact with the AR visualization on-the-fly promotes a more intuitive surgical process. Operators may adapt the visualization based on their real-time assessment of the surgical field, potentially reducing the cognitive load, and allowing for a more focused approach to the procedure.

[0080] According to another aspect of the present disclosure, there is provided a computer system for surgical assistance, the computer system comprising a processing unit, wherein the processing unit is adapted to receive image data illustrating a first surgical tool and an internal body portion of a patient, wherein the internal body portion is physically related to a marker unit, the marker unit comprising optically detectable features, identify, based on the image data, one or more geometrical features associated with the first surgical tool, determine a spatial relationship between the first surgical tool and the marker unit based on the identified geometrical features of the first surgical tool, the optically detectable features of the marker unit, and a predefined size of at least a portion of the marker unit, form a graphical visualization representing the spatial relationship between the first surgical tool and the marker unit, the graphical visualization being based on the identified features of the first surgical tool and the determined spatial relationship, and display, at a display device arranged in communication with the computer system, the graphical visualization. Such an aspect of the present disclosure provides similar advantages as discussed above in relation to the previous aspects of the present disclosure.

[0081] According to a further aspect of the present disclosure, there is provided a computer program product comprising a non-transitory computer readable medium having stored thereon computer program means for operating a computer system designed for surgical assistance, the computer system comprising a processing unit, wherein the computer program product comprises code for receiving, at the processing unit, image data illustratinga first surgical tool and an internal body portion of a patient, wherein the internal body portion is physically related to a marker unit, the marker unit comprising optically detectable features, code for identifying, using the processing unit and based on the image data, one or more geometrical features associated with the first surgical tool, code for determining, using the processing unit, a spatial relationship between the first surgical tool and the marker unit based on the identified geometrical features of the first surgical tool, the optically detectable features of the marker unit, and a predefined size of at least a portion of the marker unit, code for forming, using the processing unit, a graphical visualization representing the spatial relationship between the first surgical tool and the marker unit, the graphical visualization being based on the identified features of the first surgical tool and the determined spatial relationship, and

[0082] code for displaying, at a display device arranged in communication with the computer system, the graphical visualization. Also this aspect of the present disclosure provides similar advantages as discussed above in relation to the previous aspects of the present disclosure.

[0083] A software executed by the processing unit for operation in accordance to the present disclosure may be stored on a computer readable medium, being any type of memory device, including one of a removable nonvolatile random access memory, a hard disk drive, a floppy disk, a CD-ROM, a DVD-ROM, a USB memory, an SD memory card, a solid state drive, other non-volatile flash based storage mediums, or a similar computer readable medium known in the art.

[0084] Further features of, and advantages with, the present disclosure will become apparent when studying the appended claims and the following description. The skilled addressee realizes that different features of the present disclosure may be combined to create embodiments other than those described in the following, without departing from the scope of the present disclosure.

[0085] BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The various aspects of the present disclosure, including its particular features and advantages, will be readily understood from the following detailed description and the accompanying drawings, in which:

[0087] Fig. 1 shows an example of a computer system for surgical assistance for use in accordance with a presently preferred embodiment of the present disclosure,Fig. 2 provides an exemplary embodiment of a marker unit to be arranged in physical relation to an internal body portion,

[0088] Figs. 3 A and 3B presents conceptual illustrations of the method steps according to the present disclosure, and

[0089] Fig. 4 is a flow chart illustrating the exemplary steps of the method according to the present disclosure, aligning with Figs. 3A and 3B.

[0090] DETAILED DESCRIPTION

[0091] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the present disclosure are shown. Such a present disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness, and fully convey the scope of the present disclosure to the skilled person. Like reference characters refer to like elements throughout. The following examples illustrate the present disclosure and are not intended to limit the same.

[0092] Turning now to the drawings, and to Fig. 1 in particular, there is conceptually illustrated a computer system 100 for surgical assistance, comprising a processing unit 110. The computer system 100 is specifically arranged to facilitate graphical visualizations, including augmented reality (AR) overlays, during surgical procedures, improving spatial perception, anatomical understanding and surgical precision through enhanced visualization.

[0093] For reference, the processing unit 110 may be implemented as a general-purpose processor, a graphics processing unit, an application-specific processor, a circuit containing processing components, a group of distributed processing components, a group of distributed computers configured for processing, or a field-programmable gate array (FPGA). The processing unit 110 may include any number of hardware components for conducting data, signal, and / or image processing or for executing computer code stored in memoiy 120. It may also be possible to utilize System-on-Chip (SoC) implementations.

[0094] The memoiy 120 may comprise one or more devices for storing data and / or computer code for completing or facilitating the various methods described in the present disclosure. The memoiy 120 may include volatile memoiy, non-volatile memoiy, database components, object code components, script components, or any other type of information structure supporting the various functionalities of the computer system 100. The memoiy 120 is communicably connected to the processing unit 110 (e.g., via a circuit or any other wired,wireless, or network connection) and includes computer code for executing one or more processes described herein.

[0095] The processing unit 110 is communicatively connected to a plurality of separate components, including an imaging modality 130, an image capturing device 140 and at least one display device 150. Such connections may be wired or wireless and are designed to allow seamless data transfer between the components.

[0096] The imaging modality 130 is used to acquire a three-dimensional model of an internal body portion of a patient. Examples of imaging modalities include but are not limited to cone beam computed tomography (CBCT), ultrasound (US), computed tomography (CT), magnetic resonance imaging (MR1), positron emission tomography (PET), single-photon emission computed tomography (SPECT), and X-ray imaging (XRAY). Other suitable imaging techniques capable of providing detailed internal anatomical views are possible and within the scope of the present disclosure.

[0097] The image capturing device 140, which may be implemented as a laparoscopic camera, endoscopic camera, bronchoscopic camera, or any other suitable imaging device used in surgical settings, is designed to capture real-time images of the surgical site. Such images are received by the processing unit 110, and are used to generate graphical visualizations, including augmented reality (AR) overlays, based on the three-dimensional model acquired by the imaging modality 130.

[0098] The display device 150 is communicatively connected to the processing unit 110 to present the captured images augmented with the graphical visualization. The display device 150 may be part of a surgical navigation system, standalone monitor, or a headmounted display, providing real-time, enhanced visual guidance to the operator.

[0099] Turning further to Fig. 2, there is depicted an exemplary embodiment of the marker unit 200, generally designed for integration with the surgical assistance computer system 100 as presented in relation to Fig. 1. The marker unit 200 comprises a flat body with two main surfaces, 202 and 204 opposite each other, each tailored for specific functional roles within augmented reality (AR)-aided surgery.

[0100] The surface 202 of the marker unit 200 is preferably provided with a set of optically detectable features 210. These markers are typically arranged in a non-rotationally symmetric geometrical pattern, enabling for a determination of the rotational position of the marker unit 200 when viewed by the image capturing device 140 during surgery. The optically detectable features 210 are preferably designed to be (preferably highly) visible under the operational lighting conditions typically found in surgical environments. Suchmarkers 210 may include a variety of shapes such as QR codes, geometric figures, or custom-designed icons that may be easily recognizable by the processing unit’s 110 imaging algorithms, generally known within the technical field. Furthermore, the optically detectable features 210 on surface 202 may include coatings with hydrophobic and anti -reflective properties, reducing occlusion from bodily fluids and minimizing light reflections that could interfere with detection.

[0101] Conversely, the opposite surface 204 is preferably designed to interface directly with the internal body portion of a patient during surgery. The surface 204 may be coated with a biocompatible adhesive that enables firm attachment to the organ of interest without necessitating invasive procedures. Such an adhesive ensures that the marker unit 200 remains attached throughout the duration of the surgical procedure, even considering mechanical bending stress in relation to minimally invasive surgery.

[0102] The marker unit 200 optionally, but preferably, includes radiopaque markers 220 embedded between surfaces 202 and 204, visible by the medical imaging modality 130. The radiopaque markers 220 may generally be composed of high-density materials such as gold, bismuth, barium sulfate, steel and alloys, or tungsten, which provide excellent visibility under X-ray and CT imaging. The placement of the radiopaque markers 220 is typically designed to align with the optically detectable features 210, creating a fixed, predetermined geometrical correlation between the two types of markers. Such a dual-marker setup enhances the system’s capability to merge AR visualizations with real-time surgical images, offering a more comprehensive view of the surgical site.

[0103] The (internal) body of the marker unit 200 itself may typically be made from durable, medical-grade polymers or composite materials that offer high rigidity and resistance to deformation under typical surgical manipulations. Such materials ensure that the marker unit maintains its integrity and dimensional stability, crucial for maintaining accurate geometrical alignment of the markers.

[0104] Additionally, the design of the marker unit 200 is preferably arranged to facilitate secure placement on the targeted internal body portion. For example, the flat body of the marker unit is preferably optimized for introduction through standard laparoscopic ports, with typical dimensions ranging from 5 to 20 mm in length and width, ensuring a slim profile to avoid obstructing surgical tools or views.

[0105] Furthermore, to additionally enhance surgical utility, the marker unit 200 may include features such as handles or tabs (not explicitly shown in Fig. 2) that facilitate manipulation using surgical instraments (again not explicitly presented). Such features maybe designed to protrude from the main body without compromising the sterility or functionality of the marker unit 200 and are typically manufactured from the same medicalgrade materials as the body of the marker unit to ensure uniformity in mechanical and sterilization properties, providing surgeons with enhanced control during placement and adjustment. Such features may also be of single-use characteristics, which are detached from the main body of the marker unit after application. Additionally, a separate tool may be provided for precisely positioning and orienting the marker unit within the patient. Such a tool may be reusable or designed as a disposable, single-use component that ensures secure placement.

[0106] Turning now to Figs 3A and 3B in conjunction with Fig. 4, generally presenting an embodiment of operating the computer system 100 in line with the scheme according to the present disclosure.

[0107] In Fig. 3 A, the processing unit 110 receives, SI image data, wherein the image capturing device 140 captures real-time images of a surgical scene, illustrating at least a first surgical tool 302 and a second surgical tool 304. The internal body portion 306 of the patient is also visible within the image data, wherein the marker unit 200 is affixed to serve as a reference for spatial alignment.

[0108] Here, the first surgical tool 302 may be a laparoscopic instrament, such as a grasping forceps or an or an electrocoagulation instrument, while the second surgical tool 304 may be a robot-assisted surgical tool or another manually controlled laparoscopic instrament. The system may in such an embodiment for example apply feature extraction techniques to identify, S2, distinguishing characteristics of each tool within the image data. For instance, rigid tools may be recognized using predefined geometric patterns, while robotic tools may include unique markers or shape-based contours. The processing unit 110 extracts these features to facilitate further spatial analysis.

[0109] Following identification, the spatial relationship between the detected features of surgical tools 302 and 304 and the optically detectable features 210 of the marker unit 200 is determined, S3. By utilizing known physical dimensions of the marker unit, the system establishes a coordinate transformation, ensuring accurate depth and positional alignment. The processing unit 110 subsequently forms, S4 a graphical visualization based on the computed spatial relationship and identified features.

[0110] Furthermore, Fig. 3A presents a side view visualization 330 displayed, S5, on a display device 150, illustrating the relative positioning of the first surgical tool 302 and second surgical tool 304 within a coordinate system. Such a separate visualization provides aclear, unobstructed view of the tools’ positioning, allowing the operator to anticipate tool movements with enhanced spatial awareness, which can be beneficial in laparoscopic or robotic-assisted surgeries, where depth perception is otherwise limited.

[0111] In addition to tracking surgical tools, the computer system 100 also utilizes preoperative imaging data to generate a three-dimensional (3D) model of the internal body portion 306, such as an organ targeted for surgical intervention. The 3D model may for example be derived from imaging modality data (e.g., CT, MRI, ultrasound, or CBCT scans) and reconstructed into a point cloud representation, aligning with the marker unit’s reference position. The 3D anatomical model may then be integrated into the side view visualization, allowing the operator to accurately position surgical tools relative to the actual anatomical structure.

[0112] Furthermore, if the surgical procedure involves the removal of a pathological region, such as a tumor 308, the 3D model may include segmentation data distinguishing the tumor from surrounding healthy tissue, such as e.g. blood vessels. In Fig. 3A, the tumor 308 is illustrated separately from the organ, using a distinct graphical marker within the side view visualization. By visually differentiating the tumor, the operator can accurately assess the spatial relationship between the tumor, the first surgical tools, and the marker unit, ensuring precise tool movements for tumor resection. In accordance with the present disclosure, the organ 308 is illustrated as a liver. Any internal body portion or organ may be possible and within the scope of the present disclosure. For example, any anatomical structure within a patient’s body, including soft tissue organs (e.g., kidney, pancreas), vascular structures, and skeletal components, depending on the surgical context.

[0113] In accordance with the present disclosure, the side view visualization may greatly enhance spatial awareness by providing depth cues and reference markers, ensuring that the surgical tools 302 and 304 are correctly aligned with the targeted area of the organ. The graphical representation may incorporate various visual indicators, including overlayed depth lines that represent the distance between each tool and the tumor, as well as highlighted cutting paths that outline an optimal trajectory for tumor removal. Additionally, color-coded proximity warnings may dynamically adjust as the surgical tools approach critical anatomical structures, providing real-time visual feedback to assist the operator in maintaining precise control throughout the procedure.

[0114] Turning now to Fig. 3B, illustrating a further embodiment of the operation of the computer system 100, wherein surgical tools, including a handheld ultrasonic scanning probe 310 and a flexible instrument 312, are used within the surgical procedure. The imagecapturing device 140 acquires real-time image data illustrating the handheld ultrasonic scanning probe 310 and the flexible instrument 312, both used in proximity to an internal body portion 306. Identification of these tools may as an alternative to the discussion provided in relation to Fig. 3A involve different recognition techniques compared to rigid tools. For example, the ultrasonic scanning probe 310 may be identified using an attached QR code or barcode, allowing the system to retrieve predefined tool-specific parameters.

[0115] Similarly, the flexible instrament 312, such as a catheter or fiber-optic tool, may be recognized using contrast-based tracking or shape-sensing technologies.

[0116] Again, the spatial relationship is determined by analyzing the identified tool features relative to the marker unit 200, where the system calculates the distance and orientation between the surgical tools and predefined anatomical structures. The graphical visualization is then generated, incorporating depth and proximity indicators based on realtime tool positioning.

[0117] Fig. 3B illustrates how augmented distance indicators and depth cues are integrated as AR data within the captured image data. For instance, a virtual distance line Ml is shown between the tip of the ultrasonic probe 310 and a specific reference point on the marker unit 200, providing real-time feedback regarding tool proximity. The flexible instrament 312 may have as similar virtual distance line M2, or possibly a color-coded depth indicator, dynamically adjusting as the tool moves closer to or farther from the internal body portion. The virtual distance lines Ml and M2 may in some embodiments be annotated with a distance between e.g. the tip of the surgical tool 310, 312 and e.g. a center point of the marker 200. In accordance with the present disclosure, the system may calculate the distance using known tool dimensions and spatial transformation algorithms, optionally incorporating real-time depth-sensing techniques such as stereo imaging or structured light analysis, depending on the imaging modality available.

[0118] Accordingly, the final visualization may in such an embodiment be displayed on the display device 150, where the augmented reality visualization is overlaid directly onto the image feed from the image capturing device 140. Such an implementation enables precise, real-time spatial guidance, reducing the need for separate depth estimation and improving overall surgical precision.

[0119] It should be noted that the system according to the present disclosure may be configured to display the visualizations of Figs. 3A and 3B simultaneously or allow the operator to toggle between them as needed during the procedure.Furthermore, the control functionality of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwire system. Embodiments within the scope of the present disclosure include program products comprising machine-readable medium for carrying or having machine-executable instractions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, solid state drives or other non-volatile flash based storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instractions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a machine, the machine properly views the connection as a machine-readable medium. Thus, any such connection is properly termed a machine-readable medium.

[0120] Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instractions and data which cause a general-purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

[0121] Although the figures may show a sequence, the order of the steps may differ from what is depicted. For example, two or more steps may be performed concurrently or with partial concurrence. Such variation will depend on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations could be accomplished with standard programming techniques with rale-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps and decision steps. Additionally, even though the present disclosure has been described with reference to specific exemplifying embodiments thereof, many different alterations, modifications and the like will become apparent for those skilled in the art.

[0122] In addition, variations to the disclosed embodiments can be understood and effected by the skilled addressee in practicing the claimed present disclosure, from a study of the drawings, the disclosure, and the appended claims. Furthermore, in the claims, the word’’comprising” does not exclude other elements or steps, and the indefinite article ”a” or ”an” does not exclude a plurality.

Claims

CLAIMS1. A computer-implemented method for surgical assistance using a computer system comprising a processing unit, wherein the method comprises the steps of:receiving, at the processing unit, image data illustrating a first surgical tool and an internal body portion of a patient, wherein the internal body portion is physically related to a marker unit, the marker unit comprising optically detectable features,identifying, using the processing unit and based on the image data, one or more geometrical features associated with the first surgical tool,determining, using the processing unit, a spatial relationship between the first surgical tool and the marker unit based on the identified geometrical features of the first surgical tool, the optically detectable features of the marker unit, and a predefined size of at least a portion of the marker unit,forming, using the processing unit, a graphical visualization representing the spatial relationship between the first surgical tool and the marker unit, the graphical visualization being based on the identified features of the first surgical tool and the determined spatial relationship, anddisplaying, at a display device arranged in communication with the computer system, the graphical visualization.

2. The method according to claim 1, wherein the graphical visualization is presented separately from the image data.

3. The method according to claim 1, further comprising the step of: augmenting, using the processing unit, the image data with the graphical visualization.

4. The method according to claim 1, wherein the graphical visualization further comprises a three-dimensional representation of the internal body portion.

5. The method according to claim 4, wherein the three-dimensional representation is based on data acquired from at least one imaging modality selected from the group consisting of cone beam computed tomography (CBCT), ultrasound (US), computedtomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single-photon emission computed tomography (SPECT), and X-ray imaging (XRAY).

6. The method according to any one of the preceding claims, wherein the marker unit further comprises radiopaque features, the radiopaque features having a predefined geometrical relationship with the optically detectable features.

7. The method according to any one of the preceding claims, wherein the graphical visualization further comprises a representation of the spatial relationship between the first surgical tool and the marker unit in at least one predefined plane or a coordinate system relative to the marker unit.

8. The method according to any one of the preceding claims, wherein the graphical visualization includes distance indicators visually representing the separation between the first surgical tool and predefined features of the internal body portion associated with the marker unit.

9. The method according to any one of the preceding claims, further comprising the step of:applying, using the processing unit, a predefined noise reduction algorithm to the image data to enhance the accuracy of the spatial relationship determination and the graphical visualization.

10. The method according to any one of the preceding claims, further comprising the step of:updating, using the processing unit, the graphical visualization in real-time based on changes in the position or orientation of the first surgical tool relative to the marker unit.

11. The method according to any one of the preceding claims, wherein the graphical visualization is formed to include a visual overlay of a surgical plan.

12. The method according to any one of the preceding claims, wherein the graphical visualization further comprises a depth-adjusted representation of the internal bodyportion, the depth being determined based on the spatial relationship between the first surgical tool, the marker unit, and the image data.

13. The method according to any one of the preceding claims, wherein the optically detectable features of the marker unit include machine-readable codes, the codes being used to facilitate identification of the marker unit within the image data.

14. The method according to any one of the preceding claims, further comprising the step of:providing, using the processing unit, a user interface configured to allow an operator to adjust the graphical visualization by modifying at least one parameter selected from the group consisting of scale, transparency, and orientation.

15. The method according to any one of the preceding claims, wherein the one or more features associated with the first surgical tool are identified using feature extraction techniques applied to the image data.

16. The method according to any one of the preceding claims, wherein the one or more features associated with the first surgical tool are identified based on a physical identifier arranged on the first surgical tool.

17. The method according to any one of the preceding claims, wherein the one or more features associated with the first surgical tool are identified based on a geometric pattern, shape, or color associated with the first surgical tool.

18. The method according to any one of the preceding claims, wherein the identification of the one or more features associated with the first surgical tool comprises:detecting, using the processing unit, a machine-readable code associated with the first surgical tool, the code being optically detectable within the image data, or extracting, using the processing unit, a contour or boundary of the first surgical tool from the image data.

19. A computer system for surgical assistance, the computer system comprising a processing unit, wherein the processing unit is adapted to:receive image data illustrating a first surgical tool and an internal body portion of a patient, wherein the internal body portion is physically related to a marker unit, the marker unit comprising optically detectable features,identify, based on the image data, one or more geometrical features associated with the first surgical tool,determine a spatial relationship between the first surgical tool and the marker unit based on the identified geometrical features of the first surgical tool, the optically detectable features of the marker unit, and a predefined size of at least a portion of the marker unit,form a graphical visualization representing the spatial relationship between the first surgical tool and the marker unit, the graphical visualization being based on the identified features of the first surgical tool and the determined spatial relationship, and display, at a display device arranged in communication with the computer system, the graphical visualization.

20. A computer program product comprising a non-transitory computer readable medium having stored thereon computer program means for operating a computer system designed for surgical assistance, the computer system comprising a processing unit, wherein the computer program product comprises:code for receiving, at the processing unit, image data illustrating a first surgical tool and an internal body portion of a patient, wherein the internal body portion is physically related to a marker unit, the marker unit comprising optically detectable features, code for identifying, using the processing unit and based on the image data, one or more geometrical features associated with the first surgical tool,code for determining, using the processing unit, a spatial relationship between the first surgical tool and the marker unit based on the identified geometrical features of the first surgical tool, the optically detectable features of the marker unit, and a predefined size of at least a portion of the marker unit,code for forming, using the processing unit, a graphical visualization representing the spatial relationship between the first surgical tool and the marker unit, the graphical visualization being based on the identified features of the first surgical tool and the determined spatial relationship, andcode for displaying, at a display device arranged in communication with the computer system, the graphical visualization.

Images