Method and system for an image guided medical instrument navigation system

Abstract

In some embodiments, the invention relates to a method and a system for an image guided navigation of a trackable instrument, wherein a computation unit that is configured for generating a visual representation of the instrument together with a visual representation of tomographic image data so as to visualize the instrument in the visual representation of tomographic image data on a display, determining a distance between the instrument distal end and the target region and / or the target object, determining if the determined distance is less than a predetermined threshold distance, and changing the visual representation of the instrument together with a visual representation of tomographic image data in response to determining that the determined distance is less than the predetermined threshold distance.

Description

[0001] Method and System for an image guided medical instrument navigation system

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] The present application claims priority to U.S. provisional patent application Serial No. 63 / 722,298 filed November 19, 2024, the entire contents of which is hereby incorporated by reference.

[0004] BACKGROUND

[0005] The invention relates to a method and a system for an image guided instrument navigation system.

[0006] SUMMARY

[0007] For an image guided instrument navigation, a trackable instrument is tracked by means of an instrument tracking system and the position and orientation of the tracked instrument is viewed on a display together with a representation of an anatomical structure of an object or a person. The representation of the anatomical structure is generated from tomographic image data of the object or person that is examined or treated by means of the tracked instrument.

[0008] The visualization of the tracked instrument together with the representation of the anatomical structure of an object or a person on the display may help a user to steer and manipulate the instrument as needed.

[0009] The instrument tracking system can be an electromagnetic (EM) tracking system that comprises at least one electromagnetic field generator that generates an electromagnetic field (EM field) in a tracking space. The electromagnetic field varies over time and position.

[0010] The trackable instrument may comprise electromagnetic field sensors (EM sensors) comprising at least one, or, in some embodiments, two sensor coils that can sense the electromagnetic field. Because the electromagnetic field is controlled to vary over time and position, electric signals captured by the sensor coils of an EM sensor can be evaluated to thus determine the position of the EM sensor - and thus the instrument - in the tracking space. The

[0011] *20240750538*

[0012] *20240750538* evaluation comprises measuring the voltage induced in each sensor coil. Thus, an EM sensor can provide an output signal (hereinafter also called sensor signal) that is a voltage signal that can be evaluated to find the position of the sensor inside the EM field, e.g., in the tracking space. An EM sensor with a single sensor coil can provide a sensor signal that allows determining of the position and the orientation of the EM sensor within the tracking space with respect to five degrees of freedom (5 DoF), e.g., the position in three dimensions and the orientation in two dimensions. For determining the position and the orientation of an EM sensor in all six degrees of freedom (the position in three dimensions and the orientation in three dimensions), two sensor coils are needed that are arranged at an angle with respect to one another. Typically, a 6 DoF EM sensor comprises two sensor coils each having a longitudinal axis wherein the two longitudinal axis are arranged perpendicular with respect to each other. A 6 DoF EM sensor can produce two sensor signals including, for example, one sensor signal per sensor coil.

[0013] The EM sensors comprising the sensor coils can be very small and thus can be attached to a variety of instruments, for instance surgical instruments. Thus surgical instruments can be tracked even inside a body without a I ine- of-sight between the sensors and the field generator.

[0014] The EM sensors comprising the sensor coils are connected to an evaluation unit that evaluates the electric signals captured by the sensor coils, for instance the voltage induced into the sensor coils by the electromagnetic field generated and emitted by the EM field generator. The EM field generator is also connected to and controlled by the evaluation unit.

[0015] In order to maintain a proper registration between the EM field and a patient for whom a tomographic 3D model may exist that can be used like a three- dimensional map of the patient, a patient localizer can be attached to a pa- tient. The patient localizer may comprise sensor coils and thus allows tracking of the position and orientation of the patient in the EM field. This may, for example, enable determining a relative position of a tracked instrument relative to the (tracked) patient and in turn allows real-time visualization of the instrument in a 3D model of the patient.

[0016] EM navigation is particularly useful if a 3D model of a patient exists and the position and orientation of a tracked instrument (i.e. an instrument with an EM sensor) can be shown in the 3D model during treatment. As known in the art, a 3D model of a patient can be obtained from pre-operative images for instance by means of tomography, for instance using magnetic resonance imaging (MRI). To allow proper alignment of the coordinate system of a 3D model and a real object, for instance a living patient, registration is needed. The EM navigation (i.e. the EM tracking system) is capable of determining the position and orientation of localizers in a coordinate system of the EM navigation system. Registration may allow the visualization or other reporting of the position and orientation of the localizers in real-time images of for instance a patient. Real time images may be obtained with different modalities, for instance an X-ray device (for instance a C-arm device), a computed tomography device (CT device, for instance a CT scanner) or an optical system such as an endoscope that can capture optical images. Likewise, the position and orientation of the localizers can be displayed in 3D images that are reconstructed from the preoperatively acquired 3D data. In order to track the position and orientation of instruments, localizers may be arranged on or in an instrument to be tracked. Therefore it is possible to display an instrument's position and orientation in real time and / or reconstructed images. In particular, renderings (i.e. reconstructed 3D images) of an instrument can be super imposed onto reconstructed and / or real-time images of a patient's anatomy. One way to achieve registration is photo registration. Photo registration prior to a surgery using electromagnetic navigation refers to the process of incorporating pre-operative images, typically in the form of computer tomography scans or magnetic resonance imaging scans, into a surgical navigation system that utilizes electromagnetic technology, for instance an EM tracking system.

[0017] The steps involved in the photo registration process typically are:

[0018] 1 . Acquisition of pre-operative images:

[0019] Before the surgery, a patient undergoes tomographic imaging using computer tomography or magnetic resonance imaging, which capture detailed images of the anatomy relevant to the planned surgery.

[0020] 2. Creation of a digital 3D model:

[0021] The acquired images are converted into a digital 3D model using specialized software. This model accurately represents the patient's anatomy and serves as a reference for the surgical navigation system.

[0022] 3. Registration:

[0023] The next step involves aligning the digital 3D model with the patient's actual anatomy. Electromagnetic sensors placed on the patient's body serve as reference points for the tracking system to precisely detect and track the patient's position and orientation. The electromagnetic sensors placed on the patient's body are typically part of a patient localizer that is attached to the patient.

[0024] 4. EM tracking system calibration: The electromagnetic tracking system needs to be calibrated to establish a relationship between the position of the sensors and the corresponding location on the digital 3D model. This calibration allows the system to accurately overlay the digital model onto the patient's anatomy.

[0025] 5. Overlaying the digital 3D model:

[0026] After calibration of the EM tracking system, the digital 3D model is superimposed onto the patient's real-time anatomy. This allows the surgeon to visualize and track instruments for navigating tracked instruments through the patient's body during the surgery using the guidance provided by the registered images.

[0027] The step of EM tracking system calibration is a crucial step in the photo registration process that ensures accurate tracking and alignment of the EM tracking system with the patient's anatomy. The calibration step involves:

[0028] Taking a stereoscopic or 3D image of the patient and the patient localizer with a stereoscopic or a 3D camera. The 3D camera may be a time-of-flight (ToF) camera.

[0029] The image taken by the camera is processed to generate a point cloud. The point cloud is a dense collection of 3D data points that represent the patient's outer appearance in a 3D image together with the patient localizer.

[0030] The point cloud generated from the 3D images taken by the camera is then aligned with the patient's anatomy as represented in the pre-op- erative, digital 3D model. Once the point cloud generated from the 3D images taken by the camera and the patient's anatomy as represented in the pre-operative 3D model are aligned, the relationship between the positions and orientations of the anatomic structures as represented in the 3D model and the EM tracking system's coordinate system can be established.

[0031] The calibration process may involve additional steps to optimize the accuracy of the alignment. This may include fine-tuning the registration based on known anatomical landmarks or performing iterative adjustments to improve the system's accuracy. Once the calibration is completed, the accuracy and reliability of the EM tracking system are verified and validated. This may involve conducting calibration tests and quality control checks to ensure the system's performance meets the intended accuracy requirements.

[0032] Methods and systems for an image guided instrument navigation are inter alia known from US 10,568,713 B2, US 2010 / 0249506 A1 and US 9,641 ,808 B2, the entire disclosures of which are incorporated by reference herein.

[0033] It is an object of the invention to improve the usability of navigated instruments.

[0034] According to a first aspect of the invention, a method for an image guided instrument navigation is provided that comprises the steps of providing a trackable instrument with an instrument distal end providing tomographic image data providing target region and / or target object information that characterizes a target region and / or a target object in relation to the tomographic image data providing a display for displaying visual representations manipulating the instrument for changing at least one of the position, the orientation and the operational state of the instrument tracking the instrument by capturing signals representing the position and orientation of the instrument and generating positional information from the captured signals generating a visual representation of the instrument together with a visual representation of tomographic image data so as to visualize the instrument in the visual representation of tomographic image data on a display determining a distance between the instrument distal end and the target region and / or the target object determining if the determined distance is less than a predetermined threshold distance changing the visual representation of the instrument together with a visual representation of tomographic image data in response to determining that the determined distance is less than the predetermined threshold distance, wherein the change of the representation of the instrument together with a visual representation of tomographic image data comprises at least one of zooming on the target region of interest in the visual representation of tomographic image data, changing from orthogonal visual representations of tomographic image data to an in-line guidance representation of tomographic image data, changing from orthogonal visual representations of tomographic image data to apply an overlay of generated depth visualization on to the orthogonal views of the tomographic image data, creating a new visual representation of tomographic image data based on the instrument orientation or anatomy (e.g., generating a new 45 degree slice view or cut view for the Eustachian tube (ET), or other target anatomy like the frontal sinus), enhancing the contrast and brightness of the visual representation of the instrument or the visual representation of tomographic image data or the visual representation of the instrument together with the visual representation of tomographic image data, changing the color of the visual representation of tomographic image data, cropping the visual representation of tomographic image data to just show the target region, displaying a guidance target visual representation of tomographic image data wherein guidance target visual representation of tomographic image data comprises a distance indicator and an orientation indicator indicating the distance of the instrument distal end from target region and / or target object and the instrument orientation relative to the target region and / or target object, displaying of a preview of the image data in the projected orientation of the instrument (“look-ahead-view”) by way of stacked or overlaid image slices and / or creating a 3D preview from the 2D image slices stacked together to show to a user in which direction the instrument should be moved. changing the visualization of the instrument from 2D to 3D (e.g., from a side elevated view to a perspective view) or vice versa and / or changing an instrument representation color which target region is reached changing the visual representation of tomographic image data from 2D to 3D rendering

[0035] In some embodiments, the method further comprises: providing and manipulating an optical instrument having an image sensor, said optical instrument being configured for capturing optical images being projected on the image sensor, capturing optical images by means of the optical instrument, displaying the optical images on the display, and displaying navigation information within the displayed optical images.

[0036] According to a second aspect, a system for an image guided navigation is provided that comprises a trackable instrument having an instrument distal end, a data storage comprising tomographic image data, a data storage comprising target region and / or target object information that characterizes a target region and / or a target object in relation to the tomographic image data, a display for displaying visual representations, an instrument tracking unit that is configured for tracking the instrument by capturing signals representing the position and orientation of the instrument and generating positional information from the captured signals a computation unit that is configured for generating a visual representation of the instrument together with a visual representation of tomographic image data so as to visualize the instrument in the visual representation of tomographic image data on a display determining a distance between the instrument distal end and the target region and / or the target object determining if the determined distance is less than a predetermined threshold distance and changing the visual representation of the instrument together with a visual representation of tomographic image data in response to determining that the determined distance is less than the predetermined threshold distance, wherein the change of the representation of the instrument together with a visual representation of tomographic image data comprises at least one of zooming on the target region of interest in the visual representation of tomographic image data, changing from orthogonal visual representations of tomographic image data to an in-line guidance representation of tomographic image data, creating a new visual representation of tomographic image data based on the instrument orientation or anatomy (i.e. generating a new 45 degree slice view for the Eustachian Tube (ET)), enhancing the contrast and brightness of the visual representation of the instrument and / or the visual representation of tomographic image data, changing the color of the visual representation of tomographic image data, cropping the visual representation of tomographic image data to just show the target region, displaying a guidance target visual representation of tomographic image data wherein guidance target visual representation of tomographic image data comprises a distance indicator and an orientation indicator indicating the distance of the instrument distal end from the target region and / or target object and the instrument orientation relative to the target region and / or target object, changing the visualization of the instrument for 2D to 3D or vice versa, and / or changing an instrument representation color if the target region is reached

[0037] According to some embodiments, the computation unit is configured for generating and causing displaying of guidance views as follows: generating a new viewing plane view along the instrument axis generated from two sensor coils, displaying of the instrument shape in 3D or 2D showing the shape for the instrument in a bent or straight configuration, specifically for malleable instruments being able to show the shape of the instrument on the CT scan, displaying of the balloon dilation device shape in 3D or 2D showing the shape of the inflated and / or deflated balloon in a bent or straight configuration generating a perspective offset view from the tip of the instrument based on the instrument axis as defined by the sensor coil configuration, displaying a combination of instrument plane views and an in-line view from the perspective of the instrument tip and axis, displaying of a preview of the image data in the projected orientation of the instrument (“look-ahead-view”) by way of stacked or overlaid image slices and / or creating a 3D preview from the 2D image slices stacked together to show which direction to go. overlaying of the 3D instrument representation in a live video view instead of just on reconstructed images that are reconstructed from the preoperatively acquired 3D image dataset, determining the shape / bend angle of an instrument that is optimal for accessing a target anatomical position and instructing the user in realtime how to bend the instrument until it reaches that angle, e.g., a bending view appears as the user is bending the malleable region of the balloon until it reaches the correct bend, and / or instructing the user how to position the instrument in order to reach the target region, (e.g., go up and to the left)

[0038] The trackable instrument is, in some embodiments, a balloon dilation device. In some embodiments of the method and of the system, the trackable instrument comprises a longitudinal shaft with a malleable section and at least two five-degrees-of-freedom (5 DoF) sensor coils arranged on or in the shaft in alignment with the shaft and spaced from each other along the longitudinal direction of the shaft. In electromagnetic navigation systems, 5 DoF sensor coils are single coils with a helically wound wire and a longitudinal axis defined by the helix. With a 5 DoF sensor coil, a position and orientation of the sensor coil can be determined in 5 degrees-of-freedom, i.e. 3 degrees- of-freedom with respect to the position and 2 degrees-of-freedom with respect to the orientation because a 5 DoF sensor coil is symmetric about its longitudinal axis. With 5 DoF sensor coils that are arranged in an angled configuration, determining position and orientation in all 6 degrees-of-free- dom is possible.

[0039] If a trackable instrument has a longitudinal shaft with a malleable section that allows to bend the shaft, a bending angle between shaft sections adjacent to the malleable shaft section can be determined by means of two 5 DoF sensor coils that are arranged in straight shaft sections extending from the malleable shaft section in two directions.

[0040] According to one non-limiting embodiment, a plane defined by at least two 5 DoF sensor coils is determined and a new viewing plane view along the plane defined by the at least two 5 DoF sensor coils is generated. This nonlimiting embodiment is a further aspect of the invention that can be implemented independently from changing the visible representation of the instrument together with a visible representation of tomographic image data in response to determining that the determined distance is less than the predetermined threshold distance.

[0041] According to a further embodiment that also can be implemented independently from the other aspects of the invention, the instrument shaft is malleable in a section between the at least two sensor coils. The method comprises the steps of determining an instrument shape by processing signals provided by the at least two 5 DoF sensor coils and displaying a representation of the instrument shape in a two dimensional or a three dimensional view showing the shape of the instrument in a bent or straight configuration. A three dimensional view may be a perspective view.

[0042] According to this aspect of the invention, a user can see a representation of the instrument in its bent or straight configuration. In addition to showing the instrument, a target shape of the instrument may be displayed wherein the instrument is shown with a bending angle. A user may then bend the malleable instrument shaft until the actual bending angle corresponds to the target bending angle. The latter aspect may be implemented by purely providing an instrument with a longitudinal shaft and at least two 5 DoF sensor coils arranged in or on the shaft wherein the shaft comprises a malleable shaft section that extends between the two sensor coils. In some embodiments, the system and the method further comprises providing a target bending angle for the instrument. In some embodiments, the system and the method further comprises displaying a representation of the instrument in its actual bent or straight configuration and further displaying a representation of the instrument wherein a bent distal end of the instrument extends in a direction that corresponds to the target bending angle. A user may then bend the malleable instrument until the representation of the actual distal instrument section coincides with the representation of the distal shaft section in its target orientation.

[0043] Such method and such system may assist a user in shaping the malleable instrument by bending.

[0044] According to a further aspect of the method and the system, the method comprises the step of obtaining and displaying a CT scan (for instance obtained by means of a computed tomography scanner or by means of an x- ray device) and the step of displaying a representation of the instrument shape shown as an overlay on the CT scan. Accordingly, the computation unit of the system is, in some embodiments, configured to perform the one or more of the method steps.

[0045] In yet another embodiment, the trackable instrument is a balloon dilation device having a longitudinal, malleable shaft. According to this embodiment, the method comprises displaying a graphic representation of the balloon dilation shape in a two dimensional or a three dimensional view showing the shape of the inflated and / or deflated balloon in a bent or straight configuration of the instrument shaft. A three dimensional view may be a perspective view.

[0046] In yet another embodiment, the method comprises generating a perspective offset view from the tip of the instrument based on the instrument axis as determined from the sensor coil configuration. The sensor coil configuration is the relative special arrangement of the two sensor coils with respect to each other. The sensor coil configuration can be determined by processing signals provided by the at least two sensor coils.

[0047] In some embodiments, the method comprises displaying a combination of instrument plane views and an in-line view from the perspective of the instrument tip and axis. According to yet another embodiment, the method comprises capturing a live video stream by means of an optical instrument and overlaying of a three dimensional representation of the instrument in a live video view instead of just on reconstructed images that are reconstructed from the preoperatively acquired 3D image data set. A three dimensional representation of the instrument may be a perspective representation of the instrument. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In the following, some embodiments of the invention are described with respect to the figures. In the figures

[0049] Fig. 1 : illustrates some basic components of an electromagnetic instrument system;

[0050] Fig. 2a - c:is schematic illustration of a balloon dilation device having two sensor coils and an inflatable and deflatable balloon in different states;

[0051] Fig. 3: illustrates a machine learning algorithm model;

[0052] Fig. 4: illustrates a first representation of the instrument together with a visual representation of tomographic image data;

[0053] Fig. 5: illustrates a second representation of the instrument together with a visual representation of tomographic image data;

[0054] Fig. 6: illustrates a third representation of the instrument together with a visual representation of tomographic image data;

[0055] Fig. 7a, b: illustrate changing the visualization from an orthogonal visual representation to an in-line guidance representation of tomographic image data when reaching the target region;

[0056] Figure 7C is an expanded view of Figure 7A and shows traditional orthogonal views with an identified target region and tracked instrument approaching the target region. Figure 7D is an expanded view of Figure 7B showing the change of view from Fig. 7 to a slice view and inline views when the tracked instrument got near to the target region,

[0057] Fig. 8: shows an image data visualization comprising a slice view of the target region.

[0058] Figs. 9a, b : illustrate displaying a guidance target visual representation of tomographic image data wherein guidance target visual representation of tomographic image data comprises a distal indicator and an orientation indicator indicating the distance of the instrument distal end from the target region or the target object and the instrument orientation relative to the target region or the target object. These figures illustrates the guidance aiming view showing the distance the instrument is from the target (26.9mm in this figures) and showing when the instrument is aimed inline with the target. Figure 9a depicts where the instrument not aimed at target. Figure 9b depicts where instrument aimed at the target and on the right trajectory.

[0059] Fig. 10: shows a 3D rendering of an image data set representing a region of interest and an overlay of the 3D rendering over the visualization of the live endoscopic view image data;

[0060] Figs. 11a, b: illustrate a visualization of a 3D preview from 2D image slices being stacked together to show a direction where the instrument 12 should be moved to. The figures depict a 2D view with overlaid 3D depth information Fig. 12a, b: illustrate displaying an endoscopic live view together with a CT navigation view in an immediate neighborhood to the displayed endoscopic live view. These figures provide a virtual endoscopic view.

[0061] Fig. 13a; b; c illustrate displaying a representation of a bent malleable instrument for showing to the user an orthogonal view of the bending angle. The figures depict 3D instrument representation showing the bending of the instrument. Figure 13A depicts the instrument bend. Figure 13B depicts the visualization on the display.

[0062] Fig. 14. illustrates a scanning view along an axis of the instrument generator of a dual coil instrument;

[0063] Fig. 15. illustrates an automatic determination of a suitable bending angle for a malleable medical instrument tip. In this figure, the ballon bending is overlaid with the target anatomy to show the correct bending angle needed for accessing the target.

[0064] Fig. 16. a representation of the medical instrument in an extended instrument bending view on a display. In this figure, real-time bending view with different target bends is being shown.

[0065] Fig. 17. illustrates a malleable medical instrument with a malleable tip section along with a representation of the medical instrument in an extended instrument bending view on a display;

[0066] Fig. 18. .is an enlarged illustration of a malleable medical instrument with a malleable tip section along with the representation of the medical instrument in an extended instrument bending view on a display- DETAILED DESCRIPTION

[0067] An image guided instrument navigation system 10 as illustrated in figure 1 comprises a trackable instrument 12 having an instrument distal end 12.1 , a data storage 14 comprising tomographic image data and target region and / or target object information that characterizes a target region and / or a target object in relation to the tomographic image data a display 16 for displaying visual representations an instrument tracking arrangement 18 that is configured for tracking an instrument by capturing signals representing the position and orientation of an instrument and generating positional information from the captured signals and a computation unit 20 that is configured for generating a visual representation of the instrument together with a visual representation of tomographic image data so as to visualize the instrument in the visual representation of tomographic image data on a display determining a distance between the instrument distal end and the target region and / or the target object determining if the determined distance is less than a predetermined threshold distance and changing the visual representation of the instrument together with a visual representation of tomographic image data in response to determining that the determined distance is less than the predetermined threshold distance.

[0068] The instrument tracking arrangement 18 comprises an electromagnetic field generator 18.1 for generating an alternating electromagnetic field. The instrument tracking arrangement 18 further comprises a localizer 18.2 comprising an electromagnetic sensor 18.3. The electromagnetic sensor 18.3 of the localizer 18.2 comprises two sensor coils 18.3.1 and 18.3.2.

[0069] The instrument tracking arrangement 18 further comprises an instrument tracking unit 18.4 that is configured for processing signals provided by sensor coils that in use are arranged within an alternating electromagnet filed generated by the field generator 18.1. Induction causes the sensor coils to provide an output signal that depends on the position and orientation of a respective sensor coil within the electromagnetic field provided by the field generator 18.1 .

[0070] As known in the art, the electromagnetic signals provided by the two sensor coils 18.3.1 and 18.3.2 may be processed so as to determine the relative position of the sensor coils 18.3.1 and 18.3.2 - and thus the localizer 18.2 - with respect to the electromagnetic field generator 18.1 .

[0071] As is shown in figures 2a to 2c, the trackable instrument 12 comprises a handle 12.1 and a shaft 12.2 with a distal end 12.3. In the handle, an instrument electromagnetic sensor 12.4 is arranged. The instrument electromagnetic sensor 12.4 comprises at least one sensor coil 12.4.1 and, in some embodiments, comprises a second sensor coil 12.4.2. At or near the distal end 12.3 of the shaft 12.2, an inflatable and deflatable balloon 12.5 is arranged. The balloon 12.5 can be expanded by inflating the balloon with a fluid.

[0072] At or near the distal end 12.3 of the shaft 12.2 a tip sensor coil 12.6 is arranged that, in some embodiments, is a single coil that can act as a 5 de- grees-of-freedom (DoF) position and orientation sensor in the instrument tracking arrangement 18. In some embodiments, a shaft sensor coil 12.7 is arranged on or in the shaft 12.2 at or near a proximal end of the balloon 12.5. The signals provided by sensor coils 12.6 and 12.7 thus represent the position and orientation of the instrument 12. The electromagnetic signals provided by the two sensor coils 12.6 and 12.7 are processes for generating positional information, i.e. data that represent the relative position of the sensor coils 12.6 and 12.7 - and thus the instrument 12 - with respect to the electromagnetic field generator 18.1.

[0073] As indicated in figure 2c, the distal end section 12.2.1 of the shaft is made malleable. Accordingly, the distal end section 12.2.1 of the shaft 12.2 can be shaped by bending.

[0074] When in use, i.e. during operation, the electromagnetic field generator 18.1 emits at least one alternating electromagnetic field that can be sensed by the sensor coils 18.3.1 and 18.3.2 and that allows to determine the relative position of the sensor coils 18.3.1 and 18.3.2 with respect to the field generator 18.1.

[0075] The electromagnetic field emitted by electromagnetic field generator 18.1 is also sensed by the instrument electromagnetic sensor coils 12.4, the tip sensor coil 12.6 and the shaft sensor coil 12.7 and thus allows to determine the relative position of the trackable instrument 12 with respect to the field generator 18.1 . Processing of the signals picked-up (i.e. induced into) by the sensor coils 12.4, 12.6, 12.7, 18.31 and 18.3.2 is performed by the instrument tracking unit 18.4. As known in the art, the instrument tracking unit 18.4 is configured for determining the position and the orientation of the sensor coils - and thus of the instrument 12 - in a coordinate system of the instrument tracking system 18.

[0076] The tip sensor coil 12.6 and the shaft sensor coil 12.7 further allow displaying the trackable instrument 12 with the actual bend of the malleable shaft section 12.2.1 .

[0077] The computation unit 20 is configured for receiving position and orientation signals from the instrument tracking unit 18.4. The position and orientation signals represent the position and the orientation of - for instance - the instrument 12 in a coordinate system of the instrument tracking arrangement 18.

[0078] The computation unit 20 further has access to the data storage 14 that comprises tomographic image data, i. e. data that represents a tomographic image of, for instance, an object or a person. Via registration - as described above - the computation unit 20 is configured for determining the orientation and position of the sensor coils - thus for instance, of the instrument 12 - in the coordinate system of the tomographic image data stored in the data storage 14.

[0079] The computation unit 20 is further configured to generate visual representations of the tomographic image data and of the instrument 12. So as to visualize the instrument 12 in the visual representation of tomographic image data on the display 16.

[0080] The data storage 14 further comprises target region information or target object information or both, including, without limitation, information that characterizes a target region or a target object or both in relation to the tomographic image data. Accordingly, in the data storage 14 information is stored that characterizes the position and the extension of a target object and / or a target region. The computation unit is further configured for determining the position or orientation of the instrument 12 relative to the position (and extension) of a target region and / or a target object. Therefore, the computation unit can determine a distance between the instrument distal end 12.3 and the target region and / or target object. The computation unit 20 is configured for determining the distance between the instrument distal end and the target region and / or the target object.

[0081] For assisting a user in manipulating the trackable instrument 12 so as to best navigate the instrument distal end 12.3 towards the target region or the target object, the computation unit 20 is further configured for determining if the determined distance between the instrument distal end 12.3 and the target object or the target region is less than a predetermined threshold distance. If so, the computation unit 20 is configured for changing the visual representation of the instrument together with the visual representation of tomographic image data. Thus, the user is provided with an intuitive visual indication that the instrument distal end 12.3 (or any other part of the instrument 12) is approaching the target object or the target region and already is close enough to the target object or the target region.

[0082] Once the instrument distal end 12.3 is close enough to the target object, navigation of the instrument 12 typically must be more precise in order to achieve a desired effect and in order to avoid accidental damage. Therefore, the computation unit 20 is configured for changing the representation of the instrument together with the visual representation of the target object in way that facilitates the perception of the instrument 12 in relation to the target object as represented on the display 16. One way of changing the representation of the instrument together with the visual representation of the tomographic image data the computational unit 20 is configured for is zooming on the target region of interest and the visual representation of the tomographic image data. In other words, the tomographic image data is represented on the display 16 in a different scale, e.g., an enlarged view that allows a more detailed perception of the instrument position and orientation with respect to the target object or the target region.

[0083] Alternatively or additionally the computational unit 20 may be configured for changing from an orthogonal visual representation of tomographic image data to an in-line guidance representation of tomographic image data on the display 16 once the computational unit 20 has determined that the determined distance between the instrument 12 and the target object is less than the predetermined threshold distance.

[0084] Changing from an orthogonal visual representation to an in-line guidance representation of tomographic image data is illustrated in figures 7a and 7b. In yet another alternative embodiment, the computation unit 20 may alternatively or additionally be configured for creating a new visual representation of tomographic image data based on the instrument orientation or on the target region anatomy. For instance, the computation unit 20 may be configured for generating a slice view of the target region as illustrated in figure 8. The latter is particularly suitable in case the instrument 12 is used for the Eustachian Tube (ET). An example is shown in figures 7b and 8. Figure 7a illustrates a traditional orthogonal views with an identified target region and tracked instrument approaching the target region. Figure 7b illustrates a change of views to a slice view and inline views when the tracked instrument got near to the target region.

[0085] Alternatively or additionally, the computational unit 20 may be configured for enhancing the contrast and the brightness of the visual representation of the instrument or of the tomographic image data or of both, the instrument and the tomographic image data. Enhanced contrast and brightness facilitates the perception of a user with respect to the position and orientation of the instrument relative to the target object.

[0086] According to yet another embodiment, the computational unit 20 may be configured for changing the color of the visual representation of the tomographic image data.

[0087] Further, the computational unit 20 may alternatively or additionally be configured for cropping the visual representation of tomographic image data to just show the target region. Cropping the visual representation means that an image that only shows that target region but no further tomographic structures is displayed on the display 16. A cropped view is similar to a zoomed in view. However, cropping does not necessarily imply a change of scale, for instance an enlargement of the displayed structures.

[0088] In some embodiments, if the computational unit 20 is configured for displaying a guidance target visual representation of tomographic image data wherein guidance target visual representation of tomographic image data comprises a distal indicator and an orientation indicator indicating the distance of the instrument distal end 12.3 from the target region or the target object and the instrument orientation relative to the target region or the target object. An example is given in figures 9a and 9b. These figures illustrate a non-limiting guidance aiming view showing the distance the instrument has from the target (e.g., 26.9mm) and showing when the instrument is aimed inline with the target. Figure 9a illustrates a generated view when the instrument not aimed at target, and figure 9b illustrates a generated view when the instrument aimed at the target and on the right trajectory.

[0089] Alternatively or additionally, the computational unit 20 may be configured for changing the visualization of the instrument from a two-dimensional (2D) to a three-dimensional (3D) representation or vice versa. A two-dimensional visualization is for instance a cross-sectional visualization while the three- dimensional representation is a perspective representation of the structures that are represented by the tomographic image data and by image data representing an instrument. Image data representing an instrument allow generating a visualization of the instrument 12 on the display 16 in all the use cases and embodiments disclosed above.

[0090] In order for indicating to a user that the instrument has arrived at the target position and orientation, the color of a visualization of the instrument 12 is changed. If the instrument 12 is an instrument with an end effector like a balloon 12.8, a user can now manipulate or activate the end effector, for instance inflating the balloon 12.8.

[0091] Figure 4 illustrates a first representation of the instrument 12 together with a visual representation of tomographic image data. The visual representation of the tomographic image data are showing a Eustachian tube 50. The instrument 12 is bent near the distal end (e.g., the instrument tip 12.3) to facilitate the access to the target region. The bending angle may be adapted to the specific anatomy of the body cavities used for accessing the target region. The bending angle can be determined, for example, manually or automatically as described further below.

[0092] Figure 5 illustrates a another non-limiting representation of the instrument 12 together with a visual representation of tomographic image data representing the Eustachian tube 50 wherein the visualization of the instrument 12 and of the tomographic image data stored in the data storage 14 comprises: a plane view of the tomographic image data created between the two sensor coils 12.6' and 12.7' and a representation of the instrument shape and location. The visualization is generated by the computational unit 20.

[0093] Figure 6 illustrates a third representation of the instrument together with a visual representation of tomographic image data wherein the visualization of the instrument 12 and of the tomographic image data stored in the data storage 14 comprises a plane view created between the two sensor coils and display of the instrument shape and location and wherein the visualization is changed to show target point (marked by a target cross in green) and boundary lines / areas as marked in red. The change of visualization is triggered if the computational unit 20 determines that the distance between the distal end 12.3 of the instrument 12 and the target region is less than the predetermined threshold distance. Accordingly, the visual representation of the instrument together with a visual representation of tomographic image data is changed by the computational unit 20 in response to determining that the determined distance is less than the predetermined threshold distance.

[0094] According to a non-limiting embodiment, the computational unit 20 may be configured for automatic segmentation of anatomic structures within an image data set. For automatic segmentation, the computational unit 20 may comprise a segmenting neural network 20.1 that is configured for segmenting 3D image data; compare to figure 3. The 3D image data stored in data storage 14 may comprise density information about the density of the structures represented by the 3D image data. The computational unit 20 may be configured for using this density information for segmenting anatomic structures. For instance, the trained segmenting neural network 20.1 may have been trained with training data representing density information. This training data can be generated from computerized tomography images and / or X- ray images. The computational unit 20 may be further configured for highlighting or color coding anatomical structures after segmentation. Thus, it is possible, to indicate to a user in an easy to perceive manner the structures displayed on the display 16. For highlighting and / or color coding of anatomical structures, the segmenting neural network 20.1 may be configured as a semantic segmenting neural network that is trained with data representing anatomic structures and that is configured to assign voxels (volume elements) of the 3D image data to one of the objects the semantic segmenting neural network is trained for.

[0095] One of the objects, the semantic segmenting neural network 20.1 may be trained for are air passage ways. This enables the computational unit 20 to automatically identify air passage ways in the image data set.

[0096] Likewise, the semantic segmenting neural network 20.1 may be trained with image data representing sinus pathways and sinus ostia for enabling the computation unit 20 for automatic identification of sinus pathways and sinus ostia.

[0097] Similarly, the semantic segmenting neural network may be trained with image data representing an Eustachian tube. Thus, the semantic segmenting neural network 20.1 is configured to determine which voxel of the 3D image data are representing an Eustachian tube. Thus, the computation unit 20 is configured for automatic identification of an Eustachian tube.

[0098] The semantic segmenting neural network 20.1 , in some embodiments, has an encoder-decoder structure with a convolutional input layer for receiving 3D image data to be processed.

[0099] In yet another embodiment, the image guided navigation system 10 may comprise an endoscope 40 that comprises an image sensor and imaging objects for projecting an image on the image sensor. Thus, the endoscope can provide optical image data that represent a live endoscopic view. According to one non-limiting embodiment, image data representing the live endoscopic view are processed by the computation unit 20 in order to generate a visual representation of the live endoscopic view image data on the display 16. The computation unit 20 is further configured for generating a visualization of navigation information and to overlay the visualization of the navigation information over the visualization of the live endoscopic view of the display 16.

[0100] In yet a further embodiment, the computation unit 20 may be configured for processing the image data of the live endoscopic view in order to automatically identify the instrument in the live endoscopic view. For this purpose, the computation unit 20 may comprise a further semantic segmenting neural network 20.2 that is trained with 2D image data representing images of instruments; cf. figure 3. The second semantic segmenting neural network 20.2 may be configured for processing image data representing a live endoscopic view and for assigning pixels of the images as represented by the image data to an object class that corresponds to an instrument in case the image representing the live endoscopic view comprises such pixels that score high enough with respect to instruments representing image data the second semantic segmenting neural network 20.2 was trained with. Also the second semantic segmenting neural network 20.2, in some embodiments, has an encoder-decoder structure and convolutional input layer for receiving image data representing a two-dimensional live endoscopic view as recorded by the image sensor 40.1 of the endoscope 40.

[0101] Similarly to the previously described embodiments, the computation unit 20 is, in some embodiments, configured for a 3D visualization of an instrument shape in position in the visualized live endoscopic view image data. Further, the computation unit 20 is, in some embodiments, configured for 3D rendering of an image data set representing a region of interest and to overlay the 3D rendering over the visualization of the live endoscopic view image data. An example is shown in figures 10a, 20b and 10c. These non-limiting, re- pressentative figures illustrate a 3D rendering of CT scan representing a region of interest and an overlay of the 3D rendering over the visualization of the live endoscopic view image data.

[0102] In case the computation unit 20 is configured for automatic identification of air passage ways of sinus pathways or an Eustachian tube in the image data representing tomographic images, the computation unit 20 may be further configured to identify these pathways and to generate guidance information for accessing target anatomical structures via an identified pathway. Target anatomical structures may be an Eustachian tube or sinus ostia.

[0103] The computation unit 20 may be further configured for generating a visualization that displays an instrument position in the image data and also shows a preview of image data in a projected orientation of the instrument. The latter image data corresponds to virtually reconstructed look-ahead-view of the instrument that may be generated from the 3D tomographic image data stored in the storage unit 14. Figures 12a and 12b illustrate a virtual endoscopic view generated form the tomographic image data stored in the storage unit 14.

[0104] The computation unit 20 may be further configured to generate a visualization of stacked or overlaid image slices as shown in figures 11 a and 11 b and / or to generate a visualization of a 3D preview from 2D image slices being stacked together to show a direction where the instrument 12 should be moved to. This facilitates manipulation of the instrument 12 by a user. Figures 11 a and 11 b illustrate a two-dimensional (2D) view with overlaid three-dimensional (3D) depth information. The three-dimensional (3D) depth information is provided by gray-shaded, perspective renderings of preoperatively acquired tomographic image data stored in data storage 14. Figure 14 illustrates a scanning view created along a axis of the instrument generated from dual coil instrument.

[0105] A non-limiting method comprises:

[0106] Providing an image guided instrument navigation system (10) that comprises computation unit (20) for loading and displaying an image dataset (i.e. CT or MRI scan),

[0107] A manual or automatic identification of an anatomical region of interest in the image dataset,

[0108] A registration of the image data to the patient in the coordinate system of the position tracking system,

[0109] A position tracking of a surgical instrument,

[0110] Displaying of the instrument position in the image dataset,

[0111] Automatic recognition of when the instrument is close to the defined region on interest on the image dataset, and

[0112] Changing in the display / viewing of the image data when the instrument is close to the defined region of interest or anatomical structure.

[0113] The view-changing options once triggered by reaching the region of interest include:

[0114] Auto zooming on the region of interest in the image data when the instrument gets close, Changing of the display views (i.e. change from orthogonal views to in-line guidance views),

[0115] Creating a new slice view based on the instrument orientation or anatomy (i.e. generating a new 45 degree slice view for ET),

[0116] Enhancing the contrast and brightness of the image,

[0117] Changing of the display views from 2D to 3D or vice versa,

[0118] Changing the color of the segmented region or defined region of interest,

[0119] Cropping the image to just show the target region of critical structures,

[0120] Show a guidance target view with distance and orientation from target,

[0121] Change in instrument representation for 2D to 3D or vise versa,

[0122] Change in the instrument representation for visualization of the balloon inflated and / or deflated,

[0123] Change instrument registration color which target region is reached, and

[0124] Generating automatic feedback or alert from the instrument navigation system.

[0125] The method may further comprise at least one of:

[0126] Manual or automatic segmentation of anatomic structures within the image set, Highlighting and / or color coding of anatomical structures, automatic identification of air passageways in the image set, automatic identification of sinus pathways and sinus ostia, automatic identification of Eustachian tube, overlay of navigation information into the live endoscopic view, automatic identification and segmentation of instruments in the live endoscopic view,

[0127] 3D visualization of instrument shape and position, and / or

[0128] 3D rendering of image data set and region of interest.

[0129] Method and system for an image guided navigation system may be, for example, configured for pathway identification and guidance to access target anatomical structures, for instance an Eustachian tube or a sinus ostia.

[0130] The method may, in some embodiments, further comprise displaying instrument position on the image data and also showing a preview of the image data in the projected orientation of the instrument (“look-ahead-view”) by way of stacked or overlaid image slices and / or

[0131] Creating a 3D preview from the 2D image slices stacked together to show which direction to go. Automatic identification of air passageways in the image set, or of sinus pathways and sinus ostia or of an Eustachian tube can be done using a machine learning algorithm, i.e. a neural network that was trained using several image datasets that represent the structure to be identified, i.e. an air passageway, a sinus ostia and / or a Eustachian tube. The neural network can be trained to identify different structures like the sinus ostia or ET or pathways based on geometry, color density, other identified structures, regional mapping, or grey scale values of the different voxels from the image. Typically, air passageways show up a black on a CT scan.

[0132] Figures 10a, 10b and 10c illustrate displaying an endoscopic live view 60 together with a CT navigation view 62 in an immediate neighborhood to the displayed endoscopic live view. According to this aspect, one central endoscopic view is displayed in the center of the monitor and a representation of images reconstructed from the preoperatively recorded tomography data is displayed as a navigation view on the same monitor near the tip of navigated instrument in the displayed endoscopic live view when a region on interests is reached. The navigation view 62 is similar to a thumbnail image or how a car navigation system zooms in near an exit to show you more information about which lane to be in. Figures 10a, !0b and 10c show a 3D rendering of CT scan representing a region of interest 64 and an overlay of the 3D rendering over the visualization of the live endoscopic view image data.

[0133] The presentation of images reconstructed from the preoperatively recorded tomography data could be a single CT-like view showing the navigated instrument position within the view or a combination of orthogonal views or a zoomed in image of a certain CT-like view providing direction information.

[0134] According to a further aspect that is independent from other aspects disclosed herein, the video stream of the endoscopic live view is real-time im- age processed for determining anatomic structures. Real-time image processing may be performed by a trained segmenting neural network such as the second semantic segmenting neural network 20.2 illustrated in figure 3.

[0135] The CT-like navigation view generated from the preoperatively recorded tomographic image data is automatically placed in a desired location in the eye focus area of the user by detecting the instrument position in the live camera view and overlaying the CT view near the instrument tip. Detecting the instrument position in the live camera view is achieved by real-time image processing as referred to above.

[0136] Figure 13 illustrates displaying a representation of a bent malleable instrument 12 for showing to the user an orthogonal view of the bending angle of the instrument shaft. The representation of a bent malleable instrument for showing to the user an orthogonal view of the bending angle of the instrument shaft id further-on called "bending view".

[0137] For generating the bending view a navigation instrument 12 having at least two sensor coils where the navigation instrument (i.e. balloon or flexpointer) has a shaft with a malleable / shapable shaft section is provided. The navigation system automatically detects when the instrument shaft is being bent and displays a live shape deduction screen to guide the user to different bend angles. This is illustrated in figure 13 b wherein the target shape is represented in a broken line representation, while the actual shape is illustrated in solid lines.

[0138] If a user, for example, uses a navigated instrument 12 with a balloon and a dual coil sensor carrier, when the user starts to bend the balloon a bending view appears automatically showing the actual (live) shape / bend of the instrument with the balloon and the recommended bends for each target sinus or Eustachian tube. Accordingly, a system may comprise a navigated instru- merit with two sensor coils on a longitudinal instrument shaft with a malleable shaft section between the two sensor coils. The system may further comprise an instrument tracking unit that is configured for tracking the position and the shape of the instrument by capturing signals representing the position and orientation of the instrument by means of the at least two sensor coils and generating instrument shape information from the captured signals.

[0139] The system may further comprise a computation unit that may be configured to detected a change of the instrument shape, for instance if the instrument shaft is bended by a user.

[0140] Once the computation unit detects that the instrument is shaped - e.g., bent, curved, or straight - by a user, an instrument bending view illustrating the actual shape of the instrument is generated and displayed; see figures 16, 17 and 18.

[0141] The computation unit may be configured for using a preoperatively generated image dataset (for instance CT scans) for analyzing and automatically determining a bending angle the instrument needs to be to best access the target anatomically location; see figure 15. The computation unit may be further configured for generating an extended instrument bending view showing not only the actual shape of the instrument but also a target shape of the instrument with the determined bending angle; see figures 13a and 16 to 18. The extended instrument bending view may, in some embodiments, guide the user to shape the instrument into the recommended bend as calculated from the preoperatively generated image dataset. Reference numerals:

[0142] 10 instrument navigation system

[0143] 12 trackable instrument

[0144] 12.1 instrument distal end

[0145] 12.2 instrument electromagnetic sensor

[0146] 14 data storage

[0147] 16 display

[0148] 18 instrument tracking arrangement

[0149] 18.1 electromagnetic field generator

[0150] 18.2 localizer

[0151] 18.3 electromagnetic sensor

[0152] 18.3.1 sensor coil

[0153] 18.3.2 sensor coil

[0154] 20 computation unit

[0155] 20.1. 20.2 segmenting neural network

[0156] 40 endoscope

[0157] 40.1 image sensor

[0158] 40.2 imaging optics

[0159] 50 Eustachian tube

[0160] 52 target mark

[0161] 54 boundary mark

[0162] 60 endoscopic live view

[0163] 62 navigation view

[0164] What is claimed is:

Claims

Claims1 . A method for an image guided navigation, comprising the steps of providing a trackable instrument with an instrument distal end, providing tomographic image data, providing target region and / or target object information that characterizes a target region and / or a target object in relation to the tomographic image data, providing a display for displaying visual representations, manipulating the instrument for changing at least one of the position, the orientation and the operational state of the instrument,) tracking the instrument by capturing signals representing the position and orientation of the instrument and generating positional information from the captured signals, generating a visual representation of the instrument together with a visual representation of tomographic image data so as to visualize the instrument in the visual representation of tomographic image data on a display, determining a distance between the instrument distal end and the target region and / or the target object,determining if the determined distance is less than a predetermined threshold distance, changing the visual representation of the instrument together with a visual representation of tomographic image data in response to determining that the determined distance is less than the predetermined threshold distance, wherein the change of the representation of the instrument together with a visual representation of tomographic image data comprises at least one of zooming on the target region of interest in the visual representation of tomographic image data, changing from orthogonal visual representations of tomographic image data to an in-line guidance representation of tomographic image data, changing from the orthogonal visual representations of tomographic image data to a 3D representation of tomographic image data, creating a new visual representation of tomographic image data based on the instrument orientation or anatomy, in particular generating a new slice view for Eustachian tube or other target anatomy such as sinuses, creating a new visual representation of tomographic image data to show a preview of the image data in the projected orientation of the instrument (“look-ahead-view”),creating a stacked and / or overlay of tomographic image slices, creating a 3D preview from the 2D image slices stacked together to show which direction to go toward a target region. enhancing the contrast and brightness of the visual representation of the instrument or of the visual representation of tomographic image data or of both, the visual representation of the instrument and the visual representation of tomographic image data, changing the color of the visual representation of tomographic image data, cropping the visual representation of tomographic image data to just show the target region, displaying a guidance target visual representation of tomographic image data wherein guidance target visual representation of tomographic image data comprises a distance indicator and an orientation indicator indicating the distance of the instrument distal end from target region and / or target object and the instrument orientation relative to the target region and / or target object, changing the visualization of the instrument for 2D to 3D or vice versa, and / or changing an instrument registration color which target region is reached.

2. The method of claim 1 , further comprising the steps of:providing and manipulating a optical instrument, capturing optical images by means of the optical instrument, displaying the optical images on the display and displaying navigation information within the displayed optical images,3. The method of claim 1 or 2, wherein the trackable instrument comprises at a longitudinal shaft with a malleable section and at least two five-degrees-of-freedom (5DoF) sensor coils arranged on or in the shaft in alignment with the shaft and spaced along the longitudinal direction of the shaft.

4. The method of claim 3, further comprising the steps of determining a plane defined by the at least two 5 DoF sensor coils and the generating a new viewing plane view along the plane defined by the at least two 5 DoF sensor coils.

5. The method of claim 3 or 4, wherein the instrument shaft is malleable in a section between the at least two sensor coils and wherein the method further comprising the steps of determining an instrument shape by processing signals provided by the at least two sensor coils and displaying a representation of the instrument shape in a two-dimensional or a three dimensional view showing the shape of the instrument in a bent or straight configuration,6. The method of claim 5, further comprising the step of obtaining and displaying a CT scan and of displaying a representation of the instrument shape show as an overlay on the CT scan.

7. The method of at least one of claims 1 to 6, wherein the trackable instrument is a balloon dilation device having a longitudinal, malleable shaft and wherein the method comprises displaying a graphic representation of the balloon dilation device shape in a two-dimensional or a three-dimensional view showing the shape of the inflated and / or deflated balloon in a bent or a straight configuration of the instrument shaft.

8. The method of at least one of claims 4 to 7, wherein the method comprises generating a perspective offset view from the tip of the instrument based on the instrument axis as determined from by the sensor coil configuration.

9. The method of at least one of claims 1 to 8, wherein the method comprises displaying a combination of instrument plane views and an inline view from the perspective of the instrument tip and axis.

10. The method of at least one of claims 1 to 9, wherein the method comprises capturing a live video stream by means of an optical instrument and overlaying of a three-dimensional representation of the instrument in a live video view.

11. A system for an image guided navigation comprising a trackable instrument having an instrument distal end, a data storage comprising tomographic image data, a data storage comprising target region and / or target object information that characterizes a target region and / or a target object in relation to the tomographic image data,a display for displaying visual representations, an instrument tracking unit that is configured for tracking the instrument by capturing signals representing the position and orientation of the instrument and generating positional information from the captured signals, a computation unit that is configured for generating a visual representation of the instrument together with a visual representation of tomographic image data so as to visualize the instrument in the visual representation of tomographic image data on a display, determining a distance between the instrument distal end and the target region and / or the target object, determining if the determined distance is less than a predetermined threshold distance, and changing the visual representation of the instrument together with a visual representation of tomographic image data in response to determining that the determined distance is less than the predetermined threshold distance, wherein the change of the representation of the instrument together with a visual representation of tomographic image data comprises at least one ofzooming on the target region of interest in the visual representation of tomographic image data, changing from orthogonal visual representations of tomographic image data to an in-line guidance representation of tomographic image data, changing from the orthogonal visual representations of tomographic image data to a 3D representation of tomographic image data, creating a new visual representation of tomographic image data based on the instrument orientation or anatomy (i.e. generating a new 45 degree slice view for ET), creating a new visual representation of tomographic image data to show a preview of the image data in the projected orientation of the instrument (“look-ahead-view”), creating a stacked and / or overlay of tomographic image slices, creating a 3D preview from the 2D image slices stacked together to show which direction to go toward a target region. enhancing the contrast and brightness of the visual representation of the instrument together and / or the visual representation of tomographic image data, changing the color of the visual representation of tomographic image data, cropping the visual representation of tomographic image data to just show the target region,displaying a guidance target visual representation of tomographic image data wherein guidance target visual representation of tomographic image data comprises a distance indicator and an orientation indicator indicating the distance of the instrument distal end from target region and / or target object and the instrument orientation relative to the target region and / or target object, changing the visualization of the instrument for 2D to 3D or vice versa, and / or changing an instrument representation color when target region is reached12. A system comprising a trackable instrument having a longitudinal instrument shaft with a malleable shaft section two sensor coils arranged in or on the longitudinal instrument shaft with a malleable shaft section between the two sensor coils an instrument tracking unit that is configured for tracking the position and the shape of the instrument by capturing signals representing the position and orientation of the instrument by means of the at least two sensor coils andgenerating instrument shape information from the captured signals, and a computation unit that is configured for detecting a change of the instrument shape, and generating an instrument bending view representing the actual shape of the instrument once the computation unit detects that the instrument is bend.

13. The system according to claim 12, wherein the computation unit is configured for using a preoperatively generated tomographic image dataset for analyzing and automatically determining a bending angle for the trackable instrument so as to best access a target anatomically location.

14. The system according to claim 12 or 13, wherein the computation unit is configured for generating an extended instrument bending view showing not only the actual shape of the instrument but also a target shape of the instrument with the determined bending angle.

Images