Electromagnetic sensor mapping systems and methods for multi-nodal, flexible catheters

Abstract

Electromagnetic (EM) sensing elements are incorporated into a multi-nodal catheter system for use with endoluminal surgical systems and methods. The catheter system includes a flexible catheter, a flexible substrate coupled to the flexible catheter, first EM sensing elements coupled to the flexible substrate in different orientations at a first position of the flexible catheter, and second EM sensing elements coupled to the flexible substrate in different orientations at the second position of the flexible catheter. The surgical system includes an EM field generator with antennas emitting EM fields. The surgical system may determine EM field strengths at positions within a sensing volume, build a neural network model, receive EM field signals from the EM sensing elements, estimate positions of the EM sensing elements by processing the EM field signals with the neural network model, and display a representation of the flexible catheter based on the estimated positions.

Description

Attorney Docket No. A0012826W001ELECTROMAGNETIC SENSOR MAPPING SYSTEMS AND METHODS FOR MULTI-NODAL, FLEXIBLE CATHETERSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of the filing date of provisional U.S. Patent Application No. 63 / 737,551, filed December 20, 2024, the entire contents of which is incorporated herein by reference.FIELD

[0002] The technology of the disclosure is generally related to multi-nodal, flexible catheters employing electromagnetic (EM) sensing elements at each node, and systems and methods for estimating the positions and orientations at each node in real-time using deep learning methods.BACKGROUND

[0003] In endoluminal procedures, EM navigation systems and methods give a clinician the ability to know the location of a surgical instrument, e.g., a catheter, in the body, which helps the clinician guide the surgical instrument to a target. For example, EM navigation systems may help a clinician localize and sample a biopsy target. EM navigation systems and methods generally involve generating an EM field in which EM sensors coupled to the surgical instrument are tracked.

[0004] The EM sensors are typically embedded within the surgical instruments and respond to EM fields generated in the area of interest. A field generator creates the EM field, which induces a current within the embedded EM sensors. The induced current is then converted into a signal that an EM navigation system can process. Using the signal, the position (e.g., x, y, z coordinates) and orientation (e.g., rotational axes) of the embedded EM sensors are calculated. Accurate position and orientation data is crucial for precise navigation, which enables precise tracking and guidance of surgical instruments during procedures.

[0005] Conventional electromagnetic (EM) localization techniques use the gradient descent method to calculate position information of a distal portion of a catheter. The gradient descent method is an iterative approach that finds the local minimum. In each step, the gradient descent method: calculates expected EM field strengths; compares theAttorney Docket No. A0012826W001 expected EM field strengths with measured EM field strengths; and calculates and takes the best step multiple times until the difference between the expected and measured EM field strengths is zero.

[0006] In many clinical applications, data from a single position on the catheter or surgical tool is insufficient. For example, in structural heart procedures, the path to valve deployment is so complex that the shape of the catheter is important to the clinician. Similarly, in lung biopsy procedures, capturing the entire shape of the extended working channel and minimizing the error in targeting the biopsy site is important to the clinician.

[0007] To achieve better resolution of shape sensing, more EM sensing elements are tracked. As a result, there would be a heavier computational burden if the gradient descent is used.SUMMARY

[0008] The techniques of this disclosure generally relate to multi-nodal catheter systems incorporating electromagnetic (EM) sensing elements for use with endoluminal surgical systems and methods. The techniques include sensing EM fields by the EM sensing elements at positions within a sensing volume, building a neural network model, and estimating positions of the EM sensing elements by processing sensed EM fields signals with the neural network model.

[0009] In one aspect, the disclosure provides a catheter system. The catheter system includes a flexible catheter having at least two nodes and at least one flexible substrate coupled to the flexible catheter. The catheter system also includes at least two first electromagnetic (EM) sensing elements coupled to the at least one flexible substrate and disposed at the at least two nodes, respectively. The catheter system also includes at least two second EM sensing elements (a) coupled to the at least one flexible substrate in orientations different from the orientations of the at least two first EM sensing elements, respectively, and (b) disposed at the at least two nodes, respectively. The catheter system also includes at least two third EM sensing elements (a) coupled to the at least one flexible substrate in orientations different from the orientations of the at least two first EM sensing elements, respectively, and the at least two second EM sensing elements, respectively, and (b) disposed at the at least two nodes, respectively.Attorney Docket No. A0012826W001

[0010] Implementations of the catheter system may include one or more of the following features. The at least two first EM sensing elements, the at least two second EM sensing elements, and the at least two third EM sensing elements may be tunnel magnetoresistance (TMR) sensing elements. The at least one flexible substrate may be a polyurethane substrate or a polyimide substrate.

[0011] In aspects, the at least two first EM sensing elements may be oriented with respect to the at least two second EM sensing elements such that a first electromagnetic field direction sensed by the at least two first EM sensing elements are perpendicular to a second electromagnetic field direction sensed by the at least two second EM sensing elements. In aspects, the at least two first EM sensing elements include at least three first EM sensing elements.

[0012] In aspects, the at least one flexible substrate may include three flexible substrates to which the at least two first EM sensing elements, the at least two second EM sensing elements, and the at least two third EM sensing elements are coupled, respectively. In aspects, the at least two nodes may define inflection points of the flexible catheter. In aspects, the at least one flexible substrate may be disposed within the flexible catheter. In aspects, the flexible substrate may couple to an inner or outer surface of the flexible catheter.

[0013] In another aspect, the disclosure provides a system. The system includes a flexible catheter and a flexible substrate coupled to the flexible catheter. The system also includes first electromagnetic (EM) sensing elements coupled to the flexible substrate in different orientations at a first longitudinal position of the flexible catheter. The system also includes second EM sensing elements coupled to the flexible substrate in different orientations at a second longitudinal position of the flexible catheter. The system also includes an EM field generator. The system also includes a processor in communication with the first, second, and third EM sensing elements, and a memory having stored thereon instructions, which when executed by the processor, causes the processor to: build a model of the positions of the first and second EM sensing elements as a function of EM fields; receive EM field signals from the first and second EM sensing elements; and determine positions of the first and second EM sensing elements based on the model and the received EM field signals.Attorney Docket No. A0012826W001

[0014] Implementations of the system may include one or more of the following features. In aspects, the flexible substrate may disposed within the flexible catheter. In aspects, the instructions, when executed by the processor, may cause the processor to train the model using at least one of preoperative or intraoperative EM field signals from the first and second EM sensing elements. In aspects, the instructions, when executed by the processor, may cause the processor to preoperatively train different models using EM field signals from the first and second EM sensing elements in different operative environments introducing different EM field distortion.

[0015] In aspects, the instructions, when executed by the processor, may cause the processor to select a model trained based on a current operative environment. In aspects, the instructions, when executed by the processor, may cause the processor to determine a shape of a portion of the catheter based on the determined positions.

[0016] In another aspect, the disclosure provides another system. The system includes a flexible catheter, and a flexible substrate coupled to the flexible catheter. The system also includes first electromagnetic (EM) sensing elements coupled to the flexible substrate in different orientations at a first longitudinal position of the flexible catheter. The system also includes second EM sensing elements coupled to the flexible substrate in different orientations at a second longitudinal position of the flexible catheter. The system also includes an EM field generator. The system also includes a processor in communication with the EM field generator and the first, second, and third EM sensing elements, and a memory having stored thereon instructions, which when executed by the processor, causes the processor to: determine EM field strengths at positions within a sensing volume, yielding determined EM field strengths; build a neural network model based on the positions within the sensing volume and the determined EM field strengths; receive EM field signals from the first and second EM sensing elements; estimate positions of the first and second EM sensing elements by processing the EM field signals with the neural network model, yielding estimated positions; and display a representation of the flexible catheter based on the estimated positions.

[0017] Implementations of the system may include one or more of the following features. In aspects, determining the EM field strengths may include at least one of measuring the EM field strengths or estimating the EM field strengths. In aspects, the EM field generator includes at least two antennas.Attorney Docket No. A0012826W001

[0018] In aspects, the instructions, when executed by the processor, may cause the processor to cause the EM field generator to generate at least two different EM field signals to be emitted by the at least two antennas.

[0019] In aspects, the system may include third EM sensing elements coupled to the flexible substrate in different orientations at a third longitudinal position of the flexible catheter.

[0020] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF DRAWINGS

[0021] FIG. l is a block diagram that illustrates an example of a surgical system that uses the electromagnetic (EM) sensor mapping methods of the disclosure;

[0022] FIG. 2 is a perspective view of an example of a multi-nodal catheter that may be used in a variety of surgical systems including the surgical system of FIG. 1;

[0023] FIG. 3 is a circuit block diagram that illustrates an EM sensor circuit that may be incorporated into the multi-nodal catheter of FIG. 2;

[0024] FIG. 4 is a cross-sectional view of the multi -nodal catheter of FIG. 2 incorporating an EM sensor circuit;

[0025] FIGS. 5A-5C depict a circuit diagram of an EM sensor circuit;

[0026] FIGS. 6 A and 6B are a layout diagram of the EM sensor circuit depicted in the circuit diagram of FIG. 5;

[0027] FIG. 7 is a block diagram that illustrates EM sensor mapping using a neural network model;

[0028] FIG. 8 is a flow chart of an EM sensor mapping method using a neural network-based model;

[0029] FIG. 9 is a block diagram that illustrates an example of the EM sensor mapping method of FIG. 8;

[0030] FIG. 10 is a flow chart of an EM sensor mapping method using a convolutional neural network;Attorney Docket No. A0012826W001

[0031] FIGS. 11 A and 1 IB include a block diagram that illustrates an example of the EM sensor mapping method of FIG. 10;

[0032] FIG. 12 is a block diagram that illustrates an example of an EM sensor mapping method that takes into account the distortion introduced by different operative environments;

[0033] FIG. 13 is a block diagram that illustrates an example of a computer system that may interface with the EM sensor circuit and perform the methods of the disclosure;

[0034] FIGS. 14A and 14B are a layout diagram of an EM sensor circuit including four EM sensors;

[0035] FIG. 15A is a perspective view that illustrates a shaft for use with the flexible catheter according to aspects of the disclosure;

[0036] FIG. 15B is an exploded view of the shaft of FIG. 15 A; and

[0037] FIG. 16 is a perspective view of another example of the intermediate member of FIG. 15B that includes multiple helical tendons.DETAILED DESCRIPTION

[0038] To achieve better resolution of sensing the shape of a portion of a navigation catheter in endoluminal navigation systems, this disclosure presents a multi-nodal, flexible catheter employing electromagnetic (EM) sensing elements at each node. The flexible catheter incorporates a flexible printed circuit (FPC) on which miniature EM sensing elements, e.g., TMR sensing elements, are disposed at different nodes of the navigation catheter. TMR sensing elements are less expensive, smaller, and have a larger operating frequency band than many existing EM sensing elements. The larger operating frequency band reduces or eliminates conductive and magnetic interference.

[0039] And to reduce the computational burdens associated with the gradient descent method, this disclosure presents systems and methods for estimating the positions and orientations at each node in real-time using deep learning methods involving EM field simulation and a trained deep neural network model. The deep neural network model may receive as input the sensed EM field strengths and may output EM sensor positions and orientations.

[0040] The method for EM sensor localization using deep learning and EM field simulation shifts the real-time processing time required by current optimization algorithmsAttorney Docket No. A0012826W001 to the training phase. This may allow for the computation of multiple sensor positions much faster. Since some location boards have many antennas, e.g., nine antennas, operating at different frequencies and spatial locations, a more computationally efficient method is needed. For example, a convolutional neural network (CNN) of a graphicsbased method, e.g., an image recognition method, may be configured for use in the systems and methods of this disclosure. Lastly, EM field distortions have significant impacts on the accuracy of EM sensor localization. Accurate mapping of the clinical environment is needed to account for different distortion sources and configurations, which is too time-consuming and tedious to be realistic in clinical procedures. The systems and methods of this disclosure embed EM field distortions in the training phase so that different types of distortion can be accounted for automatically.

[0041] These improvements to catheter systems and methods may enable EM navigation for cardiac procedures, e.g., cardiac ablation, pacemaker placement, transcatheter mitral valve replacement (TMVR), and other robotic surgeries that require precise knowledge of the physiological geometry.

[0042] FIG. 1 is a perspective view of an example of a navigation system 100 for facilitating navigation of a medical tool, e.g., a catheter 102 or a camera and sensor tool lOle, to a target or target area via airways of the lungs. During navigation, the system 100 may be configured to generate a point cloud volume from stereoscopic images of at least one body lumen acquired by the camera and sensor tool lOle and register the point cloud volume to a three-dimensional (3D) model generated, for example, from preoperative radiographic images. The system 100 may also be configured to apply a machine learningbased stereoscopic image rectification algorithm to stereoscopic images acquired by the camera and sensor tool lOle to facilitate point matching in the process of generating a point cloud volume.

[0043] The system 100 may be further configured to construct radiographic-based volumetric data of a target area from intraprocedural 2D radiographic images, e.g., intraprocedural fluoroscopic and / or CBCT images, to confirm navigation of a navigation catheter 102, e.g., an extended working channel (EWC) or a smart extended working channel (sEWC), to a desired location near the target area, where a tool, e.g., the camera and sensor tool lOle, may be placed through and extending out of the navigation catheterAttorney Docket No. A0012826W001102. In aspects, the optional imaging system 124 of the system 100 may include a C-arm fluoroscope or a cone-beam computed tomography (CBCT) imaging system.

[0044] The system 100 may be further configured to facilitate approach of a medical tool to the target area and to determine the location of the medical tool with respect to the target by using electromagnetic navigation (EMN) of the navigation catheter 102. One such EMN system is the ILLUMISITE system currently sold by Medtronic PLC, though other systems for intraluminal navigation are considered within the scope of this disclosure. The navigation catheter 102 may include three nodes, which may incorporate respective EM sensors 110, 120, 130 for sensing one or more characteristics of EM fields generated by the location board 141, which, as shown, may include nine antennas 145. The EM sensors 110, 120, 130, each of which may include at least one sensing element (e.g., a TMR sensing element) may sense the frequency and / or strengths of the EM fields emitted by the antennas 145.

[0045] One aspect of the system 100 is a software component for reviewing computed tomography (CT) image scan data that has been acquired separately from the system 100. The review of the CT image data allows a user to identify one or more targets, plan a pathway to an identified target (planning phase), navigate the navigation catheter 102 (e.g., and EWC or sEWC) including the camera and sensor tool lOle to the target (navigation phase) using a user interface running on a computer system 122, and confirming placement of a distal end portion of the navigation catheter 102 near the target using the camera and sensor tool lOle and / or using electromagnetic (EM) sensor 110 disposed in or on the navigation catheter 102 at a predetermined position at or near the distal end portion of the navigation catheter 102. The target may be tissue of interest identified by review of the CT image data during the planning phase.

[0046] Following navigation of the navigation catheter 102 near the target, a medical tool, such as a biopsy tool, an access tool, or a therapy tool, e.g., a flexible microwave ablation catheter, may be inserted into and may be fixed in place with respect to the navigation catheter 102 such that a distal end portion of the medical tool extends a desired distance 107 beyond the distal end of the navigation catheter 102 and the navigation catheter 102 is further navigated using EM navigation to obtain a tissue sample, enable access to a target site, or apply therapy to the target using the medical tool.Attorney Docket No. A0012826W001

[0047] As shown in FIG. 1, the navigation catheter 102 is part of a catheter guide assembly 112. In practice, the navigation catheter 102 is inserted into a bronchoscope 108 or other suitable endoscope for access to a luminal network of the patient P. Specifically, the navigation catheter 102 of catheter guide assembly 112 may be inserted into a working channel of the bronchoscope 108 for navigation through a patient’s luminal network. A bronchoscope adapter 109 is coupled to the proximal end portion of the bronchoscope. The bronchoscope adapter 109 may be the EDGE™ Bronchoscope Adapter, which is currently marketed and sold by Medtronic PLC. The bronchoscope adapter 109 is configured either to allow motion of the navigation catheter 102 through the working channel of the bronchoscope 108 (which may be referred to as an unlocked state of the bronchoscope adapter 109) or prevent motion of the navigation catheter 102 through the working channel of the bronchoscope (which may be referred to as a locked state of the bronchoscope adapter 109). Alternatively, the camera and sensor tool lOle may be inserted into the working channel of the bronchoscope 108 in place of the navigation catheter 102.

[0048] Aspects of the disclosure may be applied to a variety of procedures including biopsy, ablation, or marker placement procedures. For example, the procedures may involve one or more of a locatable guide 101a, a microwave ablation tool 101b, a biopsy needle 101c, a forceps 101 d, or a camera and sensor tool lOle. The locatable guide (LG) 101a, which may be a catheter and which may include a sensor 104a, is inserted into the navigation catheter 102 and locked into position such that the sensor 104a extends a predetermined distance beyond the distal end portion of the navigation catheter 102. The camera and sensor tool lOle may include a sensor pack 113, which may incorporate miniaturized LEDs, and a catheter assembly 111, which may include one or more camera wires, one or more EM sensor wires, one or more IMU sensor wires, one or more pull wires, and / or one or more illumination optical fibers (in aspects that do not incorporate miniaturized LEDs).

[0049] The tools lOla-lOle may include a fixing member 103a-e (103b is not explicitly shown) such that when the fixing member 103a-103e of the tools lOla-lOle engages, e.g., snaps in, with the proximal end portion of the handle 106 of the catheter guide assembly 112, the tools lOla-lOle extend a predetermined distance 107 beyond a distal tip or end portion of the navigation catheter 102. The predetermined distance 107Attorney Docket No. A0012826W001 may be based on the length of the navigation catheter 102 and a length between the end portion of the handle 105a-e or the fixing member 103a-103e and the distal end portion of the LG 101a or the other medical tools lOlb-lOle. In aspects, the handles 105a-105e may include control objects, e.g., a button or a lever, for controlling operation of the medical tools lOla-lOle.

[0050] In some aspects, the position of the fixing member 105a-105e along the length of the medical tools lOla-lOle may be adjustable so that the user can adjust the distance by which the distal end portion of the LG 101a or the medical tools lOlb-lOle extend beyond the distal end portion of the navigation catheter 102. The position and orientation of the LG sensor 104a or the sensor pack 113 of the camera and sensor tool lOle relative to a reference coordinate system within an electromagnetic field can be derived using an application executed by the computer system 122. In some aspects, the navigation catheter 102 may function as the LG 101a, in which case the LG 101a may not be used. In other aspects, the navigation catheter 102 and the LG 101a may be used together. For example, data from the sensors 104a, 110, 120, 130 may be fused together. In aspects, one or more of the sensors 104a, 110, 120, 130 may be TMR sensors or other miniaturized sensors suitable for navigation through lumens of the body. Catheter guide assemblies 112 are currently marketed and sold by Medtronic PLC under the brand names SUPERDIMENSION® Procedure Kits, or EDGE™ Procedure Kits, and are contemplated as useable with the disclosure.

[0051] The system 100 generally includes an operating table 140 configured to support a patient P; a bronchoscope 108 configured for insertion through patient P’s mouth into patient P’s airways; monitoring equipment 114 coupled to bronchoscope 108 (e.g., a video display for displaying the video images received from the video imaging system of bronchoscope 108 or the images received from the camera and sensor tool 101 e); and a tracking system 115 including a tracking module 116, patient sensor triplet (PST) 118, and a location board 141. The location board 141 includes one or more EM transmitters for generating an EM field. The location board 141 may be in the form of a transmitter mat.

[0052] The location board 141 may also include fiducials, which may be embedded or otherwise incorporated into the location board 141, and which are designed and / or arranged to appear in radiographic images for the purpose of creating a 3D reconstruction from the radiographic images. Since the fiducials may be radiographically dense, theAttorney Docket No. A0012826W001 fiducials create artifacts on the radiographic images, e.g., intraoperative CBCT images. The system 100 further includes a computer system 122 on which software and / or hardware are used to facilitate identification of a target, planning a pathway to the target, navigating a medical tool to the target, and / or confirmation and / or determination of placement of the navigation catheter 102, or a suitable tool therethrough, relative to the target.

[0053] As noted above, an optional imaging system 124 capable of acquiring CBCT images or fluoroscopic images of the patient P is also included in the system 100. The images, sequence of images, or video captured by the imaging system 124 may be stored within the imaging system 124 or transmitted to the computer system 122 for storage, processing, and display. Additionally, the imaging system 124 may move relative to the patient P so that images may be acquired from different angles or perspectives relative to patient P to create a series of CBCT images.

[0054] The pose of the optional imaging system 124 relative to patient P and while capturing the images may be estimated via markers incorporated into the location board 141, the operating table 140, or a pad (not shown) placed between the patient P and the operating table 140. The markers are positioned under patient P, between patient P and operating table 140 and between patient P and a radiation source or a sensing unit of the imaging system 124. The markers may have a symmetrical spacing or may have an asymmetrical spacing, a repeating pattern, or no pattern at all. The imaging system 124 may include a single imaging system or more than one imaging system. When a CBCT system is employed, the captured images can be employed to confirm the location of the navigation catheter 102 and / or one of the medical tools lOla-lOle within the patient, update CT-based 3D modeling, or replace pre-procedural 3D modeling with intraprocedural modeling of the patient’s airways and the position of the navigation catheter 102 within the patient.

[0055] The computer system 122 may be any suitable computer system including a processor (e.g., the processor 1304 of FIG. 13) and storage medium (e.g., the memory 1302 of FIG. 13), such that the processor can execute instructions stored on the storage medium. The computer system 122 may further include a database configured to store patient data, CT data sets including CT images, CBCT images and data sets, fluoroscopic data sets including fluoroscopic images and video, 3D reconstructions, navigation plans,Attorney Docket No. A0012826W001 and any other such data. Although not explicitly illustrated, the computer system 122 may include inputs, or may otherwise be configured to receive, CT data sets, CBCT or fluoroscopic images or video, and other suitable imaging data. Additionally, the computer system 122 includes a display configured to display graphical user interfaces. The computer system 122 may be connected to one or more networks through which one or more databases may be accessed by the computer system 122.

[0056] With respect to the navigation phase, a five and / or six degrees-of-freedom electromagnetic locating or tracking system 115, or other suitable system for determining positions and orientations of nodes of a distal portion of the navigation catheter 102 (e.g., the EM sensors 110, 120, 130) and / or the camera and sensor tool lOle, is utilized for performing registration of pre-procedure images (e.g., a CT image data set and 3D models derived therefrom) and the pathway for navigation with the patient as they are located on the operating table 140.

[0057] In an EMN-type system, the tracking system 115 may include the tracking module 116, the PST 118, the EM sensors 110, 120, 130, and the location board 141, which includes multiple antennas 145 for emitting multiple EM fields with, in some aspects, different characteristics, e.g., different frequencies and / or amplitudes. In some aspects, the antennas may emit EM fields with different characteristics, e.g., different frequencies and / or different amplitudes. When imaging is used, the location board 141 may also include radiopaque markers (not shown). The tracking system 115 may be configured for use with a locatable guide (and the LG sensor 104a) and / or a camera and sensor tool (particularly the sensor coils). As described above, the medical tools, e.g., the locatable guide (LG) 101a with the LG sensor 104a and / or the camera and sensor tool lOle may be configured for insertion through the navigation catheter 102 into patient P’s airways (either with or without the bronchoscope 108) and are selectively lockable relative to one another via a locking mechanism, e.g., the bronchoscope adapter 109.

[0058] The location board 141 is positioned beneath patient P. The location board 141 generates an electromagnetic field around at least a portion of the patient P within which the position of the LG sensor 104a, the navigation catheter sensors 110, 120, 130, the PST 118, and the distal portion of the camera and sensor tool lOle can be determined through use of a tracking module 116. The navigation catheter sensors 110, 120, 130 may be five degree-of-freedom sensors, six degree-of-freedom sensors, or a combination of the two.Attorney Docket No. A0012826W001One or more of the reference sensors of the PST 118 are attached to the chest of the patient P.

[0059] Registration refers to a method of correlating the coordinate systems of the preprocedure images, and particularly a 3D model derived therefrom, with the patient P’s airways as, for example, observed through the bronchoscope 108 and allow for the navigation to be undertaken with accurate knowledge of the location of the LG sensor 104a within the patient and an accurate depiction of that position in the 3D model. Registration may be performed by moving the LG sensor through the airways of the patient P. More specifically, data pertaining to locations of the LG sensor, while the locatable guide is moving through the airways with the multi-nodal navigation catheter 102, is recorded using the location board 141, the PST 118, the EM sensors within the nodes of the navigation catheter, and the tracking system 115. A shape resulting from this location data is compared to an interior geometry of passages of the 3D model generated in the planning phase, and a location correlation between the shape and the 3D model based on the comparison is determined, e.g., using software application on the computer system 122.

[0060] The software may align, or register, a point cloud volume generated from stereographic images acquired by the camera and sensor tool lOle or an image representing a location of LG sensor 104a and / or EM sensors 110, 120, 130 with the 3D model and / or two-dimensional images generated from the three-dimensional model. Alternatively, a manual registration technique may be employed by navigating the bronchoscope 108 with the LG sensor 104a to pre-specified locations in the lungs of the patient P and manually correlating the images from the bronchoscope 108 to the model data of the 3D model.

[0061] Though described herein with respect to EMN systems using EM sensors (e.g., EM sensors 110, 120, 130) incorporated into nodes of the navigation catheter 102, this disclosure is not so limited and may be used in conjunction with other suitable sensors such as Fiber-Bragg gratings or ultrasonic sensors. Additionally, the methods described herein may be used in conjunction with robotic systems such that robotic actuators drive the navigation catheter 102 or bronchoscope 108 proximate the target.

[0062] At any point during the navigation process, tools such as a locatable guide 101a, a therapy tool (e.g., a microwave ablation tool 101b or a forceps 101 d), a biopsy toolAttomey Docket No. A0012826W001(e.g., a biopsy needle 101c), may be inserted into and fixed in place relative to the navigation catheter 102 to place one of the tools lOlb-lOle proximate the target or a desired location using position information from the navigation catheter 102. The position information from one or more of the sensors 110, 120, 130 of the navigation catheter 102 may be used to calculate the position of the distal tip or distal end portion of any of the tools lOlb-lOld.

[0063] To ensure the accuracy of the position calculations, the tools lOla-lOle may be designed to extend a predetermined distance from the distal end of the navigation catheter 102 and at least the distal portions of the tools lOla-lOle that extend from the navigation catheter 102 are designed to be rigid or substantially rigid. The predetermined distance may be different depending on one or more of the design of the tools lOla-lOle, the stiffnesses of the tools lOla-lOle, or how each of the tools lOla-lOle interact with different types of tissue. The tools lOla-lOle may be designed or characterized to set the predetermined distance to ensure deflection is managed (e.g., minimized) so that the virtual tools and environment displayed to a clinician are an accurate representation of the actual clinical tools and environment.

[0064] Calculating the position of the distal end portion of any of the tools lOla-lOle may include distally projecting position information from the distal EM sensor 110 according to tool information. The tool information may include one or more of the shape of the tool, the type of tool, the stiffness of the tool, the type or characteristics of the tissue to be treated by the tool, or the dimensions of the tool.

[0065] With respect to the planning phase, the computer system 122, or a separate computer system (not shown), utilizes previously acquired CT image data for generating and viewing a 3D model or rendering of patient P’s airways, enables the identification of a target (automatically, semi-automatically, or manually), and allows for determining a pathway through patient P’s airways to tissue located at and around the target. More specifically, CT images acquired from CT scans are processed and assembled into a 3D CT volume, which is then utilized to generate a 3D model of patient P’s airways. The 3D model may be displayed on a display associated with the computer system 122, or in any other suitable fashion.

[0066] Using the computer system 122, various views of the 3D model or enhanced two-dimensional images generated from the 3D model are presented. The enhanced two-Attorney Docket No. A0012826W001 dimensional images may possess some 3D capabilities because they are generated from 3D data. The 3D model may be manipulated to facilitate identification of target on the 3D model or two-dimensional images, and selection of a suitable pathway through patient P’s airways to access tissue located at the target can be made. Once selected, the pathway plan, the 3D model, and the images derived therefrom, can be saved, and exported to a navigation system for use during the navigation phase(s). The ILLUMISITE software suite currently sold by Medtronic PLC includes one such planning software.

[0067] FIG. 2 is a perspective view of an example of a multi-nodal catheter 200 that may be used in a variety of surgical systems including the surgical system 100 of FIG. 1. The multi -nodal catheter 200 includes three flexible portions: a proximal flexible portion 203, a middle flexible portion 202, and a distal flexible portion 201. In aspects, the multimodal catheter 200 may be made of stainless steel, which may at least minimize interference experienced by the EM sensing elements. In aspects, a polymer skin may be disposed over the stainless steel. The flexible portions 201-203 are terminated with nodes 210, 220, 230, respectively. In aspects, each node 210, 220, 230 includes an EM sensor circuit (not shown) for sensing the positions of each node 210, 220, 230. The distal node 210 is coupled to a working channel 240, through which a diagnostic or treatment instrument may extend. The diagnostic or treatment instrument may include a camera, an ablation tool, forceps, or a biopsy tool. In aspects, each of the nodes 210, 220, 230 may be manipulated, for example, via a cable coupled to a handle at a proximal portion of the catheter 200.

[0068] Each node 210, 220, 230 may incorporate EM sensing elements. The EM sensing elements may be tunnel magneto-resistance (TMR) sensing elements. The TMR sensing elements are cost-effective and have a small footprint. The TMR sensing elements also have a large operational bandwidth, which may be used to avoid metal inferences. For example, when a portable ultrasonic probe is placed near TMR sensing elements at 3kHz, there may be field distortion. At 100Hz, however, there may be negligible field distortion caused by the portable ultrasonic probe.

[0069] FIGS. 3 and 4 illustrate an EM sensor circuit 300 that may be incorporated into a distal portion of the multi-nodal catheter 200. As illustrated in FIG. 3, the EM sensor circuit 300 may include vertical or perpendicular sections 310, 320, 330, each of which include three TMR sensing elements. As illustrated in FIG. 4, the distal perpendicularAttorney Docket No. A0012826W001 section 310 may include three TMR sensing elements 411, 412, 413. And the other perpendicular sections 320, 330 may each include three TMR sensors. In aspects, the three TMR sensing elements of each perpendicular section 310, 320, 330 may be packaged in a single chip. In aspects, the three TMR sensing elements (referred to collectively as the TMR triplet) may be separated or spaced from each other, in which case the three TMR sensing elements may not be oriented orthogonal to each other. For example, the TMR triplet may be placed along a length of a catheter. The TMR sensing elements of each perpendicular section 310, 320, 330 may be coupled to a flexible substrate 305 forming a flex printed circuit (FPC). The flexible substrate 305 may be made of polyimide, polyurethane (e.g., thermoplastic polyurethane (TPU)), or one or more other plastic materials capable of stretching and / or flexing in one or more directions, e.g., bending and twisting. For example, the flexible substrate 305 may be made of a thin layer of polyimide or polyurethane.

[0070] The flexible substrate 305 of the FPC 300 enables the FPC 300 to bend about a longitudinal axis such that the FPC 300 can be incorporated into the catheter 200. For example, the FPC 300 may be rolled and coupled to an inner surface of the catheter 200. In some aspects, the FPC 300 may be attached to the inner surface of the catheter 200 via an adhesive. The flexible substrate 305 of the FPC 300 also enables the FPC 300 to bend about a longitudinal axis such that the magnetic field sensing directions of the three TMR sensing elements in each perpendicular section are orthogonal to each other.

[0071] For example, as shown in FIG. 4, the TMR sensing elements 411-413 are spaced from each other on the flexible substrate 305 such that the respective magnetic field sensing directions 451-453 are orthogonal to each other. In the example of FIG. 4, the TMR sensing element 411 is arranged on the flexible substrate 305 such that the TMR sensing element 411 can sense magnetic field lines in the magnetic field sensing direction 451, which is in the distal direction x (i.e., into the page). The TMR sensing element 412 is arranged on the flexible substrate 305 such that the TMR sensing element 412 can sense magnetic field lines in the magnetic field sensing direction 452, which is in the horizontal direction y. And the TMR sensing element 412 is arranged on the flexible substrate 305 such that the TMR sensing element 412 can sense magnetic field lines in the magnetic field sensing direction 453, which is in the vertical direction z.Attorney Docket No. A0012826W001

[0072] FIGS. 5 and 6 illustrate an example of an EM sensor circuit, in which three FPC sections are printed on the same polyurethane substrate. As illustrated in FIG. 6, the layout is designed to minimize cross wires and vias in the FPC. Each perpendicular section (e.g., first perpendicular section including TMR sensing elements 511, 512, 513, second perpendicular section including TMR sensing elements 521, 522, 523, or third perpendicular section including TMR sensing elements 531, 532, 533) makes up one of the nodes which includes TMR sensing elements sensitive to three orthogonal magnetic field directions when the flexible substrate is rolled into a cylindrical shape. For example, the perpendicular section 510 makes up a node which includes TMR sensing elements 511, 512, 513 sensitive to three orthogonal magnetic field directions when rolled into a cylindrical shape.

[0073] Each horizontal section (e.g., first horizonal section including TMR sensing elements 511, 521, 531; second horizontal section including TMR sensing elements 512, 522, 532; or third horizontal section including TMR sensing elements 513, 523, 533) includes TMR sensing elements sensitive to the same magnetic field direction. For example, horizontal section 501 includes TMR sensing elements 511, 521, 531 sensitive to the magnetic field direction x, e.g., in the distal direction parallel to the longitudinal axis of the catheter 200. The TMR sensing elements in each horizontal section share the one proximal end connector, e.g., one of proximal end connectors 541-543. For example, the TMR sensing elements 511, 521, 531 of the horizontal section 501 share the same proximal end connector 541.

[0074] In aspects, the EM sensor circuit may include one, two, or three FPC sections printed on the same substrate, e.g., a polyurethane substrate. For example, two FPC sections, each having four sensing elements, may be printed on the same substrate. In aspects, an FPC section may include one or more sensing elements. For example, an FPC section may include two sensing elements for a two-node catheter system or four sensing elements for a four-node catheter system.

[0075] In aspects, an EM sensor mapping method is used to process EM sensor data to determine positions of nodes of a catheter. As illustrated in FIG. 7, the EM sensor mapping method may employ a neural network model 704. The neural network model 704 may receive as inputs 702 magnetic field strengths Hx, Hy, Hz from EM sensing elementsAttorney Docket No. A0012826W001 at a node. The neural network model 704 processes and outputs 706 sensor position and / or orientation.

[0076] The neural network model 704 may include an input layer, one or more hidden layers, and an output layer. The input layer may include three neural network nodes corresponding to the magnetic field strengths Hx, Hy, Hz from EM sensing elements at one or more catheter nodes. For example, the input layer may include six nodes corresponding to two sets of magnetic field strengths Hx, Hy, Hz from EM sensing elements at two catheter nodes. In aspects, there may be other neural network nodes corresponding to other characteristics of the magnetic fields such as frequency and / or amplitude.

[0077] The neural network model 704 may be a deep neural network model employing multiple hidden layers. Each hidden layer contains neural network nodes that apply transformations to inputs of the input layer. The transformations may include weights, bias values, and activation functions. The weights are applied to every connection between neurons of the neural network model 704. The bias value is applied to the weighted sum to adjust the output layer independently of the input layer. Then, an activation function is applied to the combination of the weighted sum and the bias value to introduce nonlinearity into the neural network model. The activation function may include the Rectified Linear Unit (ReLU), which introduces non-linearity into the neural network model 704. In some aspects, the output layer may use a linear activation function to estimate position and / or orientation information of the EM sensors (e.g., EM sensors 110, 120, 130 of FIG. 1) at one or more nodes of the catheter. In other aspects, the output layer may not use an activation function. The neural network model 704 may be trained with training data sets including EM sensor data and corresponding catheter node position and / or orientation data.

[0078] FIG. 8 is a flow chart of an example of an EM sensor mapping method using a neural network-based model. The EM sensor mapping method may be implemented on a system that includes a flexible catheter, a flexible substrate coupled to the flexible catheter, first electromagnetic (EM) sensor elements coupled to the flexible substrate in different orientations at the same first position of the flexible catheter, second EM sensor elements coupled to the flexible substrate in different orientations at the same second position of the flexible catheter, and an EM field generator. The flexible substrate may beAttorney Docket No. A0012826W001 formed (e.g., rolled, bent, or flexed) such that the flexible substrate may be coupled to or inserted in a channel of the catheter. In aspects, the flexible substrate may be formed into a shape (e.g., a semi-cylindrical shape) suitable for inserting the flexible substrate into a channel of the catheter or coupling the flexible substrate to an inner or outer surface of the catheter. The EM field generator may be implemented as a location board placed beneath the patient (e.g., the location board 141 illustrated in FIG. 1).

[0079] At block 802, a model representing the positions of the EM sensor elements as a function of EM field signals is built. The model may be a neural network model. The model may be trained using EM field signals collected from EM sensor elements preoperatively, intraoperatively, or both. In aspects, different models may be preoperatively trained using EM field signals from the EM sensor elements in various operative environments that introduce distinct EM field distortions. A model trained on an operative environment similar to the current operative environment may be selected for use. If training includes a sufficiently large set of operative environments, the system may interpolate the current operative environment by using an estimate, which may be a coarse estimate. This estimate may be based on relevant operative environment parameters, e.g., C-arm angle, distance to the emitter, etc.

[0080] At block 804, EM field signals are received from the first and second EM sensor elements, each of which is disposed at a node of the flexible catheter. The EM field signals may include magnetic field strengths sensed by the first and second EM sensor elements. At block 806, the positions of the first and second EM sensor elements are determined based on the model and the sensed EM field signals. In aspects, the method 800 may include determining the shape of a portion of the flexible catheter based on the determined catheter node positions.

[0081] According to existing methods, if many sensing elements, e.g., ten, need to be localized, each sensor repeats optimization steps until a local minimum is found. For example, starting from an initial time tO, a first sensor repeats optimization steps many times until the local minimum (xl, yl, zl) is found. The gradient descent method is then repeated for the subsequent sensor until the tenth sensor, for which the optimization steps are repeated many times until the local minimum (xlO, ylO, zlO) is found.

[0082] The existing methods may be computationally inefficient and slow, especially when performing optimization processes on data from many EM sensing elements. AsAttorney Docket No. A0012826W001 illustrated in FIG. 9, to address this issue, the EM mapping methods of the disclosure may perform optimization in the training phase 902 instead of in real time after an initial time tO 904, which allows fast localization of many sensing elements. Accordingly, in aspects of the disclosure, the model f may be trained prior to the start of a medical procedure.

[0083] In aspects, a large amount of training data 906 is used during the training phase 902 to build the model f to ensure the accuracy of the position estimates output from the model f. The model f may be expressed in simple form as x, y, z = f(Bx, By, Bz), where x, y, z are the position coordinates of a catheter node and Bx, By, Bz are magnetic fields measured by the EM sensing elements located at the catheter nodes. After initial time tO 904, real-time magnetic field measurements from all EM sensing elements (Bxl, Byl, Bzl, . . ., BxlO, BylO, BzlO) are input to the model f. The model f processes the real-time magnetic field measurements (Bxl, Byl, Bzl, . . ., BxlO, BylO, BzlO) and outputs estimates of position and orientation information of each of the catheter nodes (xl, yl, zl, yawl, pitchl, . . ., xlO, ylO, zlO, yawlO, pitchlO).

[0084] In aspects of the disclosure, the estimated position and orientation information of each of the catheter nodes may be used to perform shape sensing. Fiber optic technology has been used to obtain precise location, shape, and orientation information of a catheter throughout a navigation and treatment procedure. Using trained neural network models, position and orientation information may be estimated from real-time data output from the sensing elements at the catheter nodes and processed to determine precise shape information of the catheter.

[0085] In one example implementation of EM-based shape sensing, EM sensing elements are positioned at four nodes along a catheter and a spline fitting method is applied to the position information estimated from the EM sensor data to determine the shape. The EM-based shape sensing performed by a prototype has been shown in a realtime experiment to be comparable to the fiber-optic shape sensing performed by a commercial fiber optic shape sensor inserted through the same catheter.

[0086] FIG. 10 is a flow chart of an EM sensor mapping method using a convolutional neural network. The EM sensor mapping method may be applied to a system including a flexible catheter, a flexible substrate coupled to the flexible catheter, first electromagnetic (EM) sensor elements coupled to the flexible substrate in different orientations at the same first longitudinal position of the flexible catheter, second EM sensor elements coupled toAttorney Docket No. A0012826W001 the flexible substrate in different orientations at the same second longitudinal position of the flexible catheter, and an EM field generator.

[0087] In aspects, the system may include more than the first and second EM sensor elements located at one or more other nodes of the flexible catheter. For example, the system may include third EM sensor elements coupled to the flexible substrate in different orientations at the same third longitudinal position of the flexible catheter. The EM field generator may include at least two antennas. At block 1002, EM field strengths are determined at positions within a sensing volume. Determining the EM field strengths may include at least one of measuring the EM field strengths or estimating the EM field strengths. At block 1003, a neural network model is built based on the positions within the sensing volume and the determined EM field strengths.

[0088] At block 1004, EM field signals are received from the first and second EM sensor elements. In aspects, prior to performing block 1004, the method 1000 may include causing the EM field generator to generate two different EM field signals to be emitted by the at least two antennas. At block 1006, positions of the first and second EM sensor elements are estimated by processing the received EM field signals with the neural network model. At block 1008, a representation of the flexible catheter is displayed based on the estimated positions. Block 1008 may also include generating or selecting a representation of the flexible catheter based on flexible catheter data input to the system via a user interface by a clinician.

[0089] In the case where the EM field generator generates at least two different EM fields to be transmitted by at least two antennas, a convolutional neutral network model may be used to solve the multi -frequency field strengths to position mapping. For example, as illustrated in FIG. 1, the location board 141 may include nine antennas 145 at nine different spatial locations. Also, as illustrated in FIG. 11 A, the nine antennas 145 may emit nine EM fields 1102 at nine different frequencies, respectively. To build the convolutional neural network model, the EM field strengths B may be simulated and / or measured at many positions (x,y,z) in the sensing volume and may be stored in nine multidimensional arrays B(x,y,z).

[0090] As illustrated in FIG. 1 IB, the nine multi-dimensional arrays B(x,y,z) may then be input to a convolutional neural network including an input layer, a convolutional layer, a pooling layer, dense layers, and an output layer. The input layer may be the first layerAttorney Docket No. A0012826W001 that receives the multi-dimensional arrays B(x,y,z). The input layer may prepare the multidimensional arrays B(x,y,z) for processing by the convolutional layer and may feed the result to the convolutional layer. The convolutional layers may apply filters (e.g., small matrices or kernels) to the input data to detect features such as patterns. Specifically, the filters are slid over the input data, performing element-wise multiplication, and summing the results to produce a feature map.

[0091] The feature map is then fed to the pooling layers, which may reduce the dimensions of the feature maps by summarizing local regions. The dimensions of the feature maps may be reduced by using techniques such as max pooling or average pooling. This may reduce computational complexity and prevent overfitting. After the convolutional and pooling layers, dense layers (e.g., fully connected layers) are applied, where each node is connected to every node in the previous layer. The dense layers may integrate high-level features learned by the convolutional layers to perform regression. The output layer may use a linear activation function or no activation function to provide estimates of the positions (x,y,z) of EM sensing elements in the sensing volume.

[0092] A prototype was developed to evaluate the localization accuracy and computational performance of a convolutional neural network model as applied to estimating positions of an EM sensor in an EM field generated by an antenna in a sensing volume. The convolutional neural network architecture included a fully connected layer. The inputs to the convolutional neural network model included normalized real and imaginary components of the simulated strength of an EM field emitted from an antenna, e.g., an antenna 145 of FIG. 1, on a location board, e.g., location board 141 of FIG. 1, 6400 training locations, and 1600 testing locations. The targets of the convolutional neural network model included the x, y, z positions in the simulation. The x, y, z positions estimated from the testing location dataset closely tracked the true x, y, z positions. Thus, the prototype demonstrated that good localization accuracy could be achieved with reduced computation burden using a convolutional neural network model.

[0093] If there is appropriate operating room mapping, the training of the neural network models may lead to accurate localization results. However, if there is interference in environment, e.g., interference from medical equipment including the workstation 1301 and imaging system 1315, the trained neural network model may output inaccurateAttorney Docket No. A0012826W001 localization estimates. In aspects, the systems and methods of the disclosure may embed field distortion in the neural network models to reject interference.

[0094] FIG. 12 is an example of an EM sensor mapping method that accounts for the field distortions caused by different clinical scenarios, each with different equipment configurations. For example, clinical environment mapping 1 may introduce field distortion Di, clinical environment mapping 2 may introduce field distortion D2, and clinical environment mapping 3 may introduce field distortion D3. In aspects, to account for these distortions, the systems and methods of the disclosure train neural network models with the effects of the field distortions embedded into the training data. The neural network models, for example, may be represented in simple form as follows: x, y, z, yaw, pitch =f(Di(Bx, By, Bz)), ' x, y, z, yaw, pitch =f(D2(Bx, By, Bz)),' and x, y, z, yaw, pitch = f(D3(Bx, By, Bz)), where Di, D2, and D3 represent the embedded field distortion associated with each of three possible clinical scenarios.

[0095] In some aspects, the user interface 1316 may allow a clinician to select a clinical scenario or an approximate clinical scenario from a menu of options. Based on this selection, the application 1318 may use a corresponding convolutional neural network that has been trained to handle the specific field distortions associated with the selected clinical scenario. A set of convolutional neural networks may be trained using training datasets that incorporate real-world use conditions to sufficiently capture possible clinical scenarios or environments.

[0096] Reference is now made to FIG. 13, which is a schematic diagram of a system 1300 configured for use with the methods of the disclosure including the methods of FIGS. 8 and 10. The system 1300 may include a workstation 1301, and optionally an imaging system 1315, e.g., a fluoroscopic imaging system and / or a CT imaging system for capturing image data, e.g., radiographic images. In some aspects, the workstation 1301 may be coupled with the imaging system 1315, directly or indirectly, e.g., by wireless communication. The workstation 1301 may include a memory 1302, a processor 1304, a display 1306, and an input device 1310. The processor 1304 may include one or more hardware processors. The workstation 1301 may optionally include an output module 1312 and a network interface 1308. The memory 1302 may store an application 1318 and data 1314. The data 1314 may include neural network modes, sensed position data, and estimated position data. The application 1318 may include instructions executable by theAttorney Docket No. A0012826W001 processor 1304 for executing the methods of the disclosure including the methods of FIGS. 8 and 10.

[0097] The application 1318 may further include a user interface 1316. The memory 1302 may also include image data such as CT image data, fluoroscopic image data, or CTCB image data. The processor 1304 may be coupled with the memory 1302, the display 1306, the input device 1310, the output module 1312, the network interface 1308, and the imaging system 1315. The workstation 1301 may be a stationary computer system, such as a personal computer, or a portable computer system such as a tablet computer. The workstation 1301 may embed multiple computers.

[0098] The memory 1302 may include any non-transitory computer-readable storage media for storing data and / or software including instructions that are executable by the processor 1304 and which control the operation of the workstation 1301 and, in some aspects, may also control the operation of the imaging system 1315. The imaging system 1315 may be used to capture a sequence of preoperative and / or intraoperative images of a portion of a patient’s body, e.g., the lungs or the heart. Optionally, the imaging system 1315 may include a fluoroscopic imaging system that captures a sequence of fluoroscopic images based on which a fluoroscopic 3D reconstruction is generated and / or to capture a live 2D fluoroscopic view to confirm placement of a medical tool. In one aspect, the memory 1302 may include one or more storage devices such as solid-state storage devices, e.g., flash memory chips. Alternatively, or in addition to the one or more solid-state storage devices, the memory 1302 may include one or more mass storage devices connected to the processor 1304 through a mass storage controller (not shown) and a communications bus (not shown).

[0099] Although the description of computer-readable media contained herein refers to solid-state storage, it should be appreciated by those skilled in the art that computer- readable storage media can be any available media that can be accessed by the processor 1304. That is, computer readable storage media may include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media may include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD, Blu-Ray or other optical storage, magnetic cassettes,Attorney Docket No. A0012826W001 magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store the desired information, and which may be accessed by workstation 1301.

[0100] The application 1318 may, when executed by the processor 1304, cause the display 1306 to present the user interface 1316. The user interface 1316 may be configured to present to the user a single screen including a three-dimensional (3D) view of a 3D model of a target from the perspective of a tip of a medical tool, a live two-dimensional (2D) view showing the medical tool, and a target mark, which corresponds to the 3D model of the target, overlaid on the live 2D fluoroscopic view. The user interface 1316 may be further configured to display the target mark in different colors depending on whether the medical tool tip is aligned with the target in three dimensions.

[0101] The network interface 1308 may be configured to connect to a network such as a local area network (LAN) consisting of a wired network and / or a wireless network, a wide area network (WAN), a wireless mobile network, a Bluetooth network, and / or the Internet. The network interface 1308 may be used to connect between the workstation 1301 and the imaging system 1315. The network interface 1308 may also be used to receive the data 1314. The input device 1310 may be any device by which a user may interact with the workstation 1301, such as, for example, a mouse, keyboard, foot pedal, touch screen, and / or voice interface. The output module 1312 may include any connectivity port or bus, such as, for example, parallel ports, serial ports, universal serial buses (USB), or any other similar connectivity port known to those skilled in the art. From the foregoing and with reference to the various figures, those skilled in the art will appreciate that certain modifications can be made to the disclosure without departing from the scope of the disclosure.

[0102] As described herein, each of the FPCs may include one or more sensing elements. Each of the sensing elements may be placed at a distinct node of a multi-nodal catheter. FIGS. 14A and 14B are a layout diagram of an EM sensor circuit having three FPCs 1401-1403, each including four sensing elements 1411-1413, 1421-1423, 1431— 1433, and 1441-1443. Sensing elements 1411, 1421, 1431, and 1441 are electrically coupled to a first electrical connector 1404 via traces of the FPC 1401. Similarly, sensing elements 1412, 1422, 1432, and 1442 are electrically coupled to a second electrical connector 1405 via traces of the FPC 1402, and sensing elements 1413, 1423, 1433, andAttorney Docket No. A0012826W0011443 are electrically coupled to a third electrical connector 1406 via traces of the FPC 1403.

[0103] In aspects, each sensing element triplet 1411-1413, 1421-1423, 1431-1433, and 1441-1443 may be designed to align with a node of a four-node catheter. In aspects, each of the FPCs 1401-1403 may have distinct substrates or may share a common substrate.

[0104] Aspects of the disclosure may be applied to any endoluminal procedure requiring precise navigation of a flexible catheter within the lumen of a tubular structure in the body, such as blood vessels, the gastrointestinal tract, or other hollow organs. For example, aspects of the disclosure may be applied to cardiac procedures requiring precise navigation of a flexible catheter through the heart and blood vessels. The cardiac procedures may include pacemaker placement, mitral valve repair (e.g., by deployment of a MitraClip) or replacement, cardiac catheterization (e.g., coronary angiography), percutaneous coronary intervention (PCI), electrophysiology (EP) study, EP or cardiac ablation, transcatheter aortic valve replacement (TAVR), transcatheter mitral valve replacement (TMVR), pulmonary artery catheterization (e.g., using a Swan-Ganz catheter), transseptal puncture, left atrial appendage (LAA) closure, or endomyocardial biopsy.

[0105] As used herein, “EM sensing element” may refer to any component, structure, or device that senses electromagnetic phenomena including but not limited to electric fields, magnetic fields, or any combination of the two. This may include, but is not limited to, magnetic field sensing elements (e.g., TMR elements, giant magnetoresistance (GMR) elements, Hall-effect sensors, or inductive coils), electric field sensors (such as capacitive or impedance-based sensors), sensors that sense electromagnetic waves in the radio, optical, or infrared spectra, or other magnetic field, electric field, or electromagnetic field sensing components, structures, or devices.

[0106] As used herein, “first EM sensing elements,” “second EM sensing elements,” and “third EM sensing elements” may refer to EM sensing elements in either one direction (e.g., the horizontal direction) or another different direction (e.g., the vertical direction or the perpendicular direction).

[0107] According to some examples of the present technology, the flexible catheter may comprise an elongate shaft formed of a plurality of coaxially arranged elongateAttorney Docket No. A0012826W001 tubular members that comprises strips or tendons cut from and / or otherwise formed of one of the tubular members to steer or articulate the flexible catheter. These tendons or strips eliminate the need for, and replace, the steering or pull wires for steering and / or articulating the flexible catheter. FIG. 15 A, for example, shows a shaft 1500 for use with the flexible catheter of the present technology, the shaft 1500 comprising an outer elongate tubular member 1530 (or “outer member 1530”), an intermediate elongate tubular member 1520 (or “intermediate elongate tubular member 1520”) disposed within a lumen of the outer member 1530, and an inner elongate tubular member 1510 (or “inner member 1510”) disposed within a lumen of the intermediate member 1520. In some implementations, the shaft 1500 may comprise only two elongate tubular members. In yet other examples, the shaft 1500 may comprise more than three elongate tubular members such that the shaft 1500 includes multiple intermediate members.

[0108] In any case, the shaft 1500 includes a proximal portion 1500a configured to be coupled to a steering interface (manual or robotic), a distal portion 1500b, and an intermediate portion 1500c extending therebetween. The distal portion 1500b and / or the intermediate portion 1500c is configured to be positioned in a body and includes at least one steerable region 1502 configured to be selectively deflected in response to a steering input at the proximal portion 1500a. The steerable region 1502 may be configured to deflect in a single plane (e.g., bi-directional) or multiple planes (e.g., multi-dimensional). While only a single steerable region 1502 is shown in FIG. 15 A, in some examples the shaft 1500 may have two or more steerable regions 1502, each independently controllable at the proximal portion 1500a. The intermediate portion 1500c of the shaft 1500 may be passively bendable and / or flexible (e.g., to accommodate one or more turns in the body lumen), or may be rigid (and thus not bendable).

[0109] As shown in the exploded view of the shaft 1500 in FIG. 15B, each of the inner, intermediate, and outer members 1510, 1520, and 1530 can have distal portions 1510b, 1520b, 1530b that include one or more flexible regions 1518, 1528, 1538 that are aligned with one another when the inner, intermediate, and outer members 1510, 1520, and 1530 are assembled into the shaft 1500. Each of the inner, intermediate, and outer members 1510, 1520, 1530 also include a respective intermediate portion 1510c, 1520c, 1530c that may be passively bendable and / or flexible (e.g., to accommodate one or more turns in the body lumen) but not steerable. The flexible regions 1518, 1528, 1538 mayAttorney Docket No. A0012826W001 include a plurality of cuts and / or slits, such as circumferentially extending cuts and / or helically extending cuts, etc., configured to improve the flexibility and / or bendability of the respective elongate member along that region. Likewise, each of the intermediate portions 1510c, 1520c, 1530c may include a plurality of cuts and / or slits, such as circumferentially extending cuts and / or helically extending cuts, etc., configured to improve the flexibility and / or bendability of the respective elongate member along that region.

[0110] In some embodiments, one, some, or all of the inner, intermediate, and outer members 1510, 1520, and 1530 include rigid bands at their respective distal end portions 1514, 1524, 1534. The rigid bands can be fixed to one another (and thus the inner, intermediate, and outer members 1510, 1520, and 1530 can be fixed to one another) via welding or other techniques. The inner, intermediate, and outer members 1510, 1520, 1530 can also be fixed to one another at one or more other locations along the length of the shaft 1500, such as along their respective intermediate portions 1510c, 1520c, 1530c and / or proximal portions 1510a, 1520a, 1530a. In some embodiments, one, some, or all of the inner, intermediate, and outer members 1510, 1520, and 1530 include rigid bands at their respective proximal end portions 1512, 1522, 1532.[OHl] In further embodiments, the distal portions 1510b, 1520b, 1530b of the inner, intermediate, and outer members 1510, 1520, and 1530, respectively, have only a single flexible region, corresponding to a single steerable region 1502 on the shaft 1500. In those embodiments in which the shaft 1500 has multiple distinct steerable regions 1502, each of the distal portions 1510b, 1520b, 1530b has multiple flexible regions. In such embodiments, the flexible regions along a given member may be separated from an axially adjacent flexible region by a rigid band.

[0112] As shown in FIG. 15B, the intermediate member 1520 may further include one or more tendons 1526, each associated with and / or extending through a respective flexible region 1528 such that axial movement of a tendon 1526 causes articulation of its corresponding flexible region 1528, as well as articulation of the flexible regions 1518 and 1538 of the inner and outer members 1510, 1530. As such, actuation of the tendons 1526 causes articulation of the steerable region 1502. In some embodiments, a distal end of a tendon 1526 may be coupled to (e.g., integrally formed with, or attached such as via at least one weld) a rigid band just distal of the corresponding flexible region 1528. ForAttorney Docket No. A0012826W001 example, in FIG. 15B, the distal end of tendon 1526 is integral with the rigid band comprising the distal end portion 1524 of the intermediate member 1520. In any case, axial movement of the tendon 1526 can be controlled via a steering interface directly or indirectly coupled to a proximal end of the tendon 1526.

[0113] A tendon 1526 may be formed as a longitudinal member or strip extending longitudinally along at least a portion of a wall of the intermediate member 1520, for all or a portion of the length of the intermediate member 1520. A tendon 1526 may be cut from the sidewall of the intermediate member 1520 and thus integral with the sidewall of the intermediate member 1520. A tendon 1526 may be configured to move generally in a longitudinal direction within a respective slot 1529 defined by the intermediate member 1520 (e.g., cut from the wall of the intermediate member 1520). On either side of a tendon 1526 or slot 1529 can be a tendon-adjacent portion 1527 of the sidewall that is not configured to move longitudinally and / or be actuated and is separated from the tendon 1526 by a slit. The slit may, for example, be formed by removal of material such as laser cutting, where the width of the slit corresponds to the width of the laser beam. The slot 1529 may also be formed via laser cutting or another material removal process.

[0114] In the example shown in FIG. 15B, the tendon 1526 and slot 1529 are generally linear and extend in a longitudinal direction. However, in some embodiments the tendon 1526 and slot 1529 may be any suitable shape generally extending in a longitudinal direction, such as a helical shape that wraps in a spiral manner around the wall of the intermediate member 1520. For example, FIG. 16 illustrates an example intermediate member 1620 (an example of intermediate member 1520) that includes multiple helical tendons 1626 that have been obtained after making helical slits 1625 (only a few labeled) in the wall of the intermediate member 1620. The tendon-adjacent portions 1627 of the sidewall may also extend helically. The spiraling path taken by the tendons 1626 beneficially compensates path length changes in the tendons 1626 caused by any bending of the intermediate portion of the intermediate member 1620, for instance when the shaft is inserted in a tortuous path.

[0115] Referring again to FIG. 15B, in some embodiments, a tendon 1526 may be actuated via an actuating input applied to a feature of the outer member 1530. For example, in some embodiments the outer member 1530 may include at least one slider 1536 configured to move within a respective slot 1539 defined by the outer member 1530.Attorney Docket No. A0012826W001Each slider 1536 may be coupled to an underlying tendon 1526 (e.g., via epoxy, spot welding, etc.) such that movement of the slider 1536 results in movement of its associated tendon 1526. The outer member 1530 may include multiple sliders 1536, where each slider 1536 is coupled to a respective tendon 1526, such as selective actuation of the sliders 1536 results in selective movement of tendons 1526 and thus selective articulation of the articulable regions of the members of the shaft 1500.

[0116] Although only one tendon 1526 is visible in the example shown in FIG. 15B, the intermediate member 1520 may include any suitable number of tendons 1526. For example, in some embodiments a second tendon 1526 may be located on a circumferentially opposite side of the intermediate member 1520. For example, two tendons 1526 may be circumferentially offset about 180 degrees from one another and operate in an antagonistic manner to articulate a flexible region 1528 to which the two tendons 1526 are coupled. For instance, a first tendon 1526 may be pulled proximally in a longitudinal direction (and / or a second tendon 1526 arranged circumferentially opposite to the first tendon 1526 may be pushed distally in a longitudinal direction) to cause bending of the flexible region 1528 in a first direction. The second tendon 1526 may be pulled proximally in a longitudinal direction (and / or the first tendon 1526 may be pushed distally in a longitudinal direction) to cause bending of the articulable region 1528 in a second direction (e.g., opposite the first direction).

[0117] Additionally, or alternatively, the intermediate member 1520 may include one or more tendons 1526 coupled to different flexible regions 1528. For example, a first pair of antagonistic tendons 1526 may be coupled to a first flexible region 1528 (e.g., arranged at a first axial location along the length of the intermediate member 1520), and a second pair of antagonistic tendons 1526 may be coupled to a second flexible region 1528 (e.g., arranged at a second axial location along the length of the intermediate member 1520). In those embodiments in which the intermediate member 1520 includes tendons 1526 coupled to different flexible regions 1528, the tendon(s) associated with one flexible region 1528 may have a different length than the tendon(s) associated with another flexible region 1528 since the flexible regions 1528 are disposed at different locations along the length of the shaft 1500.

[0118] The inner member 1510, the intermediate member 1520, and the outer member 1530 may be assembled to form a combined unit that defines the shaft 1500. For example,Attorney Docket No. A0012826W001 the inner member 1510 may be inserted into the intermediate member 1520, and the combined inner member 1510 and intermediate member 1520 subassembly may be inserted into the outer member 1530, although any order of insertion may be possible. Various features of the elongate members of the shaft 1500 (e.g., articulable regions, tendons, sliders, slots, etc.) may be formed through removal of material from the wall of each respective tube forming the members of the shaft 1500. For example, starting from a cylindrical tube with desired inner and outer diameters (and desired wall thickness), various features of an elongate member may be formed by removing parts of the wall of the cylindrical tube, such as by laser cutting or water cutting. However, in some embodiments the elongate members may be formed through injection molding, plating techniques, 3D printing or other material deposition process, photochemical etching, deep pressing, conventional chipping techniques such as drilling or milling, and / or any suitable technique. In some embodiments, removal of material may be performed through laser cutting, which may allow for a very accurate and clean removal of material under reasonable economic conditions.

[0119] The inner member 1510, intermediate member 1520, and / or outer member 1530 may be formed from any suitable rigid material such as stainless steel, cobaltchromium, shape memory alloy such as Nitinol®, plastic, polymer, composites and / or other materials. Additionally, or alternatively, the elongate member(s) (e.g., inner member 1510, intermediate member 1520, and / or outer member 1530) can be made by a 3D printing process or other known material deposition processes.

[0120] Various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the techniques). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a medical device.

[0121] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software,Attorney Docket No. A0012826W001 the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0122] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0123] The invention may be further described by reference to the following numbered paragraphs:1. A catheter system comprising: a flexible catheter having at least two nodes; at least one flexible substrate coupled to the flexible catheter; at least two first electromagnetic (EM) sensing elements coupled to the at least one flexible substrate and disposed at the at least two nodes, respectively; at least two second EM sensing elements (a) coupled to the at least one flexible substrate in orientations different from the orientations of the at least two first EM sensing elements, respectively, and (b) disposed at the at least two nodes, respectively; and at least two third EM sensing elements (a) coupled to the at least one flexible substrate in orientations different from the orientations of the at least two first EM sensing elements, respectively, and the at least two second EM sensing elements, respectively, and (b) disposed at the at least two nodes, respectively.2. The catheter system according to paragraph 1, wherein the at least two first EM sensing elements, the at least two second EM sensing elements, and the at least two third EM sensing elements are tunnel magnetoresistance (TMR) sensing elements.Attorney Docket No. A0012826W0013. The catheter system according to any of the preceding paragraphs, wherein the at least one flexible substrate is a polyurethane substrate or a polyimide substrate.4. The catheter system according to any of the preceding paragraphs, wherein the at least two first EM sensing elements are oriented with respect to the at least two second EM sensing elements such that a first electromagnetic field direction sensed by the at least two first EM sensing elements are perpendicular to a second electromagnetic field direction sensed by the at least two second EM sensing elements.5. The catheter system according to any of the preceding paragraphs, wherein the at least two first EM sensing elements include at least three first EM sensing elements.6. The catheter system according to any of the preceding paragraphs, wherein the at least one flexible substrate includes three flexible substrates to which the at least two first EM sensing elements, the at least two second EM sensing elements, and the at least two third EM sensing elements are coupled, respectively.7. The catheter system according to any of the preceding paragraphs, wherein the at least two nodes define inflection points of the flexible catheter.8. The catheter system according to any of the preceding paragraphs, wherein the at least one flexible substrate is disposed within the flexible catheter.9. The catheter system according to any of the preceding paragraphs, wherein the flexible substrate is coupled to an inner or outer surface of the flexible catheter.10. A system comprising: a flexible catheter; a flexible substrate coupled to the flexible catheter; first electromagnetic (EM) sensing elements coupled to the flexible substrate in different orientations at a first longitudinal position of the flexible catheter;Attorney Docket No. A0012826W001 second EM sensing elements coupled to the flexible substrate in different orientations at a second longitudinal position of the flexible catheter; an EM field generator; a processor in communication with the first and second EM sensing elements; and a memory having stored thereon instructions, which when executed by the processor, causes the processor to: build a model of the positions of the first and second EM sensing elements as a function of EM fields; receive EM field signals from the first and second EM sensing elements; and determine positions of the first and second EM sensing elements based on the model and the received EM field signals.11. The system according to paragraph 10, wherein the flexible substrate is disposed within the flexible catheter.12. The system according to any of the preceding paragraphs, wherein the instructions, when executed by the processor, further cause the processor to train the model using at least one of preoperative or intraoperative EM field signals from the first and second EM sensing elements.13. The system according to any of the preceding paragraphs, wherein the instructions, when executed by the processor, further cause the processor to preoperatively train different models using EM field signals from the first and second EM sensing elements in different operative environments introducing different EM field distortion.14. The system according to any of the preceding paragraphs, wherein the instructions, when executed by the processor, further cause the processor to select a model trained based on a current operative environment.Attorney Docket No. A0012826W00115. The system according to any of the preceding paragraphs, wherein the instructions, when executed by the processor, further cause the processor to determine a shape of a portion of the catheter based on the determined positions.16. A system comprising: a flexible catheter; a flexible substrate coupled to the flexible catheter; first electromagnetic (EM) sensing elements coupled to the flexible substrate in different orientations at a first longitudinal position of the flexible catheter; second EM sensing elements coupled to the flexible substrate in different orientations at a second longitudinal position of the flexible catheter; an EM field generator; a processor in communication with the EM field generator and the first and second EM sensing elements; and a memory having stored thereon instructions, which when executed by the processor, causes the processor to: determine EM field strengths at positions within a sensing volume, yielding determined EM field strengths; build a neural network model based on the positions within the sensing volume and the determined EM field strengths; receive EM field signals from the first and second EM sensing elements; estimate positions of the first and second EM sensing elements by processing the EM field signals with the neural network model, yielding estimated positions; and display a representation of the flexible catheter based on the estimated positions.17. The system according to paragraph 16, wherein determining the EM field strengths includes at least one of measuring the EM field strengths or estimating the EM field strengths.Attorney Docket No. A0012826W00118. The system according to any of the preceding paragraphs, wherein the EM field generator includes at least two antennas.19. The system according to any of the preceding paragraphs, wherein the instructions, when executed by the processor, further cause the processor to cause the EM field generator to generate at least two different EM field signals to be emitted by the at least two antennas.20. The system according to any of the preceding paragraphs, further comprising third EM sensing elements coupled to the flexible substrate in different orientations at a third longitudinal position of the flexible catheter.

Claims

Attorney Docket No. A0012826W001WHAT IS CLAIMED IS:

1. A catheter system comprising: a flexible catheter having at least two nodes; at least one flexible substrate coupled to the flexible catheter; at least two first electromagnetic (EM) sensing elements coupled to the at least one flexible substrate and disposed at the at least two nodes, respectively; at least two second EM sensing elements (a) coupled to the at least one flexible substrate in orientations different from the orientations of the at least two first EM sensing elements, respectively, and (b) disposed at the at least two nodes, respectively; and at least two third EM sensing elements (a) coupled to the at least one flexible substrate in orientations different from the orientations of the at least two first EM sensing elements, respectively, and the at least two second EM sensing elements, respectively, and (b) disposed at the at least two nodes, respectively.

2. The catheter system according to claim 1, wherein the at least two first EM sensing elements, the at least two second EM sensing elements, and the at least two third EM sensing elements are tunnel magnetoresistance (TMR) sensing elements.

3. The catheter system according to any of the preceding claims, wherein the at least one flexible substrate is a polyurethane substrate or a polyimide substrate.

4. The catheter system according to any of the preceding claims, wherein the at least two first EM sensing elements are oriented with respect to the at least two second EM sensing elements such that a first electromagnetic field direction sensed by the at least two first EM sensing elements are perpendicular to a second electromagnetic field direction sensed by the at least two second EM sensing elements.

5. The catheter system according to any of the preceding claims, wherein the at least two first EM sensing elements include at least three first EM sensing elements.Attorney Docket No. A0012826W0016. The catheter system according to any of the preceding claims, wherein the at least one flexible substrate includes three flexible substrates to which the at least two first EM sensing elements, the at least two second EM sensing elements, and the at least two third EM sensing elements are coupled, respectively.

7. The catheter system according to any of the preceding claims, wherein the at least two nodes define inflection points of the flexible catheter.

8. The catheter system according to any of the preceding claims, wherein the at least one flexible substrate is disposed within the flexible catheter.

9. The catheter system according to any of the preceding claims, wherein the flexible substrate is coupled to an inner or outer surface of the flexible catheter.

10. A system comprising: a flexible catheter; a flexible substrate coupled to the flexible catheter; first electromagnetic (EM) sensing elements coupled to the flexible substrate in different orientations at a first longitudinal position of the flexible catheter; second EM sensing elements coupled to the flexible substrate in different orientations at a second longitudinal position of the flexible catheter; an EM field generator; a processor in communication with the first and second EM sensing elements; and a memory having stored thereon instructions, which when executed by the processor, causes the processor to: build a model of the positions of the first and second EM sensing elements as a function of EM fields; receive EM field signals from the first and second EM sensing elements; and determine positions of the first and second EM sensing elements based on the model and the received EM field signals.Attorney Docket No. A0012826W00111. The system according to claim 10, wherein the instructions, when executed by the processor, further cause the processor to train the model using at least one of preoperative or intraoperative EM field signals from the first and second EM sensing elements.

12. The system according to any of the preceding claims, wherein the instructions, when executed by the processor, further cause the processor to preoperatively train different models using EM field signals from the first and second EM sensing elements in different operative environments introducing different EM field distortion.

13. The system according to any of the preceding claims, wherein the instructions, when executed by the processor, further cause the processor to select a model trained based on a current operative environment.

14. The system according to any of the preceding claims, wherein the instructions, when executed by the processor, further cause the processor to determine a shape of a portion of the catheter based on the determined positions.

15. A system comprising: a flexible catheter; a flexible substrate coupled to the flexible catheter; first electromagnetic (EM) sensing elements coupled to the flexible substrate in different orientations at a first longitudinal position of the flexible catheter; second EM sensing elements coupled to the flexible substrate in different orientations at a second longitudinal position of the flexible catheter; an EM field generator; a processor in communication with the EM field generator and the first and second EM sensing elements; and a memory having stored thereon instructions, which when executed by the processor, causes the processor to: determine EM field strengths at positions within a sensing volume, yielding determined EM field strengths;Attorney Docket No. A0012826W001 build a neural network model based on the positions within the sensing volume and the determined EM field strengths; receive EM field signals from the first and second EM sensing elements; estimate positions of the first and second EM sensing elements by processing the EM field signals with the neural network model, yielding estimated positions; and display a representation of the flexible catheter based on the estimated positions.

Images