Registering three-dimensional shape data to two-dimensional image data
The system registers 3D shape data to 2D image data using a flexible elongate device to predict pose and expand scanning angles, addressing limitations of fiducial marker reliance and manual segmentation in fluoroscopic imaging, thereby improving tomosynthesis accuracy and scanning range.
Patent Information
- Application Number
- PCT/US2026/028110
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-15
- Filing Date
- 2026-05-14
- Publication Date
- 2026-11-19
AI Technical Summary
Current medical imaging techniques relying on fiducial markers for pose estimation in fluoroscopic imaging are limited by the requirement that markers remain within the field of view, restricting scanning angles and image utilization, and manual segmentation methods are time-intensive and variable.
A system and method for registering 3D shape data to 2D image data using a flexible elongate device, involving extraction, pose prediction, and projection matrix application to enhance registration accuracy and expand scanning angles beyond conventional limits.
Enables accurate 3D reconstruction and expanded scanning angles, improving the quality and accuracy of tomosynthesis by predicting the pose of imaging devices without relying on fiducial markers, thus enhancing medical procedure navigation and data registration.
Smart Images

Figure US2026028110_19112026_PF_FP_ABST
Abstract
Description
Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 REGISTERING THREE-DIMENSIONAL SHAPE DATA TO TWO-DIMENSIONAL IMAGE DATA CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to and the benefit of the filing date of provisional U.S. Patent Application No. 63 / 806,328 entitled “REGISTERING THREE-DIMENSIONAL SHAPE DATA TO TWO-DIMENSIONAL IMAGE DATA,” filed on May 15, 2025. The entire contents of the provisional application are hereby expressly incorporated herein by reference.FIELD
[0002] The present disclosure relates to registering three-dimensional (3D) shape data to two-dimensional (2D) image data, and more particularly, to systems and methods for registering 3D shape sensor data of a flexible elongate device to frames of 2D image data with optimizations to reduce registration loss.BACKGROUND
[0003] In medical imaging involving fluoroscopic imaging devices, such as an x-ray imaging device having a C-arm, accurate pose estimation plays a critical role in performing, for example, tomosynthesis on image frames captured by the X-ray imaging device.Traditionally, pose estimation is facilitated through the use of fiducial markers or boards placed in the imaging field, serving as reference points to calibrate and orient the image data with respect to the patient's anatomy and the imaging device's geometry. However, these methods can be limited by the requirement that fiducials remain within the fluoroscopic imaging device’s field of view for accurate pose estimation, which can restrict the usable range of scanning angles of the fluoroscopic imaging device, in-tum limiting the number of captured images that can be utilized when performing 3D reconstruction of the image subject using tomosynthesis techniques.
[0004] Additionally, the registration of medical instruments, such as a flexible elongate device (e.g., a catheter), within the imaging data is a challenge due to the complexity of mapping 3D objects into a 2D image space. Current techniques rely heavily on manual segmentation and feature extraction processes, which can be time-intensive and subject to variability. These feature extraction techniques typically rely on visible fiducial markers for registering the frame of 2D image data to a 3D coordinate system. However, these externalIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 reference points are not available in every scenario. Thus, in these scenarios, the imaging system may be unable to accurately localize a pose of the catheter during the procedure.
[0005] Accordingly, a need exists for improved techniques for pose estimation and instrument registration in the field of fluoroscopic imaging.SUMMARY OF THE DISCLOSURE
[0006] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.
[0007] In some examples, a system for registering three-dimensional (3D) shape data to two-dimensional (2D) image data comprises one or more processors and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to obtain, via an imaging device, a frame of 2D image data depicting a flexible elongate device. The instructions, when executed by the one or more processors, may cause the one or more processors to extract the flexible elongate device from the frame of 2D image data, and predict a pose of the imaging device associated with the frame of 2D image data. The instructions, when executed by the one or more processors, may cause the one or more processors to obtain, via a shape sensor, 3D shape sensor data representative of a shape of the flexible elongate device, and generate a 2D projection of the flexible elongate device by applying, to the 3D shape sensor data, a projection matrix associated with the predicted pose. Based on the extracted flexible elongate device and the generated 2D projection of the flexible elongate device, the instructions, when executed by the one or more processors, may cause the one or more processors to register the 3D shape sensor data to the frame of 2D image data.
[0008] In other examples, a computer-implemented method for registering three-dimensional (3D) shape data to two-dimensional (2D) image data comprises obtaining, by one or more processors via an imaging device, a frame of 2D image data depicting a flexible elongate device. The computer-implemented method may comprise extracting, by the one or more processors, the flexible elongate device from the frame of 2D image data, and predicting, by the one or more processors, a pose of the imaging device associated with the frame of 2D image data. The computer-implemented method may comprise obtaining, by the one or more processors via a shape sensor, 3D shape sensor data representative of a shape of the flexible elongate device, and generating, by the one or more processors, a 2D projection of the flexible elongate device by applying, to the 3D shape sensor data, a projection matrixIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 associated with the predicted pose. Based on the extracted flexible elongate device and the generated 2D projection of the flexible elongate device, the computer-implemented method may comprise registering, by the one or more processors, the 3D shape sensor data to the frame of 2D image data.
[0009] In other examples, a tangible, non-transitory, computer readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain, via an imaging device, a frame of two -dimensional (2D) image data depicting a flexible elongate device. The instructions may further cause the one or more processors to extract the flexible elongate device from the frame of 2D image data, and predict a pose of the imaging device associated with the frame of 2D image data. The instructions may further cause the one or more processors to obtain, via a shape sensor, three-dimensional (3D) shape sensor data representative of a shape of the flexible elongate device, and generate a 2D projection of the flexible elongate device by applying, to the 3D shape sensor data, a projection matrix associated with the predicted pose. Based on the extracted flexible elongate device and the generated 2D projection of the flexible elongate device, the instructions may further cause the one or more processors to register the 3D shape sensor data to the frame of 2D image data.BRIEF DESCRIPTION OF THE DRAWINGS
[0010] FIG. 1 depicts an example system for registering 3D shape data to 2D image data, according to embodiments.
[0011] FIG. 2A is a simplified diagram of a flexible elongate device disposed within an anatomical structure, according to embodiments.
[0012] FIGS. 2B, 2C are simplified diagrams depicting an example imaging geometry for imaging a subject, according to embodiments.
[0013] FIG. 2D is a simplified three-dimensional representation of a flexible elongate device disposed within an anatomical structure within the imaging geometry of FIGS. 2B, 2C, according to embodiments.
[0014] FIG. 2E depicts a portion of a projection image of a flexible elongate device disposed within an anatomical structure overlay ed with fiducial markings, according to embodiments.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0015] FIGS. 2F, 2G schematically illustrate projections of an example fiducial plate at two projection angles, according to embodiments.
[0016] FIG. 3A schematically illustrates an example multi-projection image acquisition geometry for a flexible elongate device disposed within an anatomical structure, according to embodiments.
[0017] FIGS. 3B, 3C are example projection images for the image acquisition geometry of FIG. 3A, according to embodiments.
[0018] FIG. 4A depicts a rigid body in two example coordinate systems, according to embodiments.
[0019] FIG. 4B schematically illustrates an example coordinate transformation process, according to embodiments.
[0020] FIG. 4C depicts an example geometry for flexible elongate device extraction and coordinate registration using shape data and projection images, according to embodiments.
[0021] FIG. 5A schematically illustrates an example process for 2D / 3D registration of 2D fluoroscopic images and 3D shape sensor points of a flexible elongate device sensor data, according to embodiments.
[0022] FIG. 5B schematically illustrates an example process for regularizing a set of predicted poses of the imaging unit derived based on the registration process of FIG. 5 A.
[0023] FIG. 6 depicts a computer-implemented method 600 for registering three-dimensional (3D) shape data to two-dimensional (2D) image data, according to embodiments.
[0024] FIG. 7 is a simplified diagram of a medical system, according to embodiments.
[0025] FIG. 8A is a simplified diagram of a medical instrument, according to embodiments.
[0026] FIG. 8B is a simplified diagram of a medical instrument including a medical tool within an elongate device, according to embodiments.
[0027] FIGS. 9 A and 9B are simplified diagrams of side views of a patient coordinate space including a medical instrument mounted on an insertion assembly, according to embodiments.
[0028] Examples of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, whereinIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 showings therein are for purposes of illustrating examples of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION
[0029] In the following description, specific details are set forth describing some examples consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the examples. It will be apparent, however, to one skilled in the art that some examples may be practiced without some or all of these specific details. The specific examples disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one example may be incorporated into other examples unless specifically described otherwise or if the one or more features would make an example non-functional. In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the examples.
[0030] The disclosed technology addresses a significant challenge in the field of medical imaging, such as in the context of tomosynthesis and fluoroscopic X-ray imaging that may rely on a fiducial board to estimate the pose of an imaging sensor (and / or a C-arm to which the imaging sensor is affixed) used to capture a fluoroscopic image of a subject. In particular, the disclosed techniques provide an improved catheter-based pose estimation that expands the ability to perform accurate 3D reconstruction (e.g., tomosynthesis) to additional imaging locations that may be outside the field of view respective to the fiducial board. Moreover, when the instant techniques are utilized with a fiducial board, the instant techniques can expand the set of projection angles beyond what the fiducial board conventionally supports to further enhance the quality and accuracy of tomosynthesis techniques. It should be appreciated that while the instant disclosure generally refers applications to medical imaging, similar techniques may be implemented in similar imaging systems used to image subjects for non-medical purposes.
[0031] The disclosed techniques include obtaining a frame of 2D image data from an imaging device (e.g., fluoroscopic imaging device) depicting a flexible elongate device (e.g. catheter), and extracting the flexible elongate device from the frame of 2D image data. The techniques may include predicting a pose of the imaging device associated with the frame ofIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 2D image data. The techniques may further include obtaining, via a shape sensor, 3D shape sensor data representative of a shape of the flexible elongate device, and generating a 2D projection of the flexible elongate device by applying a projection matrix associated with the predicted pose to the 3D shape sensor data. Based on the extracted flexible elongate device and the generated 2D projection of the flexible elongate device, the techniques may register the 3D shape sensor data to the frame of 2D image data (e.g., by calculating a transformation matrix that aligns the extracted depiction of the flexible elongate device with the 2D projection of the flexible elongate device).
[0032] Conventional tomosynthesis techniques estimate the pose of the imaging device for a given frame of image data using a fiducial board placed beneath the patient during image acquisition, however, not every imaging site has fiducial boards on-hand. Moreover, fiducial boards may restrict the working range of projection angles that can be utilized when performing tomosynthesis to reconstruct the imaged subject. Thus, the catheter-based pose estimation of the disclosed techniques may increase the working range of projection angles that can be utilized during reconstruction. For example, the disclosed techniques may expand a geometric range for a fiducial board that natively supports a 100° sweep of projection angles to a 130° sweep.
[0033] FIG. 1 depicts an example system 100 for registering three-dimensional (3D) shape data to two-dimensional (2D) image data, according to embodiments. The system 100 may obtain images from a portion of an operating environment 101 disposed within a field of view F (approximately demarcated by dashed lines) of an imaging unit 110, also referred to herein at times as an imaging device. To that end, the system 100 may be in communicative connection with the imaging unit 110. Additionally, the system 100 may obtain data from one or more sensors disposed within the operating environment. To that end, the system 100 may be in communicative connection with the one or more sensors via a communicative connection with a sensing unit 115. Although in FIG. 1 the imaging unit 110 and the sensing unit 115 are depicted as being distinct from the system 100, in other examples, the system 100 may include the imaging unit 110 and / or the sensing unit 115.
[0034] The system 100 includes a processing unit 120 and a display unit 130 in communicative connection with each other. One or more processors of the processing unit 120 of the system 100 may be configured to receive images and / or processed image information from the imaging unit 110 and to receive data from the one or more sensors by way of the sensing unit 115. Throughout the disclosure, the descriptions of exampleIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 operations performed by the processing unit 120 below are to be understood to be executed by the one or more processors of the processing unit 120. In some examples, the one or more processors may include hardware specifically configured (e.g., hardwired or programmable) to carry out at least a portion of the example operations described in this disclosure.
[0035] Additionally or alternatively, the one or more processors may be configured to carry out at least a portion of the example operations described in this disclosure by carrying out a set of software instructions. To that end, the system 100 may include or be communicatively connected to a tangible, non-transitory, computer readable medium. The medium may store instructions which, when executed by the processing unit 120, perform any one or more of the example operations described below. For example, the instructions may cause the processing unit 120 to perform image processing operations on the images received from the imaging unit 110 and / or to perform computations (e.g., for coordinate registration, tomosythesis, etc.) based on the data received by way of the sensing unit 115. Furthermore, the instructions may cause the processing unit 120 to cause the display unit 130, also referred to herein at times as a display device, to display information via a GUI based on the processing of images received from the imaging unit 110 and the processing of data received by way of the sensing unit 115. For example, the processing unit 120 may send the information, or send data representing the entire GUI including the information, to the display unit 130.
[0036] An operator (e.g., a physician, another medical practitioner, or a fully- automated robotic surgery system) of a medical system may use the information displayed at the display unit 130 to perform a medical procedure (e.g., endoscopy, biopsy, pharmacological treatment, and / or treatment, such as ablation). During a medical procedure, the operator may control a flexible elongate device 140 inserted through an orifice O (or through a suitable surgical incision) into an anatomical structure A of a patient P disposed at a table T. For example, the medical procedure may include navigating the flexible elongate device 140 (indicated with solid lines outside and dashed lines inside the patient P) toward a region of interest (ROI) R within the anatomical structure A with the aid of information displayed at the display unit 130. The ROI R, for example, may be a designated procedure site for visual examination, biopsy, treatment, or any other medical procedure. Throughout the disclosure, the ROI R may be referred to as a region or a target.
[0037] One or more fiducial markings (also referred to simply as “fiducials”) may be disposed at (e.g., physically contacting, integrated within, fixedly attached to, or removablyIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 attached to in a manner that, during operation / use, forms a rigid relationship with) the flexible elongate device 140. The fiducials are configured to be visible in images obtained by the imaging unit 110 to thereby enhance visibility of the flexible elongate device 140 and / or to aid in identifying certain points along the device 140. To that end the fiducials may include elements of a variety of materials and / or structures such as metals, plastics, etched glass, dyes, radioactive or fluorescent markings, confined fluids (e.g., bubbles), etc. At least some of the fiducials may be integrated (e.g., etched, deposited, painted, or otherwise fixedly attached) onto the flexible elongate device 140. Additionally or alternatively, the fiducials may include elements removably disposed at the flexible elongate device 140. For example, the fiducials may be integrated onto a removable structure, such as a sleeve or a stylet, which in turn may be removably attached to the flexible elongate device 140.
[0038] One or more sensors may also be disposed at the flexible elongate device. The sensors may be mechanical sensors, optical sensors, electromagnetic (EM) sensors, or any other suitable sensors. The sensors may be integrated into the flexible elongate device 140, or removably attached to the flexible elongate device 140. The sensors may be configured to communicate with the sensing unit 115.
[0039] In some examples, the sensors are active sensors, configured to transmit electromagnetic (e.g., optical, radio, low-frequency magnetic) or radioactive radiation. The sensing unit 115 may include components to receive the radiation emitted by the sensors and triangulate or trilaterate locations of the sensors in the sensor system coordinates. The sensor system coordinates may be the coordinates of the robotic-assisted system configured to manipulate, control, or guide the flexible elongate device 140. As such, the sensor system coordinates may be referred to as instrument coordinates.
[0040] In other examples, the sensors are passive and emit no radiation. The passive sensors may sense radiation emitted by components of the sensing unit 115 disposed within the operating environment 101. For example, in an EM sensing system, one or more electrified coils may be disposed within the operating environment 101 to generate static or dynamic magnetic fields. The sensors may be configured to pick up variations in the dynamic magnetic fields or, as the sensors move, variations in the sensed static or quasi-static magnetic field and convert the sensed variations into electrical currents that are received by the sensing unit 115. In turn, the sensing unit 115 may compute indications of positions of the sensors from the received electrical currents.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0041] The sensors may be fiberoptic sensors disposed along the length of the flexible elongate device 140. The fiberoptic sensors may include Bragg gratings and / or materials to enhance non-linear scattering. The fiberoptic sensors may be configured to change spectral reflectivity based on material strain. Such sensors may scatter light emitted, for example, by the sensing unit 115 in a manner that indicates locations and degrees of bends in the flexible elongate device 140.
[0042] In some examples, sensors may include accelerometers, gyroscopes, and / or magnetometers. Furthermore, the point sensors may include inertial measurement units (IMUs) that combine multiple sensors (e.g., accelerometers, gyroscopes) and / or inertial and magnetic measurement units (IMMUs) that combine multiple sensors (e.g., accelerometers, gyroscopes, magnetometers). IMUs and / or IMMUs may generate signals indicative of orientation of the flexible elongate device at a given point with respect to gravitational field and / or magnetic field (e.g., of Earth). In some examples, additional magnetic fields may be introduced to the sensor environment. Additionally or alternatively, IMUs and / or IMMUs may generate signals indicative of motion of the flexible elongate device (e.g., caused by motion of an anatomical structure due to breathing and / or other factors, and / or independent motion of the flexible elongate device within the anatomical structure). In some examples, the sensing unit 115 may combine indications of orientation (e.g., up to three degrees of freedom) from IMUs with indications of position from other (e.g., EM) sensors to generate more complete data indicative of pose of the flexible elongate device. Furthermore, in some examples, the sensing unit 115 may combine data from sensors in multiple sensor coordinate systems. The sensing unit 115 and / or the processing unit 120 may register multiple sensor coordinate systems with each other.
[0043] Generally, the system 100 may use any suitable combinations of the sensors described above to obtain shape data indicative of shape of the flexible elongate device 140 (e.g., pose, including location and orientation, at different points along the length of the flexible elongate device 140).
[0044] FIG. 2A is a simplified diagram of the flexible elongate device 140 disposed within the anatomical structure A. FIG. 2A is included to give an expanded and more detailed view of a portion of the operating environment 101 disposed within the field of view F of the imaging unit 110. The anatomical structure A may be a lung of the patient P. The flexible elongate device 140 may be inserted into and navigated by the operator toward the region R (e.g., target of the medical procedure), for example, for the purpose of investigating orIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 treating a pathology in the region R. The techniques described in the present disclosure can facilitate the navigation process by generating and displaying timely and accurate imaging and sensing of the flexible elongate device 140 and imaging of the anatomical structure A. These techniques incorporate location, pose, and / or shape data in one coordinate system obtained using the sensors and location, pose, and / or shape data obtained using imaging in another coordinate system. The combined data may improve speed, accuracy, reliability and / or safety of a medical procedure. For example, the processing unit 120 may combine the data from the imaging unit 110 and / or the sensing unit 115 to quickly and accurately determine the position, orientation, and / or pose of at least a portion of the flexible elongate device 140 within reference to the anatomical structure A of the patient P and, particularly, the ROI or target R.
[0045] Furthermore, the processing unit 120 may generate a graphical user interface (GUI) or update GUI data for display on the display unit 130 to aid an operator with the medical procedure. In some examples, the processing unit 120 may generate data and / or control signals for a control unit of a robotic system configured to manipulate and / or navigate the flexible elongate device 140. Additionally or alternatively, the processing unit 120 may be configured to generate, based on the combined imaging and sensing data, one or more alerts. The alerts may include, for example, an alert indicating proximity to the region R, an alert indicating a potential navigation error, and alert indicating that confidence in location of the tip of the flexible elongate device 140 fell below a threshold level, etc. It should be noted that combining intra-operative data from the imaging unit 110 and the sensing unit 115 entails registering two coordinate systems (e.g., 2D and / or 3D coordinate systems) with respect to each other, as described with reference to FIGS. 4A-B below.
[0046] Generally, a suitably configured imaging unit (e.g., imaging unit 110) may generate the intra-operative imaging data. The intra-operative imaging data may include, for example, X-ray data. To that end, the imaging unit may include a C-arm fluoroscopic imaging system. Additionally or alternatively, the intraoperative imaging data may be obtained using computed tomography (CT), thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging or any other suitable imaging technique.
[0047] In the present disclosure, particular attention is given to three-dimensional (3D) image reconstruction from two-dimensional (2D) projection images. Such intra-operative imaging data may include fluoroscopy X-ray data (e.g., generated by a C-arm X-ray device),Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 and, particularly, tomosynthesis data (reconstructed from 2D X-ray images, such as fluoroscopy images, into a 3D volume).
[0048] FIGS. 2B and 2C are simplified diagrams depicting an example imaging geometry for imaging a subject. As in FIG. 2B, the patient P is disposed within the operating environment 101. An X-ray source 150 (which may be included in the imaging unit 110) may be disposed at a location (e.g., Cl, C2, C3, etc.) along a circular arc C and directed toward a diametrically disposed detector 155 (which, likewise, may be included in the imaging unit 110) on the opposite side of the patient P. The circular arc C may lie in a plane orthogonal to an axis A disposed lengthwise through the patient P, and the center of the arc C may be disposed at the axis A. More generally, the axis A need not run lengthwise through the patient P. Generally, the axis of rotation for the arc C need not run through the patient, but may be below or above the patient P and at any suitable angle with respect to the body of the patient P. Furthermore, the arc C need not be circular to apply the techniques of the disclosure. In some examples, the arc C may be determined by the movement of a C-arm, e.g., of a fluoroscopy imaging device. The system, based on the geometry of FIG. 2B may generate projections from a limited set of projection angles (e.g., 120°, 100°, 110°, 90°, 80°, 70°, 60°, or any other suitable span).
[0049] Besides passing through the patient P, X-rays emanating from the X-ray source 150 may pass through a fiducial plate 160 disposed, for example, under the patient P (or any other suitable location at or near the body of the patient P). In embodiments where the fiducial plate 160 is utilized, the instant techniques may enable the extension of the geometric range of projection angles of the fiducial board 160 that can be utilized in the tomosynthesis process. In FIG. 2B, the fiducial plate 160 is shown separately to illustrate example fiducial markings disposed, as an example, as a grid throughout the plate 160. The fiducial markings may be high X-ray density spheres and / or cylinders (with regular or varying diameters of 1 , 2, 5, 10 mm or any other suitable size) placed throughout the plate 160 at regular or varying spacing of 10, 20, 30, 40 mm or any other suitable spacing. Other fiducial markings, such as lines squares, or other suitable markings that have high X-ray attenuation or scattering may be included in the plate 160. As described in more detail below, the system 100 may use the fiducial plate 160 to determine projection angles for 2D images (e.g., fluoroscopy images) detected by the detector 155 of the imaging unit 110.
[0050] FIG. 2C include an example imaging coordinate system. The axis A may be referred to as the x-axis. The orthogonal horizontal axis may be referred to as the y-axis.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 Finally, the vertical axis may be referred to as the z-axis. Planes Sx, Sy, and Sz are, respectively, orthogonal to the x-axis, the y-axis, and the z-axis, and may be referred to, respectively, as axial, sagittal, and coronal planes. It should be noted that the axial plane Sx is parallel to the plane of the arc C and does not correspond to any possible projection angle. In contrast, the coronal plane Sz corresponds to the projection acquired with the X-ray source 150 at the apex of the curve C. The range of angles with respect to the apex of the curve C may not permit a projection corresponding to the sagittal plane, which would require an angle of ±90° with respect to vertical. The fluoroscopy acquisition angles may lie, for example, between +60°with respect to vertical, subtending an arc that may be less than 120°. A reconstructed tomosynthesis image, on the other hand, may be presented in slices parallel to any one or more of the three planes Sx, Sy, and / or Sz.
[0051] FIG. 2D is a simplified three-dimensional representation of a flexible elongate device 140 disposed within the anatomical structure A within the imaging geometry of FIGS.2B and 2C. It is included to help visualize projections acquired by the tomosynthesis imaging system illustrated in FIG. 2B with respect to the coordinate system of FIG. 2C. The flexible elongate device 140 may have higher X-ray density than the surrounding anatomical structure as may be seen in an example projection in FIG. 2E.
[0052] FIG. 2E depicts a portion of a projection image 180 of a flexible elongate device 140 (which may be device 140) disposed within an anatomical structure (e.g., anatomical structure A) overlayed with fiducial markings (e.g., from the fiducial board 160). The flexible elongate device 140 appears darker than the surrounding tissue, but may be converted, in the negative, to the high-intensity portion of the image. The high X-ray density fiducial marks (e.g., mark 161) are highly visible in the image 180.
[0053] FIGS. 2F and 2G schematically illustrate projections (i.e., projected images) of the example fiducial plate 160 at two projection angles. One can assume, for example, that the plate 160 has fiducial markings arranged in a regular square grid, with identical spacing along two directions. In FIG. 2F, the projection angle may then be 0° with respect to the z-axis, i.e., normal to the plate, resulting in a regular grid of fiducial markings within the projected image of the plate 160. In FIG. 2G, on the other hand, the projection angle deviates from the normal, resulting in the columns of fiducial markings being closer that the rows, as well as other distortions. Certain distortions may arise from the projection angle deviating from the normal with respect to the two symmetry axes (represented by dashed lines in FIGS. 2F, H) of the fiducial plate 160. The projection in FIG. 2G may additionally have a perspectiveIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 distortion resulting, for example, from slightly nonparallel X-rays admitted by the X-ray source 150.
[0054] The system 100 may use the geometric distortions in projections of the fiducial plate 160 to determine the X-ray incidence angle and, therefore, the projection angle and the position of the X-ray source 150 along the arc C. In some examples, the system 100 may rely on geometric formulas, such as the spacing of fiducial elements in a given direction being proportional to the cosine of the respective angle, to determine the projection angle.Additionally or alternatively, the system 100 may obtain and use calibration data from known projection angles.
[0055] However, determining the projection angle of other data based upon the fiducial plate may not be possible using such conventional methods when a requisite number of fiducials do not appear in the image data (e.g., when the fiducial board is out of the field of view of the X-ray source 150). Additionally, the fiducial plate may not be available in all scenarios.
[0056] FIG. 3A schematically illustrates an example multi-projection image acquisition geometry for a flexible elongate device 340 (which may be the flexible elongate device 140 or 240) disposed within the anatomical structure A. The geometry is analogous to the one displayed in FIG. 2B above, with the anatomical structure A replacing the patient P to more clearly illustrate the generation of projection images of the anatomical structure A. For example, one projection image of the anatomical structure A and the flexible elongate device 340 may correspond to the X-ray source 150 at the location Cl and the detector 155 at the corresponding location DI. Another projection image of the anatomical structure A and the flexible elongate device 340 may correspond to the X-ray source 150 at the location C2 and the detector 155 at the corresponding location D2.
[0057] FIGS. 3B and 3C are example projection images for the image acquisition geometry of FIG. 3 A. For example, the image 301 may correspond to the X-ray source at the location Cl, while the image 302 may correspond to the X-ray source at the location C2. A processing unit (e.g., processing unit 120) of a system (e.g., system 100) may cause a display device (e.g., display unit 130) to generate a GUI and to display within the GUI the projection image 301 and / or the projection image 302.
[0058] The system may identify within the projection image 301 a reference point 310. The reference point 310 may be the tip of the flexible elongate device 340 or another suitableIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 reference point (e.g., identifiable by a fiducial disposed at the flexible elongate device 340, as described above with reference to FIG. 1). In some examples, the system may identify the reference point 310 automatically (such as when fiducial markings are included on the flexible elongate device 340). In other examples, the system may prompt a user to identify the reference point 310 within the displayed projection image 301, using for example, the GUI generated at the display. Furthermore, the system may identify a reference point 320 within the projection image 302, which may correspond to the same physical point at the flexible elongate device 340 as the reference point 310. The system may identify the point 320 automatically or based on user input, as described above. Furthermore, the system may identify within the projection image 302 additional points 322, 324 disposed at the flexible elongate device 340. Identifying the points 310-324 may include identifying two-dimensional coordinates within their respective projection images (e.g., images 301, 302). The system may use the identified points 310-320 (e.g., the identified two-dimensional coordinates) to identify a curve corresponding to at least a portion of the flexible elongate device 340 within at least one of the projection images (e.g., image 302). Furthermore, the system may use the identified points 310-320 and / or the identified two-dimensional curve to reconstruct a corresponding portion of the flexible elongate device 340 in three dimensions. Additionally or alternatively, the system may use at least two projection images (e.g., the projection images 301, 302) to register, using shape data received from a sensing system (e.g., the sensing unit 115), an imaging coordinate system with a sensing or instrument coordinate system as described with reference to FIGS. 4A-C.
[0059] It should be appreciated that the respective projection angles of the imaging unit 110 at the time the projection images 301, 302 were generated are used to reconstruct a 3D model of the imaged anatomy A. As described above, the fiducial board 160 may not always be available and / or be associated with a limited geometric range of projection angles.Techniques described herein relate to predicting the projection angle of the imaging unit 110 for a given projection image based on shape sensing data sensed by the sensing unit 115 when no fiducial board is present and / or at angles beyond the typical geometric range associated with a fiducial board.
[0060] FIG. 4A depicts a rigid body 402 in two example coordinate systems to illustrate the concept of registration between two coordinate systems. A first coordinate system 404 may correspond to coordinates of shape data obtained or generated, for example, by the sensing unit 115. The sensing unit 115 may be a part of a robotic unit configured to actuate and / orIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 manipulate the flexible elongate device 140. The first coordinate system 404 may therefore represent the coordinate system of the robotic unit. A second coordinate system 406 may correspond to coordinates of images obtained or generated by an imaging system, e.g., the imaging unit 110.
[0061] The rigid body 402 may represent a portion (e.g., a segment of length) of the flexible elongate device (e.g., flexible elongate device 140, 240, or 340). Although the flexible elongate device 140 is flexible, a short portion (e.g., an infinitesimal segment) may be considered rigid for all practical purposes.
[0062] A state (e.g., position and orientation) of the rigid body 402 in three-dimensional space can be described using the first coordinate system 404 and / or the second coordinate system 406. A rigid body without symmetries (e.g., body 402) has six degrees of freedom (6 DOFs) and its position and orientation can be described with six coordinates. In the first coordinate system 404, the rigid body 402 may have coordinates (x, y, z, 9, (|>, a), where x, y and z are position coordinates of a center 408 of the rigid body 402 along the axes of the first coordinate system 404 with respect to origin O. In other examples, the position coordinates may be designated for any point within the rigid body 402, or, in fact, any point in a rigidly defined geometric relationship to the body 402. Coordinates 9, (|> and a can describe orientation of the body 402 with the aid of an orientation vector 409, which, in the example of FIG. 4A, originates in the center 408 and goes through the middle of one of the facets of the rigid body 402. For example, 9 may be an elevation angle with respect to z-axis, (|> may be an azimuthal angle parallel to the xy-plane, and a may be the angle of rotation of the rigid body 402 around the orientation vector 409. Alternatively, the three orientation coordinates may be roll, pitch and yaw of the rigid body 402 with respect to any suitable reference direction. Analogously to the coordinates (x, y, z, 9, (|), a) of first coordinate system 404, coordinates (x’, y’, z’, 9’, (|)’, a’) of the second coordinate system 406 describe position (x’, y’, z’) with respect to origin O’ and orientation (9’, <|>’, a’) with respect to, for example, z’-axis of the rigid body 402. Registering the first coordinate system 404 with the second coordinate system 406, at least in the vicinity of the rigid body 402, includes finding a mapping (e.g., a transformation, a mathematical relationship, etc.) at least between the coordinates (x, y, z, 9, (|>, a) and the coordinates (x’, y’, z’, 9’, (|)’, a’). Another way of looking at the registration, is that the mapping defines a corresponding point (u, v, w) in the vicinity of position (x, y, z) for any point (u’, v’, w’) in the vicinity of position (x’, y’, z’).Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0063] In the discussion above, it is assumed that the first coordinate system 404 and the second coordinate system 406 have the same scale and the transformation from the first coordinate system 404 to the second coordinate system 406 is rigid. In some examples, however, the mapping from the first coordinate system 404 to the second coordinate system 406 may include one or more scaling factors for the axes. Thus, the mapping may include three translation variables, three rotation variables, and / or three scaling variables.Furthermore, each of the variables may depend on position and the mapping may include deformations.
[0064] In some examples, each of the coordinate systems 404 and 406 are independently calibrated to have accurate and consistent scaling within a shared operating volume.Coordinate registration process may then be defined in terms of three translation constants and three rotation constants for the shared operating volume. In other examples, gradual variations in scaling within at least one of the coordinate systems 404 and / or 406 may necessitate use of up to three translation variables and up to three rotation variables, each a function of location within the shared operating volume.
[0065] FIG. 4B schematically illustrates an example coordinate transformation process. In one example, a processing unit (e.g., the processing unit 120) may obtain or determine a position and an orientation of a rigid body (e.g., rigid body 402) within the first coordinate system 404 as (x, y, z, 9, (|), a) and the second coordinate system 406 as (x’, y’, z’, 9’, <|>’, a’). The processing unit may then generate a mapping, M, between the two coordinate systems 404 and 406. The processing unit may be configured to map a new position (u’, v’, w’) within the second coordinate system 406 on to a corresponding position (u, v, w) within the first coordinate system 404. Additionally or alternatively, the processing unit may be configured to map coordinates from the first coordinate system 404 to the second coordinate system 406. The mapping may be only valid in a region around (x, y, z). By collecting rigid body coordinates in the two coordinate systems 404 and 406, the processing unit may extend the validity of mapping over any portion of a shared operating volume of the two coordinate systems 404 and 406. For example, the processing unit may implement the coordinate registration as a linear mapping:Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 -u- V-w- Sn 0 0 cosy — siny 0 ' cos p 0 sin P' 1 0 0 d-y 0220 siny cosy 0 0 1 0 0 cos a — sin a + d2- 0 0 ^33- . 0 01. .—sin p 0 cos p. .0 sin a cos a . .d3. where a, P and y are rotation parameters (e.g., roll, pitch and yaw), Sn, S22 and S33 are scaling parameters (which may be unity, as discussed above), and di, d2 and d3 are displacement factors. Thus, with scaling factors, there may be nine mapping parameters, and six mapping parameters when scaling can be ignored. As discussed above, the linear mapping may be a function of the input position coordinates (u’, v’, w’). For example, the processing unit may store and / or access a lookup table find an entry for mapping parameters corresponding to the input position coordinates. Because the lookup table can only have a limited number of recorded input coordinates (herein, recorded coordinates), the system may use the entry corresponding to the recorded coordinates nearest to the input coordinates. Alternatively, the system may interpolate mapping parameters corresponding to a set of recorded coordinates near the input coordinates. In other examples, the processing unit may store and / or access a polynomial, spline, or another suitable fit function relating input coordinates to the mapping parameters. The mapping may relate the shape data for the flexible elongate device obtained from a sensing system and the flexible elongate device shape data in the imaging coordinate system as discussed with reference to FIG. 4C. Furthermore, the system may use theA A.. mapping to update the instrument coordinate system.
[0066] FIG. 4C depicts an example geometry for flexible elongate device extraction and coordinate registration using shape data and projection images. Two-dimensional projections 410, 420 of a three-dimensional curve 430 representative of at least a portion of a flexible elongate device (e.g., flexible elongate device 140, 240, or 340). The projections 410, 420 (which may be referred to as projected curves 410, 420) may be analogous to and / or representative of the two-dimensional projections of the flexible elongate device 340 in projection images 301, 302. The projections 410, 420 are depicted on orthogonal planes - the xz-plane and the xy-plane, respectively. Although orthogonal projections may be obtained (e.g., at ±45°) in the imaging geometries described above, the discussion with reference to FIG. 4C can be generalized to non-orthogonal projections.
[0067] The three-dimensional image and coordinate system, defined by the x-, y-, and z-axes of FIG. 4C may be defined with respect to the origin located at a reference point disposed at the flexible elongate device. For example, the reference point 310 (which may beIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 a tip of the flexible elongate device 340) in the projection image 301 and the reference point 320 in the projection image 302 may be projections of the same reference point along the flexible elongate device 340 and may define the origin, without any loss of generality, of the three-dimensional imaging coordinate system. Thus, as depicted in FIG. 4C, the projections 410, 420 and the three-dimensional curve 430 representation of the section of the flexible elongate device all intersect at the origin of the coordinate system.
[0068] A system (e.g., system 100) may be configured to reconstruct the curve 430 based on two-dimensional projection images which include projected curves 410, 420. To that end, the system may first identify within at least one of the projection images (e.g., corresponding to the projection onto the xy-plane) a projected curve (e.g., the projected curve 420). When identified, the two-dimensional curve 420 may be represented as a centerline of the projection of the flexible elongate device. Identification of the centerline curve within a projection image may, in some examples be based on at least in part identifying several points (e.g., points 320, 322, and 324) along the curve. The system may identify the points along the curve based on user input via a GUI displaying the respective projection or based on automatic extraction of the two-dimensional curve corresponding to the flexible elongate device from the respective projection image using suitable image processing algorithms.
[0069] In some examples the system may extract a two-dimensional curve (e.g., curve 410) in a second projection image based on the extracted two-dimensional curve (e.g., curve 420) in the first projection image. The curves in the two projection images necessarily share the same coordinate along the axis defined by the intersection of the two projection planes (e.g., the x-axis with respect to the curves 410, 420). Consequently, for a given point along one two-dimensional projected curve, one can find a corresponding point on the second two-dimensional projected curve. For example, a point on the curve 420 may be projected along a projection ray 462 to the x-axis. The system may find the corresponding point on the curve 410 by tracing along a projection ray 464 to find the intersection with the curve 410. The system may identify the intersection as a high or a low (depending on the format of the projection image) intensity point in the projection in the projection image that intersects the projection ray 464. In an example of another point along the curve 420 projected onto the x-axis by ray 466, the corresponding gray 468 intersects the curve 410 at least two points. The system may resolve the ambiguity by selecting the point that would be contiguous with previous points detected along the curve 410.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0070] Once the system identifies two corresponding points in the projection images, the corresponding point in three dimensions may be found, e.g., at an intersection 470 of back-projected rays emanating from the two corresponding points in two-dimensional projection images along the respective projection angles. In some examples, the system may use more than two projections to identify corresponding points in each of the projections and back project the identified points to find the correspondent point in three dimensions. In the presence of measurement uncertainties, the back projected rays may not intersect. The system may then find a point in three dimensions that minimizes a metric (e.g., root mean square or L2-norm) of distances between the three-dimensional point in the back projection rays. In the manner described above, the curve 430 in three dimensions may be reconstructed from the curves 410, 420 in two-dimensional projections.
[0071] In other example, the system may reconstruct the curve 430 in three dimensions based on the curve 420 (or curve 410) by first performing a reconstruction of the imaging volume based on the full set of two-dimensional projections. The curve 430 in the resulting three-dimensional reconstructed image may be computed by back-tracing or back-projecting along rays corresponding to the projection angle from points along the curve 420. The system may identify corresponding points along the curve 430 based on suitable intensity thresholds and continuity requirements in a manner similar to identifying corresponding points on the two-dimensional curve 410, as discussed above.
[0072] In some examples, the system may use the reconstructed curve 430 to register the imaging coordinate system with a sensing coordinate system. To that end the system may receive shape data from a sensing unit (e.g., sensing unit 115) and identify, within the shape data, sensing system coordinates corresponding to the reference point (e.g., tip of the flexible elongate device) defining the origin of the imaging coordinate system in FIG. 4C. The first step of registration may be defining the translation portion of the transformation (e.g., transformation M described with reference to FIG. 4B), placing the reference point in at the common origin of the coordinate systems registered by the translation. The shape data from the sensing system may then be represented by the dashed-line curve 480, assuming the same scale in the two coordinate systems, as discussed above. The remaining portion of the registration may include finding roll, pitch, and yaw angles that align the shape data curve 480 to the curve 430. Although perfect alignment may not be possible, due to noise and distortions in shape data received from the sensing unit and computed based on the imaging data from the imaging unit, an optimization algorithm may minimize a cost functionIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 indicative of misalignment or misregistration. For example, the system may compute the registration transformation between the shape data 480 and the curve 430 in the least-square sense with or without regularization accounting for prior transformation data and / or other suitable constraints. Additionally or alternatively, the system may minimize a weighted leastsquare difference, valuing the alignment near the origin more heavily than the alignment away from the origin. Valuing the alignment near the origin more heavily may be particularly applicable to examples where the reference point at the origin of the registered coordinate systems in the tip of the flexible elongate device when the tip is in proximity to an anatomical target (e.g., target R).
[0073] In some examples, the registration between the imaging and sensing coordinate systems need not follow the extraction of the curve 430 in three-dimensions. Instead, the system may use the shape data 480 to simultaneously extract the curve 430 and register the coordinate systems. To that end, the system may compute coordinate transformations to align projections of the shape data 480 with at least one of the projected curves 410, 420. In this manner, the shape data 480 from the sensing unit may constrain the reconstruction of the curve 430.
[0074] FIGS. 5 A and 5B schematically illustrates example processes 500, 550, respectively for optimizing the pose prediction of an imaging unit (such as the imaging unit 110) as part of registration process. Starting with Figure 5A in particular, illustrated is the process 500 for 2D / 3D registration of 2D fluoroscopic images and 3D shape sensor points of a flexible elongate device sensor data, according to embodiments. A system (e.g., system 100) may execute the process 500 using one or more processors of a processing unit (e.g., processing unit 120) and a display unit (e.g., display unit 130).
[0075] The system 100 may obtain (e.g., from imaging unit 110) one or more frames of 2D image data 502 depicting a flexible elongate device. The frames of 2D image data may each be associated with a respective plurality of poses of the imaging device. In the illustrated embodiment, the 2D image data 502 includes depictions of fiducials associated with a fiducial board. Accordingly, the 2D image data may include depictions of one or more reference points (e.g., the fiducials) from which the pose of the imaging unit 110 can be derived for a first set of viewing angles (the typical geometric range associated with the fiducial board), but not a second set of viewing angles (viewing angles extending beyond the geometric range associated with the fiducial board). In other embodiments where no fiducial board is available, the 2D image data does not include depictions of fiducials. In eitherIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 scenario, the conventional pose determination techniques cannot be applied to the full set of image data 502.
[0076] In the process 500, after obtaining the 2D image data 502, the system 100 performs segmentation 504 of the one or more frames to generate a 2D mask 506 of the flexible elongate device. For example, the system 100 may implement a feature detection model (such as a convolutional neural network) to detect the image data representative of the flexible elongate device. The system 100 may perform centerline extraction 508 to extract the flexible elongate device from the 2D image frames. In some embodiments, the centerline extraction may be performed using the centerline extraction techniques described with respect to FIG. 4C.
[0077] In parallel or sequentially, the system 100 may obtain, via one or more shape sensors (e.g., via the sensing unit 115, sensors of the flexible elongate device 140), 3D shape sensor data 510 representative of a shape of the flexible elongate device. It should be appreciated that the 3D shape sensor date is defined with respect to a separate coordinate system (e.g., the robotic coordinate system 404) than the image data (e.g., the imaging coordinate system 406).
[0078] The system 100 may then perform a pose prediction optimization process 514 as part of registering the 3D shape sensing data 510 with the 2D image data 502. This optimization process may be performed in several phases. In an initial phase, the pose associated with each individual frame of image data 502 may be optimized in isolation (also called “single-view registration”) to generate an initial set of imaging unit pose predictions.
[0079] In the single-view registration process, the system 100 generates a plurality of pose predictions for the imaging device for each frame of image data 502. The system 100 then applies a projection matrix associated with each predicted pose to the 3D shape sensing data 510 to reproject the flexible elongate device into a 2D image. Then the system 100 may attempt to register a point from the reprojection of the flexible elongate device to a respective point of the frame of 2D image data 502. For example, the system 100 may implement an iterative closest point (ICP) registration process to select corresponding point pairs between the reprojection of the flexible elongate device and the frame of image data 502.
[0080] As part of the optimization process, the system 100 may reduce a loss associated with the registered point pairs. In one example, the loss is defined to be a mean distance between the registered point pairs that define the flexible elongate device. The system 100Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 may select the predicted pose associated with the minimum loss as the predicted post of the imaging unit 110 when the frame of image data 502 was captured.
[0081] It should be appreciated that because system 100 is often utilized to navigate the flexible elongate device to a ROI in a patient’s anatomy, it may be more important that the pose of the tip of the flexible elongate device is accurately registered. This ensures that the portion of the flexible elongate device that is advancing through the anatomy is more accurately registered, resulting in more accurate control of the flexible elongate device and better avoidance of obstacles in the target anatomy. Accordingly, in these embodiments, the loss function may apply a weighted mean to the point pairs, where the weights associated with the point pairs that define the tip of the flexible elongate device are given higher weights (e.g., a weight of 1) and weights for point pairs proximal from the tip decrease based on distance from the tip. In some embodiments, the system 100 may define a predetermined number of point pairs (e.g., 3, 5, 7, 9) as representing the tip of the flexible elongate device. Accordingly, in these embodiments, the loss function may result in selecting a predicted pose of the imaging unit 110 that most closely aligns the tip of the flexible elongate device as represented in the reprojection and the 2D frame of image data 502.
[0082] The system 100 may repeat this single- view registration process for each frame of image data 502 to generate a set of predicted poses of the imaging unit 110 for imaging event. It should be appreciated that in embodiments where the fiducial board is present, instead of performing the single view registration process described above, the system 100 may instead implement conventional pose determination techniques to determine the pose of the imaging unit 110 for the frames of image data 502 within the geometric range supported by the fiducial board. This range may be determined dynamically based on the ability to resolve the fiducials within the frames of image data 502. However, for frames of image data 502 in which the fiducials cannot be resolved (e.g., those exceeding the geometric range of the fiducial board), the system 100 may still implement the disclosed single- view registration techniques to predict the pose of the imaging device associated with those frames of image data 502.
[0083] It should be appreciated that performing the single- view registration ignores the relationship between successive frames of image data, namely, that the imaging unit 110 typically traverses a circular arc while rotating around the patient. Thus, the predicted poses may be further optimized by defining a circular trajectory of the imaging unit 110 and usingIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 the circular trajectory to regularize the set of predicted poses. This regularization process may be referred to as a multi-view registration process.
[0084] With reference to FIG. 5B, illustrated is a process 550 for regularizing the set of predicted poses of the imaging unit 110. As illustrated, the process 550 includes registering the 3D shape sensing data 510 to the frames of image data 502 via the process 500 to produce the optimized set of 3D poses of the imaging unit 110. To apply the knowledge of the circular trajectory, the system 100 may apply a circular fit algorithm to the set of predicted poses to define a circle that best fits the predicted poses. For example, the system 100 may implement a random sample consensus (RANSAC) or other suitable fitting algorithm to fit a circle to the set of predicted poses.
[0085] The system 100 may then implement a regularization process 522 to adjust the set of predicted poses to more closely align with the fit circle. More particularly, the system 100 may reduce an error between the set of predicted poses and the fit circle. In some embodiments, the error is calculated based on a distance a predicted pose is from the fit circle and / or an angular error between predicted poses and the fit circle. As a result, the system 100 has a new set of predicted poses associated with each frame of image data 502 that more accurately reflects the true pose of the imaging unit 110.
[0086] Using the new set of predicted poses, the system 100 may then repeat the registration process 514 to register the reprojection of the flexible elongate device with the frames of data 502 using the updated set of predicted poses when performing the reprojection. If the loss is below a threshold value and / or the loss stops decreasing by a threshold amount with respect to a prior loss calculation, the system 100 may select the current set of predicted poses as the final set of poses to utilize when reconstructing the imaged anatomy via tomosynthesis. If the loss is above the threshold or is continuing to decrease, the system 100 may instead repeat processes 520 and 522 by generating a new circular fit and regularizing the current predicted poses to reduce loss with the new circular fit. As a result, the system 100 is able to utilize the shape sensing data 510 to accurately predict the pose of the imaging unit 110 for a plurality of frames of image data 502 without the use of a fiducial board.
[0087] It should be appreciated that in embodiments that utilize a fiducial board, the fit circle should closely align with the determined poses across the geometric range of the fiducial board. Thus, the knowledge of the pose of the imaging device across this sweep ofIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 projection angles can be used to further constrain the circle fitting process. Accordingly, the system 100 is able to more accurately predict the circular trajectory for the additional sweeps of projection angles on either side the geometric range. As a result, the system 100 is able to extend the set of projection angles that produce image data usable in a tomosynthesis process.
[0088] FIG. 6 depicts a computer-implemented method 600 for registering three-dimensional (3D) shape data to two-dimensional (2D) image data, according to embodiments. The computer-implemented method 600 may be implemented by a system (e.g., system 100), one or more processors (e.g., disposed at the processing unit 120), and a display device (e.g., the display unit 130). In some embodiments, the method 600 may include similar steps as those described with respect to Figs. 5A and 5B.
[0089] The computer-implemented method 600 may include obtaining, via an imaging device (e.g., the imaging unit 110), a frame of 2D image data depicting a flexible elongate device (block 602). For example, capturing 2D fluoroscopic X-ray images that include the flexible elongate device (e.g., the flexible elongate device 140) within the field of view.
[0090] The computer-implemented method 600 may include extracting the flexible elongate device from the frame of 2D image data (block 604). This extraction process may include segmenting the flexible elongate device from the 2D X-ray image and / or include extracting the flexible elongate device’s centerline points from the segmentation mask (e.g., for registering the 3D shape data with the 2D image data).
[0091] The computer-implemented method 600 may include predicting a pose of the imaging device associated with the frame of 2D image data (block 606). Predicting the imaging device's pose may indicate the orientation and position relative to the flexible elongate device, and / or generate a projection matrix for registering the 3D shape data with the 2D image data.
[0092] The computer-implemented method 600 may include obtaining, via a shape sensor (e.g., sensing unit 115), 3D shape sensor data representative of a shape of the flexible elongate device via a shape sensor (block 608). The shape sensor data may be used for registration with the 2D image data.
[0093] The computer-implemented method 600 may include generating a 2D projection of the flexible elongate device by applying a projection matrix associated with the predicted pose to the 3D shape sensor data (block 610).Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0094] Based on the extracted flexible elongate device and the generated 2D projection of the flexible elongate device, the computer-implemented method 600 may include registering the 3D shape sensor data to the frame of 2D image data (block 612).
[0095] In at least some embodiments, the computer-implemented method 600 may include optimizing the predicted pose to reduce a loss associated with a registration algorithm applied to the 2D projection of the flexible elongate device and the extracted flexible elongate device, wherein the 2D projection is associated with a minimum loss. The registration algorithm may include an Iterative Closest Point (ICP) registration algorithm. The loss may be based on the distance between point pairs selected by the registration algorithm and / or based on a weighted mean that applies respective weights based on the distance of the point pairs from the tip of the flexible elongate device (e.g., the tip of the flexible elongate device including a predetermined number of point pairs). Optimizing the predicted pose may include predicting the pose of the imaging device with respect to six degrees of freedom (DOFs).
[0096] In at least some embodiments, the frame of 2D image data may be included in a plurality of frames of 2D image data associated with a respective plurality of predicted poses of the imaging device. In some such embodiments, optimizing the predicted pose may include predicting the plurality of predicted poses of the imaging device for the plurality of frames of 2D image data, fitting a circular plane to the plurality of predicted poses, and constraining, based on the fitted circular plane, selection of a next predicted pose when optimizing the predicted pose of a frame of 2D image data. Fitting the circular plane to the plurality of predicted poses may include applying a random sample consensus (RANSAC) algorithm to the plurality of predicted poses. Constraining the selection may include (i) reducing a distance between the predicted pose and the circular plane, and / or (ii) calculating a predicted trajectory of the imaging device based on the plurality of predicted poses and reducing an angular error between the predicted trajectory and the circular plane.
[0097] The 2D image data may include depictions of one or more reference points (e.g., fiducial markings, bony structures, etc.) from which the pose of the imaging device may be derived for a first set of viewing angles, but not a second set of viewing angles. The computer-implemented method 600 may include calculating a trajectory of the imaging device across the first set of viewing angles, extrapolating the trajectory to predict the pose of the imaging device through the second set of viewing angles, and applying the projection matrix to the 3D shape sensor data generated at a viewing angle of the second set of viewing angles based on the extrapolated trajectory.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0098] FIGS. 7-9B depict diagrams of a medical system that may be used for manipulating a medical instrument that includes a flexible elongate device according to any of the methods and systems described above, in some examples. For example, each reference above to the “system” may refer to a system (e.g., system 700) discussed below, or to a subsystem thereof.
[0099] FIG. 7 is a simplified diagram of a medical system 700 according to some examples. The medical system 700 may include at least portions of the system 100 described with reference to FIG. 1. The medical system 700 may be suitable for use in, for example, surgical, diagnostic (e.g., biopsy), or therapeutic (e.g., ablation, electroporation, etc.) procedures. While some examples are provided herein with respect to such procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. The systems, instruments, and methods described herein may be used for animals, human cadavers, animal cadavers, portions of human or animal anatomy, non-surgical diagnosis, as well as for industrial systems, general or special purpose robotic systems, general or special purpose teleoperational systems, or robotic medical systems.
[0100] As shown in FIG. 7, medical system 700 may include a manipulator assembly 702 that controls the operation of a medical instrument 704 in performing various procedures on a patient (e.g., patient P on table T, as in FIG. 1). The medical instrument 704 may include the flexible elongated device 140 of FIG. 1. Medical instrument 704 may extend into an internal site within the body of patient P via an opening in the body of patient P. The manipulator assembly 702 may be teleoperated, non-teleoperated, or a hybrid teleoperated and nonteleoperated assembly with one or more degrees of freedom of motion that may be motorized and / or one or more degrees of freedom of motion that may be non-motorized (e.g., manually operated). The manipulator assembly 702 may be mounted to and / or positioned near patient table T. A master assembly 706 allows an operator O (e.g., a surgeon, a clinician, a physician, or other user, as described above) to control the manipulator assembly 702. In some examples, the master assembly 706 allows the operator O to view the procedural site or other graphical or informational displays. In some examples, the manipulator assembly 702 may be excluded from the medical system 700 and the medical instrument 704 may be controlled directly by the operator O. In some examples, the manipulator assembly 702 may be manually controlled by the operator O. Direct operator control may include various handles and operator interfaces for hand-held operation of the medical instrument 704.
[0101] The master assembly 706 may be located at a surgeon’s console which is in proximity to (e.g., in the same room as) the patient table T on which patient P is located, suchIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 as at the side of the patient table T. In some examples, the master assembly 706 is remote from the patient table T, such as in in a different room or a different building from the patient table T. The master assembly 706 may include one or more control devices for controlling the manipulator assembly 702. The control devices may include any number of a variety of input devices, such as joysticks, trackballs, scroll wheels, directional pads, buttons, data gloves, trigger-guns, hand-operated controllers, voice recognition devices, motion or presence sensors, and / or the like.
[0102] The manipulator assembly 702 supports the medical instrument 704 and may include a kinematic structure of links that provide a set-up structure. The links may include one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place) and / or one or more servo controlled links (e.g., one or more links that may be controlled in response to commands, such as from a control system 712). The manipulator assembly 702 may include a plurality of actuators (e.g., motors) that drive inputs on the medical instrument 704 in response to commands, such as from the control system 712. The actuators may include drive systems that move the medical instrument 704 in various ways when coupled to the medical instrument 704. For example, one or more actuators may advance medical instrument 704 into a naturally or surgically created anatomic orifice. Actuators may control articulation of the medical instrument 704, such as by moving the distal end (or any other portion) of medical instrument 704 in multiple degrees of freedom. These degrees of freedom may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). One or more actuators may control rotation of the medical instrument about a longitudinal axis. Actuators can also be used to move an articulable end effector of medical instrument 704, such as for grasping tissue in the jaws of a biopsy device and / or the like, or may be used to move or otherwise control tools (e.g., imaging tools, ablation tools, biopsy tools, electroporation tools, etc.) that are inserted within the medical instrument 704.
[0103] The control system 712 may include at least portions of the processing unit 120. Additionally or alternatively, the control system 712 may be in communicative connection with the processing unit 120. In some examples, the output of the processing unit 120 according to the techniques described above may cause the control system 712 to autonomously (without input from the operator O) control certain movements of the medical instrument 704.1Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0104] The medical system 700 may include a sensor system 708 (which may include at least a portion of the sensing unit 115) with one or more sub-systems for receiving information about the manipulator assembly 702 and / or the medical instrument 704. Such sub-systems may include a position sensor system (e.g., that uses electromagnetic (EM) sensors or other types of sensors that detect position or location); a shape sensor system for determining the position, orientation, speed, velocity, pose, and / or shape of a distal end and / or of one or more segments along a flexible body of the medical instrument 704; a visualization system for capturing images, such as from the distal end of medical instrument 704 or from some other location; and / or actuator position sensors such as resolvers, encoders, potentiometers, and the like that describe the rotation and / or orientation of the actuators controlling the medical instrument 704. The subsystems may include an imaging sub-system (e.g., using a color imaging device, an infrared imaging device, an ultrasound imaging device, an x-ray imaging device, a fluoroscopic imaging device, a computed tomography (CT) imaging device, a magnetic resonance imaging (MRI) imaging device, or some other type of imaging device), such as the imaging unit 110.
[0105] It should be noted that the positions and orientations of sensors in the sensor system 708 may be determined in the sensor coordinate system. In some examples, the sensor coordinate system is integrated with or identical to the coordinate system of the manipulator assembly 702.
[0106] The medical system 700 may include a display system 710 (e.g., display unit 130) for displaying an image or representation of the procedural site and the medical instrument 704. Display system 710 and master assembly 706 may be oriented so physician O can control medical instrument 704 and master assembly 706 with the perception of telepresence. The display system 710 may include at least portions of the display unit 130.
[0107] In some examples, the medical instrument 704 may include a visualization system, which may include an image capture assembly that records a concurrent or real-time image of a procedural site and provides the image to the operator O through one or more displays of display system 710. The image capture assembly may include various types of imaging devices. The concurrent image may be, for example, a two-dimensional image or a three-dimensional image captured by an endoscope positioned within the anatomical procedural site. The visualization system may obtain intra-operative images in image system coordinates, distinct from the sensor system coordinates. In some examples, the visualization system may include endoscopic components that may be integrally or removably coupled toIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 medical instrument 704. Additionally or alternatively, a separate endoscope, attached to a separate manipulator assembly, may be used with medical instrument 704 to image the procedural site. The visualization system may be implemented as hardware, firmware, software or a combination thereof which interact with or are otherwise executed by one or more computer processors, such as of the control system 712.
[0108] Display system 710 may also display an image of the procedural site and medical instruments, which may be captured by the visualization system. In some examples, the medical system 700 provides a perception of telepresence to the operator O. For example, images captured by an imaging device at a distal portion of the medical instrument 704 may be presented by the display system 710 to provide the perception of being at the distal portion of the medical instrument 704 to the operator O. The input to the master assembly 706 provided by the operator O may move the distal portion of the medical instrument 704 in a manner that corresponds with the nature of the input (e.g., distal tip turns right when a trackball is rolled to the right) and results in corresponding change to the perspective of the images captured by the imaging device at the distal portion of the medical instrument 704. As such, the perception of telepresence for the operator O is maintained as the medical instrument 704 is moved using the master assembly 706. The operator O can manipulate the medical instrument 704 and hand controls of the master assembly 706 as if viewing the workspace in substantially true presence, simulating the experience of an operator that is physically manipulating the medical instrument 704 from within the patient anatomy.
[0109] In some examples, the display system 710 may present virtual images of a procedural site that are created using image data recorded pre-operatively (e.g., prior to the procedure performed by the medical instrument system 200) or intra-operatively (e.g., concurrent with the procedure performed by the medical instrument system 200), such as image data created using computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like. The virtual images may include two-dimensional, three-dimensional, or higher-dimensional (e.g., including, for example, time based or velocitybased information) images. In some examples, one or more models are created from preoperative or intra-operative image data sets and the virtual images are generated using the one or more models.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0110] In some examples, for purposes of imaged guided medical procedures, display system 710 may display a virtual image that is generated based on tracking the location of medical instrument 704. For example, the tracked location of the medical instrument 704 may be registered (e.g., dynamically referenced) with the model generated using the preoperative or intra-operative images, with different portions of the model correspond with different locations of the patient anatomy. As the medical instrument 704 moves through the patient anatomy, the registration is used to determine portions of the model corresponding with the location and / or perspective of the medical instrument 704 and virtual images are generated using the determined portions of the model. This may be done to present the operator O with virtual images of the internal procedural site from viewpoints of medical instrument 704 that correspond with the tracked locations of the medical instrument 704.
[0111] The display system 710 may include the display unit 130 and may display images including the position, orientation, and / or pose of the medical instrument 704 in a joint coordinate system based on registering the sensor coordinate system with the imaging coordinate system according to the techniques described above.
[0112] The medical system 700 may also include the control system 712, which may include processing circuitry (e.g., the processing unit 120) that implements some or all of the methods or functionality discussed herein. The control system 712 may include at least one memory and at least one processor for controlling the operations of the manipulator assembly 702, the medical instrument 704, the master assembly 706, the sensor system 708, and / or the display system 710. Control system 712 may include instructions (e.g., a non-transitory machine-readable medium storing the instructions) that when executed by the at least one processor, configures the one or more processors to implement some or all of the methods or functionality discussed herein. While the control system 712 is shown as a single block in FIG. 7, the control system 712 may include two or more separate data processing circuits with one portion of the processing being performed at the manipulator assembly 702, another portion of the processing being performed at the master assembly 706, and / or the like. In some examples, the control system 712 may include other types of processing circuitry, such as application-specific integrated circuits (ASICs) and / or field-programmable gate array (FPGAs). The control system 712 may be implemented using hardware, firmware, software, or a combination thereof.
[0113] In some examples, the control system 712 may receive feedback from the medical instrument 704, such as force and / or torque feedback. Responsive to the feedback, theIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 control system 712 may transmit signals to the master assembly 706. In some examples, the control system 712 may transmit signals instructing one or more actuators of the manipulator assembly 702 to move the medical instrument 704. In some examples, the control system 712 may transmit informational displays regarding the feedback to the display system 710 for presentation or perform other types of actions based on the feedback.
[0114] The control system 712 may include a virtual visualization system to provide navigation assistance to operator O when controlling the medical instrument 704 during an image-guided medical procedure. Virtual navigation using the virtual visualization system may be based upon an acquired pre-operative or intra-operative dataset of anatomic passageways of the patient P. The control system 712 or a separate computing device may convert the recorded images, using programmed instructions alone or in combination with operator inputs, into a model of the patient anatomy. The model may include a segmented two-dimensional or three-dimensional composite representation of a partial or an entire anatomic organ or anatomic region. An image data set may be associated with the composite representation. The virtual visualization system may obtain sensor data from the sensor system 708 that is used to compute an (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P. The sensor system 708 may be used to register and display the medical instrument 704 together with the pre-operatively or intra-operatively recorded images. For example, PCT Publication WO 2016 / 191298 (published December 1, 2016 and titled “Systems and Methods of Registration for Image Guided Surgery”), which is incorporated by reference herein in its entirety, discloses example systems. Additionally or alternatively, the registration may be based on the techniques discussed.
[0115] During a virtual navigation procedure, the sensor system 708 may be used to compute the (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P. The location can be used to produce both macro-level (e.g., external) tracking images of the anatomy of patient P and virtual internal images of the anatomy of patient P. The system may include one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display a medical instrument together with pre-operatively recorded medical images. For example, U.S. Patent No. 8,900,131 (filed May 13, 2011 and titled “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”), which is incorporated by reference herein in its entirety, discloses example systems.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0116] Medical system 700 may further include operations and support systems (not shown) such as illumination systems, steering control systems, irrigation systems, and / or suction systems. In some examples, the medical system 700 may include more than one manipulator assembly and / or more than one master assembly. The exact number of manipulator assemblies may depend on the medical procedure and space constraints within the procedural room, among other factors. Multiple master assemblies may be co-located or they may be positioned in separate locations. Multiple master assemblies may allow more than one operator to control one or more manipulator assemblies in various combinations.
[0117] FIG. 8 A is a simplified diagram of a medical instrument system 800 according to some examples. The medical instrument system 800 includes a flexible elongate device 802 (e.g., device 140), also referred to as elongate device 802, a drive unit 804, and a medical tool 826 that collectively is an example of a medical instrument 704 of a medical system 700. The medical system 700 may be a teleoperated system, a non-teleoperated system, or a hybrid teleoperated and non-teleoperated system, as explained with reference to FIG. 7. A visualization system 831, tracking system 830, and navigation system 832 are also shown in FIG. 8A and are example components of the control system 712 of the medical system 700. In some examples, the medical instrument system 800 may be used for non-teleoperational exploratory procedures or in procedures involving traditional manually operated medical instruments, such as endoscopy. The medical instrument system 800 may be used to gather (e.g., measure) a set of data points corresponding to locations within anatomic passageways of a patient, such as patient P.
[0118] The elongate device 802 is coupled to the drive unit 804. The elongate device 802 includes a channel 821 through which the medical tool 826 may be inserted. The elongate device 802 navigates within patient anatomy to deliver the medical tool 826 to a procedural site. The elongate device 802 includes a flexible body 816 having a proximal end 817 and a distal end 818. In some examples, the flexible body 816 may have an approximately 3 mm outer diameter. Other flexible body outer diameters may be larger or smaller.
[0119] Medical instrument system 800 may include the tracking system 830 for determining the position, orientation, speed, velocity, pose, and / or shape of the flexible body 816 at the distal end 818 and / or of one or more segments 824 along flexible body 816, as will be described in further detail below. The tracking system 830 may include one or more sensors and / or imaging devices. The flexible body 816, such as the length between the distal end 818 and the proximal end 817, may include multiple segments 824. The tracking systemIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 830 may be implemented using hardware, firmware, software, or a combination thereof. In some examples, the tracking system 830 is part of control system 712 shown in FIG. 7. The tracking system 830 may implement at least some of the techniques described, and, to that end, may include at least portions of or be in communicative connection with the processing unit 120 of FIG. 1A.
[0120] Tracking system 830 may track the distal end 818 and / or one or more of the segments 824 of the flexible body 816 using a shape sensor 822. It should be noted that, with the techniques of the instant disclosure, the shape sensor 822 may be omitted. The shape sensor 822 may include an optical fiber aligned with the flexible body 816 (e.g., provided within an interior channel of the flexibly body 816 or mounted externally along the flexible body 816). In some examples, the optical fiber may have a diameter of approximately 800 pm. In other examples, the diameter may be larger or smaller. The optical fiber of the shape sensor 822 may form a fiber optic bend sensor for determining the shape of flexible body 816. Optical fibers including Fiber Bragg Gratings (FBGs) may be used to provide strain measurements in structures in one or more dimensions. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions, which may be applicable in some examples, are described in U.S. Patent Application Publication No. 2006 / 0013523 (filed July 13, 2005 and titled “Fiber optic position and shape sensing device and method relating thereto”); U.S. Patent No. 7,772,541 (filed on March 12, 2008 and titled “Fiber Optic Position and / or Shape Sensing Based on Rayleigh Scatter”); and U.S. Patent No. 8,773,650 (filed on Sept. 2, 2010 and titled “Optical Position and / or Shape Sensing”), which are all incorporated by reference herein in their entireties. Sensors in some examples may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and Fluorescence scattering.
[0121] In some examples, the shape of the flexible body 816 may be determined using other techniques. For example, a history of the position and / or pose of the distal end 818 of the flexible body 816 can be used to reconstruct the shape of flexible body 816 over an interval of time (e.g., as the flexible body 816 is advanced or retracted within a patient anatomy). In some examples, the tracking system 830 may alternatively and / or additionally track the distal end 818 of the flexible body 816 using a position sensor system 820. Position sensor system 820 may be a component of an EM sensor system with the position sensor system 820 including one or more position sensors. Although the position sensor system 820 is shown as being near the distal end 818 of the flexible body 816 to track the distal end 818, the numberIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 and location of the position sensors of the position sensor system 820 may vary to track different regions along the flexible body 816. In one example, the position sensors include conductive coils that may be subjected to an externally generated electromagnetic field. Each coil of position sensor system 820 may produce an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field. The position sensor system 820 may measure one or more position coordinates and / or one or more orientation angles associated with one or more portions of flexible body 816. In some examples, the position sensor system 820 may be configured and positioned to measure six degrees of freedom, e.g., three position coordinates X, Y, Z and three orientation angles indicating pitch, yaw, and roll of a base point. In some examples, the position sensor system 820 may be configured and positioned to measure five degrees of freedom, e.g., three position coordinates X, Y, Z and two orientation angles indicating pitch and yaw of a base point. Further description of a position sensor system, which may be applicable in some examples, is provided in U.S. Patent No. 6,380,732 (filed August 11, 1999 and titled “Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked”), which is incorporated by reference herein in its entirety.
[0122] A processing unit (e.g., processing unit 120) may enhance the accuracy of positions obtained by the position sensor system 820 by combing data obtained by the position sensor system 820 with data obtained by an external imaging system (e.g., by way of the imaging unit 110) according to the techniques of this disclosure described above.
[0123] In some examples, the tracking system 830 may alternately and / or additionally rely on a collection of pose, position, and / or orientation data stored for a point of an elongate device 802 and / or medical tool 826 captured during one or more cycles of alternating motion, such as breathing. This stored data may be used to develop shape information about the flexible body 816. In some examples, a series of position sensors (not shown), such as EM sensors like the sensors in position sensor system 820 or some other type of position sensors may be positioned along the flexible body 816 and used for shape sensing. In some examples, a history of data from one or more of these position sensors taken during a procedure may be used to represent the shape of elongate device 802, particularly if an anatomic passageway is generally static.
[0124] FIG. 8B is a simplified diagram of the medical tool 826 within the elongate device 802 according to some examples. The flexible body 816 of the elongate device 802 mayIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 include the channel 821 sized and shaped to receive the medical tool 826. In some examples, the medical tool 826 may be used for procedures such as imaging, surgery, biopsy, ablation, illumination, irrigation, suction, electroporation, etc. Medical tool 826 can be deployed through channel 821 of flexible body 816 and operated at a procedural site within the anatomy. Medical tool 826 may be, for example, an image capture probe, a biopsy tool (e.g., a needle, grasper, brush, etc.), an ablation tool (e.g., a laser ablation tool, radio frequency (RF) ablation tool, cryoablation tool, thermal ablation tool, heated liquid ablation tool, etc.), an electroporation tool, and / or another surgical, diagnostic, or therapeutic tool. In some examples, the medical tool 826 may include an end effector having a single working member such as a scalpel, a blunt blade, an optical fiber, an electrode, and / or the like. Other end types of end effectors may include, for example, forceps, graspers, scissors, staplers, clip appliers, and / or the like. Other end effectors may further include electrically activated end effectors such as electrosurgical electrodes, transducers, sensors, and / or the like.
[0125] The medical tool 826 may be a biopsy tool used to remove sample tissue or a sampling of cells from a target anatomic location. In some examples, the biopsy tool is a flexible needle. The biopsy tool may further include a sheath that can surround the flexible needle to protect the needle and interior surface of the channel 821 when the biopsy tool is within the channel 821. The medical tool 826 may be an image capture probe that includes a distal portion with a stereoscopic or monoscopic camera that may be placed at or near the distal end 818 of flexible body 816 for capturing images (e.g., still or video images). The captured images may be processed by the visualization system 831 for display and / or provided to the tracking system 830 to support tracking of the distal end 818 of the flexible body 816 and / or one or more of the segments 824 of the flexible body 816. The image capture probe may include a cable for transmitting the captured image data that is coupled to an imaging device at the distal portion of the image capture probe. In some examples, the image capture probe may include a fiber-optic bundle, such as a fiberscope, that couples to a more proximal imaging device of the visualization system 831. The image capture probe may be single- spectral or multi-spectral, for example, capturing image data in one or more of the visible, near-infrared, infrared, and / or ultraviolet spectrums. The image capture probe may also include one or more light emitters that provide illumination to facilitate image capture. In some examples, the image capture probe may use ultrasound, x-ray, fluoroscopy, CT, MRI, or other types of imaging technology.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0126] In some examples, the image capture probe is inserted within the flexible body 816 of the elongate device 802 to facilitate visual navigation of the elongate device 802 to a procedural site and then is replaced within the flexible body 816 with another type of medical tool 826 that performs the procedure. In some examples, the image capture probe may be within the flexible body 816 of the elongate device 802 along with another type of medical tool 826 to facilitate simultaneous image capture and tissue intervention, such as within the same channel 821 or in separate channels. A medical tool 826 may be advanced from the opening of the channel 821 to perform the procedure (or some other functionality) and then retracted back into the channel 821 when the procedure is complete. The medical tool 826 may be removed from the proximal end 817 of the flexible body 816 or from another optional instrument port (not shown) along flexible body 816.
[0127] In some examples, the elongate device 802 may include integrated imaging capability rather than utilize a removable image capture probe. For example, the imaging device (or fiber-optic bundle) and the light emitters may be located at the distal end 818 of the elongate device 802. The flexible body 815 may include one or more dedicated channels that carry the cable(s) and / or optical fiber(s) between the distal end 818 and the visualization system 831. Here, the medical instrument system 800 can perform simultaneous imaging and tool operations.
[0128] In some examples, the medical tool 826 is capable of controllable articulation. The medical tool 826 may house cables (which may also be referred to as pull wires), linkages, or other actuation controls (not shown) that extend between its proximal and distal ends to controllably bend the distal end of medical tool 826, such as discussed herein for the flexible elongate device 802. The medical tool 826 may be coupled to a drive unit 804 and the manipulator assembly 702. In these examples, the elongate device 802 may be excluded from the medical instrument system 800 or may be a flexible device that does not have controllable articulation. Steerable instruments or tools, applicable in some examples, are further described in detail in U.S. Patent No. 7,316,681 (filed on Oct. 4, 2005 and titled “Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity”) and U.S. Patent No. 9,259,274 (filed Sept. 30, 2008 and titled “Passive Preload and Capstan Drive for Surgical Instruments”), which are incorporated by reference herein in their entireties.
[0129] The flexible body 816 of the elongate device 802 may also or alternatively house cables, linkages, or other steering controls (not shown) that extend between the drive unit 804Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 and the distal end 818 to controllably bend the distal end 818 as shown. In some examples, at least four cables are used to provide independent up-down steering to control a pitch of the distal end 818 and left-right steering to control a yaw of the distal end 881. In these examples, the flexible elongate device 802 may be a steerable catheter. Examples of steerable catheters, applicable in some examples, are described in detail in PCT Publication WO 2019 / 018736 (published Jan. 24, 2019 and titled “Flexible Elongate Device Systems and Methods”), which is incorporated by reference herein in its entirety.
[0130] In examples where the elongate device 802 and / or medical tool 826 are actuated by a teleoperational assembly (e.g., the manipulator assembly 702), the drive unit 804 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of the teleoperational assembly. In some examples, the elongate device 802 and / or medical tool 826 may include gripping features, manual actuators, or other components for manually controlling the motion of the elongate device 802 and / or medical tool 826. The elongate device 802 may be steerable or, alternatively, the elongate device 802 may be nonsteerable with no integrated mechanism for operator control of the bending of distal end 818. In some examples, one or more channels 821 (which may also be referred to as lumens), through which medical tools 826 can be deployed and used at a target anatomical location, may be defined by the interior walls of the flexible body 816 of the elongate device 802.
[0131] In some examples, the medical instrument system 800 (e.g., the elongate device 802 or medical tool 826) may include a flexible bronchial instrument, such as a bronchoscope or bronchial catheter, e.g., for use in visual examination and diagnosis, biopsy, and / or treatment of a lung. The medical instrument system 800 may also be suited for navigation and treatment of other tissues, via natural or surgically created connected passageways, in any of a variety of anatomic systems, including the colon, the intestines, the kidneys and kidney calices, the brain, the heart, the circulatory system including vasculature, and / or the like.
[0132] The information from the tracking system 830 may be sent to the navigation system 832, where the information may be combined with information from the visualization system 831 and / or pre-operatively obtained models to provide the physician, clinician, surgeon, or other operator with real-time position information. The tracking system 830, the navigation system 832, and the visualization system 831 may cooperatively implement, at least partially, the functionality of the system 100 in implementing the techniques described above. In some examples, the real-time position information may be displayed on the display system 710 for use in the control of the medical instrument system 800. In some examples, the navigationIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 system 832 may utilize the position information as feedback for positioning medical instrument system 800.
[0133] FIGS. 9 A and 9B are simplified diagrams of side views of a patient coordinate space including a medical instrument mounted on an insertion assembly according to some examples. As shown in FIGS. 9 A and 9B, a surgical environment 900 may include the patient P positioned on the patient table T. Patient P may be stationary within the surgical environment 900 in the sense that gross patient movement is limited by sedation, restraint, and / or other means. Cyclic anatomic motion, including respiration and cardiac motion, of patient P may continue. Within surgical environment 900, a medical instrument 904 is used to perform a medical procedure which may include, for example, surgery, biopsy, ablation, illumination, irrigation, suction, or electroporation. The medical instrument 904 may also be used to perform other types of procedures, such as a registration procedure to associate the position, orientation, and / or pose data captured by the sensor system 708 to a desired (e.g., anatomical or system) reference frame. The medical instrument 904 may be, for example, the medical instrument 704. In some examples, the medical instrument 904 may include an elongate device 910 (e.g., a catheter) coupled to an instrument body 912. Elongate device 910 may be the elongate device 140 of FIG. 1. Elongate device 910 includes one or more channels sized and shaped to receive a medical tool.
[0134] Elongate device 910 may also include one or more sensors (e.g., components of the sensor system 708). In some examples, a shape sensor 914 may be fixed at a proximal point 916 on the instrument body 912. The proximal point 916 of the shape sensor 914 may be movable with the instrument body 912, and the location of the proximal point 916 with respect to a desired reference frame may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 914 may measure a shape from the proximal point 916 to another point, such as a distal end 918 of the elongate device 910. The shape sensor 914 may be aligned with the elongate device 910 (e.g., provided within an interior channel or mounted externally). In some examples, the shape sensor 914 may optical fibers used to generate shape information for the elongate device 910.
[0135] In some examples, position sensors (e.g., EM sensors) may be incorporated into the medical instrument 904. A series of position sensors may be positioned along the flexible elongate device 910 and used for shape sensing. Position sensors may be used alternatively to the shape sensor 914 or with the shape sensor 914, such as to improve the accuracy of shape sensing or to verify shape information.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786
[0136] Elongate device 910 may house cables, linkages, or other steering controls that extend between the instrument body 912 and the distal end 918 to controllably bend the distal end 918. In some examples, at least four cables are used to provide independent up-down steering to control a pitch of distal end 918 and left-right steering to control a yaw of distal end 918. The instrument body 912 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of a manipulator assembly.
[0137] The instrument body 912 may be coupled to an instrument carriage 906. The instrument carriage 906 may be mounted to an insertion stage 908 that is fixed within the surgical environment 900. Alternatively, the insertion stage 908 may be movable but have a known location (e.g., via a tracking sensor or other tracking device) within surgical environment 900. Instrument carriage 906 may be a component of a manipulator assembly (e.g., manipulator assembly 702) that couples to the medical instrument 904 to control insertion motion (e.g., motion along an insertion axis A) and / or motion of the distal end 918 of the elongate device 910 in multiple directions, such as yaw, pitch, and / or roll. The instrument carriage 906 or insertion stage 908 may include actuators, such as servomotors, that control motion of instrument carriage 906 along the insertion stage 908.
[0138] A sensor device 920, which may be a component of the sensor system 708, may provide information about the position of the instrument body 912 as it moves relative to the insertion stage 908 along the insertion axis A. The sensor device 920 may include one or more resolvers, encoders, potentiometers, and / or other sensors that measure the rotation and / or orientation of the actuators controlling the motion of the instrument carriage 906, thus indicating the motion of the instrument body 912. In some examples, the insertion stage 908 has a linear track as shown in FIGS. 9A and 9B. In some examples, the insertion stage 908 may have curved track or have a combination of curved and linear track sections.
[0139] FIG. 9 A shows the instrument body 912 and the instrument carriage 906 in a retracted position along the insertion stage 908. In this retracted position, the proximal point 916 is at a position E0 on the insertion axis A. The location of the proximal point 916 may be set to a zero value and / or other reference value to provide a base reference (e.g., corresponding to the origin of a desired reference frame) to describe the position of the instrument carriage 906 along the insertion stage 908. In the retracted position, the distal end 918 of the elongate device 910 may be positioned just inside an entry orifice of patient P. Also in the retracted position, the data captured by the sensor device 920 may be set to a zero value and / or other reference value (e.g., 1=0). In FIG. 9B, the instrument body 912 and theIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 instrument carriage 906 have advanced along the linear track of insertion stage 908, and the distal end 918 of the elongate device 910 has advanced into patient P. In this advanced position, the proximal point 916 is at a position LI on the insertion axis A. In some examples, the rotation and / or orientation of the actuators measured by the sensor device 920 indicating movement of the instrument carriage 906 along the insertion stage 908 and / or one or more position sensors associated with instrument carriage 906 and / or the insertion stage 908 may be used to determine the position LI of the proximal point 916 relative to the position L0. In some examples, the position LI may further be used as an indicator of the distance or insertion depth to which the distal end 918 of the elongate device 910 is inserted into the passageway(s) of the anatomy of patient P.
[0140] One or more components of the examples discussed in this disclosure may be implemented in software for execution on one or more processors of a computer system. The software may include code that when executed by the one or more processors, configures the one or more processors to perform various functionalities as discussed herein. The code may be stored in a non-transitory computer readable storage medium (e.g., a memory, magnetic storage, optical storage, solid-state storage, etc.). The computer readable storage medium may be part of a computer readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded via computer networks such as the Internet, Intranet, etc. for storage on the computer readable storage medium. The code may be executed by any of a wide variety of centralized or distributed data processing architectures. The programmed instructions of the code may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein. The components of the computing systems discussed herein may be connected using wired and / or wireless connections. In some examples, the wireless connections may use wireless communication protocols such as Bluetooth, near-field communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 802.11, Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).
[0141] Various general-purpose computer systems may be used to perform one or more processes, methods, or functionalities described herein. Additionally or alternatively, various specialized computer systems may be used to perform one or more processes, methods, orIntuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 functionalities described herein. In addition, a variety of programming languages may be used to implement one or more of the processes, methods, or functionalities described herein.
[0142] While certain examples and examples have been described above and shown in the accompanying drawings, it is to be understood that such examples and examples are merely illustrative and are not limited to the specific constructions and arrangements shown and described, since various other alternatives, modifications, and equivalents will be appreciated by those with ordinary skill in the art.
Claims
Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 WHAT IS CLAIMED IS:
1. A system for registering three-dimensional (3D) shape data to two-dimensional (2D) image data, the system comprising:one or more processors; andone or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to:obtain, via an imaging device, a frame of 2D image data depicting a flexible elongate device;extract the flexible elongate device from the frame of 2D image data; predict a pose of the imaging device associated with the frame of 2D image data;obtain, via a shape sensor, 3D shape sensor data representative of a shape of the flexible elongate device;generate a 2D projection of the flexible elongate device by applying, to the 3D shape sensor data, a projection matrix associated with the predicted pose; and based on the extracted flexible elongate device and the generated 2D projection of the flexible elongate device, register the 3D shape sensor data to the frame of 2D image data.
2. The system of claim 1, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:optimize the predicted pose to reduce a loss associated with a registration algorithm applied to the 2D projection of the flexible elongate device and the extracted flexible elongate device, wherein the 2D projection is associated with a predicted pose having a minimum loss.
3. The system of claim 2, wherein the registration algorithm is an iterative closest point (ICP) registration algorithm.
4. The system of claim 2, wherein to reduce the loss the instructions, when executed by the one or more processors, further cause the one or more processors to:reduce the loss based on a distance between point pairs selected by the registration algorithm.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 5. The system of claim 4, wherein to reduce the loss the instructions, when executed by the one or more processors, further cause the one or more processors to:reduce the loss based on a weighted mean that applies respective weights to the loss associated with selected point pairs based on a distance of the selected point pairs from a tip of the flexible elongate device.
6. The system of claim 5, wherein the tip of the flexible elongate device includes a predetermined number of point pairs.
7. The system of claim 2, wherein to optimize the predicted pose of the imaging device, the instructions, when executed by the one or more processors, cause the one or more processors to:predict the pose of the imaging device with respect to six degrees of freedom (DOF).
8. The system of claim 1, wherein the frame of 2D image data is included in a plurality of frames of 2D image data associated with a respective plurality of predicted poses of the imaging device.
9. The system of claim 8, wherein to optimize the predicted pose, the instructions, when executed by the one or more processors, cause the one or more processors to:predict the plurality of predicted poses of the imaging device for the plurality of frames of 2D image data;fit a circular plane to the plurality of predicted poses; andconstrain, based on the fitted circular plane, selection of a next predicted pose when optimizing the predicted pose of a frame of 2D image data.
10. The system of claim 9, wherein to fit the circular plane to the plurality of predicted poses, the instructions, when executed by the one or more processors, cause the one or more processors to:apply a random sample consensus (RANSAC) algorithm to the plurality of predicted poses.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 11. The system of claim 9, wherein to constrain the selection, the instructions, when executed by the one or more processors, cause the one or more processors to:reduce a distance between the predicted pose and the circular plane.
12. The system of claim 9, wherein to constrain the selection, the instructions, when executed by the one or more processors, cause the one or more processors to:calculate a predicted trajectory of the imaging device based on the plurality of predicted poses; andreduce an angular error between the predicted trajectory and the circular plane.
13. The system of any one of claims 1-12, wherein the 2D image data includes depictions of one or more reference points from which the pose of the imaging device can be derived for a first set of viewing angles, but not a second set of viewing angles.
14. The system of claim 13, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:calculate a trajectory of the imaging device across the first set of viewing angles; extrapolate the trajectory to predict the pose of the imaging device through the second set of viewing angles; andapply the projection matrix to the 3D shape sensor data generated at a viewing angle of the second set of viewing angles based on the extrapolated trajectory.
15. The system of claim 13, wherein the one or more reference points are fiducial markings or represent an anatomical feature.
16. A computer-implemented method for registering three-dimensional (3D) shape data to two-dimensional (2D) image data, the computer-implemented method comprising:obtaining, by one or more processors via an imaging device, a frame of 2D image data depicting a flexible elongate device;extracting, by the one or more processors, the flexible elongate device from the frame of 2D image data;predicting, by the one or more processors, a pose of the imaging device associated with the frame of 2D image data;Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 obtaining, by the one or more processors via a shape sensor, 3D shape sensor data representative of a shape of the flexible elongate device;generating, by the one or more processors, a 2D projection of the flexible elongate device by applying, to the 3D shape sensor data, a projection matrix associated with the predicted pose; andbased on the extracted flexible elongate device and the generated 2D projection of the flexible elongate device, registering, by the one or more processors, the 3D shape sensor data to the frame of 2D image data.
17. The method of claim 16, further comprising:optimizing, by the one or more processors, the predicted pose to reduce a loss associated with a registration algorithm applied to the 2D projection of the flexible elongate device and the extracted flexible elongate device, wherein the 2D projection is associated with a predicted pose having a minimum loss.
18. The method of claim 17, wherein the registration algorithm is an iterative closest point (ICP) registration algorithm.
19. The method of claim 17, wherein calculating the loss comprises: calculating, by the one or more processors, the loss based on a distance between point pairs selected by the registration algorithm.
20. The method of claim 19, wherein calculating the loss comprises: calculating, by the one or more processors, the loss based on a weighted mean that applies respective weights to the loss associated with selected point points pairs based on a distance of the selected point pairs from a tip of the flexible elongate device.
21. The method of claim 20, wherein the tip of the flexible elongate device includes a predetermined number of point pairs.
22. The method of claim 17, wherein optimizing the predicted pose of the imaging device further comprises:predicting, by the one or more processors, the pose of the imaging device with respect to six degrees of freedom (DOF).Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 7078623. The method of claim 17, wherein the frame of 2D image data is included in a plurality of frames of 2D image data associated with a respective plurality of predicted poses of the imaging device.
24. The method of claim 23, wherein optimizing the predicted pose further comprises:predicting, by the one or more processors, the plurality of predicted poses of the imaging device for the plurality of frames of 2D image data;fitting, by the one or more processors, a circular plane to the plurality of predicted poses; andconstraining, by the one or more processors based on the fitted circular plane, selection of a next predicted pose when optimizing the predicted pose of a frame of 2D image data.
25. The method of claim 24, wherein fitting the circular plane to the plurality of predicted poses further comprises:applying, by the one or more processors, a random sample consensus (RANSAC) algorithm to the plurality of predicted poses.
26. The method of claim 24, wherein constraining the selection comprises: reducing, by the one or more processors, a distance between the predicted pose and the circular plane.
27. The method of claim 24, wherein constraining the selection comprises: calculating, by the one or more processors, a predicted trajectory of the imaging device based on the plurality of predicted poses; andreducing, by the one or more processors, an angular error between the predicted trajectory and the circular plane.
28. The method of any one of claims 16-27, wherein the 2D image data includes depictions of one or more reference points from which the pose of the imaging device can be derived for a first set of viewing angles, but not a second set of viewing angles.Intuitive Docket No.: P07007-WO Attorney Docket No.: 33685 / 70786 29. The method of claim 28, further comprising:calculating, by the one or more processors, a trajectory of the imaging device across the first set of viewing angles;extrapolating, by the one or more processors, the trajectory to predict the pose of the imaging device through the second set of viewing angles; andapplying, by the one or more processors, the projection matrix to the 3D shape sensor data generated at a viewing angle of the second set of viewing angles based on the extrapolated trajectory.
30. The method of claim 28, wherein the one or more reference points are fiducial markings or represent an anatomical feature.
31. A tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 16-30.
