Method and system for deviation compensation

Abstract

A method for deviation compensation in a tool insertion procedure includes generating a representation of a target and one or more target points disposed on the target. For each target point of the one or more target points, an associated trajectory for an elongated tool to strike the target point is determined. Prior to or during insertion of the elongated tool, a deviation is automatically detected and an alert associated with the deviation is automatically generated. Upon receiving a user confirmation responsive to the alert, a compensation based on the detected deviation is automatically performed. A robotic system implementing the method is also disclosed.

Description

METHOD AND SYSTEM FOR DEVIATION COMPENSATIONTECHNICAL FIELD

[0001] The present disclosure relates broadly, but not exclusively, to methods and systems for deviation compensation in a tool insertion procedure.BACKGROUND

[0002] Prostate biopsy, which uses special needles to collect tissue samples of the prostate gland for histologic examination, is an essential procedure in the clinical pathway of prostate cancer management. This procedure is usually performed by the guidance of transrectal ultrasound (TRUS) in the transrectal or transperineal manner. Within the past decade, transperineal methods have gradually demonstrated its advantages over transrectal method because of the much less post-biopsy infection.

[0003] In addition, robotic systems with magnetic resonance imaging (MRI) -ultrasound fusion, auto-planning and needle trajectory computing algorithms have been developed to assist the urologist in performing the biopsy procedure. The steps involved include volumetric ultrasound image data acquisition, prostate modelling and lesion recognition, multi-modality image fusion, biopsy core planning and needle positioning. Clinical studies have demonstrated that such robotic biopsy systems are able to standardize the workflow, reduce the urologist’s learning curve and workload, and more importantly, elevate the biopsy precision with higher positive rates.

[0004] In the real clinical scenario, the patient may move due to the discomfort and / or pain in the needle positioning and insertion stage, especially under the local anesthesia condition. Therefore, the prostate may also move, which produces the spatial deviation between the pre-built ultrasound prostate model and the actual prostate position at the time of the biopsy procedure. As the biopsy cores are planned in the pre-built ultrasound prostate model, such a deviation can cause deviations of the needle landing points in the prostate. It means the biopsy needlesmay not land at the desired landing spots in the prostate, which may impact the biopsy results considerably.

[0005] Currently, to tackle this problem, the live ultrasound stream is turned on and displayed at the software GUI for the user's observation during the needle positioning and insertion stage. The corresponding prostate contour from a prebuilt ultrasound prostate model is superimposed to the live ultrasound, so that the user can determine whether the spatial deviation between the pre-built prostate model and the real-time prostate location in 3D space is significant for adjustment. Figure 1 shows the deviation of the prostate contour from the pre-built prostate model 102 to the actual prostate margin 104 shown from the live ultrasound image. If the user determines that a significant deviation movement has occurred, which indicates a large movement of the prostate, he / she can choose to perform the adjustment to compensate.

[0006] This process has a few problems. Firstly, the determination of the occurrence of a significant prostate movement is subjective and user dependent. In addition, under a stressful working environment, the user may not notice the movement of the prostate or ignore the changes. Secondly, in the adjustment process, the current adjustment algorithm is based on manual adjustment in a single axial plane. If the user enters the movement adjustment window, he / she will perform manual operations in translation, rotation and scaling (zoom in / out) on the 2D transversal contour of the pre-built prostate model in a single axial plane, to fit the actual prostate margin from the live ultrasound scan. Once the adjustment is confirmed, the software will update the needle trajectory based on a transformation matrix reflecting the adjustments in translation, rotation and scaling. With such a process, the adjustment only reflects the changes in one single axial plane, which may not well reflect the change in 3D. Moreover, whether the adjusted prostate contour fits the actual prostate margin in live ultrasound is also subjective and user dependent. These problems, without proper solutions, can impact the precision of the biopsy needle landing and subsequently bring negative influence on the positive rate of prostate cancer detection.

[0007] Also, in the real clinical scenario, the inserted needle may deflect or bend in patient’s prostate due to reasons including the heterogeneity of tissue structure, tissue resistance, elastic recoil of the tissue, needle flexibility and design, insertion angle, etc. With the occurrence of needle bending or deflection, the actual needlelanding spot may deviate from the planned spot. Thus, at the step of needle insertion, the user will need to monitor the actual landing spot of the needle tip and determine whether there is substantial deviation from the planned needle core for correction. In the live ultrasound image, the landing of the needle at the scanning plane is represented by a flash of high echo mass if the needle trajectory is perpendicular to the scanning plane, or a flash of high echo mass with a tail if the needle trajectory is inclined to the scanning plane. Hence, for this process, the user may have to manually select the landing spot from the high echo flash mass by screen touching or a stylus pen. This selection is subjective from doctor to doctor and such operations may also affect the user’s concentration on the entire procedure.

[0008] It may be desirable to provide methods and devices that can address at least some of the above problems.SUMMARY

[0009] According to an aspect of the present disclosure, there is provided a method for deviation compensation in a tool insertion procedure, the method comprising generating a representation of a target and one or more target points disposed on the target; for each target point of the one or more target points, determining an associated trajectory for an elongated tool to strike the target point; prior to or during insertion of the elongated tool, automatically detecting a deviation and generating an alert associated with the deviation; upon receiving a user confirmation responsive to the alert, automatically performing a compensation based on the detected deviation.

[0010] According to another aspect of the present disclosure, there is provided a system for deviation compensation in a tool insertion procedure, the system comprising a processor; a computer-readable memory communicatively coupled to the processor; and an imaging device communicatively coupled to the processor, wherein the processor is configured to: generate a representation of a target and one or more target points disposed on the target; for each target point of the one or more target points, determine an associated trajectory for an elongated tool to strike the target point; prior to or during insertion of the elongated tool, automatically detect a deviation and generate an alert associated with thedeviation; based on a user confirmation responsive to the alert, automatically perform a compensation based on the detected deviation.

[0011] Generating the representation of the target may comprise fusing a first set of imaging data obtained by a first imaging mode with a second set of imaging data obtained by a second imaging mode different from the first imaging mode. For example, the first imaging mode may comprise magnetic resonance imaging and the second imaging mode may comprise ultrasound imaging.

[0012] The deviation may comprise a movement of the target prior to insertion of the elongated tool, and automatically detecting the deviation may comprise capturing, at a selected frequency, real-time contours of the target at a reference plane; comparing each of the real-time contours of the target at the reference plane with a stored contour of the target at the reference plane, wherein the stored contour of the target at the reference plane is based on the representation of the target; and generating the alert if a difference between the real-time contour and the stored contour is greater than a predetermined threshold.

[0013] In that case, automatically performing the compensation may comprise determining a three-dimensional (3D) transformation based on differences between real-time contours of the target at a selected plurality of planes and corresponding stored contours of the target at the selected plurality of planes, wherein the stored contours of the target at the selected plurality of planes are based on the representation of the target; applying the 3D transformation to the representation of the target to obtain an updated representation of the target; and regenerating the one or more target points and associated trajectories based on the updated representation of the target. The selected plurality of planes may include, relative to an axial direction of an imaging device, the reference plane, at least one plane fore of the reference plane, and at least one plane aft of the reference plane.

[0014] The deviation may comprise a deflection of the elongated tool during insertion, and automatically detecting the deviation may comprise determining an actual landing point of the elongated tool; and generating the alert if a difference between the actual landing point of the elongated tool and the target point is greater than a predetermined threshold. In that case, performing the compensation may comprise withdrawing the tool; and regenerating the trajectory based on thedifference between the determined actual landing point of the elongated tool and the target point

[0015] Further, determining the actual landing point of the elongated tool may comprise generating, at a selected frequency, a plurality of ultrasound images at a transverse plane passing through the target point; for each ultrasound image of the plurality of ultrasound images, detecting a flash representing a position of the elongated tool; selecting the ultrasound image having the largest flash; and determining a center of the flash in the selected ultrasound image. Detecting the flash may comprise comparing said ultrasound image with a reference ultrasound image at the transverse plane passing through the target point, the reference ultrasound image being free of the elongated tool.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Embodiments of the disclosure will be better understood and readily apparent to one of ordinary skill in the art from the following written description, by way of example only, and in conjunction with the drawings, in which:

[0017] Figure 1 is a two-dimensional (2D) transversal ultrasound image (i.e. an image taken at a transverse plane) showing a deviation of the prostate contour of the actual prostate margin from the pre-built prostate model.

[0018] Figure 2 is a 2D transversal ultrasound image of the prostate, the prostate contour and the two targeted biopsy cores.

[0019] Figure 3 is a 3D view of the prostate with two needle trajectories superimposed thereon.

[0020] Figure 4 is a flow chart illustrating a method for deviation compensation in a tool insertion procedure according to an example embodiment.

[0021] Figure 5 is a flow chart illustrating an example workflow of target movement detection and compensation.

[0022] Figure 6A is a schematic diagram of a side (i.e. sagittal) view illustrating the spatial relationship between the prostate before / after movement and theultrasound probe, with the ultrasound probe positioned to capture an image at a reference Plane O.

[0023] Figure 6B is the corresponding transversal view of Figure 6A showing the spatial relationship between the prostate before I after movement and the ultrasound probe.

[0024] Figure 7A is a schematic diagram of a side (i.e. sagittal) view illustrating the spatial relationship between the prostate before I after movement and the ultrasound probe, with the ultrasound probe positioned to capture an image at a Plane A aft of Plane O.

[0025] Figure 7B is the corresponding transversal view of Figure 7A showing the spatial relationship between the prostate before / after movement and the ultrasound probe.

[0026] Figure 8A is a schematic diagram of a side (i.e. sagittal) view illustrating the spatial relationship between the prostate before I after movement and the ultrasound probe, with the ultrasound probe positioned to capture an image at a Plane B fore of Plane O.

[0027] Figure 8B is the corresponding transversal view of Figure 8A showing the spatial relationship between the prostate before I after movement and the ultrasound probe.

[0028] Figure 9 is a flow chart illustrating an example workflow of tool deflection detection and compensation.

[0029] Figure 10A is an example transversal ultrasound image of a prostate showing a flash mass.

[0030] Figure 10B is an example transversal ultrasound image of the prostate in Figure 10A with a marker identifying the landing point of the needle added and a message displayed.

[0031] Figure 11 shows a schematic diagram of an example computer system capable of implementing the method according to the present embodiments.

[0032] Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been depicted to scale. For example, the dimensions of some of the elements in the illustrations, block diagrams or flowcharts may be exaggerated in respect to other elements to help to improve understanding of the present embodiments.DETAILED DESCRIPTION

[0033] The present disclosure provides methods and systems that use automated image recognition to detect a deviation in a tool insertion procedure, such as target movement or tool deflection. If the deviation is determined to be larger than a preset threshold, the user is prompted to make an adjustment so that the tool can strike the desired target point. If the user chooses to make the adjustment, the trajectory of the tool is automatically updated.

[0034] Embodiments will be described, by way of example only, with reference to the drawings. Like reference numerals and characters in the drawings refer to like elements or equivalents.

[0035] Figure 4 shows a flow chart 400 illustrating a method for deviation compensation in a tool insertion procedure. At step 402, a representation of a target and one or more target points disposed on the target is generated. At step 404, for each target point of the one or more target points, an associated trajectory for an elongated tool to strike the target point is determined. In some example procedures, only one target point is sufficient. In other procedures similar to the example shown in Figures 2 and 3, multiple target points spatially distributed on the target are used. At step 406, prior to or during insertion of the elongated tool, a deviation is automatically detected and an alert associated with the deviation is generated. At step 408, upon receiving a user confirmation responsive to the alert, a compensation based on the detected deviation is automatically performed. The compensation may include updating the trajectory of the elongated tool.

[0036] In the examples that follow, the tool insertion procedure is described in relation to a prostate biopsy. However, it will be appreciated by a person skilled in the art that the methods and systems of the present disclosure can be applied to other types of biopsies (e.g. liver biopsy), other minimally-invasive surgeries, ormore generally, other procedures to access and / or extract an object that is occluded. The prostate biopsy procedure may be robot-assisted and controlled by a computer communicatively coupled to at least one imaging device and at least one robotic arm. In other words, the biopsy may be carried out with the help of a robotic system.

[0037] The typical workflow of prostate biopsy includes following stages:

[0038] Stage 1: Annotation of prostate contour and suspicious lesion(s) on patient’s pre-operative MRI data, to build the MRI-based prostate model.

[0039] Stage 2: In the intra-operative setting, TRUS scan on patient's prostate is performed. During the scan, the ultrasound probe, controlled by a motor, moves in patient’s rectum from prostate base to apex with a pre-defined scan range and internal. The video output from the ultrasound scanner is fed into a frame grabber to record and save as three-dimensional (3D) volumetric ultrasound data.

[0040] Stage 3: Annotation of prostate contour on the intra-operative ultrasound data acquired in Stage 2 to build the ultrasound-based prostate model.

[0041] Stage 4: MRI - ultrasound fusion for the registration of the MRI prostate model to the ultrasound prostate model. This serves to map the suspicious lesions marked on MRI space to the ultrasound space, which is the physical space under the human-machine coordinate, so as to highlight the regions in ultrasound-based prostate model for targeted biopsy.

[0042] Stage 5: Planning of the biopsy cores. Targeted cores (for targeted biopsy), systematics cores (for systematics biopsy) or / and saturation cores (for saturation biopsy) are generated automatically according to the prevailing clinical guidelines, and / or interactively by the user via the graphical user interface (GUI). Once the biopsy plan is confirmed, the trajectory of each needle path in the physical humanmachine coordinate will be calculated based on the geometric properties of the biopsy needle, the prostate model and the planned landing point. Figure 2 shows a two-dimensional (2D) transversal ultrasound image of the prostate, the prostate contour and the two targeted biopsy cores (T1 and T2). Figure 3 shows the 3Dview of the prostate model, the ultrasound probe and the two needle trajectories for the two planned biopsy cores respectively.

[0043] Stage 6: Needle positioning and insertion. Based on the trajectory of each needle path, the robot carrying a needle guide will move in 3D space for needle positioning. After the robot has completed the positioning, the user can insert the biopsy needle into the patient’s prostate through the needle guide and the perineal skin.

[0044] In example embodiments, a deviation is detected and compensated at Stage 6 of the biopsy workflow.

[0045] Some portions of the description which follows are explicitly or implicitly presented in terms of algorithms and functional or symbolic representations of operations on data within a computer memory. These algorithmic descriptions and functional or symbolic representations are the means used by those skilled in the data processing arts to convey most effectively the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities, such as electrical, magnetic or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

[0046] Unless specifically stated otherwise, and as apparent from the following, it will be appreciated that throughout the present specification, discussions utilizing terms such as “scanning”, “calculating", “determining”, “applying”, “extracting", “generating”, “initializing”, “outputting”, or the like, refer to the action and processes of a computer system, or similar electronic device, that manipulates and transforms data represented as physical quantities within the computer system into other data similarly represented as physical quantities within the computer system or other information storage, transmission or display devices.

[0047] The present specification also discloses apparatus for performing the operations of the methods. Such apparatus may be specially constructed for the required purposes, or may comprise a computer or other device selectively activated or reconfigured by a computer program stored in the computer. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various machines may be used with programs in accordance with theteachings herein. Alternatively, the construction of more specialized apparatus to perform the required method steps may be appropriate. The structure of a conventional computer will appear from the description below.

[0048] In addition, the present specification also implicitly discloses a computer program, in that it would be apparent to the person skilled in the art that the individual steps of the method described herein may be put into effect by computer code. The computer program is not intended to be limited to any particular programming language and implementation thereof. It will be appreciated that a variety of programming languages and coding thereof may be used to implement the teachings of the disclosure contained herein. Moreover, the computer program is not intended to be limited to any particular control flow. There are many other variants of the computer program, which can use different control flows without departing from the scope of the disclosure.

[0049] Furthermore, one or more of the steps of the computer program may be performed in parallel rather than sequentially. Such a computer program may be stored on any computer readable medium. The computer readable medium may include storage devices such as magnetic or optical disks, memory chips, or other storage devices suitable for interfacing with a computer. The computer readable medium may also include a hard-wired medium such as exemplified in the Internet system, or wireless medium such as exemplified in the GSM, GPRS, 3G, 4G or 5G mobile telephone systems, as well as other wireless systems such as Bluetooth, ZigBee, WiFi. The computer program when loaded and executed on such a computer effectively results in an apparatus that implements the steps of the preferred method.

[0050] The present disclosure may also be implemented as hardware elements. More particularly, in the hardware sense, an element is a functional hardware unit designed for use with other components or elements. For example, an element may be implemented using discrete electronic components, or it can form a portion of an entire electronic circuit such as an Application Specific Integrated Circuit (ASIC) or Field Programmable Gate Array (FPGA). Numerous other possibilities exist. Those skilled in the art will appreciate that the system can also be implemented as a combination of hardware and software elements.

[0051] According to various embodiments, a “circuit” may be understood as any kind of a logic implementing entity, which may be special purpose circuitry or aprocessor executing software stored in a memory, firmware, or any combination thereof. Thus, in an embodiment, a “circuit” may be a hard-wired logic circuit or a programmable logic circuit such as a programmable processor, e.g. a microprocessor (e.g. a Complex Instruction Set Computer (CISC) processor or a Reduced Instruction Set Computer (RISC) processor). A “circuit” may also be a processor executing software, e.g. any kind of computer program, e.g. a computer program using a virtual machine code such as e.g. Java. Any other kind of implementation of the respective functions which may be described in more detail herein may also be understood as a “circuit” in accordance with an alternative embodiment.

[0052] Figure 5 shows a flow chart 500 of an example workflow for target movement adjustment, i.e. compensation for target movement. In this example, the target is a prostate gland of a patient and the tool used is an elongated biopsy needle. Hereinafter, the expression “target movement adjustment” (TMA) is used interchangeably with “patient movement adjustment” (PMA), as movement of the prostate is generally caused by patient movement. The workflow includes automated image recognition algorithm for the detection of significant prostate movement, automated in-screen prompt to the user for adjustment if a significant prostate movement is detected, automated approximation of the prostate movement in a 3D region by ultrasound probe motion control, image recognition algorithm and 3D Non-Uniform Rational Basis Splines (NURBS) shape fitting, and generation of the transformation matrix reflecting the adjustment in 3D for needle trajectory updating.

[0053] First, a start distance is computed. A start distance is used to establish a baseline when the patient has not moved. Based on testing observations, the start distance may vary from case to case and can be up to 2.5 mm. Factors affecting the start distance include, but are not limited to, how the modelling is done, system calibration, and overall accuracy of the system. The distance computed from the first frame of the first needle is defined as the start distance, as shown by step 502 in Figure 5. Then, the detection starts at step 504. Typically, detection is only active at positioning stage, and is disabled during motor movement between needle positions or during PMA execution. Steps 502, 504, 506, 508 and 510 in Figure 5 are described in more detail below. Subsequent movement checks are evaluated against the start distance plus a predefined limit. In other words, aftercompensation (i.e. PMA in Figure 5) is applied, the start distance is recalculated using the frame immediately following PMA, as shown by step 512 in Figure 5.

[0054] TMA Stage 1: TMA enabling - In an example implementation, after the user completes the needle planning and starts the needle positioning by the robot, the system switches to the live ultrasound at a reference plane, e.g. Plane O in Figures 6A, 7A and 8A, and enables automatic TMA. For example, Plane O is a position where the imaging device, e.g. in the form of an ultrasound probe / transducer 602, is scanning from the patient’s rectum to a cross-section of the prostate, as shown in Figure 6A. In one implementation, Plane O passes through the target point where the needle is expected to land.

[0055] TMA Stage 2: Detection of significant target movement (step 506 in Figure 5) — The robotic system captures a 2D transversal ultrasound image via the frame grabber, segments the prostate by a deep learning model to extract its 2D contour (named as PCLo) at Plane O, computes the average symmetric surface distance (ASSD) between PCLo and the 2D prostate contour PCMO from the pre-built prostate model at Plane O, and compares ASSD(PCLO, PCMO) with a predefined threshold. Figure 6B is an example transversal view showing the spatial relationship between the segmented contour PCLO and the model contour PCMO at Plane O.

[0056] Example steps to compute the distance between the two contours are as follows:i. Compute bounding box BB1 of the model contour PCMO- II. Compute bounding box BB2 of the segmented contour PCLO.iii. Calculate1. Dx1 = fabs(BB1.min_x- BB2.min_x)2. Dx2 = fabs(BB1.max_x - BB2.max_x)3. Dy1 = fabs(BB1.min_y - BB2.min_y)4. Dy2 = fabs(BB2.max_y - BB2.max_y)iv. Distance = max(Dx1 , Dx2, Dy1, Dy2)v. If Distance > (Start Distance + Limit), return True; otherwise, return False.

[0057] In example embodiments, other metrics like distance between centroids of contours, Hausdorff distance of contours are ruled out as centroid distance does not detect z-direction movement and Hausdorff distance is too sensitive to error insegmentation of the contour. The limit may be 2.0 mm, for example, and can be adjusted anytime during the positioning stage by the user.

[0058] The frame capturing, prostate segmentation, ASSD computing and comparison steps are performed at a predefined rate (e.g. 1 frame per second). If ASSD(PCi_o, PCMO) is larger than the threshold, a prompt to the user for adjustment will be flashed at the image window displaying the live ultrasound image (step 508 in Figure 5). If the user decides to proceed with adjustment, the workflow will proceed to TMA Stage 3.

[0059] TMA Stage 3: Movement estimation in a 3D region - Firstly, the needle positioning stops. A motor drives the ultrasound probe 602 backwards to the apex and the transducer stops at Plane A, which has a certain distance (e g. 10 mm) to Plane O. Then the system captures a 2D transversal ultrasound image via the frame grabber, segments the prostate by a deep learning model to extract its 2D contour (named as PCLA) at Plane A, and in addition obtains the 2D prostate contour PCMA from the pre-built prostate model at Plane A as shown in Figures 7A-7B. After this step, the motor drives the ultrasound probe 602 forwards to the base and the transducer stops at Plane B, which has a certain distance (e g. 10 mm) to Plane O. Similarly, the system captures a 2D transversal ultrasound image via the frame grabber, segments the prostate by a deep learning model to extract its 2D contour (named as PCLB) at Plane B, and in addition obtains the 2D prostate contour PCMB from the pre-built prostate model at Plane B, as shown in Figures 8A-8B. By such steps, two sets of cross-sectional contours are generated:- Set 1: [PCMB PCMO PCM ]T, which represents a portion of the pre-built 3D prostate model, andSet 2: [PCLB PCLO PCLA]T, which represents a portion of the current 3D prostate.

[0060] In other words, relative to a longitudinal axis of the elongated ultrasound probe 602, Plane A is aft of the reference Plane O while Plane B is fore of the reference Plane O. It will be appreciated that, in alternate embodiments, more than one plane fore of aft of the reference Plane O may be used. Also, the distance between Plane A and Plane O, or the distance between Plane B and Plane O may be varied in alternate embodiments. The choice of the number of planes and theirrespective distances from the reference Plane O may be based on practical considerations such as speed and accuracy.

[0061] In this example, by creating NURBS points in each contour and giving the 3D coordinates, a transformation matrix TRM can be optimized to registerTRM contains translation in the X- / Y- / Z-directions, rotation in X- / Y- / Z-directions and Zoom in / out. That is:

[0062] TMA Stage 4: Movement adjustment and needle core updating (step 510 in Figure 5) - Although the prostate at the current location (after movement) is scanned at three transversal planes O, A and B, which only represents a portion of the prostate, it is sufficient and reasonable to apply the TRM into the whole prostate, to transform the pre-built prostate model (before movement) to an updated model, which represents the prostate at the latest position (after movement). Therefore, the prostate model is updated to reflect the latest shape, location and orientation of the prostate in 3D space. Once the prostate model is updated, the planned need cores will also be updated, followed by the recalculating and updating of needle trajectories.

[0063] TMA Stage 5: Resumption of needle positioning - The needle positioning resumes after biopsy cores and needle trajectories are updated.

[0064] Figure 9 shows a flow chart 900 of an example workflow for tool deflection adjustment, i.e. compensation for deflection of the tool during insertion. In this example, the target is again a prostate gland of a patient and the tool used is an elongated biopsy needle. The workflow includes automated image recognition for the detection of needle landing; automated image processing to mark the needle landing spot in the live ultrasound window; prompt to the user for adjustment if the deviation is larger than a pre-set threshold; automated needle adjustment including re-calculation of needle trajectory, needle removal from the prostate and needle repositioning.

[0065] A detailed description of the workflow and steps of needle deflection adjustment (NDA) according to an example implementation is as follows:

[0066] NDA Stage 1: Needle positioning and insertion - This stage is the same as Stage 6 of the biopsy workflow, such that after the user's clicking on 'Position' on a user interface, the live ultrasound is on, the robot carrying a needle guide moves in 3D space for needle positioning and it stops when reaching the target point. Then the user inserts the biopsy needle into the prostate through the needle guide and the perineal skin.

[0067] NDA Stage 2: Flash mass detection and marking of needle landing spot in live ultrasound - The system captures the live ultrasound frame at a reference plane and save as the reference ultrasound image at step 902. For example, the reference plane is a transverse plane passing through the target point. The reference ultrasound image is well taken before the elongated needle reaches the desired target point, so the reference ultrasound image is free of any trace of the needle. Thereafter, in the backend, the system detects a flash mass by comparing the current ultrasound frame with the reference image, as shown by step 904 in Figure 9. If a high echo flash mass is detected, a marker, e g. a red cross, will be marked at the center of the flash mass, as shown by step 906 in Figure 9. As a non-limiting example, these steps are performed at a frequency of once per second. The detailed steps in NDA Stage 2 according to an example implementation include:2.1) Given the target point, i.e. the planned biopsy core in Stage 5 of the biopsy procedure, find the bounding box with the target point at the center and width and height of 10mm, for example. Detection of the flash mass will only be within this bounding box;2.2) Apply threshold of 100 to 255 to the bounding box of both reference image and current live image frame;2.3) Calculate average of differences AvgD for each corresponding pixel within the bounding box;2.4) If AvgD < threshold (e.g. 70), consider as no flash mass; If AvgD > threshold, find the difference image ImgD within the bounding box;2.5) Find the largest connected region in ImgD;2.6) If the size of the largest connected region < a certain threshold size (e.g. 40 pixels), consider as no flash mass; If size S the threshold size, find the center of the flash mass and mark it by a marker, e g. a red cross. The center of the flash mass is considered as the actual needle landing spot.

[0068] The flash mass is indicated by an arrow in Figure 10A. The marker is indicated by a “+” mark near the tip of the upper arrow in Figure 10B, while the desired target point is indicated bynear the tip of the lower arrow in Figure 10B. Additionally, a message such as “needle deflection detected" may be displayed.

[0069] In other words, across successive frames, the detection algorithm tracks the sum of intensity differences and retains the flash position corresponding to the maximum accumulated difference. This can ensure that only the brightest flash — typically occurring when the needle is fully inserted — is reported, while suppressing transient or weaker flashes that may occur during needle insertion and withdrawal.

[0070] NDA Stage 3: Prompt of deviation to the user and triggering of NDA — At step 908, the distance between the detected needle landing spot and the target point is determined. If the distance is more than a pre-defined threshold, which can be pre-set in the application settings, the system will prompt the user for adjustment, as shown by step 910 in Figure 9.

[0071] NDA Stage 4: Needle trajectory updating, needle removal and repositioning — If the user chooses to proceed with adjustment, the system will recalculate the needle trajectory to compensate for the deviation of the flash mass center to the planned target point, remove the needle from the prostate and the perineal skin, and the robot will re-position the needle guide for needle re-insertion, as shown by step 912 in Figure 9.

[0072] In alternate embodiments, if the user clears the marker or marks the marker manually, this indicates that the automated detection is not accurate and the user prefers to overwrite it. Thus, the automatic detection will be turned off. It will only be turned on again when the user goes to the positioning stage for the next needle.In other words, manual intervention by the user is possible in certain cases based on specific circumstances or the user’s training and experience.

[0073] The automatic detection of a deviation in the tool insertion procedure and automatic compensation for the deviation as described above can reduce the manual effort of the user and allow a timely adjustment of the tool trajectory so that the tool can strike the desired target point. In particular, in the example of a biopsy procedure, the method and system of the present disclosure allow real-time, automated detection of deviation, such as target movement or biopsy needle deflection, provide image-based feedback, e.g. in the form of ultrasound images, and provide closed-loop update of needle trajectory during the procedure.

[0074] Figure 11 depicts an exemplary computing device 1100, hereinafter interchangeably referred to as a computer system 1100, where one or more such computing devices 1100 may be used as the computer in the present system. The following description of the computing device 1100 is provided by way of example only and is not intended to be limiting.

[0075] As shown in Figure 11, the example computing device 1100 includes a processor 1104 for executing software routines. Although a single processor is shown for the sake of clarity, the computing device 1100 may also include a multi-processor system. The processor 1104 is connected to a communication infrastructure 1106 for communication with other components of the computing device 1100. The communication infrastructure 1106 may include, for example, a communications bus, cross-bar, or network.

[0076] The computing device 1100 further includes a main memory 1108, such as a random access memory (RAM), and a secondary memory 1110. The secondary memory 1110 may include, for example, a hard disk drive 1112 and / or a removable storage drive 1114, which may include a floppy disk drive, a magnetic tape drive, an optical disk drive, or the like. The removable storage drive 1114 reads from and / or writes to a removable storage unit 1118 in a well-known manner. The removable storage unit 1118 may include a floppy disk, magnetic tape, optical disk, or the like, which is read by and written to by removable storage drive 1114. As will be appreciated by persons skilled in the relevant art(s), the removable storage unit 1118 includes a computer readable storage medium having stored therein computer executable program code instructions and / or data.

[0077] In an alternative implementation, the secondary memory 1110 may additionally or alternatively include other similar means for allowing computer programs or other instructions to be loaded into the computing device 1100. Such means can include, for example, a removable storage unit 1122 and an interface 1120. Examples of a removable storage unit 1122 and interface 1120 include a program cartridge and cartridge interface (such as that found in video game console devices), a removable memory chip (such as an EPROM or PROM) and associated socket, and other removable storage units 1122 and interfaces 1120 which allow software and data to be transferred from the removable storage unit 1122 to the computer system 1100.

[0078] The computing device 1100 also includes at least one communication interface 1124. The communication interface 1124 allows software and data to be transferred between computing device 1100 and external devices via a communication path 1126. In various embodiments of the disclosure, the communication interface 1124 permits data to be transferred between the computing device 1100 and a data communication network, such as a public data or private data communication network. The communication interface 1124 may be used to exchange data between different computing devices 1100 which such computing devices 1100 form part an interconnected computer network. Examples of a communication interface 1124 can include a modem, a network interface (such as an Ethernet card), a communication port, an antenna with associated circuitry and the like. The communication interface 1124 may be wired or may be wireless. Software and data transferred via the communication interface 1124 are in the form of signals which can be electronic, electromagnetic, optical or other signals capable of being received by communication interface 1124. These signals are provided to the communication interface via the communication path 1126.

[0079] As shown in Figure 11, the computing device 1100 further includes a display interface 1102 which performs operations for rendering images to an associated display 1130 and an audio interface 1132 for performing operations for playing audio content via associated speaker(s) 1134.

[0080] As used herein, the term "computer program product" may refer, in part, to removable storage unit 1118, removable storage unit 1122, a hard disk installed in hard disk drive 1112, or a carrier wave carrying software over communication path 1126 (wireless link or cable) to communication interface 1124. Computer readable storagemedia refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computing device 1100 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray™ Disc, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computing device 1100. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and / or data to the computing device 1100 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

[0081] The computer programs (also called computer program code) are stored in main memory 1108 and / or secondary memory 1110. Computer programs can also be received via the communication interface 1124. Such computer programs, when executed, enable the computing device 1100 to perform one or more features of embodiments discussed herein. In various embodiments, the computer programs, when executed, enable the processor 1104 to perform features of the above-described embodiments. Accordingly, such computer programs represent controllers of the computer system 1100.

[0082] Software may be stored in a computer program product and loaded into the computing device 1100 using the removable storage drive 1114, the hard disk drive 1112, or the interface 1120. Alternatively, the computer program product may be downloaded to the computer system 1100 over the communications path 1126. The software, when executed by the processor 1104, causes the computing device 1100 to perform functions of embodiments described herein.

[0083] It is to be understood that the embodiment of Figure 4 is presented merely by way of example. Therefore, in some embodiments one or more features of the computing device 1100 may be omitted. Also, in some embodiments, one or more features of the computing device 1100 may be combined together. Additionally, in some embodiments, one or more features of the computing device 1100 may be split into one or more component parts.

[0084] It will be appreciated that the elements illustrated in Figure 11 function to provide means for performing the various functions and operations of the servers as described in the above embodiments.

[0085] In an implementation, a server may be generally described as a physical device comprising at least one processor and at least one memory including computer program code. The at least one memory and the computer program code are configured to, with the at least one processor, cause the physical device to perform the requisite operations.

[0086] It will be appreciated by a person skilled in the art that numerous variations and / or modifications may be made to the present disclosure as shown in the specific embodiments without departing from the scope of the disclosure as broadly described. The present embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive.

Claims

CLAIMS1. A method for deviation compensation in a tool insertion procedure, the method comprising:generating a representation of a target and one or more target points disposed on the target;for each target point of the one or more target points, determining an associated trajectory for an elongated tool to strike the target point;prior to or during insertion of the elongated tool, automatically detecting a deviation and generating an alert associated with the deviation;upon receiving a user confirmation responsive to the alert, automatically performing a compensation based on the detected deviation.

2. The method as claimed in claim 1, wherein generating the representation of the target comprises fusing a first set of imaging data obtained by a first imaging mode with a second set of imaging data obtained by a second imaging mode different from the first imaging mode.

3. The method as claimed in claim 1 or 2, wherein the deviation comprises a movement of the target prior to insertion of the elongated tool, and wherein automatically detecting the deviation comprises:capturing, at a selected frequency, real-time contours of the target at a reference plane;comparing each of the real-time contours of the target at the reference plane with a stored contour of the target at the reference plane, wherein the stored contour of the target at the reference plane is based on the representation of the target; andgenerating the alert if a difference between the real-time contour and the stored contour is greater than a predetermined threshold.

4. The method as claimed in claim 3, wherein automatically performing the compensation comprises:determining a three-dimensional (3D) transformation based on differences between real-time contours of the target at a selected plurality of planes and corresponding stored contours of the target at the selected plurality of planes,wherein the stored contours of the target at the selected plurality of planes are based on the representation of the target;applying the 3D transformation to the representation of the target to obtain an updated representation of the target; andregenerating the one or more target points and associated trajectories based on the updated representation of the target.

5. The method as claimed in claim 4, wherein the selected plurality of planes include, relative to an axial direction of an imaging device, the reference plane, at least one plane fore of the reference plane, and at least one plane aft of the reference plane.

6. The method as claimed in claim 1 or 2, wherein the deviation comprises a deflection of the elongated tool during insertion, and wherein automatically detecting the deviation comprises:determining an actual landing point of the elongated tool; and generating the alert if a difference between the actual landing point of the elongated tool and the target point is greater than a predetermined threshold.

7. The method as claimed in claim 6, wherein determining the actual landing point of the elongated tool comprises:generating, at a selected frequency, a plurality of ultrasound images at a transverse plane passing through the target point;for each ultrasound image of the plurality of ultrasound images, detecting a flash representing a position of the elongated tool;selecting the ultrasound image having the largest flash; and determining a center of the flash in the selected ultrasound image.

8. The method as claimed in claim 7, wherein detecting the flash comprises comparing said ultrasound image with a reference ultrasound image at the transverse plane passing through the target point, the reference ultrasound image being free of the elongated tool.

9. The method as claimed in any one of claims 6 to 8, wherein performing the compensation comprises:withdrawing the tool; andregenerating the trajectory based on the difference between the determined actual landing point of the elongated tool and the target point.

10. A system for deviation compensation in a tool insertion procedure, the system comprising:a processor;a computer-readable memory communicatively coupled to the processor; andan imaging device communicatively coupled to the processor, wherein the processor is configured to:generate a representation of a target and one or more target points disposed on the target;for each target point of the one or more target points, determine an associated trajectory for an elongated tool to strike the target point;prior to or during insertion of the elongated tool, automatically detect a deviation and generate an alert associated with the deviation;based on a user confirmation responsive to the alert, automatically perform a compensation based on the detected deviation.

11. The system as claimed in claim 10, wherein the processor is configured to fuse a first set of imaging data obtained by a first imaging mode with a second set of imaging data obtained by a second imaging mode different from the first imaging mode to generate the representation of the target.

12. The system as claimed in claim 10 or 11, wherein the deviation comprises a movement of the target prior to insertion of the elongated tool, wherein the imaging device is configured to capture, at a selected frequency, real-time contours of the target at a reference plane, and wherein the processor is further configured to:compare each contour of the real-time contours of the target at the reference plane with a stored contour of the target at the reference plane to detect the deviation, wherein the stored contour of the target at the reference plane is based on the representation of the target; and generate the alert if a difference between the real-time contour and the stored contour is greater than a predetermined threshold.

13. The system as claimed in claim 12, wherein the processor is further configured to:determine a three-dimensional (3D) transformation based on differences between real-time contours of the target at a selected plurality of planes and corresponding stored contours of the target at the selected plurality of planes, wherein the stored contours of the target at the selected plurality of planes are based on the representation of the target;apply the 3D transformation to the representation of the target to obtain an updated representation of the target; andregenerate the one or more target points and associated trajectories based on the updated representation of the target.

14. The system as claimed in claim 13, wherein the selected plurality of planes includes, relative to an axial direction of the imaging device, the reference plane, at least one plane fore of the reference plane, and at least one plane aft of the reference plane.

15. The system as claimed in claim 10 or 11, wherein the deviation comprises a deflection of the elongated tool during insertion, and wherein the processor is further configured to:determine an actual landing point of the elongated tool; andgenerate the alert if a difference between the actual landing point of the elongated tool and the target point is greater than a predetermined threshold.

16. The system as claimed in claim 15, wherein the imaging device is configured to generate, at a selected frequency, a plurality of ultrasound images at a transverse plane passing through the target point; andwherein the processor is further configured to:for each ultrasound image of the plurality of ultrasound images, detect a flash representing a position of the elongated tool;select the ultrasound image having the largest flash; and determine a center of the flash in the selected ultrasound image, wherein the center of the flash is the actual landing point of the elongated tool.

17. The system as claimed in claim 16, wherein the processor is further configured to compare said ultrasound image with a reference ultrasound image at the transverse plane passing through the target point, the reference ultrasound image being free of the elongated tool, to detect the flash.

18. The system as claimed in any one of claims 15 to 17, further comprising a robotic arm communicatively coupled to the processor, wherein the robotic arm is configured to withdraw the tool, and the processor is further configured to regenerate the trajectory based on the difference between the determined actual landing point of the elongated tool and the target point.

Images