Conformal treatment planning using three-dimensional ultrasound

Abstract

Three dimensional (3D) image data is generated from an imaging device, and a two dimensional (2D) image section of the tissue from the 3D image data is provided on a display with a real time image from the imaging device. A 3D model of the tissue is generated in response to one or more tissue structures of the 3D image data. A user interface is configured to allow the user to view corresponding portions of the 3D model overlaid on corresponding 2D sections and the real time image. A conformal 3D treatment profile is generated in response to the 3D model, and the user interface is configured to allow the user to view corresponding portions of the 3D model and the 3D treatment plan on corresponding 2D tissue sections and the real time image.

Description

PRCPT.145WO PATENT CONFORMAL TREATMENT PLANNING USING THREE-DIMENSIONAL ULTRASOUND RELATED APPLICATIONS

[0001] The present application claims priority to U. S. Provisional Patent Application No. 63 / 773,913, filed March 18, 2025, entitled “3D TREATMENT PLANNING WITH REAL-TIME ULTRASOUND”, and U. S. Provisional Patent Application No. 63 / 822,028, filed June 11, 2025, entitled “CONFORMAL TREATMENT PLANNING USING THREE-DIMENSIONAL ULTRASOUND”, and U. S. Provisional Patent Application No. 63 / 825,875, filed on June 18, 2025, entitled “CONFORMAL TREATMENT PLANNING USING THREE-DIMENSIONAL ULTRASOUND,” the entire disclosures of which are incorporated herein by reference.

[0002] The subject matter of the present application is related to U. S. Patent Application No 18 / 539,023, filed December 13, 2023, entitled “ERGONOMIC SURGICAL ROBOTIC SYSTEM”, U. S. Patent Application No. 18 / 539,048, filed December 13, 2023, entitled “USER INTERFACE FOR SURGICAL ROBOTIC SYSTEM”, and U. S. Patent Application No. 19 / 089,260, filed March 25, 2025, entitled “IMAGE GUIDED CANCER TREATMENT SYSTEM AND METHODS”, the entire disclosures of which are incorporated herein by reference.BACKGROUND

[0003] Prior methods and apparatus for treating patients can be less than ideal in at least some respects. Although surgical systems can be at least partially automated to perform at least some aspects of a surgical procedure, work in relation to the present disclosure suggests that the prior surgical systems can be less than ideal. The prior user interfaces of prior systems can be somewhat less intuitive than would be ideal, and the system set up may require more user adjustments than would be ideal, in at least some instances. For example, some surgical approaches rely on more than one probe inserted into the patient, and the adjustment of these probes in preparation for treatment may be more complicated than ideal in at least some instances.

[0004] Although three dimensional (3D) treatment planning has been proposed, the prior approaches can be less than ideal in at least some respects. For example, 3D imaging of a volume can take longer than would be ideal, and the prior approaches to 3D treatment planning may not be well suited for monitoring a patient during treatment, in at least some instances.

[0005] Although artificial intelligence (Al) algorithms such as neural networks have been proposed to at least partially automate aspects of a surgical procedure, work in relation to the present disclosure suggests that the prior approaches to using Al may be less than ideally suited for use with treatment planning, probe alignment, and aligning treatment plans and images, in at least some instances.

[0006] In light of the above, improved methods and apparatus are needed which would ameliorate at least some of the above mentioned aspects of prior surgical systems.SUMMARY

[0007] In some embodiments, three dimensional image data is generated from an imaging device, and a 2D image section of the tissue from the 3D image data is provided on a display with a real time image from the imaging device. In some embodiments, a 3D model of the tissue is generated in response to one or more tissue structures of the 3D image data. In some embodiments, a user interface is configured to allow the user to view corresponding portions of the 3D model overlaid on corresponding 2D sections and the real time image. In some embodiments, a 3Dtreatment profile is generated in response to the 3D model, and the user interface is configured to allow the user to view corresponding portions of the 3D model and the 3D treatment plan on corresponding 2D tissue sections and the real time image.INCORPORATION BY REFERENCE

[0008] All patents, applications, and publications referred to and identified herein are hereby incorporated by reference in their entirety, and shall be considered fully incorporated by reference even though referred to elsewhere in the application.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A better understanding of the features, advantages and principles of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, and the accompanying drawings, by way of non-limiting examples.

[0010] FIGS. 1A and 1B show a surgical system, in accordance with some embodiments of the present disclosure.

[0011] FIG. 2 shows alignment of a user interface, a treatment probe and an imaging probe of the system of FIGS. 1A and 1B, in accordance with some embodiments.

[0012] FIGS. 3A to 3F show a user interface comprising display, in accordance with some embodiments.

[0013] FIG. 4A shows a treatment probe configured for use with a surgical system, in accordance with some embodiments.

[0014] FIG. 4B shows a motor-pack configured to couple to a treatment probe, in accordance with some embodiments.

[0015] FIG. 5 shows treatment probe and corresponding movements of an energy source, in accordance with some embodiments,

[0016] FIG, 6A shows an arm configured to support a treatment probe, in accordance with some embodiments.

[0017] FIG, 6B shows mounting device configured to couple to a motor-pack in an unlocked configuration, in accordance with some embodiments.

[0018] FIG, 6C shows the mounting device of FIG. 6B in a locked configuration, in accordance with some embodiments.

[0019] FIG, 7 shows an ultrasound imaging device, in accordance with some embodiments,

[0020] FIG. 8 shows an arm configured to support the ultrasound imaging device of FIG. 7, in accordance with some embodiments;

[0021] FIG. 9 shows a user interface method, in accordance with some embodiments.

[0022] FIG. 10 shows a method of assisted treatment planning, in accordance with some embodiments.

[0023] FIG. 11 shows an imaging probe and a treatment probe skewed in relation to each other, in accordance with some embodiments.

[0024] FIG. 12A shows a first location of a treatment probe in a first transverse image, in accordance with some embodiments.

[0025] FIG, 12B shows a second location of the treatment probe of FIG, 12A in a second transverse image, in accordance with some embodiments.

[0026] FIG, 12C shows a third location of the treatment probe of FIGS. 12A and 12B in a third transverse image, in accordance with some embodiments,

[0027] FIG, 13A shows an output of an Al analysis of alignment between a treatment probe and an imaging probe, in accordance with some embodiments,

[0028] FIG. 13B shows a graphical instruction to move a treatment probe into alignment with an imaging probe, in accordance with some embodiments.

[0029] FIG. 14A shows a first transverse image and a location of a hyperechoic shadow from a treatment probe with the imaging probe in a first rotational configuration, in accordance with some embodiments.

[0030] FIG. 14B shows the imaging probe in the first rotational configuration to generate an image as in FIG. 14A, in accordance with some embodiments.

[0031] FIG. 14C shows a second transverse image and the location of a hyperechoic shadow from a treatment probe with the imaging probe in a second rotational configuration, in accordance with some embodiments.

[0032] FIG. 14D shows the imaging probe in the second rotational configuration to generate an image as in FIG.14C, in accordance with some embodiments.

[0033] FIG. 15 shows alignment of an energy source on a treatment probe with a treatment plan in response to image data, in accordance with some embodiments.

[0034] FIG. 16A shows a user interface configured to receive a user input to activate an Al algorithm to detect a location of an energy source such as a nozzle on a treatment probe and a proximal location of the treatment probe, in accordance with some embodiments.

[0035] FIG. 16B shows a user interface as in FIG. 16A with the location of the energy source and the proximal portion of the probe identified, in accordance with some embodiments.

[0036] FIG. 17A shows a user interface configured to receive a user input to activate an imaging probe to generate 3D image data, in accordance with some embodiments,

[0037] FIG. 17B shows a user interface during image acquisition to generate the 3D image, in accordance with some embodiments.

[0038] FIG. 17C shows a user interface with a real time image, a digitally reconstructed image and a 3D model, in accordance with some embodiments.

[0039] FIGS. 18A and 18B show treatment planning in 2D while the treatment profile is visualized in 3D, in accordance with some embodiments,

[0040] FIG. 19 shows a user interface configured to plan a plurality of passes of an energy treatment, in accordance with some embodiments.

[0041] FIG. 20A shows an initial configuration of a protection zone, in accordance with some embodiments.

[0042] FIG. 20B shows an adjusted configuration of the protection zone as in FIG. 20A.

[0043] FIG. 21 A shows a user interface configured for a user to add a transverse treatment profile in accordance with some embodiments.

[0044] FIG. 21 B shows a user interface as in FIG. 21 A with an additional transverse treatment profile.

[0045] FIG, 210 shows the user interface as in FIGS, 21 A and 21 B configured to receive an input from the user to adjust the additional transverse treatment profile,

[0046] FIG, 22 shows the user interface configured for a user to review and accept a treatment plan, in accordance with some embodiments.

[0047] FIG, 23 shows a method of training an artificial intelligence (Al) algorithm, in accordance with some embodiments.

[0048] FIG. 24 shows a method of generating 3D image data, in accordance with some embodiments.

[0049] FIG. 25 shows a method of generating a tissue model, in accordance with some embodiments.

[0050] FIG. 26 shows a method of planning a treatment, in accordance with some embodiments.

[0051] FIG. 27 shows a method of assisting a user to align a treatment probe and an imaging probe, in accordance with some embodiments.

[0052] FIG. 28 shows a method of automatically aligning a treatment probe and an imaging probe, in accordance with some embodiments.DETAILED DESCRIPTION

[0053] The following detailed description provides a better understanding of the features and advantages of the improvements described in the present disclosure in accordance with the embodiments disclosed herein. Although the detailed description includes many specific embodiments, these are provided by way of example only and should not be construed as limiting the scope of the improvements disclosed herein.

[0054] The presently disclosed systems, methods and user interface may provide improved imaging with an imaging device configured to generate 3D image data and provide real time images. The 3D image data can be generated with the imaging device, and the imaging device provides real time images while portions of the 3D image data can be viewed by a user. In some embodiments, one or more arms support the imaging device while the imaging device is coupled with the patient. The one or more arms may comprise movable arms that remain substantially fixed while the 3D image data is acquired, and a real time image is provided. Image sections from the 3D image data, such as 2D image sections, e.g. slices, are provided on the display with the real time images, which can assist a user in assessing the quality of the 3D image data, for example. In some embodiments, a 3D model of the imaged tissue is generated from the 3D image data, and corresponding portions of the 3D model provided on image sections and the real time image, which can allow the user to verity the 3D model, for example by scanning through a plurality image sections with corresponding portions of the 3D model overlaid thereon, for example. In some embodiments, a 3D treatment profile of the imaged tissue is generated from the 3D model, and corresponding portions of the 3D treatment profile and the 3D model provided on image sections and the real time image, which can allow the user to verify the 3D treatment profile in relation to the 3D model, for example by scanning through a plurality image sections with corresponding portions of the 3D model overlaid thereon, for example. In some embodiments, the user interface is configured to allow the user to scan through a plurality of image sections with corresponding portions of the 3D treatment profile and the 3D model overlaid thereon, by providing a plurality of inputs associated with locations of the real time image, for example by sliding a marker along the real time image.

[0055] In some embodiments, a treatment probe is visible in the real time image and the plurality of image sections with corresponding portions of the treatment profile and 3D image overlaid thereon, which can allow the userto detect unintended movement of one or more of the treatment probe or the imaging probe, which may be beneficial with automated treatments and planning as described herein. In some embodiments, one or more locations of a treatment probe are registered with one or more images, and providing real time images from the imaging device that is used to generate 3D image data may provide improved registration of the treatment probe with the 3D image data, for example,

[0056] Work in relation to the present disclosure suggests that 3D image data can take longer to generate that real time images. In some embodiments, a two dimensional imaging device is used to generate 3D image data and provide real time 2D images, for example by moving the 2D imaging device with one or more of rotation or translation. In some embodiments, the real time images are generated with a latency of less than about one second, and the 3D image data of an image of an organ such as the prostate takes at least 10 seconds to acquire, for example from about 15 seconds to one minute. In some embodiments, the time to generate the 3D image data depends on the distance or rotational angle between image sections such as slices, and 3D image data with higher resolution may take 30 to 40 seconds whereas 3D image data with a lower resolution may take a shorter time, such as 20 seconds. In some embodiments, the imaging probe such as an ultrasound imaging probe generates image sections such as slices at a rate of 15 to 40 Hertz (Hz), e.g. 25 to 66.7 milliseconds per slice.

[0057] In some embodiments, it is helpful to view real time images with 2D imaging during the treatment, to allow the user to view the treatment of tissue in real time. This can allow the user to monitor and evaluate the performance of an energy source such as a waterjet in real time and allow the user to make adjustments to the treatment if needed. In some embodiments, real time 2D images are provided with a stationary imaging probe such as an ultrasound imaging probe to allow the user such as a surgeon to monitor the treatment while the treatment probe moves in accordance with a treatment profile such as a 3D treatment profile.

[0058] The imaging device may comprise any suitable imaging device such as one or more of an ultrasound array, a one dimensional ultrasound array, a two dimensional ultrasound array, an imaging probe, an ultrasound imaging probe, a one dimensional imaging probe, a two dimensional imaging probe, a three dimensional imaging probe, a transrectal ultrasound (TRUS) imaging probe, a Positron Emission Tomography (PET) probe, a Single-Photon Emission Computed Tomography (SPECT) probe, a magnetic resonance imaging (MR! Probes), an x-ray imaging device, a fluoroscopy imaging device, a computed tomography (CT) imaging device, an MRI imaging device, an optical imaging device such as an endoscope, or a fluorescence imaging device to image fluorescence from a dye such as Indocyanine Green (ICG), for example,

[0059] In some embodiments, the treatment probe comprises an endoscope that is used to identify tissue. The endoscope can be advanced and retracted with the probe placed in the patient. In some embodiments, the probe tip is configured to contact tissue, and a user selection indicates that the probe location corresponds to a selected tissue such as a verumontanum of the prostate or other tissue. In some embodiments, the endoscope is located on a carriage with sensors such as encoders on a handpiece as described herein, and the position of the endoscope can be used to determine the location of tissue, for example when the distal tip of the probe or the endoscope contacts the tissue. Alternatively or in combination, the probe may comprise impedance sensors, for example. In some embodiments, a position sensor such as a coil or magnetic element is used to contact and label tissue, which is shown in the image, for example.

[0060] In some embodiments, an endoscope such as a cystoscope is inserted with the treatment probe, and the location of the endoscope is visible in the ultrasound image. When the endoscope is near the verumontanum, the verumontanum is visible in the endoscope image and the endoscope is visible in the intraoperative image. For example, the verumontanum may be within about 5 mm to about 10 mm from the tip of the endoscope. In some embodiments, the position of the endoscope in the ultrasound image is used to determine the position of the verumontanum. in some embodiments, the user interface is configured for the user to provide an input when the verumontanum is visible in the endoscope image. Alternatively or in combination, the Al can be configured to identify the verumontanum in the endoscope image. The location of the endoscope in the intraoperative image can be used to determine the location of the verumontanum in the intraoperative image, in response to the verumontanum being within the field of view of the endoscope.

[0061] In some embodiments, a TRUS imaging probe is rotated longitudinally to acquire a plurality of longitudinal images as described herein and create a 3D model of the prostate. In some embodiments, the treatment probe comprise an endoscope such as a cystoscope. In some embodiments, an endoscopic view, e.g. a cystoscopic view is available as the treatment probe is inserted, and there are both live ultrasound images from the TRUS probe as well as live cystoscopic images of the urethra. In some embodiments, items of interest (e.g. markers) may be identified from the endoscopic view, e.g. cystoscopic view, such as the external sphincter, the verumontanum or the bladder neck, for example. As the treatment probe and endoscope (e.g. cystoscope) can be seen in a longitudinal (e.g. sagittal) ultrasound view created by the TRUS imaging probe, the location of items viewed by the endoscope (e.g. cystoscope) can be mapped to a location in the ultrasound image and the 3D model. In some embodiments, the user interface is configured for the user to input the location of the items of interest on the one or more images such as one or more ultrasound images by moving one or more markers on the user interface as described herein.

[0062] The presently disclosed systems and methods are well suited for use with many probes and diagnostic and surgical procedures. Although reference is made to a treatment probe comprising an energy source for prostate surgery and a transrectal ultrasound (“TRUS”) probe, the present disclosure is well suited for use with many types of probes inserted into many types of tissues, organs, cavities and lumens, such as brain, heart, lung, intestinal, eye, skin, kidney, liver, pancreas, stomach, uterus, ovaries, testicles, bladder, ear, nose, mouth, tumors, cancers, soft tissues such as bone marrow, adipose tissue, muscle, glandular and mucosal tissue, spinal and nerve tissue, cartilage, hard biological tissues such as teeth, bone, and lumens such as vascular lumens, nasal lumens and cavities, sinuses, colon, urethral lumens, gastric lumens, airways, esophageal lumens, trans esophageal, intestinal lumens, anal lumens, vaginal lumens, trans abdominal, abdominal cavities, throat, airways, lung passages, and surgery such as kidney surgery, ureter surgery, kidney stones, prostate surgery, tumor surgery, cancer surgery, brain surgery, heart surgery, eye surgery, conjunctival surgery, liver surgery, gall bladder surgery, bladder surgery, spinal surgery, orthopedic surgery, arthroscopic surgery, liposuction, colonoscopy, intubation, minimally invasive incisions, minimally invasive surgery, and others. More generally the present disclosure can be used for any suitable medical procedure, which can involve one or more of treatment of diagnostics. Certain disclosed examples are described with reference to endourological procedures.

[0063] The presently disclosed systems and methods are well suited for combination with prior probes such as imaging probes and treatment probes. Examples of such probes include laser treatment probes, waterjet probes, RFtreatment probes, radiation therapy probes, ultrasound treatment probes, phaco emulsification probes, imaging probes, endoscopic probes, resectoscope probes, ultrasound imaging probes, A-scan ultrasound probes, B-scan ultrasound probes, 3D ultrasound probes, Doppler ultrasound probes, transrectal ultrasound probes, transvaginal ultrasound probes, longitudinal plane ultrasound imaging probes, sagittal plane ultrasound imaging probes, transverse plane ultrasound imaging probes, and transverse and longitudinal plane (e.g. sagittal plane) ultrasound imaging probes, for example,

[0064] The one or more images described herein can be generated in many ways, and may comprise one or more of a longitudinal image, a sagittal image, a parasagittal or a transverse image. In some embodiments, a longitudinal image comprises an image generated with an elongate imaging probe, in which the longitudinal image extends along a plane corresponding to an elongate axis of the imaging probe, and the transverse images extend along a plane that is transverse to the elongate axis of the probe, for example substantially perpendicular to the elongate axis of the probe and the corresponding longitudinal images. The elongate probe can be inserted into the patient with any suitable orientation. In some embodiments, the probe is inserted into the patient substantially along a midline of the patient, such that the longitudinal images correspond to sagittal images of the patient. In some embodiments, the elongate imaging probe comprises a TRUS probe, and the longitudinal images comprise sagittal images, although other probes with different orientations can be used to generate images in accordance with the present disclosure. Although reference is made to ultrasound probes inserted into the patient, in some embodiments the imaging device comprises an external imaging probe in which the longitudinal and transverse images can be referenced to one or more arrays of the external imaging probe. In some embodiments, the one or more images are generated from a 3D tomographic image data set such as a Digital Imaging and Communications in Medicine (DICOM) image data set.

[0065] Although reference is made to ultrasound 3D image, the 3D image may comprise any suitable image, such as a tomogram from any suitable imaging device such as one or more of an X-ray image; a fluoroscopic image, a computed tomography (CT) image; a magnetic resonance imaging (MRI) image; positron emission tomography (PET) image, single-photon emission computed tomography (SPECT) image, for example. In some embodiments, 3D imaging data is generated with the ultrasound probe such as a TRUS probe and fused with other 3D or two dimensional (2D) image data,

[0066] The presently disclosed systems, methods and apparatuses are well suited for combination with many procedures, such as waterjet enucleation of the prostate, transurethral resection of the prostate (TURP), holmium laser enucleation of the prostate (HOLEP), prostate brachytherapy, and with surgical robotics systems and automated surgical procedures. The following patent applications describe examples of systems, methods, probes and procedures suitable for incorporation in accordance with the present disclosure: PCT / US2013 / 028441, filed February 28, 2013, entitled “AUTOMATED IMAGE-GUIDED TISSUE RESECTION AND TREATMENT, published as WO 2013 / 130895; PCT / US2014 / 054412, filed September 5, 2014, entitled “AUTOMATED IMAGE-GUIDED TISSUE RESECTION AND TREATMENT”, published as WO 2015 / 035249; PCT / US2015 / 048695, filed September 5, 2015, entitled “PHYSICIAN CONTROLLED TISSUE RESECTION INTEGRATED WITH TREATMENT MAPPING OF TARGET ORGAN IMAGES”, published as WO2016037137; PCT / US2019 / 038574, filed June 21, 2019, entitled “ARTIFICIAL INTELLIGENCE FOR ROBOTIC SURGERY”, published as WO2019246580A1 on December 26, 2019; PCT / US2020 / 021756, filed March 9, 2020, entitled “ROBOTIC ARMS AND METHODS FOR TISSUE RESECTIONAND IMAGING”, published as WO / 2020 / 181290; PCT / US2020 / 058884, filed on November 4, 2020, entitled “SURGICAL PROBES FOR TISSUE RESECTION WITH ROBOTIC ARMS”, published as WO / 2021 / 096741; PCT / US2021 / 070760, filed on June 23, 2021, entitled " INTEGRATION OF ROBOTIC ARMS WITH SURGICAL PROBES”, published as WO / 2021 / 263276; and PCT / US2021 / 038175, filed on June 21, 2021, entitled “SYSTEMS AND METHODS FOR DEFINING AND MODIFYING RANGE OF MOTION OF PROBE USED IN PATIENT TREATMENT”, published as WO / 2021 / 262565; the entire disclosures of which are incorporated herein by reference.

[0067] In some embodiments, an energy source is used to treat tissue. The energy source may comprise any suitable energy source, such as one or more of an electrode, a loop electrode, a laser source, a thermal energy source, a mechanical energy source, a mechanical sheer, an ultrasound probe, a cavitating ultrasound probe, a waterjet, e.g. a fixed pressure waterjet, a plasma source, a steam source, a morcellator, a trans urethral needle, a photo ablation source, a radiation energy source, a microwave energy source or a water jet evacuation source. The energy source can be combined with other treatments and compounds, such as photochemical treatment agents. The imaging probe may comprise any suitable probe, such as endoscopic probes, resectoscope probes, ultrasound imaging probes, A-scan ultrasound probes, B-scan ultrasound probes, Doppler ultrasound probes, transrectal ultrasound probes, transvaginal ultrasound probes, sagittal plane ultrasound imaging probes, transverse plane ultrasound imaging probes, and transverse and sagittal plane ultrasound imaging probes, for example.

[0068] FIG. 1A shows a surgical system 100, in accordance with some embodiments. Thesystem 100 comprises a console 120 which comprises several components of the system 100, such as electronics, circuitry and one or more processors to control a treatment probe 150, which has been inserted into the patient 101. The system 100 comprises a user interface 103 comprising a monitor, such as a touch screen display 140. The display 140 is coupled to an adjustable arm 145, which allows the display to be positioned above a midline of patient near an insertion location of the probe 150, which can allow the user such as a surgeon to control the surgery when seated in front of the display 140 and near the surgical instruments, such as the treatment probe 150 and an imaging probe 160.

[0069] The console 120 can be configured and sized and shaped in many ways. In some embodiments, the console 120 comprises a tower, in which the horizontal dimensions of the console are less than a vertical height of the console. Alternatively or in combination, the console 120 may comprise a cart, for example with lockable wheels, which allow the console to be moved,

[0070] The user interface 103 can be configured in many ways and may comprise a display, such as a touch screen display, for example. Although reference is made to a touch screen display, the user interface 103 positioned above the patient may comprise one or more components that are not shown on the display, such as a positioning pad, one or more input buttons, a pointing device such as a track pad, a knob, or other suitable component. These components of the user interface 103 can be located to the side of the display 140, for exampie. Alternatively or in combination, the display may comprise one or more input features shown on the touch screen display touch screen display, such as buttons, a positioning pad, and features configured for the user to swipe the display to provide user input.

[0071] In some embodiments, the system 100 comprises a patient support 101. The patient support may comprise any suitable support, such as a bed, for example. The support 101 can be configured to support the patient such that a midline of the patient is approximately aligned with a midline of the support 101. The support 105 maycomprise one or more stirrups 107, which allow the legs and feet of the patient to be elevated during surgery. In some embodiments, the support 101 is configured to place the patient in a lithotomy position, for example. The support may comprise adjustable features such as one or more motors to raise and lower the patient for example.

[0072] In some embodiments, the system 100 comprises one or more probes inserted into the patient. The one or more probes may comprise a treatment probe 150, While the treatment probe 150 can be configured in many ways, in some embodiments the treatment probe 150 comprises an energy source configured to deliver energy to the patient to treat tissue as described herein. In some embodiments, the treatment probe is configured to be inserted into an external opening of the urethra of the patient, although other access locations can be used, such as surgical opening to introduce the probe, in some embodiments, the one or more probe inserted into the patient comprises an imaging probe, such as an ultrasound probe, for example. In some embodiments, the imaging probe inserted into the patient comprises a transrectal ultrasound (TRUS) probe 160. Although reference is made to a separate ultrasound probe, in some embodiments the ultrasound imaging transducer array is inserted into the patient through the same opening as the treatment probe and may located on the same probe as the energy source. Also, the ultrasound imaging device may comprise an external ultrasound imaging array, for example.

[0073] The one or more probes inserted into the patient may be coupled to one or more arms to support the one or more probes. The one or more arms may comprise a treatment probe arm 156 configured to support the treatment probe 150 during treatment, for example with the treatment probe 150 inserted into the patient. Alternatively or in combination, the one or more arms may comprise an arm to support the imaging probe, such as an arm 166 configured to support a TRUS imaging probe. In some embodiments, the TRUS probe is coupled to the arm 166 with a stepper 164, which allows the probe 166 to be advanced along the rectum or the colon of the patient and to be drawn proximally toward the surgeon with rotation of a knob with the TRUS probe 160 inserted into the patient 101. In some embodiments, the stepper 164 comprises a cradle for rotationally adjusting the TRUS probe 160 to align with patient anatomy.

[0074] In some embodiments, one or more motors 165 and one or more encoders 167 are coupled to the imaging probe 160 to one or more of rotate or translate the imaging probe 160. in some embodiments, the one or more motors 165 comprises a motor to rotate the imaging probe around a longitudinal axis and the one or more encoders 167 are configured to measure angular rotation of the imaging probe 160 around the longitudinal axis, in some embodiments, the one or more motors 165 comprises a motor to translate the imaging probe along a longitudinal axis and the one or more encoders 167 are configured to measure translation of the imaging probe 160 along the longitudinal axis,

[0075] In some embodiments, the one or more motors 165 comprises a motor rotate the imaging probe to obtain longitudinal images and 3D image data as described herein, and the one or more encoders 167 comprises an encoder to measure rotation of the imaging probe in response to rotation provided by the motor. Alternatively or in combination, the one or more encoders 167 may comprise an encoder to measure translation of the imaging probe, for example translation in response to a user rotating a knob to advance and retract the imaging probe.

[0076] The treatment probe 150 can be configured in many ways, and may comprise a handpiece 152 configured to allow the probe to be inserted into the patient. The handpiece 152 can be configured to couple to a source of mechanical energy, such as a motor-pack 154, which is configured to drive one or more components of the treatment probe. In some embodiments, the treatment probe is configured to receive an endoscope such as a cystoscope inorder to allow visualization of the treatment area and to facilitate insertion of the treatment probe. In some embodiments, the endoscope is incorporated into the probe as an integral part of the probe.

[0077] The console 120 can be configured in many ways, and may comprise internal electronics, processors, circuitry and connectors configured to couple to other components of the surgical system. In some embodiments, the console 120 comprises one or more handles 122 and wheels such as locking wheel 138, which allow the console to be moved by up to two people at a time, for example to be stored or brought into an operating room for surgery and locked into position. The console 120 may comprise additional features to facilitate surgery and use of the system, such one or more of a motor-pack holder 124, a TRUS probe holder 126, a drawer 128, or a power button 129, for example. In some embodiments, the motor-pack holder 124 and the TRUS probe holder can located on either side of the console 120, depending on user needs. The console 120 may comprise connectors to couple to one or more components of the surgical system 100, such as one or more of a TRUS probe connection 132 configured to couple the console to the TRUS probe 160, a motor-pack connection 134 configured to couple to the motor-pack 154, or a scope connection 136 configured to couple to an endoscope such as a cystoscope.

[0078] In some embodiments, the system 100 comprises a water jet pump cartridge 170, which is received in a receptacle of console 120. In some embodiments, a waterjet line 172 extends from the pump cartridge 170 to the treatment probe 150 to provide a waterjet from the treatment probe. In some embodiments, the console 120 comprises an aspiration pump 174, which is coupled to probe 150 with an aspiration line 174.

[0079] In some embodiments, the system 100 comprises a waste line 178 configured to couple to a waste container 179. The waste line 178 can be configured to receive fluids and other material from the patient via the treatment probe 150, for example. In some embodiments the waste line 178 is coupled to the aspiration pump 176, although line 178 may receive fluids from other sources, for example.

[0080] In some embodiments, the system 100 comprises a foot pedal 190 configured for a user such as a physician to control treatment, such as starting, pausing, resuming or stopping the treatment for example.

[0081] In some embodiments, the console 120 comprises a front side 121 in which a plurality of connectors are plugged into the console 120, which allows the user such as a surgeon to readily view the connectors to ensure that these and the associated lines have been properly connected. The plurality of connectors located on the front side 121 may comprise one or more of the TRUS probe connection 132, the motor-pack connection 134, the scope connection 136, the waterjet pump cartridge 170, the waterjet line 172, the aspiration pump 174, the waste line 178, or the connector of the foot pedal 190. This location of the connectors and lines allows a user such as a surgeon to readily view the connectors to ensure that the appropriate connections have been made, and may help diagnose system 100 if a fault is detected, for example.

[0082] In some embodiments, system 100 comprises a source of fluids, such as a liquid, e.g, saline, although other fluids such as gasses for insufflation may be provided. In some embodiments, a saline source 112 is coupled to waterjet pump cartridge 170. In some embodiments, a saline source 114 is coupled to treatment probe 150 in order to flush the surgical site with a fluid such as saline, for example. In some embodiments, one or more of the saline sources is supported with a pole 116 to provide saline to one or more of the pump or the probe 150 with gravity assisted flow, for example. The pole 116 may comprises a pole that is separate from the console 120 or a pole that is integrated with the console 120, for example so as to extend upwardly from the console.

[0083] In some embodiments, the system 100 comprises a second display 125, which is supported with the console 120, The second display 125 can be helpful for staff to view the surgery and to assist the surgeon. In some embodiments, the second display 125 mirrors display 140, The second display 125 may have similar user input features as the display 140, or decreased functionality to ensure the physician retains control of the treatment.

[0084] In some embodiments, the screen of the second display 125 comprises annotation input features for staff or a proctor to provide annotation inputs for the surgeon to view on the first display 140, for example without controlling system functions from the second display.

[0085] FIG. 1B shows a schematic diagram of some components of system 100. In some embodiments, the console 120 comprises one or more processors 180. The one or more processors 180 can be configured with instructions to perform various functions of system 100. In some embodiments, the one or more processors 180 comprise an operating system 181, an authentication database 182 to provide user authentication, a medical database to store medical data of patients, an application block 184 to run one or more applications of the system, such as treatment planning and user interface ( U I ) processes, instructions to the motors to move the linkage of treatment probe 150, and system monitoring functions as describe herein. In some embodiments, the console 120 comprises imaging circuitry such as ultrasound circuitry 185 housed within the console 120. In some embodiments, the one or more processors 180 is configured to give different users different levels of access 188 such as full system access or partial access to system 100. The access 188 can be provided through a console user interface, for example via the console display 125, which may comprise a touch screen display, for example. In some embodiments, the system access is provided to administrators, such as a urology administrator of the local health care provider, hospital information technology administrators, and representatives of the manufacturer of system 100, such as field service representatives. In some embodiments, the system 100 comprises communication circuitry 187 housed within console 120, such as universal serial bus (USB) circuity and wireless fidelity (WiFi) circuitry, for example.

[0086] While the console 120 can be configured in many ways, in some embodiments, the console 120 comprises internal components, such as an internal computer system and an internal ultrasound imaging system housed within console 120. In some embodiments, the console 120 comprises a tower configuration, in which the console comprises a height greater than a width of the console.

[0087] The console 120 can be coupled to the components of system 100 with one or more connectors such as electrical connectors or fluidic connectors as described herein. In some embodiments, the console 120 is coupled to the console display 125 with an electrical connector such as an HDMI cable. The console can be connected to the endoscope such as cystoscope 189 with an electrical line to transmit electrical signals, in order to display images on the displays. The motor-pack 124 can be coupled to the console 120 with one or more electrical lines such as cables, to transmit movement signals and provide feedback to the one or more processors within console 120. The motor-pack 154 can be coupled to the handpiece 152 with one or more electrical lines to transmit electrical signals between the handpiece and the motor-pack 154 and the one or more processors 180, such as encoder signals and feedback signal related to movement of the energy source on the probe provided by a linkage housed within the handpiece 152. The one or more sensors 186 can be coupled to the one or more processors 180 with internal circuitry housed within console 120 to monitor fluid flow, for example. The one or more sensors 186 can be coupled to the aspiration pump, for example. In some embodiments, the one or more sensors 186 is coupled to the handpiece 152 with aspirationtubing 176 to monitor the removal of material from handpiece 152. In some embodiments, the saline source 114 such as the saline bag is coupled to the handpiece 152,

[0088] In some embodiments, the ultrasound imaging device such as TRUS probe 160 is coupled to the console 120 with one or more electrical lines, such as an electrical cable to couple the ultrasound imaging device such as TRUS probe 150 to the ultrasound circuitry 185.

[0089] In some embodiments, the console 120 comprises one or more sensors 186 coupled to the aspiration line 176 to measure fluid flow from the handpiece 152.

[0090] In some embodiments, the system 100 comprises an endoscope such as cystoscope 189, configured to view optical images of tissue at the surgical site. The endoscope such as cystoscope 189 may be coupled to the handpiece 152 and the probe 150 as described herein.

[0091] In some embodiments, a pump cartridge 170 is insertable into a receptacle of the console, in order to provide pressurized water to the energy source such as a waterjet on the treatment probe. In some embodiments, the high pressure line 172 couples the pump cartridge to the waterjet on the handpiece.

[0092] In some embodiments, the foot pedal 190 is coupled to the console 120 with one or more electrical lines to transmit signals, such as a cable, in order to control system 100 as described herein.

[0093] The arm 145 can be configured in any way to support the surgeon display 140 above the patient as described herein. In some embodiments, the arm 145 comprises a plurality of joints and one or more of a locking arm, a clutch arm, a manual arm, a gas-spring arm, a robotic arm, a zero-gravity arm or a zero gravity robotic arm, for example. In some embodiments, the arm 145 comprises a gas spring arm with a gas strut and a spring.

[0094] FIG. 2 shows alignment of a user interface 103, a treatment probe 150 and an imaging probe 160 of the system 100, in accordance with some embodiments. In some embodiments, the patient comprises a midline 205, and the user interface 103 and the treatment probe 150 are arranged with respect to the midline of the patient. In some embodiments, the monitor is poisoned such that a plane corresponding to the midline of the patient extends through the monitor, which can help with visualization of the images shown on the display. This location is also helpful because it allows the surgeon to readily reach the input controls of the user interface 103, which comprises the display 140, Readily reaching the input controls can be helpful to decrease fatigue associated with controlling the surgical system 100, for example when several surgeries are performed in a day or with long surgeries lasting more than an hour, for example.

[0095] In some embodiments, the arm 145 is configured to support the user interface 103 comprising the surgeon display 140 coupled to the positioning arm, in which the positioning arm 145 is configured to support the display at a first location above a midline of the patient with a first configuration. In some embodiments, the arm 145 is configured to place the surgery display 140 at a second location away from the midline of the patient with a second configuration for storage as described herein.

[0096] While the arm 145 can be configured in many ways, in some embodiments the arm is configured to allow movement of the surgeon display 140 for visualization and interaction by the surgeon while surgeon may be on one knee for insertion of the ultrasound probe such as TRUS probe 160, for example. In some embodiments, the display is configured to receive user inputs when covered with a sterile drape, and the inputs are provided through the sterile drape, and the sterile drape may comprise a substantially transparent drape to allow the user to see the display throughthe drape and provide inputs through the drape. The arm 145 can be configured for the surgeon to move the display 140 out of the way during handpiece insertion, for example by pushing on display 140 to rotate the arm 145 relative to the console 120 to position the display away from the surgeon and superiorly above the patient for example while using sterile technique with display at least partially draped and the surgeon wearing surgical gloves. The display 140 can then pulled in toward the physician while seated or standing for touch-screen intensive steps after insertion and alignment of probes as described herein,

[0097] In some embodiments, the imaging probe such as TRUS probe 150 is arranged with respect to the midline of the patient, the treatment probe 150 and the display 140. In some embodiments, this arrangement can assist the physician with understanding the physical locations of the probes in front of them and coordinating movements of the probes in relation to internal images of the patient shown on the display. For example, the physician can move a proximal end of a probe in order to manually move a distal portion of the probe, e.g. manually translate or rotate the probe, or use instrument controls from the user interface 103 in order to move one or more of the probes to a desired location within the patient, such as with translation or rotation of the probe.

[0098] In some embodiments, the imaging device such as TRUS probe 160 is configured to generate longitudinal images such as sagittal images and transverse images. In some embodiments, the longitudinal images correspond to an imaging plane along with a longitudinal image of the probe, and the transverse images correspond to images along plane transverse to the elongate axis of the imaging probe. In some embodiments, the transverse images correspond to a transverse field of view 262, which corresponds to a transverse image plane. In some embodiments, the transverse field of view corresponds to an angle of rotation about an elongate axis of the imaging probe such as TRUS probe 260. With the monitor placed over the midline of the patient, the physician can more readily associate the transverse images shown on the display with the physical locations of the corresponding tissue that is shown in the images.

[0099] In some embodiments, the imaging device such as TRUS probe 260 is configured to generate longitudinal images, e.g. sagittal images, along a longitudinal image plane within a longitudinal image field of view 624, which may comprise a sagittal image along a sagittal image field of view. In some embodiments, the monitor is aligned with the patient at a location above the sagittal image plane, such that the plane of the sagittal image extends through the monitor, for example. This alignment of the display with the longitudinal image plane allows the physician to associate the position of the probes more readily with the longitudinal images. Alternatively or in combination, the monitor can be located inferiorly or superiorly, with respect to the patient, such that the plane of the transverse image is within about 10 inches of the image display area of the monitor. In some embodiments, this positioning of the monitor can improve the physician’s ability to associate the transverse and sagittal images shown on the display with the physical locations of the probes and corresponding tissue that is shown in the images.

[0100] In some embodiments, the transverse image shown on the display approximately parallel to the transverse image plane. In some embodiments, the transverse image plane is parallel to the image shown on the display to within 30 degrees, or within 10 degrees, for example. In some embodiments, the ultrasound imaging device comprises an ultrasound array configured to generate a longitudinal image along a longitudinal image plane, the arm configured to support the display with longitudinal image displayed on the display above the patient.

[0101] In some embodiments, the longitudinal image plane corresponds to a plane extending from the ultrasound transducer array and through a portion of the display. In some embodiments, the transducer device is rotatable abouta longitudinal axis in order to rotate an angle of the longitudinal image plane to view the treatment probe with the plane extending through the transducer array, the treatment probe and the monitor.

[0102] In some embodiments, the treatment probe extends through at least a portion of the longitudinal image plane, in some embodiments, the longitudinal image shown on the display is approximately perpendicular to the longitudinal image plane and optionally to within 30 degrees of perpendicular.

[0103] In some embodiments, the ultrasound imaging device such as TRUS probe 160 comprises a transverse array and a longitudinal array as described herein, and the arm 145 is configured to position the image shown on the display to within 30 degrees of parallel to a long axis of the longitudinal array and to within 30 degrees of perpendicular to a long axis of the transverse array.

[0104] In some embodiments, patient support 105 is configured to support the patient with the patient support below patient, the midline 205 of the patient corresponding to a plane extending through the display and the support.

[0105] In some embodiments, the stirrups 107 are configured to support the legs of the patient splayed and the feet of the patient above a torso of the patient, and wherein the surgeon display 140 is sized to fit between the legs of the patient.

[0106] In some embodiments, the arm 145 is configured to support the surgeon display 140 above the patient and superiorly relative to a penile fenestration of a drape covering the patient, which the treatment probe 150 enters the patient.

[0107] While the surgeon user interface 103 and display 140 can be configured in many ways, in some embodiments, display 140 has a height and a width, wherein the height is greater than the width. In some embodiments, the positioning arm 145 is configured to align a vertical centerline 230 of the display 140 with a midline of the patient 205 and a corresponding midline of the patient support. In some embodiments, the positioning arm 145 is configured to place a vertical centerline 230 of the display within about 2 inches of a midline 205 of the patient with the patient on the support 105.

[0108] In some embodiments, the positioning arm 145 is configured to place a centerline of the touchscreen display 140 to within 2 inches of a centerline of the treatment probe, for example a longitudinal axis of probe 150.

[0109] In some embodiments, the surgeon display 140 comprises a touch screen display configured to receive a plurality of user inputs through a drape placed over the touch screen display to provide sterility. The arm 145 is configured to resist movement in the extended configuration in response to the user input through the drape. In some embodiments, the touchscreen display 140 is configured to receive a plurality of user inputs from a finger of a user covered with a glove transmitted from the finger through the glove and the drape to the touchscreen display, and the arm is configured to resist movement in response to the user input transmitted through the glove and the drape. In some embodiments, the drape comprises a sterile drape and the glove comprises a sterile glove,

[0110] While the arm 145 can be configured in many ways, in some embodiments the arm comprises a locking arm configured to lock the arm in place in the extended configuration above the patient. In some embodiments, movement of the display is inhibited with a locking mechanism, such as a turn screw, lever or clamp, for example.

[0111] Alternatively or in combination, the arm can be configured not to move substantially when subjected to forces associated with inputs to display 140, such as touch inputs, and to move in response to greater amounts of force to reposition the display 140 supported with arm 145. In some embodiments, the positioning arm is configured toresist movement of the touchscreen display when placed above the midline 205 and touched by a user with a first amount of force to input data and to allow movement of the display with a second mount of force greater than the first amount of force. In some embodiment, the first amount force is within a range from about 0 (e.g. 0.01 or 0.1) pounds to about 5 pounds and optionally within a range from about 0 (e.g. 0.01 or 0.1) pounds to about 2 pounds, in some embodiments, the second amount of force is greater than about 2 pounds and optionally greater than about 5 pounds. The arm can be configured with stiction to inhibit movement of the display when the first amount of force is applied touch screen when the touch screen is positioned above the midline of the patient and to allow movement with the second amount of force greater than the first amount of force, for example. The arm 145 may comprise frictional pads configured to resist movement with forces below the threshold amount of force and to allow movement with forces above the threshold amount.

[0112] FIGS. 3Ato 3F show a user interface 103 comprising a display 140 of the system 100, in accordance with some embodiments. In some embodiments, the display 140 comprises a portrait orientation to improve placement of the monitor above the patient as described herein. In some embodiments, the user interface 103 comprises an image display area 310 and a control panel 302, in which the control panel 302 is partitioned from the image display area. In some embodiments, the control panel 302 is located on an area of the display that is separate from the image display area 310, such that the control panel and image display area appear on different areas of the display. In some embodiments, the control panel 302 is located below the image display area 310. Alternatively, the control panel can be located above the image display area or alongside the image display area, for example with the display in a landscape configuration.

[0113] Although reference is made to a control panel 302 in the context of a touch screen display, one of ordinary skill in art will recognize that the control panel 302 may comprise one or more hardware components instead of a touchscreen area of the display 140 as described herein. The control panel 302 located below the image control area can facilitate user control and decrease the likelihood that the user will reach as far upward to provide inputs to user interface 103 as compared to the image control panel being located above the image display area.

[0114] In some embodiments, the user interface 103 is configured for the user to touch the user interface such as the touchscreen display a total number of times at a plurality of locations to set up the treatment, plan the treatment and perform the treatment, and at least half of the plurality of locations touched the total number of times is located below a centerline of the display, which can improve ergonomics and facilitate the user providing input to the user interface, in some embodiments, at 75% of the plurality of locations touched the total number of times is located below a centerline of the display. In some embodiments, at 90% of the plurality of locations touched the total number of times is located below a centerline of the display, in some embodiments, the total number of times the user touches the user interface at the plurality of locations to set up the treatment, plan the treatment and perform the treatment comprises a total number of touches at least 50 times, a total number of touches of at least 100 times, a total number of touches of at least 150 times, a total number of touches of at least 200 times, or a total of number of touches of at least 250 times.

[0115] In some embodiments an upper area 307 of the display is located above a horizontal centerline 305 and a lower area 309 of the display is located below the horizontal centerline 305. In some embodiments, at half of the image display area 310 is located above the centerline and at least half of the area of the control panel 302 is located below the centerline 305, which can improve ergonomics because the control panel can be more readily reached bythe user as compared to the control panel located above the image display area. In some embodiments the user will touch the control panel at least twice as often as the image display area, and in some embodiments the user will touch the control panel at least four times as often as the image display area. In some embodiments, at least 75% of the image display area 310 is located above the horizontal centerline and at least 75% of the control panel is located above the image display area. In some embodiments, at least 90% of the image display area 310 is located above the horizontal centerline and at least 90% of the control panel is located above the image display area. In some embodiments, the entire image display area 310 is located above the horizontal centerline and the entire control panel is located above the image display area.

[0116] The image display area 310 can be configured in many ways and may comprise a primary image area 312 and a secondary image area 314. The secondary image display area can be located outside the primary image display area. Alternatively or in combination, the secondary image may be shown within a portion of the primary image, for example with a picture in picture configuration. In some embodiments, the primary image display area is larger than the secondary image display area.

[0117] The control panel 302 can be configured in many ways, and may comprise a tool bar 321, for example. The tool bar 321 may comprise one or more of a plane toggle icon 322 for a user to toggle between sources of the primary image and the secondary image in response to a user input to the icon, an image setting icon 324 for a user to select to adjust image settings, a longitudinal control icon 326 for a user to adjust a longitudinal position of an energy source on the treatment probe, a measurement icon 398 for a user to select to measure distances between structures shown in an image, or an information icon 328 for a user to select to display information such as information about the surgical system and the patient, for example. Although reference is made to plane toggle icon 302, image setting icon 324, longitudinal control icon 326, measurement icon 398 and information icon 328 in the context of tool bar 321, one or more of these icons and controls can be located elsewhere on the display, for example.

[0118] The icons shown on the display may comprise any suitable symbol to indicate the functionality of the icon. In some embodiments, the icon is configured for a select the icon with a user input such as touching the display at a location corresponding to the icon. In some embodiments, the plane toggle icon 322 is configured to receive a user input such as the user touching the icon to toggle the primary image and the secondary image source.

[0119] The settings icon such as the image setting icon 324 can be configured in many ways, and may comprises any suitable icon such as a gear, slider or other icon that conveys to the user that settings can be adjusted with the icon. In some embodiments, the image setting icon 324 shows a plurality of sliders to indicate to the user that the user can adjust the image settings. In some embodiments, the user touching an icon to provide user input results in a pop up window, such as a pop up window for the user to adjust the imaging settings, which may comprise imaging setting of one or more of the endoscope or the TRUS probe. In some embodiments, the icon may comprise a plurality of controls associated with the icon. For example, the longitudinal control icon 326 may comprise a first icon such as an arrow pointing in a first direction to indicate to the user that an energy source is moved in the first direction in an image shown on the display, e.g. to advance a probe carrying the energy source, and a second icon such as an arrow pointing in a second direction to indicate to the user that the energy source is moved in the second direction in an image shown on the display, e.g. to retract the probe carrying the energy source. The longitudinal control 326 may comprise a third icon, such as a bar, to show the position of the energy source on the probe, for example.

[0120] In some embodiments, the user interface is configured to work with a first display 140 such as a surgeon display and a second display 125 for staff and other people associated with the procedure. In some embodiments, a display control icon 391 allows a user to adjust input settings of the first display 140 and the second display 125 to allow limited inputs with the second display 125 while the surgeon controls the inputs with surgeon display 140. In some embodiments, the display control icon 391 comprises an annotation enable toggle that allows annotations to be input from the second display 125 when enabled and does not allow annotations to be input from the second display 125 when disabled. In some embodiments, the display control icon 391 is configured to provide a pop-up window that allows the user of the first display 140 to allow at least some user input from the second display 140, such as annotations. In some embodiments, the first display 140 comprises a touch screen display configured for a user such as a surgeon to control and plan the surgery with inputs as described herein, and the second display 125 is configured to substantially mirror the first display 140, which allows the operating room staff to readily view the procedure on the second display 125 in synchrony with the surgeon viewing the first display 140.

[0121] In some embodiments, only the first display 140 is configured to accept and confirm the treatment plan from the surgeon touchscreen display 140, even when annotations at the second display are enabled. In some embodiments, the second display 125 is configured to allow limited user inputs such as annotations without adjusting treatment parameters or accepting the treatment plan, and the annotations are mirrored on both the first display 140 and the second display 125, such that the annotations can be viewed on both displays without allowing control of the procedure or input parameters from the second display 125. The annotations provided with input to the second display 125 can be helpful to communicate with the surgeon and can be used for telestration and tele-proctoring, for example. Although reference is made to a first display and a second display, additional displays can be provided. For example a remote display can be configured to allow annotations that are viewed on displays 125 and 140, which can allow remote telestration and remote proctoring, for example. In some embodiments, the second display 125 comprises a remote display, such as a display in another room, another building, another state, or another country, for example.

[0122] In some embodiments, the control panel 302 provides one or more instructions 304 to the user. The instructions may comprise any suitable instruction, such as an instructions to be perform at next step at a current stage in the procedure, or an instruction to view a video, for example.

[0123] In some embodiments, the control panel comprises a navigation bar 330. The navigation bar may comprise icons for the user to navigate through stages of the procedure. In some embodiments, the navigation bar 330 comprises a previous icon for the user to go back to a prior screen and step of the procedure, and a next icon for the user to advance to a next screen and step of the procedure.

[0124] In some embodiments, the navigation bar 330 comprises one or more indicators for a user to assess progress of the navigation through different stages of the procedure. In some embodiments, the user navigation bar 330 is configured to display a plurality of modes 340 of the system 100, such as one or more of a set up mode, a plan mode or a treatment mode. Each of the modes may comprise a plurality of sub-modes, in which each of the corresponding sub-modes corresponds to a stage of the associated mode. In some embodiments, the setup mode comprises one or more of a TRUS sub-mode to insert the ultrasound probe such as a TRUS probe, a handpiece mode for the user to insert the treatment mode with the handpiece, and an align mode for the user to align the treatment probe and the TRUS probe with each other to visualize the treatment probe with the TRUS probe. In someembodiments, the plan mode comprises an angle and depth sub-mode for the user to determine a plurality of angles and depths of treatment from the probe on a plurality of transverse images, a registration sub-mode for the user to register the position of the energy source along a longitudinal image such as a sagittal image, and a profile sub-mode for the user to adjust markers associated with a positions of the treatment plan on the longitudinal image such as a sagittal image.

[0125] Generally, an icon of the user interface 103 is highlighted it appears differently relative to other icons, for example if its contrast, boldness, color or shape on the display relative to other icons is different than the contrast, boldness, color or shape of other, non-highlighted icons. In some embodiments, a highlighted icon comprises an icon that is more visible than non-highlighted icons.

[0126] Referring to FIG. 3A, the set up mode is highlighted relative to other modes of the plurality of modes 340, such as relative to the plan mode icon and the treat mode icon. The handpiece sub-mode 343 is highlighted relative to the other sub-modes of the plurality of sub-modes 342, such as the TRUS sub-mode and the Align sub-mode.

[0127] Although reference is made to the plurality of modes and corresponding sub-modes in the context of a navigation bar, these modes and corresponding sub-modes can be presented on the display outside the navigation bar.

[0128] In some embodiments, the Ul 103 is configured for the user to press the next icon 334 to proceed to the next stage of the procedure, although the user may press the previous icon 332 to proceed to a prior stage of the procedure.

[0129] Referring to FIGS. 3B and 3C, the Ul 103 is configured to align the treatment probe 150 and ultrasound device such as TRUS probe 160.

[0130] As shown in FIG. 3B, the primary image 312 comprises a longitudinal ultrasound image such as sagittal image from TRUS probe 150, and the secondary image 314 comprises the endoscope image such as cystoscope image. The one or more modes 340 comprises the setup mode and the one or more sub-modes 342 comprises the align mode with the align sub-mode highlighted in relation to the other sub-modes 342, such as the hand piece mode 343. One or more markers 303 shows a position of the display corresponding to an end of the ultrasound probe such as TRUS probe 160. In some embodiments, the path of the energy 306 from the energy source is visible on the primary image 312 shown on the display.

[0131] In some embodiments, the processor is configured with instructions to automatically change the source of the primary image 312 shown on the display in accordance with the current task to be performed by the user, for exampie a task associated with the current mode of the one or more modes 340 and the current sub-mode of one or more sub-modes 342. As shown in FIG. 3B, the Ul has changed the primary image 312 to a real time ultrasound image and the secondary image 314 to the real time endoscope image. In some embodiments, the user interface is configured to change the primary image 312 from the real time endoscopic image such as the cystoscope image to the real time ultrasound image, for example to a longitudinal image such as the sagittal image in response to the stage of treatment progressing from the handpiece sub-mode 343 to the alignment sub-mode. In some embodiments, the processor is configured to automatically change the primary image 312 of the user interface 103 in response to the user progressing to another stage of a mode or sub-mode, such as changing the primary image from a longitudinal image such as a sagittal image, for example. In some embodiments, the secondary image 314 comprises the real time endoscopeimage such as the cystoscope image while the user interface or the user changes the ultrasound image shown as the primary image 312.

[0132] In some embodiments, the one or more user instructions 304 are configured to instruct the user. The one or more user instructions 304 may comprise a single displayed instruction or a plurality of instructions, for example. In some embodiments, the user is instructed to press on the foot pedal to release energy from the energy source, in order to detect the position of the energy source such as a water jet, although other energy sources can be detected on the primary image as described herein. In some embodiments, the user is instructed to translate the imaging device such as the TRUS probe along a longitudinal axis of the imaging probe to align the one or more markers 303 with the energy source such as the waterjet. In some embodiments, the user is instructed to provide an input such as a click or touch on the information icon 328 to obtain additional information about the current mode and sub-mode, for example, in some embodiments, the user is instructed to change the view from the imaging device from a longitudinal mode such as a sagittal mode to a transverse mode.

[0133] In some embodiments, the Ul 103 is configured for the user to press the next icon 334 to proceed to the next stage of the procedure, although the user may press the previous icon 332 to proceed to a prior stage of the procedure.

[0134] As shown in FIG. 3C, the primary image 312 comprises a transverse ultrasound image such as transverse image from TRUS probe 150, and the secondary image 314 comprises the endoscope image such as cystoscope image. The one or more modes 340 comprises the setup mode and the one or more sub-modes 342 comprises the align mode with the align sub-mode highlighted in relation to the other sub-modes 342, such as the hand piece mode 343. One or more markers 303 shows a position of the display corresponding to an end of the ultrasound probe such as TRUS probe 160. In some embodiments, path of energy 306 from the energy source is visible on the primary image 312 shown on the display.

[0135] In some embodiments, the processor is configured with instructions to automatically change the primary image 312 shown on the display in accordance with the current task to be performed by the user. As shown in FIG.3C, the Ul has automatically changed the primary image 312 to a real time ultrasound transverse image from the real time longitudinal image as shown in FIG. 3B, while the secondary image 314 remains the real time endoscope image,

[0136] The one or more user instructions 304 are configured to instruct the user at this stage of the align sub¬ mode, In some embodiments, the user is instructed to press on the foot pedal to release energy from the energy source, in order to detect the position of the energy source such as a waterjet, although other energy sources can be detected on the primary image as described herein. In some embodiments, the user is instructed to rotate the imaging device such as the TRUS probe about a longitudinal axis of the probe to position a hyperechoic shadow of the treatment probe ata desired position, such as 12 o’clock, which corresponds to a substantially vertical position above the imaging probe as shown in the image, e.g. to within about 10 degrees of vertical. In some embodiments, the user is instructed to provide an input to the foot pedal to visualize the position and orientation of the energy 306 from energy source.

[0137] In some embodiments, the user is instructed to rotate the handpiece about the longitudinal axis of the probe and the handpiece, for example to rock the handpiece, such that the orientation of the energy source such as a waterjet comprises a substantially vertical orientation, e.g. to within about 10 degrees of vertical, although any suitable orientation can be used. In some embodiments, the positioning of the display above the probes facilitates alignmentof the probes., for example with the active light emitting area of the display oriented such that the display is substantially parallel to the transverse image and the longitudinal axis of one or more of the probes substantially perpendicular to the active light emitting area of the display as described herein. In some embodiments, the user is instructed to provide an input such as a click or touch on the information icon 328 to obtain additional information about the current mode and sub-mode, for example. In some embodiments, the user is instructed to change the view from the imaging device from a longitudinal mode such as a sagittal mode to a transverse mode.

[0138] In some embodiments, the Ul 103 is configured for the user to press the next icon 334 to proceed to the next stage of the procedure, although the user may press the previous icon 332 to proceed to a prior stage of the procedure.

[0139] Referring to FIG. 3D, the user interface 103 can be configured to receive user input to plan the treatment with a plurality of images that are transverse to the probe 150, such as a plurality of transverse images from the TRUS probe. In some embodiments, the mode of the one or more modes 340 comprises a plan mode. In some embodiments, the plan mode comprises a plurality of sub-modes 342, in which the plurality of sub-modes comprises an angle and depth mode, a registration mode, and a profile mode. As shown in FIG. 3D, the plurality of modes 340 comprises the plan mode and the plurality of sub-modes 342 comprises an angle and depth mode 346. In some embodiments, the Ul is configured to highlight the current mode such as the plan and the sub-mode such as the angle and depth mode 346. With the angle and depth sub-mode 346, the user is able to adjust the angle of treatment and the radial distance of the treatment from the probe such as a depth of treatment from the probe 150.

[0140] With the plan mode and the angle and depth sub-mode, the user is provided with an instruction 304 to select an anatomical tab and adjust the plurality of markers 353.

[0141] In some embodiments, in the plan mode the user interface 103 is configured for the user to provide input related to the angles of treatment delivered with the energy source such as a directional energy source on the treatment probe 150. In some embodiments, the primary image 312 comprises a transverse image such as a transverse ultrasound image, and the secondary image 314 comprises an endoscope image such as a cystoscope image. The image display area 310 shows the primary image and the secondary image, A plurality of markers 353 is, shown overlaid on the primary image. The plurality of markers 353 is configured to allow the user to adjust the treatment.

[0142] In some embodiments, the primary image 312 shown on the display comprises an image among a plurality of images at different tissue depths, and the user interface 103 is configured for the user to select the primary image to display among a plurality of images along image planes from different depths of the tissue, such as transverse images from different longitudinal locations along probe 150. A plurality of planning tabs 390 is shown above the primary image shown on the display, which allows the user to select the image to view for planning the treatment, although the plurality of planning tabs may be displayed at other locations. The plurality of planning tabs may comprise a first tab corresponding to a first depth along a longitudinal axis of the probe, such as a median lobe of the prostate, a second tab corresponding to a second longitudinal depth along the probe, such as a bladder neck, and a third tab corresponding to a second longitudinal depth along the probe, such as a mid-prostate, for example. The plurality of markers 353 is shown on each of the plurality of images selected with a corresponding tab.

[0143] In some embodiments, the plurality of markers 353 are shown at initial positions for each of the plurality of planning tabs and subsequently adjusted by the user. The initial positions of the plurality of markers may comprisedefault values, or values suggested with an artificial intelligence (Al) algorithm, such as a machine learning model, which can be trained neural network. In some embodiments, the UI is configured to display an assisted planning icon 399 for the user to turn on assisted planning. In some embodiments, the assisted planning is configured to indicate one or more tissue boundaries with a marker, such a dashed line. In some embodiments, the tissue boundary comprises a prostate boundary such as a boundary of the prostate. Alternatively or in combination, the assisted planning may provide initial locations for the plurality of markers 353 on the display, in which the locations of the plurality of markers 353 correspond to locations determined with the Al algorithm, such as a machine learning model, which can be a trained neural network.

[0144] In some embodiments, a reset input 396 is shown with an icon and configured for the user to reset the markers shown in the currently selected tab to previous locations that the user has entered, a default locations, or locations determined with the Al algorithm, such as a machine learning model, which can be a trained neural network.

[0145] The Ul is configured for the user to plan the treatment at a plurality of depths with the markers 353 shown on each of the plurality of images corresponding to different depths in the tissue. This allows the user to plan the treatment angles at different depths of the tissue. In some embodiments, the user selects the planning tab corresponding to an image, and adjusts the locations of the one or more markers 353 on the image shown with the corresponding planning tab.

[0146] The Ul 103 is configured for the user to select a marker among the plurality of markers 353 to adjust the location of the selected marker on the primary image 312. In some embodiments, the Ul comprises a plurality of marker select icons such as buttons 350, which allows the user to select a marker to adjust among the plurality of markers 353. In some embodiments, the plurality of markers 353 comprises a first marker (1), a second marker (2), and a fourth marker (4). With a marker selected, the user is able to adjust the position of the selected marker on the primary image 312, which results in a corresponding change to one or more of the depth of treatment from the probe or the angular location of the treatment. The selected marker among the plurality of markers can be moved with user input in any way. In some embodiments, the display comprises a touch screen display, in which the Ul 103 is configured for the user to drag the marker to a desired location. Alternatively or in combination, the marker can be moved with an input to directional icon, such as directional pad 374, for example. In some embodiments, the Ul is configured for the user to make a coarse adjustment to the selected marker shown on the display by dragging the marker to a different location on the touch screen display, and to make a fine adjustment by pressing a button on the directional icon such as directional pad 374. In some embodiments, the directional pad is configured to move the location of the selected marker by an incremental amount in response to the user depressing a directional icon, such as an amount within a range from about 0,05 mm to about 0,2 mm in response to the user pressing on the directional icon, for example. In some embodiments, the amount of movement corresponds to a scale of the image shown on the display, such as a scale in millimeters (mm).

[0147] As shown in FIG. 3D, for example, marker three (3) has been selected and the user is allowed to adjust the position of marker three. The selected marker can be highlighted on the display, for example with a color, or other indication such as geometric shape around the marker. The geometric shape may comprise any suitable shape such a bracket, circle, square or diamond, for example.

[0148] In some embodiments, the Ul 103 is configured is configured to constrain the direction of movement of the selected marker and to indicated to the user that the direction of movement of the marker has been constrained, for example by highlighting the allowable directions on the icon. As shown in FIG. 3D, the movement of the selected marker among the plurality of markers is limited to up and down movement on the display, and the directionai icon such as direction pad 374 shows the up and down arrows highlighted with respect to the lateral arrows,

[0149] In some embodiments, the Ul 103 is configured for the user to select each of the plurality of markers, e.g. markers 1, 2, 3 and 4, and to adjust the position of each of the selected markers. For example, the user can select marker 1 to align the treatment profile with the image of the treatment probe 150 shown in the primary image 312. in some embodiments, the directional icon such as directional pad 374 shows four directional icons highlighted, e.g. left, right, up and down, to indicate to the user that the marker can be moved in each of the corresponding directions. The user can select marker 2 to adjust the depth of the treatment. The user can select marker 3 to adjust an angle of treatment and a depth of the treatment with respect to treatment probe 150 corresponding to the position of marker 1. In some embodiments, the directional icon such as directional pad 374 shows four directional icons highlighted, e.g. left, right, up and down, to indicate to the user marker 3 can be moved in each of the corresponding directions. The user can select marker 4 to adjust an angle of treatment and a depth of the treatment with respect to treatment probe 150 corresponding to the position of marker 1. In some embodiments, the directional icon such as directional pad 374 shows four directional icons highlighted, e.g. left, right, up and down, to indicate to the user marker 4 can be moved in each of the corresponding directions. The adjustment to markers 3 and 4 can be used to adjust the planned angle of treatment 392, which corresponds to an angle between marker 3 and marker 4. The planned angle of treatment 392 can be shown on the display, and may comprise any suitable angle, such that the user can adjust the displayed angle of treatment 392 from 145.5 degrees to another angle in response to user preferences.

[0150] If the user is not satisfied with a position of the selected marker after adjustment, the user can select the undo icon 351 with a user input such as touching the display at the location of the undo icon, which will move the marker to a location of the marker prior to user adjustment.

[0151] In some embodiments, the Ul 103 is configured for the user to press the next icon 334 to proceed to the next stage of the procedure, although the user may press the previous icon 332 to proceed to a prior stage of the procedure.

[0152] Referring to FIG. 3E, the user interface can be configured to receive user input to plan the treatment with one or more longitudinal images such as sagittal images. The one or more longitudinal images is aligned with treatment probe 150, such that treatment probe 150 extends a distance along the one or more longitudinal images. The one or more longitudinal images may be acquired with the TRUS probe, an ultrasound probe located on the treatment probe 150, or an external probe coupled to a skin of the patient, for example. As shown in FIG. 3E, the plurality of modes 340 comprises the plan mode and the plurality of sub-modes 342 comprises the profile mode 344. In some embodiments, the Ul is configured to highlight the current mode such as the plan mode and the sub-mode such as the profile sub-mode 344. With the profile sub-mode 344 highlighted, the user is able to adjust the distance from the treatment probe, such as a depth of treatment from the probe 150.

[0153] In some embodiments, in the plan mode the user interface 103 is configured for the user to provide input related to the depth and longitudinal position of treatment delivered with the energy source such as a directional energysource on the treatment probe 150. In some embodiments, the primary image 312 comprises the longitudinal image such as a sagittal ultrasound image, and the secondary image 314 comprises an endoscope image such as a cystoscope image. A plurality of markers 353 is, shown overlaid on the primary image. The plurality of markers 353 is configured to allow the user to adjust the treatment.

[0154] In some embodiments, the Ul 103 is configured for the user to perform a registration step at a registration stage of the work flow. The registration step may be performed as part of the planning mode of the one or more modes 340. In some embodiments, the user performs the registration step at a registration sub-mode of the one or more sub¬ modes 342. In some embodiments, the registration sub-mode is performed between the angle and planning sub-mode and the profile sub-mode, although the registration can be performed at any suitable stage of the procedure, for example with the alignment stage.

[0155] In some embodiments, the user views the treatment probe with a longitudinal image such as a sagittal image, such that the treatment probe 150 extends along a plane of the image. With the registration sub-mode, the user indicates a location of the energy source with a user input with the energy source located at a distal position. In some embodiments, the user inputs a second location corresponding to a proximal location of the energy source. In some embodiments, the user inputs the proximal location on the display. While the proximal location of the energy source can be input in many ways, in some embodiments the user places a marker along a pathway of the energy source, for example at a location along the probe. Alternatively or in combination, the user can move the energy source to the proximal location and input the proximal location. In some embodiments, the energy source such as a waterjet is activated at the first location to allow the user to accurately identify the position of the energy source shown on the display and provide an appropriate input at the location. In some embodiments, the first location and the second location are shown with a line extending therebetween, and the user is allowed to adjust one or more of the first location or the second location with a coarse or fine input to the display, for example with the directional pad 374, for example. The first location and the second location can be used to register the plurality of transverse images with the position of the probe, which can be helpful for generating a treatment plan such as a 3D treatment plan.

[0156] As shown in FIG, 3E, the treatment probe 150 is shown on the image with a plurality of markers 353. In some embodiments, a longitudinal scale 380 is shown with a radial scale from the longitudinal axis of the probe such as a depth scale 382, The primary image 312 is with a plurality of markers 353 for the user to adjust the treatment on the longitudinal image. In some embodiments, one or more treatment limits 355 is show on the display to show the maximum depth of treatment. In some embodiments, the plurality of markers shown on the display correspond to plurality of regions of tissue to be protected. The markers may correspond to any tissue, such a verumontanum (veru), which can be identified veru protection zone (VPZ) marker 362, or a bladder neck (BN) identified with a BN marker 364. Each of the plurality of protection zones can be adjust with a user input to a corresponding icon. In some embodiments, the treatment at the VPZ can be adjusted with an input to a VPZ icon 366. In some embodiments, the treatment at the BN can be adjusted with an input to a BN icon 368.

[0157] With the plan mode and the profile sub-mode, the user is provided with an instruction 304 to adjust the plurality of markers 353 as needed for treatment.

[0158] In some embodiments, the Ul 103 is configured with the plurality of selection markers 350, in which a user can select a marker to be adjusted as described herein. The selected marker can be adjusted with coarse or fineadjustment as described herein, in some embodiments, the starting location of the treatment is selected for adjustment with a treatment start (TS) icon 352, Once the TS icon 352 has been selected, the location of the start of the treatment can be adjusted with coarse or fine adjustment as described herein. In some embodiments, the ending location of the treatment is selected for adjustment with a treatment end (TE) icon 354, Once the TE icon 354 has been selected, the location of the end of the treatment can be adjusted with coarse or fine adjustment as described herein. In some embodiments, the user interface 103 comprises a plurality of user selectable markers 353, such as a first selectable marker (1), a second selectable marker (2), a third selectable marker (3), a fourth selectable marker (4) and a fifth selectable marker (5). Each of the plurality of markers can be selected with a user input to a corresponding icon among the plurality of icons 350, and the location of the corresponding marker adjusted on the primary image 312, for example with coarse or fine movement as described herein. For example a first marker can be selected with a first input to a first icon 356, such as a user touching the first icon 356. Additional markers can be selected and adjusted as described herein.

[0159] As shown in FIG. 3E, for example, marker four (4) has been selected and the user is allowed to adjust the position of marker four, for example with coarse or fine adjustment as described herein. The selected marker can be highlighted on the display, for example with contrast, a color, or other indication such as geometric shape around the marker as described herein. The geometric shape may comprise any suitable shape such a bracket, circle, square or diamond, for example. The directional icon such as directional pad 374 can be highlighted to indicate the allowable movements of the marker, for example.

[0160] The plurality of treatment markers shown in the plurality of transverse images and the one or more longitudinal images such as one or more sagittal images can be used to generate a three-dimensional treatment plan.

[0161] In some embodiments, the Ul 103 is configured for the user to press the next icon 334 to proceed to the next stage of the procedure, although the user may press the previous icon 332 to proceed to a prior stage of the procedure.

[0162] Referring to FIG. 3F, the Ul 103 is shown in a treatment mode. A treat mode 347 is shown highlighted among the plurality of modes 340, and the treatment sub-mode is show highlighted. The primary image is shown on the display with a locked icon 397 to indicate that the treatment plan has been accepted and locked. The primary image and the secondary image are shown on the display as described herein. The user instruction 304 shows an instruction for the user to begin treatment by pressing and holding the foot pedal and to pause treatment by releasing the foot pedal. In some embodiments, the instruction comprises an instruction not to move the TRUS probe or the handpiece, for example. In some embodiments, the longitudinal image such as a sagittal image is shown as the primary image on the display. With the treatment mode, information related to the treatment is shown on the display, such as an intended position of the energy source along the longitudinal axis of the probe overlaid on the image. In some embodiments, the longitudinal scale 380 and the radial scale such as depth scale 382 are shown on the longitudinal image such as the sagittal image during treatment.

[0163] Although FIGS. 3A to 3F show a user interface in accordance with some embodiments, one of ordinary skill in the art will recognize many variations. For example, some of the screens may be modified, the order of the screens can be provided in sequence as shown or the order changed. Some of the screens may not be used and someof the screens can be modified in accordance with the present disclosure. Some of the modes may comprises sub¬ modes of other modes., and some of the sub-modes may comprises one or more modes with additional sub-modes.

[0164] The system 100 can be configured in many ways in accordance with the present disclosure. In some embodiments, the one or more processors of the system 100 is configured to provide one or more safety features. In some embodiments, the one or more processors is configured to prime the energy source initially at a first flow rate with a first source of saline prior to insertion of the treatment probe into the patient, and the processor is configured to decrease the flow rate to the energy source to prime the energy source a second time at a second flow rate less than the first flow rate with a second source of saline if the first source has been depleted with the treatment probe inserted into the patient. In some embodiments, the system is configured to prime the energy source such as a waterjet emitted from the probe 150. In some embodiments the system is configured to prime the energy source at over 90% of the treatment flow rate initially, e.g. at 100% of the treatment flow rate and to provide instructions to the user to prime the water jet with the water jet outside the patient. If system runs low on saline or out of saline to the energy source after the initial priming and prior to completion of treatment, the one or more processors of the system 100 are configured to automatically reduces the priming of the energy source to no more than 80%, for example to 50% or less. While this safety feature can be provided in many ways, in some embodiments the system is configured to automatically reduce the priming flow rate if the system runs low on saline or out of saline after the probe such as the TRUS probe has been inserted into the patient.

[0165] In some embodiments, the treatment probe comprises a handpiece and a motor-pack, and the one or more processors is configured to detect decoupling of the motor-pack from the handpiece. The processor is configured to disable treatment and direct the user to repeat one or more of an alignment mode or a planning mode of treatment prior to allowing the user to resume treatment, for example. In some embodiments, if the motor-pack and handpiece have become disconnected, the system is configured to bring the user back to the handpiece step, e.g. the handpiece sub-mode of the plurality of modes, for the user to complete one or more previously completed steps such as alignment and the planning steps. In some embodiments, the user interface is configured force the user to complete one or more of these previously completed steps.

[0166] FIG. 4A shows a treatment probe 150 configured for use with a surgical system. The treatment probe 150 comprises a distal portion sized for insertion into the patient. In some embodiments, the treatment probe comprises a handpiece 152 configured to manipulate the probe during insertion of the distal end into the patient. In some embodiments, the handpiece 152 is configured to couple to one or more actuators such as motors configured to move the treatment probe in response to computer control as described herein. In some embodiments, the handpiece is configured to couple to a motor-pack 154, which is configured to move an energy source such as directional energy source to a plurality of positions and orientations as described herein. In some embodiments, the handpiece 152 comprises a handpiece release button 410 configured to release the handpiece 152 from the motor-pack 154. In some embodiments, the motor-pack comprises one or more controls 412 to control an amount of energy of the energy source, such as an amount of waterjet power to a waterjet.

[0167] While the handpiece 152 can be configured in many ways, in some embodiments, the handpiece 152 comprises a linkage to rotate and translate the energy source in response to movements such as rotational movements transmitted from the motor-pack.

[0168] While the arm 156 that supports treatment probe 150 can be configured in many ways, in some embodiment the arm comprises lever and a motor-pack release knob 414 configured to release the motor-pack from arm 156, and to securely couple the motor-pack to the arm. While motor-pack 154 can be coupled to the arm 156 in many ways, in some embodiments, the arm 156 comprises a magnetic mount 416 configured to couple to the motor¬ pack to the arm. In some embodiments, the arm comprises a leveling button configured for a user to level the motor¬ pack, for example by pressing the button and then releasing the button to lock in the angle of inclination of the motor¬ pack, e.g. to level the motor-pack and the probe.

[0169] FIG. 4B shows the motor-pack 154 configured to couple to the handpiece 152 of treatment probe 150 and the arm 156. In some embodiments, the motor-pack 154 comprises a mounting plate 420 configured to couple to the arm. In some embodiments, the motor-pack 154 comprises a handpiece interface 422 configured to interface the motor-pack with the handpiece and may comprise one or more gears and an electrical connector to couple to the handpiece.

[0170] FIG. 5 shows treatment probe 150 and corresponding movements of an energy source 510. In some embodiments, the energy source 510 is carried on a carrier 520 that is configured to carry the energy source move the energy source. In some embodiments, the carrier 520 is configured to carry the energy source to a plurality of angles and longitudinal positions corresponding to a treatment plan such as a three dimensional treatment plan as described herein. In some embodiments, energy source 510 is configured to rotate to a plurality of angles 522 in accordance an angle of the treatment plan as described herein. In some embodiments, the energy source is configured to translate as shown with arrows 524. The rotational movement and translational movement of the energy source can be combined in accordance with the 3D treatment plan as described herein, for example.

[0171] The energy source 510 may comprise any suitable energy source, such as such as one or more of an electrode, a loop electrode, a laser source, a thermal energy source, a mechanical energy source, a mechanical sheer, an ultrasound probe, a cavitating ultrasound probe, a water jet, e.g. a fixed pressure water jet, a plasma source, a steam source, a morcellator, a trans urethral needle, a photo ablation source, a radiation energy source, a microwave energy source or a waterjet evacuation source, for example.

[0172] While the treatment probe 150 can be configured in many ways, in some embodiments the treatment probe comprises an opening 530 configured to receive an endoscope such as a cystoscope to view the treatment area. In some embodiments, the probe 150 comprises a support portion 530 that is configured to support the carrier and the endoscope. The support portion 530 may comprise one or more openings 552 configured to release a fluid such as saline from a saline source as described herein. In some embodiments, the probe comprises a portion 540 comprising sufficient stiffness to advance the probe into the patient. In some embodiments, the stiff portion 540 extends to a curved distal end 542 to allow the probe to be inserted into tissue, such as along a lumen. In some embodiments, the stiff portion 540 comprises one or more openings 544 to remove material such as fluids from the surgical site. In some embodiments, the one or more openings 544 is coupled to an evacuation pump such as the aspiration pump as described herein.

[0173] FIG. 6A shows an arm 156 configured to support the treatment probe 150. The arm comprises a bedrail clamp 612 configured to couple to a bedrail of the patient support as described herein. The arm 156 comprises a clamp lever 610 configured to securely couple the bedrail clamp 610 onto the bedrail of the patient support. In someembodiments, the arm comprises an articulation release lever 614 configured to allow the arm to articulate to adjust the position of the arm. The arm may comprise a plurality of locking joints 650 that are configured to move freely when the articulation release lever 614 is pressed down by a user, and then locked in place when the lever is released. The joints of the arm may comprise locking joints, such as joints with clutches, or magnetic joints, that can be moved into position prior to surgery and lock into place during surgery with computer controlled movement of the probe as described herein. Although reference is made to a manually manipulated arm, in some embodiments the arm comprises a robotic arm.

[0174] FIG. 6B shows mounting device 660 configured to couple to the motor-pack, in which the mounting device 660 comprises an unlocked configuration. The mounting device 660 comprises a rotation guide rail 622, a mounting lock indicator 624, and a motor-pack release knob 626. As shown in FIG. 6B, the release knob 626 is shown in an unlocked configuration.

[0175] FIG. 6C shows the mounting device of FIG. 6B, in which the release knob 626 comprises a locked configuration, in in which the motor-pack release knob has been rotated to a locked configuration.

[0176] FIG. 7 shows an ultrasound imaging device 700 to image tissue of the patient. The imaging device comprises one or more arrays to image tissue of the patient. The imaging device 700 may comprise an ultrasound imaging array, and may comprise a one dimensional (1 D), a two dimensional (2D) or a three dimensional (3D) imaging device. In some embodiments the one or more arrays comprises a transverse array 710 configured to image tissue along a plane that is transverse to an elongate axis of the probe 750. The probe 160 extends in an elongate direction along an axis 750 to a distal tip sized for inversion into a patient. The transverse array 710 extends in a direction transverse to the longitudinal axis 750 of the probe 160 to generate images that are transverse to the longitudinal axis 750, for example perpendicular to the axis 750. In some embodiments, the ultrasound imaging device is configured to generate transverse images within a field of view 262. In some embodiments, the ultrasound imaging device comprises a longitudinal array of transducers, such as a sagittal array 712 of transducers. The longitudinal array of transducers 712 extends in a longitudinal direction along probe 150 corresponding to axis 750. In some embodiments, the longitudinal array of transducers 712 is configured to generate a longitudinal image of tissue within a longitudinal image field of view, such as a sagittal plane image field of view 264,

[0177] In some embodiments, the ultrasound imaging device comprises a T-shaped marking to allow the user to identify the longitudinal axis of the array and the transverse axis of the array. In some embodiments, the transverse array 710 corresponds to a transverse imaging plane of the T-shaped marking, e.g, the top of the T-shaped marking, and the and the longitudinal array such as sagittal array 712 corresponds to the longitudinal imaging plane of the T- shaped marking, e.g. the bottom part of the T-shaped marking.

[0178] In some embodiments, the imaging device 700 comprises a probe configured to be inserted into the patient, such as a TRUS probe 160. In some embodiments, the TRUS probe 160 comprises plane switch button 714 to toggle between longitudinal and transverse images shown on the display as described herein. In some embodiments, the TRUS probe comprises a probe cradle and latch 716 configured to couple the probe 150 to a stepper 722. The stepper comprises a knob 720 that allows the stepper to advance and retract the probe from the patient. The stepper is configured to couple to the TRUS arm as described herein and move the probe 160 while the arm remains fixed, for example with rotation of the knob 720. By rotating the knob to a plurality of positions, the probe isplaced at a plurality of locations corresponding to a plurality of depths of the transverse images, which can be used for treatment planning as described herein. In some embodiments, the TRUS probe 160 comprises a cable 718 extending from a proximal portion of the TRUS probe 160,

[0179] In some embodiments, the location of the one or more of the longitudinal array such as sagittal array 712 or the transverse array 710 of the ultrasound device 700 is visible to the user. In some embodiments, the one or more of the longitudinal array or the transverse array is covered with a material which is visibly different from other areas of the distal end of the probe, such as with a different color or shading for example. The visible portion of the probe corresponding to the locations of the one or more transducer arrays can be associated with the position of the display. In some embodiments, the transverse array is approximately parallel to an active light emitting area of the display, and the visible portion of the transverse array can help the user associate the physical location of the tissue shown in transverse images with the location and orientation of the transverse array 710. In some embodiments, the longitudinal array such as the sagittal array 712 is approximately perpendicular to an active light emitting area of the display, and the visible portion of the longitudinal array can help the user associate the physical location of the tissue shown in longitudinal images such as sagittal images with the location and orientation of the longitudinal array such as sagittal array 712. In some embodiments, the ultrasound imaging device comprises TRUS probe 160, in which the locations of transverse array 710 and the longitudinal array such as sagittal array 712 are visible to the user in relation to the longitudinal axis 750 of the probe, for example.

[0180] FIG. 8 shows an arm 166 configured to support an ultrasound imaging device as in FIG. 7. The arm comprises a bedrail clamp 730 is configured to couple to a bedrail of the patient support as described herein. The arm 166 may comprises a clamp lever 732 configured to securely couple the bedrail clamp 730 onto the bedrail of the patient support. In some embodiments, the arm comprises an articulation release lever 726 configured to allow the arm to articulate to adjust the position of the arm. The arm may comprise a plurality of locking joints 750 that are configured to move freely when the articulation release lever 726 is pressed down by a user, and then lock in place when the lever is released. The joints of the arm may comprise locking joints, such as joints with clutches, or magnetic joints, that can be moved into position prior to surgery and locked into place during surgery with computer controlled movement of the probe as described herein. Although reference is made to a manually manipulated arm, in some embodiments the arm comprises a robotic arm.

[0181] FIG. 9 shows a user interface method 900. The method 900 may utilize one or more components of the system and features of the user interface as described herein.

[0182] At a setup mode 902, the user interface is configured to set up the system for surgery, for example with an insertion mode, in which the system is configured to insert a probe into a patient, for example as described herein.

[0183] At a plan mode 904, the user interface is configured with a plan mode to allow a user to plan a treatment, for example as described herein.

[0184] At a treatment mode 906, the user interface is configured with a treat mode, for example as described herein.

[0185] Each of modes 902, 904 and 906 may comprise or be associated with additional steps and modes or sub¬ modes, for example as described herein. Further, each of these steps, modes, or sub-modes may correspond to oneor more configurations and processes of the system as described herein. Any of these modes can be configured to provide one or more features of the user interface as described herein, for example with reference to FiGS, 3A to 3F.

[0186] In some embodiments, the instructions implemented with the processor are configured to automatically select a primary image source for a primary image display area and a secondary image source corresponding to a secondary image display area in response to the current mode and the current sub-mode of the system, which can make it easier for the user to proceed among the different modes and sub-modes of the user interface,

[0187] In some embodiments, the setup mode 902 comprises one or more modes or sub-modes, such one or more of as an imaging probe insertion mode 910, a treatment probe insertion mode 920 or a probe alignment mode 930. In some embodiments, one or more of imaging probe insertion mode 910, treatment probe insertion mode 920 or probe alignment mode 930 comprises a sub-mode of another mode, such as setup mode 902.

[0188] In some embodiments, the plan mode 904 comprises one or more modes or sub-modes, such as one or more of an angle and depth mode 940, a registration mode 950, or a profile mode 960.

[0189] In some embodiments, the treatment mode 906 comprises one or more modes or sub-modes such as one or more of a treatment mode 990 or a prime mode 980.

[0190] In some embodiments, the imaging probe insertion mode 910 comprises one or more of a user interface configuration step 912, a display primary and secondary images step 913, a display current mode step 914, a display instructions step 915, or a receive user input step 918. In some embodiments, the imaging probe insertion mode 910 configures the Ul for an imaging probe to be inserted into the patient. In some embodiments, the imaging probe comprises an ultrasound probe such as a TRUS imaging probe as described herein.

[0191] At the user interface configuration step 912, one or more features of the user interface is displayed, such one or more of the user input features, the image display area, the primary image, the secondary image, the tool bar, the control panel, the displayed mode, the displayed sub-mode, the navigation bar as described herein, for example with reference to FIG. 3A. In some embodiments, the instructions executed by the one or more processors are configured to provide the one or more features of the configuration step 912 in response to a user input such as a selection indicating completion of a prior mode or sub-mode, e.g, the next icon 334 as described with reference to FIGS. 3A to 3F.

[0192] At the display primary and secondary images step 913 of the imaging probe insertion mode 910, the user interface is configured to automatically display a real-time image from the ultrasound probe in the primary image display area. In some embodiments, the primary image display area comprises a real time transverse ultrasound image or a real time longitudinal ultrasound image, such as a sagittal image, and the user interface is configured for the user to toggle between the real time transverse image and the real time longitudinal image being shown in the primary image display area with a user input as described herein.

[0193] In some embodiments, the user interface is configured to automatically display an image from the ultrasound probe in the primary image display area and an image from the endoscope or other imaging device in the secondary image display area during the display primary and secondary images step 913 of the imaging probe insertion mode 910. In some embodiments, the user interface is configured to toggle between a transverse ultrasound image and a longitudinal, e.g. sagittal, ultrasound image being shown in the primary image display area, and the secondaryimage display area, depending on the ultrasound imaging probe inserted into the patient during imaging probe insertion mode 910, for example,

[0194] At a display current mode step 914 of the imaging probe insertion mode 910, the user interface is configured to display the current mode of the system such as the set up mode 902 and optionally a current sub-mode, such as the insert imaging probe mode 910. In some embodiments, the displayed set up mode 902 is highlighted with relative to other displayed modes as described herein, e.g. modes 904 and 906. In some embodiments, the current insert imaging probe insertion sub-mode 910 is displayed with the current mode 902 as described herein. In some embodiments, the current sub-mode 910 is highlighted with respect to other sub-modes 920, 930, and these are displayed with the current mode 910. At a display user instructions step 915, the user interface is configured to automatically display one or more instructions for the user to insert the ultrasound treatment probe and optionally display the one or more instructions in a user input area of the control panel, for example. In some embodiments, the one or more instructions to the user comprise instruction to 1) in transverse view, insert the ultrasound probe into the patient, ensure the prostate is centered, and with appropriate amount of compression, advance the ultrasound probe to visualize bladder space; 2) switch to sagittal view (using plane switch or button on the display screen to view the prostate; and 3) center the ultrasound probe from mid-prostate to bladder in the transverse view.

[0195] At a receive user input step 918, the user provides an input corresponding to one or more features of the user interface. In some embodiments, the input comprises an instruction to toggle a view from an image source such as an ultrasound probe as described herein. In some embodiments, the input comprises an instruction to proceed to a next stage, e.g. to one or more of a next mode, a next sub-mode, or a next step, for example.

[0196] In some embodiments, the user interface is configured to automatically proceed to the treatment probe insertion mode 920 in response to a user input indicating that the imaging probe insertion step has been completed, for example by the user selecting the next icon 334, e.g. the right arrow as described herein with reference to FIGS.3A to 3F.

[0197] In some embodiments, the treatment probe insertion mode 920 comprises one or more of a user interface configuration step 922, a display primary and secondary images step 923, a display current mode step 924, a display instructions step 925, or a receive user input step 928.

[0198] At a user interface configuration step 922, one or more features of the user interface is displayed, such one or more of the user input features, the image display area, the primary image, the secondary image, the tool bar, the control panel, the displayed mode, the displayed sub-mode, the navigation bar as described herein, for example with reference to FIG. 3A. In some embodiments, the instructions executed by the one or more processors are configured to provide the one or more features of the configuration step 922 in response to a user input such as a user input selection indicating completion of a prior mode or sub-mode, e.g. the “next” icon of a prior mode or sub-mode such as mode or sub-mode 910.

[0199] At a display primary and secondary images step 923 of treatment probe insertion mode 920, the primary image comprises an endoscope image such as the cystoscope image and the secondary image comprises an ultrasound image such as an image from a TRUS probe. In some embodiments, the treatment probe comprises an endoscope such as a cystoscope as described herein. The real time ultrasound image may comprise a transverse image or a longitudinal image such as a sagittal image from the TRUS probe. In some embodiments, the real-timeimage from the TRUS probe shown in the secondary image display area and can be toggled between sagittal and transverse views in response to user input, for example. In some embodiments, user interface is configured to automatically display a real-time image from an endoscope in the image display area in the treatment probe insertion mode 920, e.g. in the primary image display area as described herein. In some embodiments, user interface is configured to automatically display a real-time image from the ultrasound probe in the secondary image display area in the treatment probe insertion mode 920, e.g, in the secondary image display area while the real-time image from the endoscope is displayed in the primary image display area.

[0200] At a display current mode step 924 of the treatment probe insertion mode 920, the user interface is configured to display the current setup mode 902 of the system and optionally the current treatment probe insertion sub-mode of the system, such as the treatment probe insertion mode 920, in which the treatment probe is inserted with a handpiece as described herein, for example with reference to FIG. 3A. In some embodiments, the displayed mode 902 is highlighted with respect to other displayed modes as described herein, e.g. plan mode 904, and treatment mode 906. In some embodiments, the current sub-mode 920 is displayed with the current mode 902 as described herein. In some embodiments, the current sub-mode 920 is highlighted with respect to other sub-modes 910, 930, that are also displayed with the current mode 902, for example with reference to FIG. 3A and the plurality of modes 340 and handpiece sub-mode 343, which corresponds to treatment probe insertion mode 920.

[0201] At a display instructions step 925, the user interface is configured to automatically display one or more instructions for the user to insert the treatment probe and optionally display the one or more instructions in a user input area of the control panel, for example as described herein with reference to FIG. 3A.

[0202] At a receive user input step 928, the user provides an input corresponding to one or more features of the user interface. In some embodiments, the input comprises an instruction to toggle a view from an image source such as an ultrasound probe as described herein. In some embodiments, the input comprises an instruction to proceed to a next stage, e.g. to one or more of a next mode, a next sub-mode, or a next step, for example for example by the user selecting the next icon 334, e.g. the right arrow, as described herein with reference to FIG. 3A.

[0203] In some embodiments, the probe alignment mode 930 comprises one or more of a user interface configuration step 932, a display primary and secondary images step 933, a display current mode step 934, a display instructions step 935, a display markers step 936, or a receive user input step 938. In some embodiments, the alignment mode 930 comprises a first configuration of the user interface at a first step of mode 930 to longitudinally align the treatment probe with the imaging probe, for example with reference to FIG. 3B, and a second configuration of the user interface at a second step of mode 930 to rotationally align the imaging probe with the treatment probe, for example with reference to FIG. 3C.

[0204] At a user interface configuration step 932, one or more features of the user interface is displayed, such as one or more of the user input features, the image display area, the primary image, the secondary image, the tool bar, the control panel, the displayed mode, the displayed sub-mode, the navigation bar as described herein, for example with reference to FIGS. 3B and 3C. In some embodiments, the instructions executed by the one or more processors are configured to provide the one or more features of the configuration step 932 in response to a user input such as a user input selection indicating completion of a prior mode or sub-mode, e.g. the “next” icon of a prior mode or sub¬ mode such as mode or sub-mode 920.

[0205] At a display primary and secondary images step 933, the user interface is configured to automatically display a real-time image from an ultrasound probe in the image display area in the alignment sub-mode 930, In some embodiments, the user interface is configured to automatically display the real time ultrasound image in the primary display area and the real time image from the endoscope such as a cystoscope in the secondary image display area, for example with reference to FIGS. 3B and 3C. In some embodiments, the user interface is configured for the user to toggle the real time ultrasound image between the longitudinal view, e.g. sagittal view, and the transverse view in response to a user input. Alternatively, the user interface can be configured to automatically switch from a longitudinal view, e.g. a sagittal view, to a transverse view in response to a user input such as an input to proceed to the next stage.

[0206] In some embodiments, the real time image that is automatically displayed comprises a longitudinal image in the primary image area, which can facilitate alignment of the longitudinal axis of the imaging probe with the treatment probe, for example by moving the imaging probe along the longitudinal axis of the imaging probe. In some embodiments, the real time image that is automatically displayed comprises a transverse image such as a transverse ultrasound image, for example. In some embodiments, the real time endoscope image is displayed as the secondary image in the alignment mode 930 as shown in FIGS. 3B and 3C, for example.

[0207] In some embodiments, user interface is configured to automatically display, in sequence, a longitudinal image and a transverse image in the alignment sub-mode. In some embodiments, the alignment mode 930 is configured to automatically display the longitudinal image first as the primary image to complete a first step of longitudinal alignment, as shown in FIG. 3B. In some embodiments, the user interface is configured to automatically display the transverse image as the primary image for the rotational alignment step as shown in FIG. 3C. In some embodiments, the transverse image is automatically displayed in response to a user input such as a next input, for example when a first step of the alignment mode 930 has been completed. Although reference is made to completing longitudinal alignment prior to rotational alignment, the order can be reversed, for example.

[0208] At a display current mode step 934, the user interface is configured to display the current set up mode 902 of the system and optionally the current alignment sub-mode 930 of the system, such as the alignment mode 930. In some embodiments, the displayed mode 902 is highlighted with respect to other displayed modes as described herein, e.g. plan mode 904, and treatment mode 906. In some embodiments, the current sub-mode 930 is displayed with the current mode 902 as described herein. In some embodiments, the current sub-mode 930 is highlighted with respect to other sub-modes 910, 920, that are displayed with the current mode 902, for example with reference to FIGS. 3B and 3C.

[0209] At a display user instructions step 935, user interface is configured to automatically display one or more alignment instructions to move one or more of the treatment probe or the ultrasound probe in the alignment sub-mode 930, and optionally to display the one or more instructions in a user input area of the control panel. In some embodiments, the one or more alignment instructions comprises an instruction to rotate an energy source of the probe about a longitudinal axis of the treatment probe to rotate an orientation of the energy source toward an elongate axis of the ultrasound probe in a transverse ultrasound image, for example.

[0210] In some embodiments, the one or more alignment instructions comprises a first instruction for the user to translate the imaging probe and a second instruction for the user to rotate one or more of the treatment probe or the imaging probe.

[0211] In some embodiments, the first user instruction comprises an instruction to translate the ultrasound probe along a longitudinal axis of the ultrasound probe to align a marker corresponding to an end of the ultrasound probe overlaid on a longitudinal ultrasound image with a position of an energy source of the treatment probe shown in the longitudinal image in the image display area, for example as shown in FIG, 3B. The first instruction may comprise an instruction to provide an input to proceed to a next step, such as by pressing a next icon as described herein.

[0212] In some embodiments, the second user instructions comprise an instruction to rotate the imaging probe such as a TRUS probe, so that the handpiece is located vertically above the imaging probe shown in a transverse image as shown in FIG. 3C. In some embodiments, the second instruction comprises an instruction to activate the energy source such as a waterjet, and an instruction to rotate the energy source so that the direction of energy released from the energy source is pointed toward the imaging probe, for example to point vertically downward toward the imaging probe as shown in the transverse image of FIG. 3C.

[0213] At a display markers step 936, the user interface is configured to display one or more markers to aid in the alignment of the treatment probe and the imaging probe such as the TRUS probe. In some embodiments, the user interface is configured to display a flashing marker, such as a flashing colored marker, e.g. a flashing blue marker. In some embodiments, the marker is placed at a location corresponding to an end of the imaging probe. In some embodiments, the imaging probe is configured to translate along a longitudinal axis of the probe to bring the marker into alignment with the treatment probe. While the imaging probe can be translated in many ways, in some embodiments, a stage, such as stage with a stepper is moved with rotation of a knob as described herein. In some embodiments, the imaging probe is supported with a motorized stage, which allows the user to input one or more instructions to move the probe with the user interface, for example.

[0214] At a receive user input step 938, the user provides an input corresponding to one or more features of the user interface. In some embodiments, the input comprises an instruction to proceed to a next stage, e.g. to one or more of a next mode, a next sub-mode, or a next step, for example.

[0215] In some embodiments, upon completion of the setup mode 902, the user interface proceeds to the plan mode 904, which may comprise one or more of angle and depth mode 940, a registration mode 950, or a profile mode 960, for example.

[0216] In some embodiments, the angle and depth mode 940 comprises one or more of a user interface configuration step 942, a display primary and secondary images step 943, a display current mode step 944, a display instructions step 945, a display markers step 946, or a receive user input step 948,

[0217] At a user interface configuration step 942, one or more features of the user interface is displayed, such one or more of the user input features, the image display area, the primary image, the secondary image, the tool bar, the control panel, the displayed mode, the displayed sub-mode, the navigation bar as described herein, for example as described herein with reference to FIG. 3D. In some embodiments, the instructions executed by the one or more processors are configured to provide the one or more features of the configuration step 942 in response to a user input such as a user input selection indicating completion of a prior mode or sub-mode, e.g. the “next' icon of a prior mode or sub-mode such as mode or sub-mode 930. In some embodiments, the user interface comprises an input feature for a user to select an assisted planning mode. In some embodiments, the input is located in the control panel, for example.

[0218] Examples of tissue recognition algorithms suitable for use with assisted planning are described in PCT / US2019 / 038574, filed June 21, 2019, entitled “ARTIFICIAL INTELLIGENCE FOR ROBOTIC SURGERY”, published as WO / 2019 / 246580 on June 26, 2019, and U. S, App, No. 18 / 163,187, filed on February 1, 2023, “USER INTERFACE FOR THREE DIMENSIONAL IMAGING AND TREATMENT”.

[0219] At a display primary and secondary images step 943 of the angle and depth mode 940, the user interface is configured to automatically display a real-time image from an imaging probe in the image display area, such as an image from an ultrasound probe, for example the TRUS probe. In some embodiments, the user interface is configured to automatically display a real time image from an ultrasound probe in the primary image display area and to automatically display the real time endoscope image such as the cystoscope in the secondary image display area, for example with reference to FIG. 3D. In some embodiments, the real time ultrasound image comprises a transverse image in a primary image display area and the real time endoscope image is displayed in the secondary image display area. In some embodiments, a transverse image is automatically displayed in the primary display area, for example a real time transverse image from TRUS probe as described herein.

[0220] In some embodiments, the user interface comprises a plurality of inputs such as tabs for a user to select a transverse image to display. In some embodiments, the user interface is configured to highlight a selected tab among the plurality tabs, in which each of the plurality of tabs corresponds to a selected transverse image.

[0221] In some embodiments, the user interface is configured to automatically display a transverse image among a plurality of transverse images. In some embodiments, the user interface is configured to automatically display a transverse image such as a transverse image of a median lobe of a prostate, for example. In some embodiments, the automatically generated image comprise a different image, such as a mid-prostate, if a “no median lobe” input has been selected by the user.

[0222] In some embodiments, the user interface is configured to automatically display another transverse image for planning in response to the user selecting the next button, which can allow the user to sequentially plan the treatment profile and adjust the treatment markers on each of the plurality of transverse images. In some embodiments, each of the plurality of displayed transverse images is overlaid with one or more treatment markers, which can be adjusted with user input as described herein, for example with reference to FIG, 3D,

[0223] At a display current mode step 944, the user interface is configured to display the current plan mode 904 of the system and optionally the current sub-mode 940 of the system, such as the angle and depth mode 940. In some embodiments, the displayed mode 904 is highlighted with respect to other displayed modes as described herein, e.g, set up mode 902, and treatment mode 906. In some embodiments, the current sub-mode 940 is displayed with the current mode 904 as described herein. In some embodiments, the current sub-mode 940 is highlighted with respect to other sub-modes, such as registration sub-mode 950 and the profile sub-mode 960 that are displayed with the current mode 904, for example with reference to FIG. 3D.

[0224] At a display instructions step 945, user interface is configured to automatically display one or more angle and depth instructions to move one or more treatment markers overlaid on an image, and optionally to display the one or more instructions in a user input area of the control panel, such as between the tool bar and marker select inputs as described herein. In some embodiments, the instructions comprise instructions to adjust one or more of an angle of a treatment plan or a depth of a treatment plan, for example. In some embodiments, the user interface is configured toprovide one or more instructions to adjust the plurality of treatment markers to adjust one or more of an angle or a depth of treatment from the treatment probe, for example with reference to FIG, 3D,

[0225] At a display markers step 946, the user interface is configured to display one or more treatment markers, for example as shown with reference to FIG, 3D, In some embodiments, the user interface comprises one or more marker select inputs for a user to select a treatment marker to be adjusted as described herein. The angle and depth of a treatment marker can be adjusted by the user as described herein.

[0226] In some embodiments as part of the display markers step 946, the user interface is configured to display one or more treatment planning markers and an associated one or more treatment limit markers overlaid on one or more images, such as one or more transverse images. Although reference is made to one or more transverse images, the one or more treatment limit markers can be overlaid on longitudinal images as described herein. In some embodiments, the user interface is configured to provide feedback to the user in response to the one or more planning markers approaching the associated limit marker, the feedback comprising one or more of highlighting the marker, changing a color of the marker, flashing the marker, bracketing the marker, providing a pop-up window, or an auditory cue, for example. In some embodiments, the feedback can be provided to the selected treatment marker by limiting movement of the treatment marker beyond the limit marker. Alternatively or in combination, the treatment marker can be configured to bounce back to the limit marker when the treatment marker is moved by the user beyond the limit marker.

[0227] At a receive user input step 948, the user provides an input corresponding to one or more features of the user interface. The user input may comprise one or more of an input to select a treatment marker or to adjust one or more markers corresponding to an angle of treatment or a depth of treatment as shown on the one or more transverse images, for example as show with reference to FIG. 3D. In some embodiments, the input comprises an instruction to proceed to a next stage, e.g. to one or more of a next mode, a next sub-mode, or a next step, for example.

[0228] In some embodiments, the user interface proceeds to a registration mode 950 in response to the user selecting a next input.

[0229] In some embodiments, the registration mode 950 comprises one or more of a user interface configuration step 952, a display primary and secondary images step 953, a display current mode step 954, a display instructions step 955, a display markers step 956, or a receive user input step 958,

[0230] At user interface configuration step 952, one or more features of the user interface is displayed, such one or more of the user input features, the image display area, the primary image, the secondary image, the tool bar, the control panel, the displayed mode, the displayed sub-mode, the navigation bar as described herein, for example with reference to FIG. 3E. In some embodiments, the instructions executed by the one or more processors are configured to provide the one or more features of the configuration step 952 in response to a user input such as a user input selection indicating completion of a prior mode or sub-mode, e.g. the “next” icon of a prior mode or sub-mode such as mode or sub-mode 940. In some embodiments, the user interface comprises an input feature for a user to select an assisted planning mode. In some embodiments, the input is located in the control panel, for example.

[0231] At a display primary and secondary images step 953, the user interface is configured to automatically display a real-time image from a probe such as an ultrasound probe, for example the TRUS probe, in the image display area. In some embodiments, the user interface is configured to automatically display a real time image from anultrasound probe in the primary image display area and to automatically display the real time endoscope image such as the cystoscope in the secondary image display area, for example with reference to FIG. 3E, in some embodiments, the automatically displayed real time ultrasound image comprises a longitudinal image in a primary image display area and the real time endoscope image is displayed in the secondary image display area. In some embodiments, a longitudinal image is automatically displayed in the primary display area, for example a real time transverse image from TRUS probe as described herein.

[0232] At a display current mode step 954, the user interface is configured to display the current plan mode 904 of the system and optionally the current sub-mode 950 of the system, such as the registration mode 940. In some embodiments, the displayed mode 904 is highlighted with respect to other displayed modes as described herein, e.g. set up mode 902, and treatment mode 906. In some embodiments, the current sub-mode 950 is displayed with the current mode 904 as described herein. In some embodiments, the current sub-mode 950 is highlighted with respect to other sub-modes, such as angle and depth sub-mode 940 and the profile sub-mode 960 that are displayed with the current mode 902.

[0233] At a display instructions step 955, the user interface is configured to automatically display one or more registration instructions for the user to move one or more registration markers overlaid on an image, and optionally to display the one or more instructions in a user input area of the control panel, such as between the tool bar and navigation bar. In some embodiments, in the registration mode, the user interface is configured to provide instructions to input a first location corresponding to a location of the energy source in the longitudinal image and to input a second location corresponding to a second location along a path of the energy source in the longitudinal image, for example with a first movable marker and a second movable marker. In some embodiments, the instructions comprise instructions to move the imaging probe such as the TRUS probe to see a desired anatomical tissue structure such as the bladder in a longitudinal image. In some embodiments, the instructions comprise instructions to place a first marker at a position corresponding to a position of the energy source such as a nozzle of a waterjet, and a second marker at a position corresponding to a path of the energy source as described herein.

[0234] At a display markers step 956, the user interface is configured to display one or more markers, such a first marker showing a position of the energy source at a first location and second marker showing a path of the energy source at a second location. In some embodiments, the user interface is configured to display a user first user adjustable marker at the first location and a second user adjustable marker at a second location. In some embodiments, the user interface comprises one or more marker select inputs for a user to select the first marker or the second marker, which can be adjusted as described herein, e.g. with coarse or fine adjustment as described herein,

[0235] In some embodiments as part of the display markers step 956, the user interface is configured to display one or more treatment planning markers and an associated one or more treatment limit markers overlaid on one or more images, such as one or more longitudinal images. The associated one or more treatment limit markers may be displayed as a limit overlaid on the primary image, for example with reference to FIG. 3E. Although reference is made to one or more longitudinal images, the one or more treatment limit markers can be overlaid on transverse images as described herein.

[0236] In some embodiments, the associated one or more treatment limit markers is shown on a longitudinal image at a location corresponding to a maximum treatment depth of a corresponding transverse image. In someembodiments, the maximum treatment depth is located away from a plane of the longitudinal image on the corresponding transverse image in a transverse planning mode as described herein with reference to FIG 3D, in which the second treatment marker (treatment marker 2) may be located at a greater distance from the probe than other markers such as the third and fourth treatment markers (treatment markers 3, 4, respectively),

[0237] In some embodiments, the user interface is configured to provide feedback to the user in response to the one or more planning markers approaching the associated limit marker, the feedback comprising one or more of highlighting the marker, changing a color of the marker, flashing the marker, bracketing the marker, providing a pop¬ up window, or an auditory cue, for example. In some embodiments, the feedback can be provided to the selected treatment marker by limiting movement of the treatment marker beyond the limit marker. Alternatively or in combination, the treatment marker can be configured to bounce back to the limit marker when the treatment marker is moved by the user beyond the limit marker.

[0238] At a receive user input step 958, the user provides an input corresponding to one or more features of the user interface. The user input may comprise an input to place one or more markers, or an input to adjust the one or more markers corresponding to a location of the energy source and a path of the energy source shown on the one or more longitudinal images. In some embodiments, the input comprises an instruction to proceed to a next stage, e.g. to one or more of a next mode, a next sub-mode, or a next step, for example.

[0239] In some embodiments, the user interface proceeds to a profile mode 960 in response to the user selecting a next input.

[0240] In some embodiments, the profile mode 960 comprises one or more of user interface configuration step 962, a display primary and secondary images step 963, a display current mode step 964, a display instructions step 965, a display markers step 966, or a receive user input step 968.

[0241] At a user interface configuration step 962, one or more features of the user interface is displayed, such one or more of the user input features, the image display area, the primary image, the secondary image, the tool bar, the control panel, the displayed mode, the displayed sub-mode, or the navigation bar as described herein, for example with reference to FIG. 3E. In some embodiments, the instructions executed by the one or more processors are configured to provide the one or more features of the configuration step 962 in response to a user input such as a user input selection indicating completion of a prior mode or sub-mode, e.g. the “next” icon of a prior mode or sub-mode such as mode or sub-mode 950.

[0242] At a display primary and secondary images step 963, the user interface is configured to automatically display a real-time image from a probe such as an ultrasound probe, for example the TRUS probe, in the image display area. In some embodiments, the user interface is configured to automatically display a real time image from an ultrasound probe in the primary image display area and to automatically display the real time endoscope image such as the cystoscope in the secondary image display area, for example with reference to FIG. 3E. In some embodiments, wherein the user interface is configured to automatically display a longitudinal ultrasound image in the profile sub-mode 960. In some embodiments, the automatically displayed real time image comprises a longitudinal image in a primary image display area and the real time endoscope image is displayed in the secondary image display area. In some embodiments, a longitudinal image is automatically displayed in the primary display area, for example a real time longitudinal image from TRUS probe as described herein.

[0243] At a display current mode step 964, the user interface is configured to display the current plan mode 904 of the system and optionally the current sub-mode 960 of the system, such as the profile mode 960. In some embodiments, the displayed mode 904 is highlighted with respect to other displayed modes as described herein, e.g. set up mode 902, and treatment mode 906, In some embodiments, the current sub-mode 960 is displayed with the current mode 904 as described herein. In some embodiments, the current sub-mode 960 is highlighted with respect to other sub-modes, such as angle and depth mode 940, registration sub-mode 950 that are displayed with the current mode 904.

[0244] At a display user instructions step 965, the user interface is configured to automatically display instructions to move one or more treatment markers overlaid on an image, and optionally to display the one or more instructions in a user input area of the control panel, such as between the tool bar and marker select inputs as described herein. In some embodiments, the user interface is configured to automatically provide one or more instructions to adjust the plurality of user adjustable treatment markers so as to adjust a profile of the treatment along the longitudinal image.

[0245] At a display markers step 966, the user interface is configured to display one or more treatment markers, for example as shown with reference to FIG. 3E. In some embodiments, in the profile sub-mode 960, the user interface is configured to provide one or more longitudinal images in the image display area with a plurality of user adjustable treatment markers for the user to plan the treatment. In some embodiments, a location of each of the plurality of user adjustable treatment markers shown on the longitudinal image corresponds to a location of a treatment profile of a corresponding transverse image.

[0246] In some embodiments, the user interface comprises one or more marker select inputs for a user to select a treatment marker to be adjusted as described herein. In some embodiments, the one or more markers comprise markers corresponding to a bladder neck marker or a veru protection zone as described herein, for example.

[0247] At a receive user input step 968, the user provides an input corresponding to one or more features of the user interface. The user input may comprise an input to adjust one or more markers as shown on the one or more longitudinal images, for example as show with reference to FIG. 3E. In some embodiments, the input comprises an instruction such as a user input selection to proceed to a next stage, e.g. to one or more of a next mode, a next sub¬ mode, or a next step, for example.

[0248] In some embodiments, the input comprises an input to accept the treatment plan, with an associated message and instructions. For example, the input to accept the treatment plan can be located adjacent a message to accept the treatment plan. In some embodiments, the input to accept the treatment plan and the corresponding message are located in a control panel as described herein.

[0249] In some embodiments, upon completion of the profile mode 904, the user interface proceeds to the treatment mode 906 in response to a user input such as a selection indicating completion of the profile mode. In some embodiments, the treatment mode 906 may comprises a treatment sub-mode 990, and a prime sub-mode 980 configured to prime the treatment probe after the treatment has started if appropriate, for example.

[0250] In some embodiments, the treatment mode or sub-mode 990 comprises one or more of a user interface configuration step 992, a display primary and secondary images step 993, a display current mode step 994, a display instructions step 995, a display markers step 996, or a receive input step 998.

[0251] At a user interface configuration step 992, one or more features of the user interface is displayed, such one or more of the user input features, the image display area, the primary image, the secondary image, the tool bar, the control panel, the displayed mode, the displayed sub-mode, the navigation bar as described herein, for example with reference to FIG, 3F, In some embodiments, the instructions executed by the one or more processors are configured to provide the one or more features of the configuration step 992 in response to a user input such as a user input selection indicating completion of a prior mode or sub-mode, e.g, the “next” icon of a prior mode or sub-mode such as mode or sub-mode 980. In some embodiments, a lock or other icon is displayed to indicate that the treatment plan has been accepted and locked subsequent to user acceptance.

[0252] In some embodiments, the user interface is configured to display one or more treatment parameters in the treatment mode, the one or more parameters comprising one or more of an elapsed treatment time, a treatment time remaining, a power level of the energy source, a sweep angle of the energy source, or an amount of progress of the treatment.

[0253] At a display primary and secondary images step 993, the user interface is configured to automatically display a real-time image from a probe such as an ultrasound probe, for example the TRUS probe, in the image display area. In some embodiments, the user interface is configured to automatically display a real time image from an ultrasound probe in the primary image display area and to automatically display the real time endoscope image such as the cystoscope in the secondary image display area, for example with reference to FIG. 3F. In some embodiments, the real time image comprises a longitudinal ultrasound image in a primary image display area and the real time endoscope image is displayed in the secondary image display area. In some embodiments, a longitudinal image is automatically displayed in the primary display area, for example a real time longitudinal image from TRUS probe as described herein.

[0254] At a display current mode step 994, the user interface is configured to display the current treat mode 906 of the system and optionally the current sub-mode 990 of the system, such as the treatment mode 990. In some embodiments, the displayed mode 906 is highlighted with respect to other displayed modes as described herein, e.g. set up mode 902, and plan mode 904, In some embodiments, the current sub-mode 990 is displayed with the current mode 906 as described herein. In some embodiments, the current sub-mode 990 is highlighted with respect to other sub-modes. Alternatively, the current sub-mode can be displayed without other sub-modes, for example as shown with reference to FIG. 3F.

[0255] At a display instructions step 995, the user interface is configured to automatically display instructions to provide one or more inputs related to the treatment of the patient, and optionally to display the one or more instructions in a user input area of the control panel, such as between the tool bar and the navigation as described herein with reference to FIG, 3F. The displayed instructions may comprise an instruction to press a foot pedal to begin the treatment, for example. The displayed instruction may comprise an instruction to lift the foot pedal to pause the treatment, for example.

[0256] At a display markers step 996, the user interface is configured to display one or more markers related to treatment, for example one or more markers as shown with reference to FIG. 3F. In some embodiments, one or more markers comprises a marker related to a longitudinal position of an energy source such as a water jet nozzle, for example.

[0257] At a receive input step 998, the user provides an input corresponding to one or more features of the user interface. The user input may comprise an input to adjust a position of an energy source or an amount of power from the energy source, such as a flow rate from a nozzle, for example. In some embodiments, one or more of the inputs can also be provided the handpiece as described herein,

[0258] In some embodiments, the user interface is configured to provide instructions to the user to prime the energy source, such as a waterjet or a laser, for example. In some embodiments, the energy source is primed outside the patient and prior to insertion of the energy source into the patient. In some instances, a source of a fluid such as saline may not be sufficient to complete a treatment, and it can be helpful to prime the energy source such as a water jet a second time. In such situations, it can be helpful to leave the probe inside the patient when the energy source is primed a second time, for example so as to maintain longitudinal placement of the energy source when the treatment is paused.

[0259] In some embodiments, the user instructions are configured to display an instruction to prime an energy source prior to providing an instruction to insert a treatment probe comprising the energy source into a patient.

[0260] In some embodiments, the processor instructions are configured to reduce an amount of energy from the energy source if the system has entered a treatment mode or sub-mode and the energy source is primed while within a patient.

[0261] In some embodiments, the processor instructions are configured to automatically orient the energy source toward a portion of the probe if the system has entered a treatment mode or sub-mode and the energy source is primed while within the patient.

[0262] In some embodiments, the prime mode or sub-mode 980 comprises one or more of a user interface configuration step 982, a display primary and secondary images step 983, a display current mode step 984, a display instructions step 985, a display markers step 986, or a receive input step 988, which can be similar to the treatment sub-mode 990.

[0263] In some embodiments, the processor instructions are configured to automatically orient the energy source toward a portion of the probe if the system has entered a treatment mode or sub-mode and the user interface enters the prime mode or sub-mode 980 after the user interface has entered the treatment mode or sub-mode 990, In some embodiments, the processor instructions are configured to automatically decrease the amount of energy of the energy source to an amount less than the amount used during the treatment if the system has entered a treatment mode or sub-mode and the user interface enters the prime mode or sub-mode 980 after the user interface has entered the treatment mode or sub-mode 990,

[0264] At a user interface configuration step 982, one or more features of the user interface is displayed, such one or more of the user input features, the image display area, the primary image, the secondary image, the tool bar, the control panel, the displayed mode, the displayed sub-mode, the navigation bar as described herein, for example with reference to FIG. 3F. in some embodiments, the instructions executed by the one or more processors are configured to provide the one or more features of the configuration step 982 in response to a user input such as a user input selection indicating completion of a prior mode or sub-mode, e.g. the “next" icon as described herein. In some embodiments, a lock or other icon is displayed to indicate that the treatment plan has been accepted and locked while the energy source is primed.

[0265] In some embodiments, the user interface is configured to display one or more energy source priming parameters in the prime mode, the one or more parameters comprising one or more of a power level of the energy source, an angle of the energy source, or an amount of progress of the treatment when the treatment was paused, and priming mode initiated.

[0266] At a display primary and secondary images step 983, the user interface is configured to automatically display a real-time image from a probe such as an ultrasound probe, for example the TRUS probe, in the image display area. In some embodiments, the user interface is configured to automatically display a real time image from an ultrasound probe in the primary image display area and to automatically display the real time endoscope image such as the cystoscope in the secondary image display area, for example with reference to FIG. 3E. In some embodiments, the real time image comprises a longitudinal image in the primary image display area and the real time endoscope image is displayed in the secondary image display area. In some embodiments, a longitudinal image is automatically displayed in the primary display area, for example a real time longitudinal image from TRUS probe as described herein.

[0267] At a display current mode step 984, the user interface is configured to display the current treat mode 906 of the system and the current prime sub-mode 980 of the system. In some embodiments, the displayed mode 906 is highlighted with respect to other displayed modes as described herein, e.g. set up mode 902, and plan mode 904. In some embodiments, the current sub-mode 980 is displayed with the current treat mode 906 as described herein. In some embodiments, the current sub-mode 980 is highlighted with respect to other sub-modes, e.g. the treatment submode 990. Alternatively, the current sub-mode can be displayed without other sub-modes, for example as shown with reference to FIG. 3F.

[0268] At a display instructions step 985, the user interface is configured to automatically display instructions to provide one or more inputs related to the treatment of the patient, and optionally to display the one or more instructions in a user input area of the control panel, such as between the tool bar and the navigation as described herein with reference to FIG. 3F. The displayed instructions may comprise an instruction to press a foot pedal to begin the priming of the energy source, for example. The displayed instruction may comprise an instruction to lift the foot pedal when priming of the energy source has been completed, for example.

[0269] At a display markers step 986, the user interface is configured to display one or more markers related to treatment, for example one or more markers as shown with reference to FIG. 3F, In some embodiments, one or more markers comprises a marker related to a longitudinal position of an energy source such as a waterjet nozzle during the priming of the energy source, for example.

[0270] At a receive input step 988, the user provides an input corresponding to one or more features of the user interface. The user input may comprise an input to adjust a rotational angle of an energy source or an amount of power from the energy source, such as a flow rate from a nozzle, for example. In some embodiments, the one or more of the inputs can also be provided with the handpiece as described herein. In some embodiments, the user interface is configured for the user to adjust a rotational orientation of the energy source to direct energy from the energy source such as a waterjet toward a portion of the probe, e.g. a stiff portion of the probe, to reduce likelihood of tissue being treated, e.g. resected, during the prime mode.

[0271] In some embodiments, the user interface is configured to automatically select an image source for an image to display in the image display area, the image source and the image corresponding to the current mode and acurrent sub-mode of the system, in some embodiments, this is performed each of a plurality of the display first and second images steps, such as two or more of steps 913, 923, 933, 943, 953, 963, 983, or 993, for example. In some embodiments, the user interface is configured to automatically select one or more of an endoscope image, an ultrasound image, a transverse ultrasound image or a longitudinal image to display in the image display area. In some embodiments, the user interface is configured to automatically display a primary image from a first image source and a secondary image from a secondary image source in the image display area, in some embodiments, the primary image source and the secondary image source correspond to the current mode and a current sub mode of the mode as described herein.

[0272] In some embodiments, the user interface is configured to automatically display a real time endoscope image from an endoscope in the primary image display area and a real time ultrasound image from an ultrasound device in the secondary image display area in a first mode, e.g. step 923 of treatment probe insertion mode 920, and to automatically display the real time ultrasound image in the primary image display area and the endoscope image in the secondary image display area in a second mode, for example with one or more of steps 933, 943, 953, 963, 983 or 993 of corresponding modes or sub-modes 930, 940, 950, 980 or 990, respectively, for example.

[0273] In some embodiments, the second mode the user interface is configured to automatically display a real time transverse ultrasound image in a first sub-mode 940 of the second mode and to automatically display a real time longitudinal ultrasound image in a second sub-mode 960 of the second mode, e.g. the plan mode 904.

[0274] In some embodiments, the user interface is configured to toggle between the real time transverse ultrasound image and the real time longitudinal ultrasound image in the first sub-mode and between the real time longitudinal ultrasound image and the real time transverse image in the second sub-mode of the second mode.

[0275] In some embodiments, the real time transverse ultrasound image and the real time longitudinal ultrasound image shown in the image display area remain partitioned from the control panel when toggled, e.g. above the control panel when toggled.

[0276] In some embodiments, wherein the user interface is configured to highlight a current mode of the plurality of modes, for example with reference one or more the display current mode steps 914, 924, 934, 944, 954, 964, 984 or 994.

[0277] In some embodiments, the user interface is configured to highlight each of the plurality of modes in a sequence. In some embodiments, the sequence corresponds to an order in which each of the modes and sub-modes are provided. In some embodiments each of the plurality of modes is highlighted in a predetermined sequence corresponding to a current mode of operation of the system, in some embodiments, the user interface is configured to sequentially highlight said each of the plurality of modes in a left to right order on the display, for example.

[0278] In some embodiments, an order the plurality of modes shown on the display remains fixed. In some embodiments, a location of each of the plurality of modes remains substantially fixed, for example at a location within a fixed area of the display. In some embodiments, each of the displayed modes 902, 904 and 906 can remain at the same location on the display while the highlighting changes to indicate which of the plurality of modes comprises the current mode, for example with reference to FIGS. 3A to 3F. In some embodiments, the user interface is configured to display the plurality of modes on a navigation bar, for example. In some embodiments, the plurality of modes is displayed between a previous icon and a next icon of the navigation bar.

[0279] In some embodiments, the user interface is configured to display the plurality of modes at a fixed area of the display, such as a fixed area between a previous icon and a next icon of a user interface. In some embodiments, the fixed area comprises a first fixed area to display the plurality of modes such as modes 902, 904 and 906, and a second fixed area to display a plurality of sub-modes adjacent the first fixed area, such as sub-modes 910, 920, 930, 940, 950, 960, 980 and 990, for example. In some embodiments, the interface is configured to display one or more sub-modes of the plurality of sub-modes at the second fixed area of the display when the displayed one or more sub¬ modes is associated with the current mode highlighted in the fixed area.

[0280] In some embodiments, each the plurality of modes 902, 904, 906 comprises one or more sub-modes, such as two or more of sub-modes 910, 920, 930, 940, 950, 960, 980 or 990. In some embodiments, each of the one or more sub-modes is shown on the display when the one or more sub-modes is associated with the current mode highlighted on the display, for example with reference to FIGS. 3A to 3F. In some embodiments, the one or more sub¬ modes shown on the display is highlighted with the current mode highlighted on the display. In some embodiments, the one or more sub-modes comprises a plurality of sub-modes associated with the current mode highlighted on the display. In some embodiments, the plurality of sub-modes associated with the current highlighted mode shown on the display is arranged in a sequence and sequentially highlighted. In some embodiments, the plurality of sub-modes is sequentially highlighted in a left to right sequence, for example with reference to FIGS. 3A to 3C and FIGS. 3D and 3E. In some embodiments, the one or more sub-modes comprises a plurality of sub-modes located on a navigation bar adjacent the plurality of modes. In some embodiments, the one or more sub-modes comprises a plurality of sub-modes located between a previous icon and a next icon.

[0281] In some embodiments, the user interface is configured to automatically select the primary image source displayed in the primary image display area and the secondary image source displayed in the secondary image display area so as to correspond with a current mode or sub-mode of the user interface. This automatic selection of the primary image source and the secondary image source can make it easier for the user to proceed through the modes and submodes and can improve safety of the procedure. In some embodiments, the primary image source comprises the ultrasound image source such as an ultrasound probe and the secondary image source comprises the endoscope image source such as an endoscope on a treatment probe, which can improve safety of the procedure because the user can view the endoscope image in the secondary display area while using the ultrasound image as the primary image, for example while aligning the probes, planning the angle and depth, registration of the treatment profile and probe, planning the treatment profile and treating the patient as described herein.

[0282] In some embodiments, the instructions executed by the one or more processors are configured to select in sequence: 1) an ultrasound image source as the primary image source for the primary image display area in a first mode or sub-mode to insert an ultrasound probe into a patient, such as during imaging probe insertion mode 910; 2) an endoscope image source as the primary image source for the primary display area and the ultrasound image as the secondary image source for the secondary image display area in a second mode or sub-mode to insert a treatment probe comprising an endoscope into the patient, such as treatment probe insertion mode 920: and 3) the ultrasound image source as the primary image source for the primary image display area and the endoscope image source as the secondary image source for the secondary image display area for each of a plurality of subsequent modes or sub¬ modes, such as two or more of probe alignment mode or sub-mode 930, angle and depth mode or sub-mode 940,regi stration mode or sub-mode 950, profile mode or sub-mode 960, prime mode or sub-mode 980 or treatment mode or sub-mode 990, In some embodiments, the plurality of subsequent modes or sub-modes comprises a planning mode or sub-mode 904 and a treatment mode or sub-mode 906, for example. In some embodiments, the plurality of subsequent modes or sub-modes comprises three or more of a probe alignment mode or sub-mode 930, an angle and depth mode or sub-mode 940, a registration mode or sub-mode 950, a profile mode or sub-mode 960, a prime mode 980, or a treatment mode or sub-mode, 990, for example.

[0283] In some embodiments, the user interface is configured to select an ultrasound image source as the primary image source in a mode corresponding to insertion of an ultrasound probe into the patient,.e.g. mode 910, and an endoscope image source for the primary image display area in a mode corresponding to insertion of a surgical treatment probe into the patient, e.g. mode 920. In some embodiments, the processor is configured to select the ultrasound probe as the primary image source and the endoscope as the secondary image source for a plurality of modes subsequent to the mode corresponding to insertion of the treatment probe into the patient, such as two or more of probe alignment mode or sub-mode 930, angle and depth mode or sub-mode 940, registration mode or sub-mode 950, profile mode or sub-mode 960, prime mode or sub-mode 980 or treatment mode or sub-mode 990. In some embodiments, the plurality of modes subsequent to the mode of inserting the treatment probe comprises a planning mode 904, and a treatment mode 906, for example. In some embodiments, the plurality of modes subsequent to the mode of inserting the treatment probe comprises two or more of an alignment mode or sub-mode 930, an angle and depth mode or sub-mode 940, a registration mode or sub-mode 940, a profile mode or sub-mode 960, a prime mode or sub-mode 980, or a treatment mode or sub-mode 990. In some embodiments, the plurality of modes subsequent to inserting the treatment probe comprises at least four of the modes or sub-modes in which the primary image source comprises the ultrasound probe and the secondary image source comprises the endoscope.

[0284] Although FIG. 9 shows a user interface method 900 in accordance with some embodiments, one of ordinary skill in the art will recognize many adaptations and variations. For example, the steps, modes and sub-modes can be performed in a sequence as shown, or in a different order. Some of the steps, modes and sub-modes can be omitted and some of the steps repeated, for example.

[0285] Although the present disclosure makes reference to modes and sub-modes, any of the sub-modes described herein may comprise a mode, which can be performed independently of other modes and sub modes,

[0286] FIG. 10 shows a method 1000 of assisted tissue planning. The method 1000 can be combined with the user interface and method 900 as described herein.

[0287] Ata step 1010, an assisted planning user input feature is displayed. The user input feature may comprise any suitable user input as described herein. In some embodiments, the input feature comprises one or more of a label, an icon, a toggle, a button, or a slider configured to receive the user input. In some embodiments, the user input comprises a toggle, for example as described with reference to FIGS. 3D and 3D. While the input feature to turn on assisted planning can be located anywhere on the display, in some embodiments the input is located on the control panel.

[0288] In some embodiments, the user interface is configured to present the assisted planning input in each of a plurality of transverse views to allow the user to select assisted planning in said each of the plurality of transverse views.

[0289] At a step 1025, a user input is received to activate assisted planning. In some embodiments, the input is configured to default to a setting from a previously completed treatment from a last treatment session of the system. For example, the input can default to assisted planning turned on if assisted planning was used for the last treatment session, or to default to off if assisted planning was not used during the last treatment session.

[0290] At a step 1030, assisted planning is activated in response to the user input. The activation of the assisted planning can trigger an image recognition process to be activated, such as an artificial intelligence algorithm configured to identify tissue structures from an image, such as a neural network, a convolutional neural network, or a machine learning algorithm, for example. Examples of suitable image recognition processes are described in PCT / US2019 / 038574, filed June 21, 2019, entitled “ARTIFICIAL INTELLIGENCE FOR ROBOTIC SURGERY”, published as WO / 2019 / 246580 on June 26, 2019, and U. S. App. No. 18 / 163,187, filed on February 1, 2023, “USER INTERFACE FOR THREE DIMENSIONAL IMAGING AND TREATMENT”, the entire disclosures of which have been previously incorporated herein by reference.

[0291] At a step 1035, an assisted planning message is activated. The message may comprise any message indicated that an assisted planning process has been activated, such as a message that image recognition or tissue recognition is in process, for example. In some embodiments, a user input is provided to cancel assisted planning.

[0292] At a step 1040, a tissue structure is identified. The identified tissue structure may comprise any suitable tissue structure, such as a tissue boundary, a boundary of an organ such as the prostate, a capsule of an organ, a capsule of a prostate, a verumontanum of a prostate, a mid-prostate, a median lobe of a prostate, or a bladder neck, for example. In some embodiments in the assisted planning mode, the instructions of an artificial intelligence (Al) algorithm such as a machine learning model, which can be a neural network, are configured to identify one or more anatomical tissue structures from one or more of a longitudinal image or a transverse image and to mark a location of the one or more tissue structure on the one or more of the longitudinal image or the transverse image.

[0293] In some embodiments, an alert is provided to the user if the boundary is not identified from the processed image, such as a transverse image or a longitudinal image, for example.

[0294] At a step 1050, at a step one or more tissue markers is overlaid on one or more images. The one or more markers may comprise a marker along a boundary of a tissue, such as a boundary of a prostate capsule, for example. In some embodiments, the marker comprises a color, change in contrast or other indicia such as a line to indicate the tissue structure. In some embodiments, the user interface is configured to mark a boundary of a tissue in a transverse image, for example.

[0295] At a step 1060, one or more assisted planning treatment markers is overlaid on one or more images. The assisted planning markers may comprise treatment markers overlaid on one or more transverse or longitudinal images, for example with reference treatment to FIGS. 3D and 3E. In some embodiments, processor instructions are configured to determine a treatment start location and a treatment end location and display a treatment start marker and a treatment end marker on the longitudinal image, for example as described herein with reference to FIG. 3E.

[0296] In some embodiments, the user interface is configured to display a plurality of treatment markers of a treatment plan overlaid on the image in response to a tissue structure such as a boundary. In some embodiments, the user interface is configured to display a treatment profile with the treatment markers, for example. In some embodiments, the treatment profile is determined in response to the tissue boundary.

[0297] In some embodiments., the processor instructions of the tissue recognition algorithm such as a machine learning model, which can be a neural network, are configured to determine a plurality of locations of a plurality of treatment markers and display the plurality of treatment markers at the plurality of locations of a longitudinal image.

[0298] At a step 1065, one or more assisted planning treatment markers is adjusted. The one or more assisted planning treatment markers can be adjusted as described herein, for example with reference to FIGS. 3D and 3E. In some embodiments, in the assisted planning mode, the user interface is configured for the user to adjust the plurality of markers of the treatment plan overlaid on the image, for example.

[0299] At a step 1070, one or more treatment planning markers placed by the user is overwritten. In some embodiments, a user may start to plan a treatment and then decide to use assisted planning part way through the planning process. The treatment markers placed by the user can be overwritten with the assisted planning markers in response to the user selecting assisted planning, in some embodiments, the locations of the assisted planning markers placed by the user are overwritten with new locations determined with the assisted planning software. In some embodiments, the user interface is configured to overwrite one or more treatment planning markers overlaid on an image at one or more locations in response to a first user input with a plurality of assisted planning treatment markers in response to a second user input to select assisted planning after the user has positioned the one or more treatment planning markers.

[0300] At a step 1075, one or more displayed treatment markers reverts to user determined markers. In some embodiments, the user interface comprises an input to undo the assisted planning markers and revert to the one or more markers overlaid on the image at the one or more locations in response to the first user input. In some embodiments, the user may not be satisfied with the results of the assisted planning. In response to the user cancelling or otherwise turning off the assisted planning, for example with an input to a toggle, the one or more displayed treatment markers reverts to user determined markers. In some embodiments, the locations of the treatment markers are reverted to the locations of the user placed treatment markers. For example, the markers may comprise the same markers and only the locations of the markers reverts back to the user defined locations.

[0301] At a step 1080, one or more treatment markers is adjusted with user input as described herein,

[0302] At a step 1090, the treatment plan is accepted by the user. In some embodiments the treatment plan is accepted with an acknowledgement that assisted planning was used to plan the treatment. In some embodiments, the user interface is configured to display a notification that assisted planning was used with the input to accept the treatment plan upon completion of a planning mode 904 and prior to the treatment mode 906 as described with reference to FIG. 9.

[0303] Although FIG, 10 shows the method 1000 of assisted planning in accordance with some embodiments, one of ordinary skill in the art will recognize many adaptations and variations. For example, the steps, modes and sub¬ modes can be performed in a sequence as shown, or in a different order. Some of the steps, modes and sub-modes can be omitted and some of the steps repeated, for example.

[0304] FIG. 11 shows an imaging probe 160 and a treatment probe 150 skewed in relation to each other, from a surgeon’s perspective. In some embodiments, a skew angle 1110 extends between a longitudinal axis 1150 of the treatment probe 150 and a longitudinal axis 1160 of the imaging probe 160. During set up of the treatment, these probes can be inserted into the patient into an initial configuration, and in some instances, it may be helpful to bringthe probes into better alignment with each other, for example to decrease skew angle 1110, The imaging probe 160 may comprise an ultrasound imaging probe such as a TRUS probe, in which longitudinal images are generated along a plane within a field of view as described herein. In some embodiments, it may be helpful to align treatment probe 150 with imaging probe 160 so that the treatment probe remains visible along a longitudinal image such as a sagittal image from a TRUS probe, for example by decreasing skew angle 1110. In some embodiments, the treatment probe remaining within a longitudinal image of the imaging probe is referred to as a co-planar configuration. In other words, an axis of the treatment probe that extends substantially along a plane corresponding to a longitudinal image of the imaging probe and that remains within the field of view of the imaging probe in the longitudinal image may be referred to as a co-planar configuration.

[0305] In some embodiments, the imaging probe 160, such as a TRUS probe is operatively coupled to a processor with one or more motors configured to rotate and translate the imaging probe 160. In some embodiments, the processor is configured to rotate the TRUS probe around the longitudinal axis 1160 to generate a plurality of longitudinal images and generate a 3D ultrasound image from the plurality of longitudinal ultrasound images. In some embodiments, the processor is configured to rotate the TRUS probe in increments within a range from about 0.5 degrees to about 5 degrees to generate the 3D image, for example. Alternatively or in combination, the processor can be configured to move the imaging probe 160 to a plurality of longitudinal locations to generate a plurality of longitudinal images, which are used to generate the 3D image data, such as a 3D image. Work in relation to the present disclosure suggests that in some embodiments rotation of an imaging probe such as a TRUS probe about the elongate axis to generate the 3D image may result in less tissue movement than other types of movement, such as translation.

[0306] FIG. 12A shows a first location 1212 of a treatment probe in a first transverse image 1210 from an image section, e.g. a slice or a plane, of a 3D image of tissue an organ, such as a prostate 1205. The first location 1212 of the treatment probe may be identified by a trained Al algorithm such as a trained machine learning model, which can be a neural network, for example with a first output as described herein, for example comprising one or more one or more of a bounding box, a contour, a key point, a segmentation or a classification.

[0307] FIG. 12B shows a second location 1222 of the treatment probe of FIG, 12A in a second transverse image 1220 from the 3D image. The second location 1222 of the treatment probe may be identified by the trained Al algorithm and output, for example with a second output as described herein, for example comprising one or more of a bounding box, a contour, a key point, a segmentation or a classification.

[0308] FIG. 120 shows a third location 1232 of the treatment probe of FIGS, 12A and 12B in a third transverse image 1230 from the 3D image. The third location 1232 of the treatment probe may be identified by the Al algorithm and output, for example with the third output as described herein, for example comprising one or more of a bounding box, a contour, a key point, a segmentation or a classification.

[0309] Although reference is made to discrete image sections, such as slices, in some embodiments, the trained Al algorithm is trained to analyze a volume of the 3D image data corresponding to different image planes and output the one or more of a bounding box, a contour, a key point, a segmentation or a classification in a 3D format.

[0310] In some embodiments, an Al algorithm is trained on a plurality of images to identify locations of the probe with the 3D image data and output the one or more of a bounding box, a contour, a key point, a segmentation or a classification, for example with 3D image data as described herein.

[0311] As will appreciated with reference to FIGS, 12A to 12C, the location of the probe in the transverse images changes, and the processor can be configured with instructions to assist the surgeon to bring the probes into alignment. In some embodiments, the processor is configured to determine an angle of the treatment probe in relation to the treatment probe in response to the location of the probe in the plurality of transverse images, for example transverse images from the 3D image. In some embodiments, the angle comprises a 3D vector indicating alignment between the treatment probe and the imaging probe. The processor can be configured to determine the extent to which the treatment probe is aligned with a plane of a longitudinal image of the treatment probe, for example coplanarity. Alternatively or in combination, the processor can be configured to determine an angle of inclination of the treatment probe along a longitudinal image from the 3D image. These angles can be used to determine output to the user interface to instruct the user.

[0312] For example, with reference to FIG. 11, the user interface can be configured to provide an instruction to move the proximal portion of the probe comprising the handpiece to the left.

[0313] In some embodiments, the user interface is configured to receive an input to “review coplanarity”. In response to the input, the processor is configured to review the probe location in a plurality of transverse images.

[0314] If assistance is requested, e.g. in response to a user input, a 3D scan triggered, and the system will identify the probe position and the anatomy position in the 3D volume from the ultrasound coordinates. In some embodiments, the processor is configured to detect a lack of coplanarity, for example in the plurality of transverse images, e.g. slices, the treatment probe 150 of the handpiece position moves laterally through a series of transverse images, e.g. slices, with the yaw.

[0315] In some embodiments, The user interface is configured to recommend adjustments to one or more of the ultrasound imaging probe, such as the TRUS probe, or to the treatment probe such as a treatment probe of the handpiece to provide substantial coplanarity. In some instances, an animation of the recommended movement may be provided.

[0316] FIG. 13A shows an output 1320 of an Al analysis of alignment between a treatment probe and an imaging probe. The output may be provided on the user interface as describe herein. Alternatively or in combination, the output may comprise one or more of a bounding box, a contour, a key point, a segmentation or a classification, for example shown on the display. The output may comprise instructions to move a proximal portion of a treatment probe, for example a handpiece. While the instructions can be configured in many ways, the instructions can be configured to provide an instruction to move the handpiece left or right, or up or down, for example.

[0317] FIG. 13B shows a graphical instruction 1330 to move a treatment probe into alignment with an imaging probe. The graphical instruction may comprise a representation 1350 of the treatment probe and a graphical representation 1360 of the imaging probe, at an angle to each other as described herein, with a visual indicator 1332 such as an arrow or animation indicating a direction to move a proximal portion of the treatment probe, for example. Alternatively or in combination, the graphical instruction 1330 may comprise a visual indicator to move the imaging probe, for example.

[0318] In some embodiments, the processor is configured to determine a lateral angle between the treatment probe and the imaging probe, for example a skew angle 1110 as shown in FIG. 11, and to output the visual indicatorwith a corresponding angle shown between the representation of the treatment probe and the representation of the imaging probe on the visual indicator.

[0319] This alignment of the treatment and imaging probes may comprise a sequential activity, for example when one or more of the treatment probe or the imaging probe is moved and another set of 3D image data is obtained and the Al analysis of alignment repeated.

[0320] Referring again to FIGS. 3A to 3F, the alignment can be performed with the user interface and appropriate user prompts as described herein, and may be incorporated into the user interface method of FIG. 9 and the method of assisted treatment planning of FIG. 10, for example.

[0321] FIG. 14A shows a first transverse image 1410 with a first hyperechoic shadow 1412 from a treatment probe with the imaging probe 160 in a first rotational configuration. In the first rotational configuration, a central portion the ultrasound transducer array 1460 faces slightly away from the treatment probe as described herein, and the hyperechoic shadow 1412 of the treatment probe appears offset, e.g. displaced laterally, with respect to a location 1430 corresponding to a longitudinal axis of the imaging probe 160. In some embodiments, a vertical centerline along the transverse ultrasound image corresponds to a rotational angle of the longitudinal axis of the imaging probe beneath the centerline of the image.

[0322] FIG. 14B shows the imaging probe 160 and the transducer array 1460 in the first configuration at a first rotational angle to generate an image as in FIG. 14A. The imaging probe 160 comprises a longitudinal axis 1464. In some embodiments, a rotational motor 1470 and encoders 1472 are coupled to the ultrasound transducer array to rotate the ultrasound transducer array around the longitudinal axis as indicated by arrow 1480. In some embodiments, these motors and encoders are coupled to a processor to control rotation and translation of the imaging probe such as the TRUS probe and may comprise components of system 100.

[0323] FIG. 14C shows a second transverse image 1420 and a second location of the hyperechoic shadow 1410 from the treatment probe with the imaging probe in a second rotational configuration. A treatment energy 306 from an energy source, such as a waterjet 1490 is visible in the image.

[0324] FIG. 14D shows the imaging probe in the second configuration at a rotational angle to generate an image as in FIG. 1 C, in which the transducer array 1462 is aligned with the treatment probe,

[0325] In some embodiments, the processor is configured to determine a rotational angle of the imaging probe 160 with respect to the treatment probe and to rotate the imaging probe to align the transducer array 1462 of the imaging probe with the treatment probe, e.g. the hyperechoic shadow of the treatment probe. In some embodiments, the hyperechoic shadow 1462 appears vertically above a location 1430 of the ultrasound image corresponding to a longitudinal axis of the imaging probe,

[0326] In some embodiments, the trained Al algorithm such as a machine learning model, which can be a trained neural network is configured to output data corresponding to the location of the probe and the energy source as described herein. In some embodiments, the trained Al algorithm outputs one or more of a bounding box, a contour, a key point, a segmentation, or a classification, for example shown on the display. In some embodiments, the output corresponds to a first location of the probe 1412 in the first image 1410 and the second location of the probe 1412 in the second image.

[0327] In some embodiments, the processor is configured with instructions to release an energy source as described herein, such as a laser or a waterjet 1490 to treat the tissue. The energy source can be imaged to determine a rotational angle of the energy source in relation to the imaging probe as described herein. In some embodiments, the energy source releases an energy stream from the energy source, a waterjet from the energy source, or light from a laser energy source. Although reference is made to propagating energy, the rotation angle can be measured for any structure associated with an energy source, such as an angle of an electrode, for example.

[0328] FIG. 15 shows alignment of an energy source on a treatment probe 150 with a treatment plan in response to image data. The energy source on treatment probe 150 may comprise a directional energy source configured to direct treatment energy 306 at an angle, which may comprise any suitable energy from any energy source as described herein. In some embodiments, the motors and control circuitry are configured to rotate the energy source on treatment probe 150 around the longitudinal axis of the treatment probe to a plurality of angles in accordance with a treatment plan as described herein. The treatment probe may comprise a maximum sweep angle 1510 of the rotation of the energy source around the longitudinal axis of the treatment probe. The treatment plan such as a treatment profile may comprise a maximum sweep angle 1520, for example. In some embodiments, the maximum sweep angle 1510 of the energy source on the probe 150 is greater than the maximum sweep angle 1520 of the treatment plan, such that the treatment probe comprises a first extra sweep range 1512 beyond the maximum sweep angle 1520 of the treatment plan on a first side of the treatment plan and a second extra sweep range 1514 on a second side of the treatment plan. In some instances, the processor is programmed to release treatment energy 306 from the energy source at an intended angle 1530, and the energy from the energy source appears in the image at different angle 1540, with a difference 1550 between the intended angle 1530 and the measured angle 1540.

[0329] In some embodiments, the treatment probe is configured to rotate the energy source around the longitudinal axis of the treatment probe so as to sweep an angle of the energy source over a maximum sweep angle 1510, e.g. 270 degrees. In some embodiments, the processor is configured to rotationally offset the angle of the energy source such as the waterjet, in response to an angle of the energy source in a transverse image, for example with reference to FIG. 14C.

[0330] In some embodiments, the treatment probe 150 comprises motors and circuitry such as encoders housed within a handpiece to rotate and the energy source in accordance with a treatment plan, and these may comprise components of system 100 as described herein.

[0331] In some embodiments, the angle 1540 at which the energy 306 is released from the energy source comprises an angle that is slightly different from an intended angle 1530, such as 180 degrees, e.g, vertically downward, the processor can be configured with instructions to determine the angular difference 1550 and make adjustments to the treatment plan accordingly. This angular difference may occur if the handpiece comprising the motors, encoders and control circuitry is rotated at an angle when placed, or the treatment probe is offset laterally from the imaging probe and the imaging probe is rotated to align the treatment probe with a central portion of an ultrasound image, e.g. along a center line of an ultrasound image. Work in relation to the present disclosure suggests that in some instances, this difference may be related to manufacturing tolerances.

[0332] In some embodiments, a trained Al algorithm, such as a machine learning model, which can be a trained neural network, is configured to determine the angle 1540 at which energy is released from the energy source. Insome embodiments, the output of the neural network (or another machine learning model) comprises one or more one or more of a bounding box, a contour, a key point, a segmentation or a classification. The output can be used to determine the angle and may be provided on a user interface as described herein.

[0333] In some embodiments, an Al algorithm is trained on a plurality of images to identify the angle with image data and output the one or more of a bounding box, a contour, a key point, a segmentation or a classification, for example with 3D image data as described herein,

[0334] In some embodiments, the user interface is configured to receive an input confirming that the angle is acceptable. In some embodiments, the user interface shows output on the image corresponding to output of the trained neural network (or another Al algorithm), an arrow or other indicia indicating the angle for the user to confirm that output angle determined by the trained neural network.

[0335] Although reference is made to 3D images and 3D image data, such as 3D ultrasound images, in some embodiments the images as shown in FIGS. 14A and 14C comprise real time transverse ultrasound images, in which the imaging probe such as a TRUS probe is configured to generate real time ultrasound images along a single plane, such as a longitudinal plane or a transverse plane, with a latency of no more than about a one second, for example.

[0336] Referring to FIGS. 16A and 16B, the user interface be configured to receive a user input to automatically determine alignment locations of the treatment probe in the image such as an ultrasound image, for example with Al algorithm such as a machine learning model, which can be a trained neural network, as described herein.

[0337] In some embodiments, and Al algorithm is trained on a plurality of images to identify the location of the energy source and the proximal location of the probe with the 3D image data and output the one or more of a bounding box, a contour, a key point, a segmentation or a classification, for example with 3D image data as described herein.

[0338] The user interface may comprise any of the user interface components as described herein, for example with reference to FIGS. 3A to 3F. The alignment can be performed with the user interface and appropriate user prompts as described herein, and may be incorporated into the user interface method of FIG. 9 and the method of assisted treatment planning of FIG. 10, for example.

[0339] FIG. 16A shows a user interface configured to receive a user input 1610 to activate an Al algorithm such as a machine learning model, which can be a trained neural network, to detect a location of an energy source such as a nozzle on a treatment probe and a location of the treatment probe, such as a proximal location on the treatment probe.

[0340] FIG. 16B shows a user interface as in FIG. 16A with a first marker (N) at a first location 1622 of the energy source and the proximal portion of the probe (P) at a second location 1624 of the probe identified by the trained Al algorithm. In some embodiments, a line 1620 extends between the location 1622 and the location 1624. These can be identified with a trained Al algorithm and output to the user interface as described herein. The output markers can be overlaid on real time images, such as real time longitudinal images, e.g. real time sagittal images from a TRUS for example.

[0341] In some embodiments, the first location 1622 and the second location 1624 are used to register the treatment probe with the image. This registration of the treatment probe with the image can be helpful with treatment planning, for example when a user provides an input to an image location to establish a boundary of a treatment plan, for example an input to establish boundary of a treatment profile as described herein.

[0342] In some embodiments, the user interface comprises an input for the user to confirm that the location of marker for the energy source and the marker for the proximal location of the probe as shown on the image. In some embodiments, the proximal location of the probe corresponds to a path of the energy source when the energy source is translated proximally along the probe, for example drawn proximally.

[0343] In some embodiments, the trained Al algorithm such as a machine learning model, which can be a trained neural network, is configured to output one or more one or more of a bounding box, a contour, a key point, a segmentation or a classification, which corresponds to location 1622. In some embodiments, the trained Al is configured to identify a path of energy 306 and generate the output. In some embodiments, the trained Al is configured to identify the proximal location 1624 of the energy source on the probe.

[0344] In some embodiments, and Al algorithm is trained on a plurality of images to identify location 1622 and location 1624 and output the one or more of a bounding box, a contour, a key point, a segmentation or a classification, for example.

[0345] FIGS. 17A to 17C show a user interface configured to move an imaging probe, such as a TRUS probe, to generate a 3D image data of tissue. The user interface may comprise any of the user interface components as described herein, for example with reference to FIGS. 3A to 3F. The 3D image data acquisition can be performed with the user interface and appropriate user prompts as described herein, and may be incorporated into the user interface method of FIG. 9 and the method of assisted treatment planning of FIG. 10, for example.

[0346] FIG. 17A shows a user interface configured to receive a user input 1716 to activate an imaging probe to generate a 3D image. The imaging probe may comprise any suitable imaging probe as described herein.

[0347] In some embodiments, the user interface comprises a first input 1712 to adjust an angle of the imaging probe, for example to rotate the imaging probe counterclockwise. In some embodiments, the user interface comprises a second input 1714 to adjust an angle of the imaging probe in an opposite direction, for example counterclockwise. In some embodiments the user interface comprises a first input to adjust a starting angle of the sweep and a second input to adjust an ending angle of the sweep to generate the 3D data. In some embodiments, the sweep generates a plurality of longitudinal images, such as a plurality of sagittal TRUS images. In some embodiments, the user interface is configured for the user to activate the 3D image data acquisition with input 1716 after the starting angle of the sweep and the ending angle of the sweep have been set,

[0348] While the user interface can be configured in many ways, in some embodiments the user interface comprises a rotating marker 1718, which rotates around an axis 1719 that corresponds to a rotational axis of the imaging probe. In some embodiments, the input 1712 and the input 1714 are used to adjust the angle of the imaging probe, and the rotational marker 1718 rotates in accordance with the angle of the imaging probe.

[0349] In some embodiments, a real time image 1750, such as a real time longitudinal image is shown in a first image display area and acquired 3D image data 1707 are shown in a second image display area. As shown in FIG.17A, the real time image comprises an image with a rotation of the imaging probe corresponding to rotation alignment of the imaging probe and the treatment probe, for example as described herein with reference to FIG. 14C.

[0350] FIG. 17B shows a user interface during image acquisition to generate the 3D image. In some embodiments, the user interface comprises an input 1722 for the user to end the. In some embodiments, a message or other indication is shown indicating that the sweep is in progress.

[0351] The real time longitudinal image 1750 is shown with an image at an angle corresponding to the angle of the imaging probe. The rotating marker 1718 is shown at an angle corresponding to the real time longitudinal image and the progress of the sweep to generate 3D image data 1707 shown on the display. In some embodiments., the 3D image data 1707 shown on the display comprises a section of the 3D image data, such as a slice of the 3D image data. In some embodiments, the 3D image data 1707 comprises a transverse image from the 3D data, such as a transverse slice or transverse section of the 3D image data.

[0352] In some embodiments, the imaging probe such as a TRUS probe is configured to capture real time images such as longitudinal images. The imaging probe is rotated in response to user input until the tissue of interest is on longer is visible in real time image shown on the display. In some embodiments, the processor is configured to rotate the imaging probe a pre-set number of degrees, e.g., 60 degrees.

[0353] With the imaging probe rotated to the angle corresponding to the start of the sweep, e.g. the starting boundary that defines a starting boundary of 3D data acquisition, the sweep is started. During the sweep a plurality of real time images, e.g. longitudinal images such as slices are acquired at plurality of rotational offsets until the full range of the sweep is reached, and the user is able to view the real time images during the acquisition process on a first area of the display with the partially completed 3D data shown in a different area of the display as shown in FIG. 17B.

[0354] The full range of the sweep can be determined by user inputs. In some embodiments, the user interface is configured to receive a first input corresponding to a first rotational angle of the and a second user input corresponding to the end of the rotational sweep. In some embodiments, the user interface is configured to allow the user to view a first real time image to define the start of the sweep with a first input with the imaging probe rotated to the angle corresponding to the start of the sweep, and the user interface is configured to allow the user to view a second real time image to define the end of the sweep with a second input with the imaging probe rotated to the angle corresponding to the end of the sweep. Alternatively or in combination, the start of the sweep and the end of the sweep may be defined by a maximum rotational angle of the imaging probe.

[0355] In some embodiments, the user interface is configured to receive a user input to define an incremental angle between each image that is acquired to generate 3D image data with the sweeping. The angle between images can be any suitable angle for example within a range from about 0.2 degrees to about 5 degrees, for example about 0.5 degrees,

[0356] In some embodiments, the processor is configured with instructions to acquire 3D data with different angular resolution in accordance with the use of the 3D data. For example, when determining an angle of a treatment probe with respect an imaging probe as described with reference to FIGS, 12A to 12C, the incremental angle may be greater than the angle used to develop a 3D tissue model as described with reference to FIGS. 17A to 17C,

[0357] Although reference is made to the acquisition of images acquired at each of a plurality of rotational angles of the probe to generate the 3D image data, one of ordinary skill in the art will recognize that the image 3D image data can be acquired without rendering each of the 2D acquired with the imaging probe.

[0358] In some embodiments the 3D image data is rendered to generate a digitally reconstructed 3D image.

[0359] FIG. 17C shows a user interface with a real time image 1750, an image section 1760 such as a slice of a digitally reconstructed image, and a 3D model 1770 of the tissue. Generation of the 3D model 1770 can assist with conformal treatment planning because it facilitates creation of a treatment plan in any portion of an anatomical regionrepresented by the 3D model. Advantageously, conformal treatment planning can facilitate treatment of the entirety of the anatomical region. In some embodiments, the real time image 1750 comprises a longitudinal image. In some embodiments, the image section 1760 comprises a transverse image from the 3D image data. The 3D model may comprise any 3D model as described herein, and may comprise one or more components generated in response to output from an Al algorithm, such as a machine learning model, which can be a trained neural network, as described herein.

[0360] In some embodiments, the 3D model 1770 is displayed as a 3D representation of the 3D model. While the 3D representation can be displayed in many ways, in some embodiments, the 3D model appears as a 3D object on the display. The 3D representation can be displayed in many ways, and may comprise one or more of an object, a 3D object, a view of a 3D object, a shaded 3D object, a perspective view of an object, a 3D view of an object, contour lines, or a 3D mesh, for example. In some embodiments, the representation of the 3D model is shown in a 3D view on the display.

[0361] In some embodiments, the real time image is provided on a first image display area, an image section is provided on a second image display area, and a representation of the 3D model is provided on a third image display area. In some embodiments, the first image display area, the second image display area and the third image display area are different, for example shown on different areas of the display.

[0362] In some embodiments a user input 1762 is received that corresponds to a location 1764 along the real time image 1750, and the image section 1760 shown on the display corresponds to the location and a corresponding portion of a representation of the 3D model is marked in response to the location along the real time image. For example, the 3D model may be marked with a marker similar to marker 1764.

[0363] Although reference is made to a TRUS probe, the imaging device used to generate any suitable imaging device as described herein.

[0364] In some embodiments, the user interface comprises instructions 1756 for the user to review the 3D image data and the 3D model 1770, e.g. with a prompt. In some embodiments, the user interface comprises an input 1754 for the user to accept the 3D model and the 3D image data and an input 1752 for the user to rescan the tissue with the imaging probe.

[0365] The 3D model 1770 shown on the display can be viewed with the real time image 1750 and 3D image data such as image section 1760, In some embodiments, the 3D model 1770 is shown in a 3D view.

[0366] In some embodiments, the 3D model 1770 comprises boundary 1774, such as a boundary corresponding to a contour of an organ such a contour of a capsule of a prostate. In some embodiments, a corresponding portion of the boundary 1774 is shown on each of the real time image 1750 and the image section 1760. In some embodiments, the corresponding portion of the boundary 1774 shown on the real time image comprises section, e.g. a slice, of the boundary corresponding to the one or more of a location or rotation of the real time image in the 3D model. In some embodiments, the corresponding portion of the boundary 1774 shown on the image section 1760 comprises a section of the boundary, e.g. a slice of the boundary corresponding to a location of a marker 1764 on the real time image.

[0367] In some embodiments, a user input 1762 is configured for a user to select a location along the real time image with the user input 1762. In some embodiments, the marker 1764 moves in accordance with a location of the user input 1762 shown on the display. The user interface can be configured for the user to select a first location of thereal time image and a second location of the real time image, and the image section 1760 and corresponding portion of the boundary 1774 of the 3D model 1770 update accordingly. While the user interface can be configured in many ways, in some embodiments, a slider bar 1766 is configured to allow the user input 1762 to slide along the real time image, and the marker 1764 and the image section 1760 and the corresponding portion of the 3D model 1770 such as boundary 1774 update accordingly.

[0368] In some embodiments, an Al algorithm (such as, a neural network or another machine learning model) is trained to identify tissue structures from the 3D image data for incorporation into the 3D model, and to output one or more of a bounding box, a contour, a key point, a segmentation or a classification as described herein.

[0369] In some embodiments, the 3D model is configured to show one or more markers corresponding to an identified tissue structure on the display. The one or more markers can be shown at a location in the image section 1760 corresponding to the location of the tissue structure from the 3D image data, such as the 3D image. The one or more markers may be associated with key points or bounding boxes from the trained neural network (or another trained Al algorithm), for example. In some embodiments, the key point comprises one or more of a key point of a verumontanum, a key point of a mid-prostate region, a key point of a mid-prostate region corresponding to a thickness of the mid-prostate region, a key point of a bladder neck, a key point of a median lobe of a prostate, a key point of a median lobe of a prostate extending into a bladder, a key point of a trigone tissue region, a key point of a ureteral orifice, a first key point of a first ureteral orifice, a second key point of a second ureteral orifice, a key point of an external sphincter, a key point of a surgical capsule, a key point of a peripheral zone, a key point a transition zone, a key point of ejaculatory glands, a key point of neurovascular bundles, a key point of a cancerous tissue, or a key point of a tumor. In some embodiments, the bounding box comprises one or more of a bounding box of a verumontanum, a bounding box of a mid-prostate region, a bounding box of a mid-prostate region corresponding to a thickness of the mid-prostate region, a bounding box of a bladder neck, a bounding box of a median lobe of a prostate, a bounding box of a median lobe of a prostate extending into a bladder, a bounding box of a trigone tissue region, a bounding box of a ureteral orifice, a first bounding box of a first ureteral orifice, a second bounding box of a second ureteral orifice, a bounding box of an external sphincter, a bounding box of a surgical capsule, a bounding box of a peripheral zone, bounding box of a transition zone, a bounding box of ejaculatory glands, a bounding box of neurovascular bundles, or a bounding box of a cancerous tissue, or a bounding box of a tumor.

[0370] In some embodiments, the user interface is configured to prompt the user to accept the 3D data, e.g. the scan with a user input 1754, Alternatively or in combination, the user interface can be configured to allow the user to accept the 3D model with user input. Work in relation to the present disclosure suggest that the user viewing the 3D image sections and tissue structures associated with the 3D model prior to acceptance may assist the user in planning the treatment, for example. In response to the user accepting the 3D image data and the 3D model of the tissue, the user interface is configured to proceed to a next step by providing treatment planning images and data on the display of the user interface, for example.

[0371] FIGS. 18A to 18B show a user interface configured to allow a user to plan a treatment with assistance of trained Al algorithm, such as a machine learning model, which can be a trained neural network. The user interface 103 may comprise any of the user interface components as described herein, for example with reference to FIGS. 3A to 3F. The 3D treatment planning can be performed with the user interface and appropriate user prompts as describedherein, and may be incorporated into the user interface method of FIG, 9 and the method of assisted treatment planning of FIG. 10, for example,

[0372] In some embodiments, the processor is configured with instructions process the 3D model to generate the 3D treatment plan. Alternatively or in combination, the processor can be configured to generate the 3D treatment plan in response to the 3D image data. In some embodiments, the processor is configured to generate the 3D treatment plan in response to output from a trained Al algorithm, such as a machine learning model, which can be a trained neural network, as described herein. The Al algorithm that generates the 3D treatment plan can be same or different than the Al algorithm that generates the 3D model. For instance, two specialized machine learning models can be used for generating the 3D model and the 3D treatment plan or a general machine learning model can be used for generating the 3D model and the 3D treatment plan.

[0373] Work in relation to the present disclosure suggest that in some instances it may be easier for a user such as a surgeon to plan a treatment with 2D images while visualizing the treatment in 3D.

[0374] FIGS. 18A and 18B show treatment planning in 2D while the treatment profile is visualized in 3D. In some embodiments, the 3D model 1770 is displayed as a 3D representation of the 3D model. While the 3D representation can be displayed in many ways, in some embodiments, the 3D model appears as a 3D object on the display. The 3D representation can be displayed in many ways, and may comprises one or more of an object, a 3D object, a view of a 3D object, a shaded 3D object, a perspective view of an object, a 3D view of an object, contour lines, or a 3D mesh, for example. In some embodiments, the representation of the 3D model is shown in a 3D view of the on the display.

[0375] In some embodiments, the 3D treatment profile 1810 is displayed as a 3D representation of the 3D treatment profile. While the 3D representation can be displayed in many ways, in some embodiments, the 3D treatment profile appears as a 3D object on the display. The 3D representation can be displayed in many ways, and may comprises one or more of an object, a 3D object, a view of a 3D object, a shaded 3D object, a perspective view of an object, a 3D view of an object, contour lines, or a 3D mesh, for example. In some embodiments, the representation of the 3D treatment profile is shown in a 3D view of the on the display.

[0376] In some embodiments, the real time image 1750, the image section 1760 and the 3D model 1770 are shown on the display with the 3D treatment profile 1810 overlaid thereon. In some embodiments, the real time image 1750, the image section 1760, the 3D model 1770 are shown on different areas of the display with the 3D treatment profile 1810 overlaid thereon. In some embodiments, the 3D model 1770 and the 3D boundary 1774 are shown in a 3D view 1705 overlaid on a 3D treatment profile 1810. In some embodiments, the 3D treatment profile 1810 comprises a boundary 1812, which is overlaid on the 3D model 1770, for example overlaid on a boundary 1774 of the 3D model. In some embodiments, the 3D treatment profile 1810 is configured to update in response to a user input to adjust the 3D treatment profile, for example to adjust a boundary 1812 of the 3D treatment profile shown on the display,

[0377] In some embodiments, the user interface comprises an input 1860 to add a pass of a treatment, such as an additional longitudinal pass, for example.

[0378] FIG. 18A shows a real time image 1750 and marker 1764 on the real time image. The marker 1764 corresponds to a section 1760 from the 3D image shown on the display. The section corresponding section 1760 is shown with a corresponding portion of the 3D model, such as boundary 1774 of the 3D model. The section 1760 is shown with a corresponding portion of the 3D treatment profile, such as a corresponding portion of boundary. Acorresponding portion of the 3D treatment profiie 1810.. such as a corresponding portion of boundary 1812 is shown on the real time image 1750, In some embodiments., the image section 1760 is shown with a corresponding portion of the 3D treatment profile such as a corresponding portion of the boundary 1812 overlaid thereon,

[0379] In some embodiments, the user interface 103 is configured to receive a user input to select a transverse image section for treatment planning. In some embodiments, a user input 1850 is configured to receive a user input corresponding to a location along the real time image 1750. in response a marker 1852 appears on the treatment profile overlaid on the real time image. The marker 1852 corresponds to the image section 1760. In some embodiments, the real time image 1750 comprises a real time longitudinal image such as a real time TRUS image and the section 1760 comprises a transverse image section.

[0380] In some embodiments, the real time image is provided on a first image display area, an image section is provided on a second image display area, and a representation of the 3D model and the 3D treatment profile is provided on a third image display area, in some embodiments, the first image display area, the second image display area and the third image display area are different, for example shown on different areas of the display.

[0381] In some embodiments a user input 1762 is received that corresponds to a location 1764 along the real time image 1750, and the image section 1760 shown on the display corresponds to the location and a corresponding portion of a representation of the 3D model 1770 overlaid with the 3D treatment profile 1810 is marked in response to the location along the real time image. For example, the 3D model and the 3D treatment profile may be marked with a marker similar to marker 1764.

[0382] Although reference is made to three image display areas, the user interface 103 can be configured to provide any suitable number of display areas. In some embodiments, the 3D treatment profile is not overlaid the 3D model 1770, for example provided on a fourth image display area. In some embodiments, the real time image 1750 is provided on a first image display area, an image section 1760 is provided on a second image display area, a representation of the 3D model 1770 is provided on a third image display area and the 3D treatment profile 1810 is shown on a fourth image display area. In some embodiments, the first image display area, the second image display area, the third image display area and the fourth image display area are different, for example on different areas of the display.

[0383] In some embodiments, a user input 1762 is received that corresponds to a location of the marker 1764 along the real time image. The image section 1760 shown on the display corresponds to the location and a corresponding portion of a 3D treatment profiie 1810 is marked in response to the location along the real time image, for example with a marker similar to marker 1764, In some embodiments, a corresponding portion of a representation of the 3D model 1770 is marked in response to the location along the real time image,

[0384] In some embodiments, corresponding portions of the 3D treatment profile 1810 and the 3D model 1760 are provided on corresponding image sections such as image section 1760 and on the real time image 1750.

[0385] In some embodiments, the user interface 103 is configured to receive a user input to confirm that the user has verified the 3D treatment profile in relation to the 3D model.

[0386] In some embodiments, the user interface is configured to toggle between the image section 1760 and the real time image 1750 in response to an input to the toggle icon 322. In some embodiments, the real time image 1750 is shown in a larger image display area, e.g. a primary image display area, and the image section 1760 is shown in asmaller image display area., e.g, a secondary image display area, for example as shown with reference to FIG, 18A. In response to a user input to the toggle icon, the image section 1760 is shown in a larger image display area and the real time image 1750 is shown in a smaller image display area, for example as shown with reference to FIG. 18B.

[0387] Although reference is larger and smaller image display areas, in some embodiments the image two image display areas for the real time image 1750 and the image section 1760 may be of equal size, for example,

[0388] In some embodiments, a user such as a surgeon can sweep through the entire 3D treatment plan in the real time image 1750, such as a real time longitudinal image, and choose any arbitrary image section 1860 to view a reconstructed image section with corresponding contours of the 3D model 1770 and planned angles of the 3D treatment profile 1810.

[0389] Work in relation to the present disclosure suggests that toggling between the two views allows planning in either image as shown, with changes reflecting the changes in the other image.

[0390] In some embodiments, the user interface is configured to proceed to a next step of treatment planning, for example in response to a user input to the next icon 334.

[0391] FIG. 17B shows the user interface 103 configured for the user to make review the 3D treatment plan on two dimensional image section 1760 and to make adjustments to the treatment profile. In some embodiments, the image section 1760 is shown with a corresponding portion of the 3D model 17750 such as boundary 1774 and a corresponding portion of the 3D treatment profile 1810, such as boundary 1812.

[0392] In some embodiments, the user interface is configured to highlight the selected input 1850 indicating the selected location of the image section 1850 along the real time image that corresponds to the image section 1760. In some embodiments, the input 1850 comprises a plurality of user inputs configured to receive user inputs corresponding to a plurality of locations along the real time image.

[0393] In some embodiments, the user interface is configured to plan the treatment on the image section 1760 corresponding to discrete locations such as Median Lobe, Bladder Neck, Mid Prostate, Veru Protection Zone, and then interpolated by the processor in between these discrete locations to generate the 3D treatment profile. In some embodiments, the user input 1850 is configured to receive an input corresponding to each of these locations.

[0394] In some embodiments, the user interface comprises one or more components of the user interface shown and described with reference to FIGS, 3A to 3F, for example. In some embodiments, the image section 1850 comprises a transverse image section, e.g. a transverse slice, and the real time image 1750 comprises a longitudinal image. The user interface 103 can be configured for the user to review the treatment plan and make adjustments to the treatment plan overlaid on image section 1760 shown on the display similarly to the transverse images and treatment planning described with reference to FIGS, 3A to 3F.

[0395] In some embodiments, the 3D treatment profile 1810 is updated in response to changes to the treatment to the treatment profile shown on the image section 1760, for example in real time, e.g. within about 1 second. In some embodiments, the user adjusts one or more of a radial distance or an angle of the boundary 1812 of the 3D treatment profile on the 2D image section 1760 and the 3D treatment profile 1812 shown in the 3D view 1805 is updated accordingly.

[0396] In some embodiments, the user interface is configured to show objects of interest within the 3D model, such as one or more tissue structures as described herein. Should other objects be of interest in the planning cycle,for instance, cancerous lesions, these could also be located within the 3D model showing the impact of the resection on the object of interest.

[0397] In some embodiments, the trained Al algorithm such as a machine learning model, which can be a neural network, is configured to generate the 3D treatment profile in a conformal manner at a plurality of locations along the 3D model such as at 0,5 mm to 10 mm along the treatment profile, for example at 0.5 mm to 5 mm along the treatment profile, which can increase the volume of the 3D treatment profile such as a 3D resection profile, for example,

[0398] In some embodiments, each of a plurality of angular coordinate references and each of a plurality of radial distances are configured to treat a longitudinal distance of no more than about 10 mm along the 3D treatment profile, for example no more than about 5 mm along the 3D treatment profile. In some embodiments, the longitudinal distance is within a range from 0.5 mm to 10 mm, for example withing a range from 0.5 mm to 5 mm. In some embodiments, the processor is configured to smooth the 3D treatment profile between each of the plurality of angular coordinate references and each of the plurality of radial distances.

[0399] In some embodiments, the user interface is configured to allow a user such as a surgeon to adjust anywhere along the 3D treatment profile, for example by sliding the marker 1762 along the real time image and making appropriate adjustments to the treatment profile shown on the image section 1760.

[0400] In some embodiments, the interface is configured to show objects of interest within the 3D model, such as one or more tissue structures as described herein. In some embodiments, objects of interest in the planning cycle, for instance, cancerous lesions, can also be located within the 3D model showing the effect of the treatment, e.g. resection on the object of interest.

[0401] In some embodiments, the user interface is configured to receive a user input to rotate and zoom the 3D model to review from any angle and at any zoom. In some embodiments, the 3D image is shown in the 3D view, for example. In some embodiments, the user interface is configured to toggle the locations of the images. For example an input to the toggle icon 322 can move the 3D view 1805 of the 3D model 1770 and 3D treatment profile 1810 to a different image display, for example to a lower image display area of increased size.

[0402] In some embodiments, the user interface 103 receives an input to add a pass of treatment. In some embodiments, the 3D treatment profile comprises a plurality of passes. In some embodiments, the 3D treatment profile comprises a plurality of 3D passes. Any suitable number of passes can added, such as a second, third or addition pass. Each of these passes can be planned with its own separate profile and angles, for example. In some embodiments, these passes overlap onto each other, e.g. are superimposed on each other, in some embodiments, at least a portion of each pass does not overlap with another pass. In some embodiments, the passes do not overlap along a first portion of the treatment and overlap along a second portion of the treatment. Each of these passes can be planned independently and combined during a single treatment for cumulative effect.

[0403] FIG. 19 shows a user interface configured to plan a plurality of passes of a treatment energy. In some embodiments, each of the plurality of passes corresponds to longitudinal movement of a treatment probe as described herein. In some embodiments, the first pass comprises a first treatment profile 1910 and pass comprises second treatment profile 1920. In some embodiments a boundary 1774 of the 3D model 1770 is shown on the display with each of the treatment profiles. While the first treatment profile 1910 and the second treatment profile 1920 can beconfigured in many ways, in some embodiments, the first treatment profiie is configured to overlap with the second treatment profile along at least a portion of the treatment profile such as a 3D treatment profile,

[0404] In some embodiments, the user interface is configured to adjust each of the treatment profiles independently. For example, the user interface can be configured to adjust the first treatment profile in response to a first input and to adjust the second treatment profile with a second user input. Each of the treatment profiles can be adjusted in accordance with the treatment planning shown and described with reference to FIGS, 18A and 18B and FIGS. 3A to 3F, for example.

[0405] In some embodiments, the tissue to be treated such as an organ may be asymmetrical, for example with reference to a longitudinal axis of the treatment probe. For example, prostates can be asymmetrical, and may have a bulge in some instances. In some embodiments the processor is configured to perform a first pass on a first side along the first treatment profile and a second pass along a second side of the treatment profile.

[0406] Referring again to FIG. 19, the user interface 103 may comprise any of the user interface components as described herein, for example with reference to FIGS. 3A to 3F. The 3D treatment planning for the plurality of passes, e.g. the first pass and the second pass, can be performed with the user interface and appropriate user inputs and prompts as described herein, and may be incorporated into the user interface method of FIG. 9 and the method of assisted treatment planning of FIG. 10, for example. In some embodiments, the trained Al algorithm is configured to generate plurality of passes in response to the 3D model as described herein.

[0407] In some embodiments, the 3D model of the tissue comprises one or more delicate tissue structures, and the 3D treatment profile is generated in response to the one or more delicate tissue structures.

[0408] FIG. 20A shows an initial configuration of a protection zone 2020 of a 3D treatment profile to protect a delicate tissue structure 2010 such as verumontanum of a prostate. In some embodiments, the protection zone 2020 is configured to decrease treatment of the delicate tissue structure and in some instances avoid the delicate tissue structure. In some embodiments, the delicate tissue structure 2010 is shown on an image section 1760 and a boundary 1774 of a 3D model of the tissue. In some embodiments the delicate tissue structure comprises a boundary 2012.

[0409] In some embodiments, the treatment profiie comprises a first profile 2022 on a first side of the delicate tissue structure and a second profile 2032 on a second side of the delicate tissue structure. In some embodiments, the first treatment profile comprises a first radial distance 2024 and a first treatment angle 2026, In some embodiments, the second treatment profile comprises a second radial distance 2034 and a second treatment angle 2036. In some embodiments, the radial distance 2024 and the radial distance 2034 extend substantially to the boundary 1774 of the 3D model of the tissue. In some embodiments, the shape of the first treatment profile and the second treatment profile may comprise a shape similar to butterfly wings and be referred to as a butterfly cut.

[0410] In some embodiments, the treatment profile may overlap with the delicate tissue structure and the treatment profile adjusted accordingly as described herein. In some embodiments, the user interface 103 is configured to modify the first treatment profile or the second tissue profile to decrease interaction of the treatment with the delicate tissue structure as described herein.

[0411] The delicate tissue structure 2010 can be identified in many ways, for example by a user such as a surgeon or with a trained Al algorithm such as a machine learning model, which can be a trained neural network, as described herein, and combinations thereof.

[0412] FIG, 20B shows an adjusted configuration of the protection zone 2030 as in FIG. 20A. The first treatment profile 2022 on the first side is configured so as not to overlap with the delicate tissue structure 2010, and the second treatment profile 2032 on second side is configured so as not to overlap with the delicate tissue structure 2010, in some embodiments one or more of the first radial distance 2024 or the first angle 2026 is adjusted with a user input. In some embodiments one or more of the second radial distance 2034 or the second angle 2036 is adjusted with a user input.

[0413] In some embodiments one or more of angle 2026 or angle 2036 is adjusted in response to a user input. For example, the angle may be adjusted so that the boundary of the treatment profile does not overlap with the delicate tissue structure. In some embodiments, one or more of the first radial distance 2024 or the second radial distance 2034 is adjusted in response to the user input.

[0414] In some embodiments, the first profile 2022 and the second profile 2032 are independently adjustable. In some embodiments, the user interface is configured for a user to adjust a plurality of markers of the first treatment profile 2022 and a plurality of markers of the second treatment profile 2032, for example by selecting one or more markers on the user interface and adjusting an angle or a radial distance.

[0415] In some embodiments, a central angle of the protection zone 2020 between the first profile 2022 and the second profile 2032 can be adjusted. In some embodiments, a central portion of the protection zone is angled as shown with reference to FIG. 20B. In some embodiments a width of the protection zone can be increased or decreased in response to inputs from one or more markers as described herein. In some embodiments, the user interface is configured to plan the first treatment profile 2022 on the first side of the protection zone 2020 independently of the second profile 2032 on the second side of the protection zone.

[0416] In some embodiments, the user interface is configured to adjust angle 2026 in response to an input corresponding to a marker at a location 2052 to adjust radial distance 2024 and angle 2026 on a first side of treatment profile 2022. In some embodiments, the user interface is configured to adjust angle 2026 in response to an input corresponding to a marker at a location 2054 to adjust to radial distance 2024 and angle 2026 on a second side of treatment profile 2022,

[0417] In some embodiments, the user interface is configured to adjust angle 2036 in response to an input corresponding to a marker at a location 2056 to adjust to radial distance 2034 and angle 2036 on first side of treatment profile 2032. In some embodiments, the user interface is configured to adjust angle 2036 in response to an input corresponding to a marker at a location 2056 to adjust to radial distance 2034 and angle 2036 on a second side of treatment profile 2032,

[0418] In some embodiments, and Al algorithm is trained to identify locations of the delicate tissue structures with the 3D image data and output the one or more of a bounding box, a contour, a key point, a segmentation or a classification, for example with 3D image data as described herein. Although reference is made to a verumontanum, the delicate tissue structure may comprise any delicate tissue structure such as one or more of a verumontanum, a trigone tissue region, a ureteral orifice, a first ureteral orifice, a second ureteral orifice, or an external sphincter, for example.

[0419] The user interface 103 as shown in FIGS. 20A and 20B may comprise any of the user interface components as described herein, for example with reference to FIGS. 3A to 3F and 18A to 18B, for example. The 3Dtreatment planning for the first treatment profile 2022 and the second treatment profile 2032 can be performed with the user interface and appropriate user prompts as described herein, and may be incorporated into the user interface method of FIG. 9 and the method of assisted treatment planning of FIG. 10, for example.

[0420] Referring to FIGS. 21 A to 21 C, in some embodiments the user interface is configured to add a transverse treatment profile and to adjust the treatment transverse treatment profile. In some embodiments, the user interface can be configured to adjust the treatment profile at a plurality of preset locations, such as one or more of the bladder neck, Although the user interface can be configured to allow the user to adjust the 3D treatment profile at a plurality of locations such as the median lobe, bladder neck, mid prostate and verumontanum (“veru”) as shown with reference to FIGS 18A to 18B, in some embodiments it may be helpful to configure the user interface to allow the user to add a transverse treatment profile.

[0421] FIG. 21A shows a user interface configured for a user to add a transverse treatment profile. The user interface 103 shown in FIG. 21A may comprise components and features similar to the user interface of FIGS 18A to 18C, for example. The user interface may comprise a marker 1764 to indicate a location along the real time image 1750 and an input to identify a location along the real time image.

[0422] FIG. 21 B shows a user interface as in FIG. 21 A with an additional transverse treatment profile. In some embodiments, the user interface is configured to add an additional transverse treatment plan, such as a profile in response to a user input 2110. In some embodiments, the location of the additional treatment plan is overlaid on the real time image with a marker 2112 to identify the location of the additional treatment plan, and the marker may comprise an appropriate label, e.g. U6.

[0423] FIG. 21C shows the user interface as in FIGS. 21A and 21B configured to receive one or more users to adjust the additional transverse treatment profile, which can be similar to the user inputs shown with reference to FIG 18B, for example. In some embodiments, the additional treatment profile 2120 may be selected in response to a user input and adjusted as described herein. In some embodiments, the user input 1850 is configured to receive a user input corresponding to a selection of the additional treatment profile 2120, for example. In some embodiments, the user input 1850 comprises a plurality of user inputs configured to receive user inputs corresponding to a plurality of locations along the real time image as described herein.

[0424] In some embodiments, treatment angles are planned at discrete locations such as Median Lobe, Bladder Neck, Mid Prostate, Veru Protection Zone, and then interpolated by the processor in between these discrete locations.

[0425] In some embodiments, the user interface is configured for a user such as a surgeon to identify a specific plane to plan angles in can be created.

[0426] In some embodiments, a version where a plurality of specific planes are prepopulated by the system such as at Median Lobe, Bladder Neck, Mid Prostate and Veru is provided on the user interface.

[0427] In some embodiments, the user interface is configured to allow a user such as a surgeon to identify a new plane that they would like to explicitly plan is provided providing input to an icon such as a “+ Plane” icon. In some embodiments, in response to the input a new user-defined location is added to the real time image with an identifier such as “U6”. which can then be modified in a manner similarly to other locations along the real time image, such as a longitudinal image, for example a real time sagittal view.

[0428] In some embodiments., the 3D model 1770 is displayed as a 3D representation of the 3D model. While the 3D representation can be displayed in many ways, in some embodiments, the 3D model appears as a 3D object on the display. The 3D representation can be displayed in many ways, and may comprises one or more of an object, a 3D object, a view of a 3D object, a shaded 3D object, a perspective view of an object, a 3D view of an object, contour lines, or a 3D mesh, for example. In some embodiments, the representation of the 3D model is shown in a 3D view of the on the display.

[0429] In some embodiments, the 3D treatment profile 1810 is displayed as a 3D representation of the 3D treatment profile. While the 3D representation can be displayed in many ways, in some embodiments, the 3D treatment profile appears as a 3D object on the display. The 3D representation can be displayed in many ways, and may comprises one or more of an object, a 3D object, a view of a 3D object, a shaded 3D object, a perspective view of an object, a 3D view of an object, contour lines, or a 3D mesh, for example. In some embodiments, the representation of the 3D treatment profile is shown in a 3D view of the on the display.

[0430] The user interface 103 as shown in FIGS. 21 A to 21 C may comprise any of the user interface components as described herein, for example with reference to FIGS. 3A to 3F and 18A to 18B, for example. The 3D treatment planning for the additional treatment profile can be performed with the user interface and appropriate user prompts as described herein, and may be incorporated into the user interface method of FIG. 9 and the method of assisted treatment planning of FIG. 10, for example.

[0431] FIG. 22 shows the user interface configured for a user to review and accept a treatment plan. In some embodiments, the real time image 1750, the image section 1760 and the 3D model 1770 are provided on the display. In some embodiments, the 3D model 1770 and the 3D treatment profile 1810 are shown in the 3D view 1805. The real time image 1750 can be shown on with the corresponding portion of the 3D treatment profile 1810 overlaid thereon, for example. The real time image section 1760 can be shown on with the corresponding portion of the 3D treatment profile 1810 overlaid thereon, for example. In some embodiments, the real time image 1750, the image section 1760 and the 3D model are shown on different areas of the display with an input 2210 to accept the treatment plan.

[0432] In some embodiments, the user interface is configured to receive a user input to rotate and zoom the 3D model to review from any angle and at any zoom,

[0433] The user interface 103 shown in FIG. 22 may comprise any of the user interface components as described herein, for example with reference to FIGS. 3A to 3F, The 3D visualization of the planned treatment can be performed with the user interface and appropriate user inputs and prompts as described herein, and may be incorporated into the user interface method of FIG, 9 and the method of assisted treatment planning of FIG. 10, for example.

[0434] FIG. 23 shows a method 2300 of training an artificial intelligence (Al) algorithm, such as a machine learning model. In some cases, a single Al algorithm can be trained to perform different tasks described herein, such as identifying one or more tissue structures, generating a 3D model, or generating a treatment profile. Different sets of training data (which can include 3D images) can be used to train the single Al algorithm to perform the different tasks. In some implementations, different specialized Al algorithms can be used to separately perform such different tasks.

[0435] At a step 2310, images are collected and grouped. The images may comprise a collection of images acquired prior to and during treatment and can include 3D images. The images can be appropriately classified andgrouped, for example classified and grouped with respect to a type of tissue. In some embodiments., the images are group according to a type of tissue., such as types of prostate tissue as described herein, for example.

[0436] At a step 2320, the Al model receives the grouped images from step 2310, The machine assisted labeling (MAL) is used to label the grouped images in accordance with each group of the images. In some embodiments, each group of images may be segmented and classified and labeled with a MAL. In some embodiments, the images are segmented and annotated with labels so as to identify tissue structures as described herein, for example,

[0437] At a step 2330, the MAL images are received by a user interface that allows an expert to review and cleanup of the MAL image data. The initial set of MAL images may be reviewed an expert such as a radiologist, for example. The expert review and cleanup of the MAL images generates high quality labeled image data. The high quality labeled image data can be added to a pool of high quality image data. This high quality image data can be used as a ground state or truth state to further train and refine the classifier and contains annotated data with appropriate labels identifying the type of tissue.

[0438] At a step 2340, the high quality labeled data is received by an Al algorithm as described herein and used for training and validation of the model. The annotated images can be used to train and validate the Al algorithm and develop model parameters of the Al algorithm. The Al algorithm may comprise any suitable algorithm as described herein, such as a neural network (for example, a deep neural network).

[0439] At a step 2350, the trained model parameters generated at step 2340 are received to refine and tune the model. While this can be performed in many ways, in some embodiments model inference speed improvements are performed on the model to increase the speed of the model without substantially compromising the output of the model. This can be helpful to increase throughput of the model and decrease processing bottle necks in the model.

[0440] At a step 2360, the model is released and deployed in the field. This field deployed model can be used to process images to generate one or more tissue structures as described herein.

[0441] In some embodiments, the model is further refined prior to the field deployment at step 2360. For example, it can be helpful to iterate and refine the model by repeating steps 2320, 2330, 2340 and 2350 to generate acceptable model parameters for field deployment at step 2360. In some embodiments, steps 2320, 2330, 2340 and 2350 comprise elements of a feedback loop. In some embodiments, the new model parameters developed at step 2350 are provided to the grouped images for MAL at step 2320 and then MAL images provided to the expert for review at step 2330. In some embodiments, additional images are provided for testing and validation at step 2320 and MAL images generated and provided to the expert for review at step 2330, and these images added to the pool of image data. In some embodiments, the new model parameters generated at step 2350 can be provided to the Al algorithm at step 2340 and used to evaluate the images and used to further refine and develop the Al algorithm at step 2340. Once the Al algorithm training and development has been completed at step 2340, this trained model can be refined at step 2350 for model inference speed improvement, for example. The steps, 2320, 2330, 2340 and 2350 can be performed as many times as appropriate to further refine and improve the model prior to field deployment at step 2360.

[0442] The Al algorithm may comprise any Al algorithm, which can be trained as described herein. In some embodiments, the artificial intelligence algorithm comprises one or more of image enhancement, image segmentation, a neural network, a convolutional neural network, a transformer, a transformer machine learning model, supervisedmachine learning, unsupervised machine learning, edge detection, feature recognition, segmentation, 3D model reconstruction, or a multi-modality image fusion, for example.

[0443] In some embodiments, the trained Al algorithm comprises one or more components of a neural network. In some embodiments, the trained neural network comprises You Only Look Once (YOLO) neural network, such as a family of real-time object detection algorithms based on convolutional neural networks.

[0444] The trained Al algorithm can be configured to output any suitable data in accordance with the present disclosure. In some embodiments, the trained Al algorithm is configured to output one or more of a bounding box, a contour, a key point, a segmentation, or a classification, and combinations thereof.

[0445] In some embodiments, the Al algorithm is trained to identity one or more tissue and non-tissue structures, such as a verumontanum, mid-prostate region, a thickness of the mid-prostate region, a median lobe of a prostate, a bladder neck, a median lobe of a prostate, median lobe of a prostate extending into a bladder, a trigone tissue region, ureteral orifice, a first ureteral orifice, a second ureteral orifice, an external sphincter, a treatment probe, cancerous tissue, a tumor, or a waterjet (such as, a waterjet nozzle) and combinations thereof. The Al algorithm can be trained with images of these objects, as described herein, and configured to generate appropriate output. In some implementations, the Al algorithm can identify one or more of a location or position of one or more non-tissue structures, such as a position and location of a treatment probe or a location of an energy delivery region of the treatment probe on the treatment probe (such as, a location of a water jet nozzle positioned on the treatment probe), and use this additional information to generate a treatment plan as described herein.

[0446] In some embodiments, the output of the trained Al algorithm is used to generate a 3D model of the tissue as described herein.

[0447] Although a method 2300 of training an Al algorithm is shown and described in accordance with some embodiments, one of ordinary skill in the art will recognize many adaptations and variations in accordance with the present disclosure. For example, the steps can be performed in any order. Some of the steps repeated, and some of the steps omitted. Some of the steps may comprise sub steps of other steps. Also, one or more of the steps of method 2300 can be combined with any step of any method described herein.

[0448] FIG. 24 shows a method 2400 of generating 3D image data.

[0449] At a step 2410 a user input is received to generate the three dimensional (3D) image data of the tissue with an imaging device. In some embodiments, an imaging probe is rotated to a plurality of angular positions to acquire a plurality of longitudinal images to generate the 3D image data. Alternatively or in combination, the imaging probe may be translated to a plurality of longitudinal positions to acquire a plurality of transverse images to generate the 3D image data.

[0450] At a step 2420, an image section of the 3D image data is provided on a display visible to a user. The 3D image data may comprise any suitable data, such as a 3D image. In some embodiments, the 3D image data remains static while the real time image is provided to the display. In some embodiments, the image section from the 3D data comprises a two dimensional (2D) transverse image, such as a transverse image slice. Alternatively or in combination, the image section from the 3D data comprises a two dimensional (2D) longitudinal image, such as a longitudinal image slice.

[0451] At a step 2430 a real time image of the tissue from the imaging device is provided on the display. In some embodiments, the real time image comprises a real time longitudinal image, such as a real time longitudinal image from an ultrasound imaging device such as TRUS probe. Alternatively or in combination, the real time image may comprise a real time transverse image, such as a real time transverse image from an ultrasound imaging device such as a TRUS probe,

[0452] In some embodiments, the real time image is provided on a first image display area and the image section is provided on a second image display area different from the first image display area.

[0453] At a step 2440, a user input is received in which the user input corresponds to a location along the real time image, and the image section shown on the display corresponds to the location. In some embodiments, the real time image comprises a real time longitudinal image and the image section comprises a transverse image.

[0454] In some embodiments, the user input location is shown with a marker on the real time image and the image section corresponds to a location of the marker along the real time image.

[0455] At a step 2450, the user interface provides a plurality of images to verify the 3D image data.

[0456] In some embodiments, a first image section is shown on the display in response to the marker at a first location on the real time image and a second image section is shown on the display in response to the marker at a second location on the real time image. In some embodiments, the user input location is configured to slide along the real time image and update the image section in response to the user input location sliding along the real time image, for example.

[0457] In some embodiments, the image section comprises a longitudinal image section and the real time image comprises a real time longitudinal image from an imaging probe. In some embodiments, the longitudinal image section corresponds to an angle of rotation of the imaging probe around a longitudinal axis of the imaging probe. In some embodiments, the imaging probe has been rotated to the angle in response to a user input.

[0458] In some embodiments, the real time longitudinal image is shown on the display at a first area of the display and the longitudinal image section is shown on a second area of the display.

[0459] At a step 2460, the user interface receives a user input to adjust an angle of rotation of an imaging probe to generate the 3D image data and a real image is generated from the imaging probe. In some embodiments, the user input comprises a first input to define a first angle of rotation of the imaging probe to define a first boundary of the 3D image data and a second input to define a first angle of rotation of the imaging probe to define a second boundary of the 3D image data.

[0460] In some embodiments, a rotating a marker is provided on the display to show an angle of rotation of the imaging probe. In some embodiments, the marker rotates with rotation of the imaging probe and a partially completed portion of the 3D image data is displayed while the 3D image data is generated. In some embodiments the marker rotates around an axis of rotation corresponding to an axis of the imaging probe.

[0461] At a step 2470, a user input is received indicating that the user has accepted the 3D image data.

[0462] FIG. 25 shows a method 2500 of generating an interactive tissue model.

[0463] At a step 2502 a three dimensional (3D) model of the tissue is generated in response to one or more tissue structures of the 3D image data.

[0464] In some embodiments, the 3D model is generated using a machine learning model (such as, a trained neural network) configured to identify the one or more tissue structures from the 3D image data. The machine learning model can be trained on 3D image data as described herein. In some embodiments, the output comprises one or more of a bounding box, a contour, a key point, a segmentation, or a classification of the one or more tissue structures,

[0465] In some embodiments, the key point comprises one or more of a key point of a verumontanum, a key point of a mid-prostate region, a key point of a mid-prostate region corresponding to a thickness of the mid-prostate region, a key point of a bladder neck, a key point of a median lobe of a prostate, a key point of a median lobe of a prostate extending into a bladder, a key point of a trigone tissue region, a key point of a ureteral orifice, a first key point of a first ureteral orifice, a second key point of a second ureteral orifice, a key point of an external sphincter, a key point of a surgical capsule, a key point of a peripheral zone, a key point a transition zone, a key point of ejaculatory glands, a key point of neurovascular bundles, a key point of a cancerous tissue, a key point of an endoscope (such as, a treatment probe or an energy delivery region of the treatment probe, such as a nozzle), or a key point of a tumor, and combinations thereof.

[0466] In some embodiments, the bounding box comprises one or more of a bounding box of a verumontanum, a bounding box of a mid-prostate region, a bounding box of a mid-prostate region corresponding to a thickness of the mid-prostate region, a bounding box of a bladder neck, a bounding box of a median lobe of a prostate, a bounding box of a median lobe of a prostate extending into a bladder, a bounding box of a trigone tissue region, a bounding box of a ureteral orifice, a first bounding box of a first ureteral orifice, a second bounding box of a second ureteral orifice or a bounding box of an external sphincter, a key point of a surgical capsule, a key point of a peripheral zone, a key point a transition zone, a key point of ejaculatory glands, a key point of neurovascular bundles, or a key point of a cancerous tissue, a key point of an endoscope (such as, a treatment probe or an energy delivery region of the treatment probe, such as a nozzle), or a key point of a tumor, and combinations thereof.

[0467] In some embodiments, the 3D model is generated in response to a plurality of neural network (or another Al algorithm) outputs from 3D image data from an imaging probe as described herein.

[0468] In some embodiments, a 3D reconstruction of the 3D image data such as imaging data from a probe generates a 3D image of the tissue as described herein. In some embodiments, a processor is configured to instructions to create a plurality of image sections, such as slices, in any suitable dimension. In some embodiments, the image sections, e.g. slices, comprise the transverse image sections that are transverse, e.g, orthogonal, to the handpiece as described herein, for example. In some embodiments, the image sections are generated with a suitable spacing, such as within a range from about 0,5 to 2 mm between sections. For example, an 8 cm long reconstructed 3D image would have from about 40 to about 160 sections, e.g. slices, depending on the spacing.

[0469] In some embodiments, a machine learning model (such as, a trained neural network) is configured to detect and segment a boundary of a tissue structure such as a capsule in each of a plurality of transverse image sections. In some embodiments, the plurality of image sections comprise a series of image sections. In some embodiments, the boundary comprise a boundary of a capsule, such as a boundary of prostate capsule.

[0470] In some embodiments, the resulting series of 3D model segments can then be used to create a 3D mesh, which comprises a boundary of the 3D model, such as a boundary of the prostate capsule.

[0471] In some embodiments, a location of the energy source such as a nozzle on the treatment probe is registered on the 3D image data, and the 3D treatment profile is generated in response the location of the energy source in the 3D image data. In some embodiments the treatment profile comprise a plurality of 3D grid segments, and the 3D treatment grid segments are registered with respect to an inner boundary of the 3D model, such as an inner boundary of a 3D prostate capsule.

[0472] Alternatively or in combination, the plurality of images may comprise a plurality of image slices generated from movement of an imaging probe such as one or more or rotational movement or translational movement, and the boundary determined and interpolated.

[0473] In some embodiments, a plurality of sections of the 3D model are generated from the plurality of neural network outputs (or machine learning model outputs). In some embodiments, each of the plurality of sections is generated from a corresponding neural network output among the plurality of neural network outputs. In some embodiments, the 3D model is generated from the plurality of 3D model sections. In some embodiments, the plurality of 3D model sections are fused, for example with interpolation among adjacent image sections.

[0474] The 3D model can be construed in response to the neural network output with one or more of Implicit Neural Representations (INRs), Diffusion Models, Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANS), or Back-Propagation Techniques, for example, as will be understood by one of ordinary skill in the art.

[0475] At a step 2504, the 3D model is provided on the display for verification by the user.

[0476] In some embodiments the image section of the 3D image provided on the display comprises one or more tissue structures from the 3D model overlaid on the image section and the real time image comprises the one or more structures from the 3D model overlaid on the real time image.

[0477] In some embodiments, the 3D model comprises a boundary overlaid on the image section, and the image section and the boundary correspond to a user input location on the real time image.

[0478] In some embodiments, the one or more tissue structures comprise one or more of a boundary of a tissue structure, a contour of the tissue structure, boundary of an organ, a contour of an organ or a delicate tissue structure,

[0479] In some embodiments, the boundary of the organ comprises a capsule of a prostate and the delicate tissue structure comprises one or more of a verumontanum of the prostate, a trigone region of the bladder, or an external sphincter around a urethra,

[0480] In some embodiments, the image section comprises a corresponding portion of the 3D model overlaid thereon.

[0481] At a step 2506 a user input corresponding to a location of a marker on the real time image is received. In some embodiments, the corresponding portion of the 3D model and the image section correspond to a location of the user input on the real time image. In some embodiments, the location is shown with a marker on the real time image.

[0482] In some embodiments, a first image section shown on the display comprises a first corresponding portion of the 3D model overlaid thereon, and a second image section shown on the display comprises a second corresponding portion of the 3D model overlaid thereon. In some embodiments, the first image section and the second image section are shown sequentially.

[0483] In some embodiments, the first image section corresponds to a marker at a first location on the real time image and the second image section corresponds to a second location of the marker on the real time image. In some embodiments, the first corresponding portion of the 3D model comprises first corresponding a boundary from the 3D model and the second portion of the 3D model comprises a second corresponding boundary of the 3D model. In some embodiments, the marker is configured to slide along the real time image and update the image section in response the marker sliding along the real time image from a first location to a second location,

[0484] At a step 2508, a longitudinal image section from the 3D image data is provided on the display with a corresponding portion of the 3D model overlaid on the longitudinal image section. In some embodiments, the longitudinal image section shown on the display corresponds to a rotational angle of an imaging probe. In some embodiments, a real time longitudinal image is acquired from an imaging probe at the rotational angle with the corresponding portion of the 3D model overlaid on the real time longitudinal image shown on the display.

[0485] At a step 2510 a user input is received indicating acceptance of the 3D model.

[0486] FIG. 26 shows a method 2600 of planning a treatment.

[0487] At a step 2602, a 3D treatment plan is generated in response to the 3D model of the tissue using any of the approaches described herein.

[0488] In some embodiments, the 3D model comprises a 3D boundary, and a 3D treatment profile is generated in response to the 3D boundary. In some embodiments, the 3D treatment profile varies in accordance with the 3D boundary. In some embodiments, the 3D treatment profile comprises a plurality of angular coordinate references and a plurality of radial distances that vary in accordance with the 3D boundary. In some embodiments, the plurality of angular coordinate references and the plurality of radial distances correspond to a plurality of locations along the real time image. In some embodiments, each of the plurality of angular coordinate references and each the plurality of radial distances are configured to treat a longitudinal distance of no more than 10 mm along the 3D treatment profile, for example at a longitudinal distance of no more than 5 mm along the 3D treatment profile. In some embodiments, the longitudinal distance is within a range from 0.5 mm to 10 mm, for example within a range from 0.5 mm to 5 mm.

[0489] At a step 2604, the real time image is provided on a first area of the display, and the image section is simultaneously provided on a second area of the display. In some embodiments, the real time image comprises a corresponding portion of the 3D model and a corresponding portion of the 3D treatment profile. The image section comprises a corresponding portion of the 3D model and a corresponding portion of the 3D treatment profile. In some embodiments, the image section is provided on a display in response to a user input corresponding to a location on the real time image,

[0490] In some embodiments, the 3D model of the tissue and the 3D treatment profile are overlaid in a 3D view provided on the display. In some embodiments, the real time image is provided on a first area of the display with a corresponding portion of the 3D treatment profile overlaid thereon, and the image section is provided on a second area of the display with a corresponding portion of the 3D treatment profile overlaid thereon, and the 3D view provided on a third area of the display.

[0491] At a step 2606 as user input is received, and the 3D model and the 3D treatment profile move together in the 3D view in response to the user input. In some embodiments, the 3D treatment profile shown in the 3D view updates in response to a user input to adjust a portion of the 3D treatment profile shown in the image section. In someembodiments, the 3D treatment profile shown in the 3D view updates in response to a user input to adjust a portion of the 3D treatment profile shown in the image section. In some embodiments., the 3D treatment profile shown in the 3D view updates in response to a user input to adjust a portion of the 3D treatment profile shown in the image section.

[0492] In some embodiments, the 3D view is provided on the display with the real time image and the image section, and the real time image comprises a corresponding portion of the 3D model and a corresponding portion of the 3D treatment profile. The image section comprises a corresponding portion of the 3D model and a corresponding portion of the 3D treatment profile.

[0493] At a step 2608, a user input is received, and the transverse image is provided in response to the user input indicating a location along the real time longitudinal image.

[0494] At a step 2610, a user input is received to add a pass of a treatment energy.

[0495] In some embodiments, a second 3D treatment profile is shown on the image section with the 3D treatment profile in response to a user input to add the pass of a treatment energy. In some embodiments, the first 3D treatment profile and the second 3D treatment profile are shown together on the image section with a plurality of inputs to adjust the first 3D treatment profile and the second 3D treatment profile independently. In some embodiments, the first 3D treatment profile corresponds to a first longitudinal pass of an energy source and the second 3D treatment profile corresponds to a second longitudinal pass of the energy source.

[0496] In some embodiments, the first 3D treatment profile and the second 3D treatment profile are displayed on a plurality of image sections, each of the plurality of image sections corresponding to a location of a marker on the real time image. In some embodiments, said each of the plurality of image sections comprises a corresponding portion of the first 3D treatment profile and a corresponding portion of the second 3D treatment profile in response to the location of the marker.

[0497] At a step 2612, a user input is received to adjust the 3D treatment profile in relation to a delicate tissue structure. In some embodiments, the 3D treatment profile comprises a first 3D treatment profile on a first side of the delicate tissue structure and second treatment profile on a second side of the delicate tissue structure, and the first treatment profile.

[0498] In some embodiments, a boundary corresponding to the delicate tissue structure, a first 3D treatment profile and the second 3D treatment profile are shown together on the image section with a plurality of inputs to adjust the first 3D treatment profile and the second 3D treatment profile independently. In some embodiments, the boundary corresponding to the delicate tissue structure, the first 3D treatment profile and the second 3D treatment profile are displayed on a plurality of image sections, each of the plurality of image sections corresponding to a location of a marker on the real time image. In some embodiments, said each of the plurality of image sections comprises a corresponding portion of the boundary of the delicate tissue structure, the first 3D treatment profile, and the second 3D treatment profile in response to the location of the marker.

[0499] At a step 2614, a user input is received to add a treatment profile.

[0500] In some embodiments, a user input is received to add a treatment profile on an image section corresponding to a location of a marker on the real time image. In some embodiments, a corresponding portion of the 3D treatment profile and the boundary are shown on the image section in response to the location of the maker on thereal time image. In some embodiments, the corresponding portion of the 3D treatment profile is adjusted with one or more markers shown on the image section.

[0501] At a step 2616, a user input is received accepting the treatment plan.

[0502] FIG, 27 shows a method 2700 of assisting a user to align a treatment probe and an imaging probe,

[0503] At a step 2702, 3D ultrasound image data is generated with an imaging probe;

[0504] At a step 2704, a plurality of locations of the treatment probe is determined from the 3D ultrasound image data.

[0505] At a step 2706, an angle between the treatment probe and the imaging probe is determined in response to the plurality of locations of the treatment probe in the 3D image data.

[0506] At a step 2708, an output is provided to a user interface in response to the angle.

[0507] In some embodiments, the plurality of locations is determined with an Al algorithm such as machine learning model, which can be a trained neural network. In some embodiments, the trained neural network is configured to generate an output corresponding to the plurality of locations, the output comprising one or more of a plurality of bounding boxes, a plurality of contours, a plurality of key points, a plurality of segmentations, or a plurality of classifications, In some embodiments, the plurality of key points comprise a plurality of key points of a hyperechoic shadow of the treatment probe. In some embodiments, the plurality of bounding boxes comprise a plurality of bounding boxes of a hyperechoic shadow of the treatment probe. In some embodiments, the output comprises a plurality of key points and a plurality of bounding boxes corresponding to the angle of the treatment probe.

[0508] In some embodiments, the plurality of locations of the treatment probe is determined in response to a plurality of locations of the treatment probe in a plurality of 2D image sections from the 3D image data.

[0509] In some embodiments, the plurality of locations of the treatment probe comprise hyperechoic locations of the treatment probe in the 3D image data.

[0510] In some embodiments, the output to the user interface comprises instructions to a user to move a proximal end of one or more of the treatment probe or the imaging probe in response to the angle. In some embodiments, the instructions indicate a direction to move the proximal end of the one or more of the treatment probe or the imaging probe. In some embodiments, the instructions comprise an animation of the direction to move the proximal end of the one or more of the treatment probe or the imaging probe.

[0511] In some embodiments, the treatment probe comprises a handpiece and the instructions indicate a direction to move the handpiece.

[0512] In some embodiments, the angle comprises an angle between an imaging plane of the imaging probe and a longitudinal axis of the treatment probe. In some embodiments, the angle between the imaging plane and the longitudinal axis corresponds to a coplanarity between the imaging plane and the treatment probe.

[0513] In some embodiments, the angle comprises an angle of inclination of the treatment probe along a longitudinal image from the imaging probe.

[0514] In some embodiments, the angle comprises a three dimensional (3D) vector corresponding to an angle between a longitudinal axis of the imaging probe and a longitudinal axis of the treatment probe. In some embodiments, the 3D vector is generated in response to one or more of a plurality of key points or a plurality of bounding boxes output by a trained neural network.

[0515] FIG, 28 shows a method of automatically aligning a treatment probe and an imaging probe.

[0516] At a step 2802, an angle of a treatment probe is determined in a first transverse image with an Al algorithm such as a machine learning model, which can be a trained neural network.

[0517] At a step 2804, an angle of emission from an energy source is determined in a second transverse image with the trained neural network.

[0518] At a step 2806, a location of an energy source and a location of the treatment probe in a longitudinal image is determined with the trained neural network.

[0519] In some embodiments, the first transverse image is acquired before the second transverse image is acquired, and the longitudinal image is acquired before the second image transverse image.

[0520] At a step 2808, an imaging probe is rotated in response to the angle in first transverse image. In some embodiments, the imaging probe is rotated before the second transverse image is acquired.

[0521] In some embodiments, the wherein imaging probe is rotated to align the treatment probe with an array of the imaging probe.

[0522] In some embodiments, the trained neural network outputs one or more of a key point or a bounding box corresponding to a location of the treatment probe in the first transverse image.

[0523] In some embodiments, the key point corresponds to a location of a hyperechoic shadow of the treatment probe in the first transverse image.

[0524] In some embodiments, the trained neural network outputs one or more of a key point or a bounding box corresponding to the angle of emission from energy source in the second transverse image. In some embodiments, the one or more of the key point or the bounding box corresponds to a location of one or more of an energy stream from the energy source, a waterjet from the energy source, or light from a laser energy source.

[0525] In some embodiments, the trained neural network outputs one or more of a key point or a bounding box corresponding to a location of the energy source and one or more of a key point or a bounding box corresponding to the location of the probe in response to the longitudinal image. In some embodiments, the key point corresponding to the location of the energy source in the longitudinal image corresponds to a location of the energy emitted from source. In some embodiments, the key point corresponding to the location of probe corresponds to a location of the energy source when the energy source has been drawn proximally.

[0526] At a step 2810, a user input is received to confirm alignment of the treatment probe with the imaging probe,

[0527] At a step 2812, a transverse image is acquired after the imaging probe has been rotated, and a user input is received to confirm alignment of the treatment probe and the imaging probe,

[0528] At a step 2814, a user input is received to confirm the location of the energy source and the location of the probe.

[0529] At a step 2816, one or more angular treatment parameters of a treatment plan are offset in response to the angle of emission of the energy source. In some embodiments, the emission angle is compared to an intended angle of emission of the energy source to determine a difference angle, and the treatment plan is offset in accordance with the difference angle. In some embodiments, the treatment plan comprises a maximum sweep angle corresponding to a maximum angular rotation of the treatment probe, the treatment probe is configured to provide a maximum angularrotation, and the difference angle is determined to less than a difference between the maximum angular rotation of the treatment probe and the maximum angular rotation of the treatment probe.Other Variations

[0530] Although some examples refer to tissue, the disclosed implementations more broadly relate to an anatomical region, which can be one or more an organ, tissue, or structure. In some cases, an organ can be the prostate. Any reference herein to a term is intended to encompass one or more instances of the referenced term. For example, any reference to a cancerous tissues encompasses one or more cancerous tissues.

[0531] Any mode, sub-mode or step of any method or user interface disclosed herein can be combined with any mode, sub-mode or step of any other method as disclosed herein. Further one of ordinary skill in the art will recognize many adaptations and variations of the methods disclosed herein. For example, the steps can be performed in any suitable order. Any step can be repeated or deleted. The steps may comprise sub-steps of other steps, for example.

[0532] At least one processor can be configured to perform the step of any method or process disclosed herein. One of ordinary skill in the art will recognize that at least one processor can be configured with instructions to perform one or more of modes, sub-modes, steps of any method or process as disclosed herein. Instructions can be stored in one or more computer-readable media. For example, various of the functionality described herein, in some implementations, is performed as software instructions that are executed by, and / or in response to software instructions being executed by, one or more hardware processors and / or any other suitable computing devices. The software instructions and / or other executable code may be read from one or more computer-readable storage mediums.

[0533] As described herein, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configuration, these computing device(s) may each comprise at least one memory device and at least one physical processor.

[0534] The term “memory” or “memory device,” as used herein, generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device may store, load, and / or maintain one or more of the modules described herein. Examples of memory devices comprise, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives, caches, variations or combinations of one or more of the same, or any other suitable storage memory.

[0535] In addition, the term “processor” or “physical processor,” as used herein, generally refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor may access and / or modify one or more modules stored in the above-described memory device. Examples of physical processors comprise, without limitation, microprocessors, microcontrollers, Central Processing Units (CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcore processors, Application-Specific Integrated Circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, or any other suitable physical processor. The processor may comprise a distributed processor system, e.g. running parallel processors, or a remote processor such as a server, and combinations thereof.

[0536] Although illustrated as separate elements, the method steps described and / or illustrated herein may represent portions of a single application. In addition, in some embodiments one or more of these steps may represent or correspond to one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks, such as the method step.

[0537] In addition, one or more of the devices described herein may transform data, physical devices, and / or representations of physical devices from one form to another. Additionally or alternatively, one or more of the modules recited herein may transform a processor, volatile memory, non-volatile memory, and / or any other portion of a physical computing device from one form of computing device to another form of computing device by executing on the computing device, storing data on the computing device, and / or otherwise interacting with the computing device.

[0538] The term “computer-readable medium,” as used herein, generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media comprise, without limitation, transmission-type media, such as carrier waves, and non-transitory-type media, such as magnetic-storage media (e.g., hard disk drives, tape drives, and floppy disks), optical-storage media (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), and BLU-RAY disks), electronic-storage media (e.g., solid-state drives and flash media), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), and other distribution systems. The computer-readable storage medium(s) is a tangible device that retains and stores data and / or instructions for use by an instruction execution device, e.g., a processor.

[0539] The “user interface” (Ul) as described herein may comprise one or more components of Ul as will be understood by one of ordinary skill in the art in computer science. The Ul may comprise one or more components of a graphical user interface (GUI), and the GUI may comprise one or more input devices such as a display and one or more pointing devices such as touch pads, directional pads, push buttons, track pads, mice, joy sticks configured for a user to input commands to the computing device in association with the display. In some embodiments, the display comprises a touch screen display configured for the user to provide touch inputs to the computing device with the display. Alternatively or in combination, the Ul may comprise voice commands or gesture sensing to allow the user to provide input such as input commands to the computing device. The processor can be configured to provide one or more features such as icons or controls on a display and to receive user input associated with the one or more features such as icons as described herein,

[0540] A person of ordinary skill in the art will recognize that any process or method disclosed herein can be modified in many ways. The process parameters and sequence of the steps described and / or illustrated herein are given by way of example only and can be varied as desired. For example, while the steps illustrated and / or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed.

[0541] The various exemplary methods described and / or illustrated herein may also omit one or more of the steps described or illustrated herein or comprise additional steps in addition to those disclosed. Further, a step of any process or method as disclosed herein can be combined with any one or more steps of any other process or method as disclosed herein.

[0542] The processor as described herein can be configured to perform one or more steps of any method disclosed herein. Alternatively or in combination, the processor can be configured to combine one or more steps of one or more methods as disclosed herein.

[0543] Unless otherwise noted, the terms “connected to” and “coupled to” (and their derivatives), as used in the specification and claims, are to be construed as permitting both direct and indirect (i.e., via other elements or components) connection. In addition, the terms “a” or “an,” as used in the specification and claims, are to be construed as meaning “at least one of” or “one or more of. For example, as used herein the term “a processor” refers to “one or more processors”. For ease of use, the terms “including” and “having” (and their derivatives), as used in the specification and claims, are interchangeable with and shall have the same meaning as the word “comprising.

[0544] It will be understood that although the terms “first,” “second,” “third”, etc. may be used herein to describe various layers, elements, components, regions or sections without referring to any particular order or sequence of events. These terms are merely used to distinguish one layer, element, component, region or section from another layer, element, component, region or section. A first layer, element, component, region or section as described herein could be referred to as a second layer, element, component, region or section without departing from the teachings of the present disclosure.

[0545] Conjunctive language such as the phrase “at least one of X, Y, and Z,” or “at least one of X, Y, or Z,” unless specifically stated otherwise, is to be understood with the context as used in general to convey that an item, term, and / or the like is either X, Y, or Z, or a combination thereof. For example, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to provide a list of elements, the term “of means one, some, or all of the elements in the list. Thus, such conjunctive language is not generally intended to imply that certain implementations require at least one of X, at least one of Y, and at least one of Z to each be present.

[0546] As used herein, characters such as numerals refer to like elements.

[0547] As used herein, the term “e.g.” means for example.

[0548] Language of degree used herein, such as the terms “approximately,” “about,” “generally,” and “substantial ly,” represent a value, amount, or characteristic close to the stated value, amount, or characteristic that still performs a desired function or achieves a desired result. For example, the terms “approximately”, “about”, “generally,” and “substantially,” according to various implementations, refer to an amount that is within less than 10% of, within less than 5% of, within less than 1% of, within less than 0,1% of, and within less than 0,01% of the stated amount.

[0549] Embodiments of the present disclosure have been shown and described as set forth herein and are provided by way of example only. One of ordinary skill in the art will recognize numerous adaptations, changes, variations and substitutions without departing from the scope of the present disclosure. Several alternatives and combinations of the embodiments disclosed herein may be utilized without departing from the scope of the present disclosure and the improvements disclosed herein. Therefore, the scope of the presently disclosed improvements shall be defined solely by the scope of the appended claims and the equivalents thereof.Example Clauses

[0550] The present disclosure includes the following numbered clauses. The features recited in the below example implementations are combinable with additional features disclosed herein. Furthermore, additional inventivecombinations of features are disclosed herein, which are not specifically recited in the below example implementations, and which do not include the same features as the specific implementations below. For sake of brevity, the below example implementations do not identify every inventive aspect of this disclosure. The below example implementations are not intended to identify key features or essential features of any subject matter described herein. Any of the example clauses below, or any features of the example clauses, are combinable with any one or more other example clauses, or features of the example clauses or other features of the present disclosure,

[0551] Clause 1. A method of planning a medical treatment procedure of a prostate, the method comprising: obtaining a three-dimensional (3D) image data of the prostate of a patient with an imaging device; identifying a plurality of anatomical (or tissue) structures in the 3D image data: generating a 3D model of the prostate using the plurality of anatomical structures, the generating comprising determining a 3D boundary of the prostate: generating a treatment profile for a medical procedure to treat the prostate using the 3D model; displaying an image section of the prostate from the 3D image data; and displaying a real time image of the prostate obtained by the imaging device. The method can be performed under control of at least one processor.

[0552] Clause 2. The method of clause 1, wherein the plurality of anatomical structures comprises one or more of a verumontanum, a mid-prostate region, a bladder neck, a median lobe of a prostate, a trigone region, a ureteral orifice, or an external sphincter.

[0553] Clause 3. The method of any one of clauses 1-2, wherein the treatment profile covers an entirety of the prostate.

[0554] Clause 4. The method of any one of clauses 1-3, wherein the treatment profile is displayed as a 3D representation of the treatment profile.

[0555] Clause 5. The method of clause 4, wherein the 3D representation of the treatment profile comprises one or more of an object, a 3D object, a view of a 3D object, a shaded 3D object, a perspective view of an object, a 3D view of an object, contour lines, or a 3D mesh.

[0556] Clause 6. The method of any one of clauses 1-5, wherein the real time image is displayed in a first image display area, the image section is displayed in a second image display area, a representation of the 3D model is displayed in a third image display area, and the treatment profile is displayed in a fourth image display area.

[0557] Clause 7. The method of clause 6, wherein the first image display area, the second image display area, the third image display area and the fourth image display area are different.

[0558] Clause 8. The method of any one of clauses 1-7, further comprising receiving a user input corresponding to a location along the real time image, wherein the image section corresponds to the location, and wherein a corresponding portion of the treatment profile is marked using the location along the real time image.

[0559] Clause 9. The method of clause 8, wherein a corresponding portion of a representation of the 3D model is marked using the location along the real time image.

[0560] Clause 10. The method of any one of clauses 1-9, wherein corresponding portions of the treatment profile and the 3D model are provided on corresponding image sections and on the real time image.

[0561] Clause 11. The method of clause 10, further comprising receiving a user input to confirm that a user has verified the treatment profile in relation to the 3D model.

[0562] Clause 12, The method of any one of clauses 1-11, wherein the treatment profile is generated using the 3D boundary.

[0563] Clause 13. The method of clause 12, wherein the treatment profile varies in accordance with the 3D boundary.

[0564] Clause 14, The method of any one of clauses 1-13, wherein the treatment profile comprises a plurality of angular coordinate references and a plurality of radial distances that vary in accordance with the 3D boundary,

[0565] Clause 15. The method of clause 14, wherein the plurality of angular coordinate references and the plurality of radial distances correspond to a plurality of locations along the real time image.

[0566] Clause 16. The method of any one of clauses 14-15, wherein each of the plurality of angular coordinate references and each the plurality of radial distances are configured to treat an area of no more than 5 mm along a longitudinal distance of the treatment profile.

[0567] Clause 17. The method of clause 16, wherein each of the plurality of angular coordinate references and each the plurality of radial distances are configured to treat an area of no more than 5 mm and no less than 0.5 mm along the longitudinal distance.

[0568] Clause 18. The method of any one of clauses 1-17, wherein the real time image is displayed in a first area and the image section is displayed in a second area, the real time image comprising a corresponding portion of the 3D model and a corresponding portion of the treatment profile, and the image section comprises a corresponding portion of the 3D model and a corresponding portion of the treatment profile.

[0569] Clause 19. The method of clause 18, wherein the image section is displayed in response to receiving a user input corresponding to a location on the real time image.

[0570] Clause 20. The method of any one of clauses 1-19, wherein the 3D model and the treatment profile are overlaid in a 3D view.

[0571] Clause 21. The method of clause 20, wherein the real time image is displayed in a first area with a corresponding portion of the treatment profile overlaid thereon, the image section is displayed in a second area with a corresponding portion of the treatment profile overlaid thereon, and the 3D view is displayed in a third area,

[0572] Clause 22. The method of any one of clauses 20-21, wherein the 3D model and the treatment profile move together in the 3D view in response to a user input,

[0573] Clause 23. The method of any one of clauses 20-22, wherein the treatment profile in the 3D view updates in response to a user input to adjust a portion of the treatment profile shown in the image section.

[0574] Clause 24. The method of any one of clauses 20-23, wherein the treatment profile shown in the 3D view updates in response to a user input to adjust a portion of the treatment profile shown in the image section,

[0575] Clause 25. The method of any one of clauses 20-24, wherein the 3D view is displayed with the real time image and the image section, the real time image comprising a corresponding portion of the 3D model and a corresponding portion of the treatment profile, and the image section comprising a corresponding portion of the 3D model and a corresponding portion of the treatment profile.

[0576] Clause 26. The method of any one of clauses 1-25, wherein: the real time image comprises a real time longitudinal image; the image section comprises a transverse image; and the transverse image is provided in response to a user input indicating a location along the real time longitudinal image.

[0577] Clause 27, The method of any one of clauses 1-26, further comprising receiving a user input to add a pass of a treatment energy and displaying an adjusted treatment profile in the image section along with the treatment profile.

[0578] Clause 28, The method of clause 27, wherein the treatment profile and the adjusted treatment profile are shown together in the image section with a plurality of inputs to adjust the treatment profile and the adjusted treatment profile independently.

[0579] Clause 29, The method of clause 28, wherein the treatment profile corresponds to a first longitudinal pass of an energy source and the adjusted treatment profile corresponds to a second longitudinal pass of the energy source.

[0580] Clause 30. The method of any one of clauses 28-29, wherein the treatment profile and the adjusted treatment profile are displayed in a plurality of image sections, each of the plurality of image sections corresponding to a location of a marker on the real time image.

[0581] Clause 31. The method of clause 30, wherein each of the plurality of image sections comprises a corresponding portion of the treatment profile and a corresponding portion of the adjusted treatment profile based on the location of the marker.

[0582] Clause 32. The method of any one of clauses 1-31, further comprising receiving a user input to adjust the treatment profile to avoid an anatomical structure.

[0583] Clause 33. The method of clause 32, wherein the treatment profile comprises a first treatment profile on a first side of the prostate and a s...

Claims

WHAT IS CLAIMED IS:

1. A method of planning a medical treatment procedure of a prostate, the method comprising: obtaining a three-dimensional (3D) image data of the prostate of a patient with an imaging device; identifying a plurality of anatomical structures in the 3D image data;generating a 3D model of the prostate using the plurality of anatomical structures, the generating comprising determining a 3D boundary of the prostate;generating a treatment profile for a medical procedure to treat the prostate using the 3D model; displaying an image section of the prostate from the 3D image data; anddisplaying a real time image of the prostate from the imaging device.

2. The method of claim 1, wherein the plurality of anatomical structures comprises one or more of a verumontanum, a mid-prostate region, a bladder neck, a median lobe of a prostate, a trigone region, a ureteral orifice, or an external sphincter.

3. The method of any one of claims 1 -2, wherein the treatment profile covers an entirety of the prostate.

4. The method of any one of claims 1-3, wherein the treatment profile is displayed as a 3D representation of the treatment profile.

5. The method of claim 4, wherein the 3D representation of the treatment profile comprises one or more of an object, a 3D object, a view of a 3D object, a shaded 3D object, a perspective view of an object, a 3D view of an object, contour lines, or a 3D mesh.

6. The method of any one of claims 1-5, wherein the real time image is displayed in a first image display area, the image section is displayed in a second image display area, a representation of the 3D model is displayed in a third image display area, and the treatment profile is displayed in a fourth image display area.

7. The method of claim 6, wherein the first image display area, the second image display area, the third image display area and the fourth image display area are different.

8. The method of any one of claims 1-7, further comprising receiving a user input corresponding to a location along the real time image and wherein the image section corresponds to the location and wherein a corresponding portion of the treatment profile is marked in response to the location along the real time image.

9. The method of claim 8, wherein a corresponding portion of a representation of the 3D model is marked in response to the location along the real time image.

10. The method of any one of claims 1-9, wherein corresponding portions of the treatment profile and the 3D model are provided on corresponding image sections and on the real time image.

11. The method of claim 10, further comprising receiving a user input to confirm that a user has verified the treatment profile in relation to the 3D model.

12. The method of any one of claims 1-11, wherein the treatment profile is generated using the 3D boundary.

13. The method of claim 12, wherein the treatment profile varies in accordance with the 3D boundary.

14. The method of any one of claims 1-13, wherein the treatment profile comprises a plurality of angular coordinate references and a plurality of radial distances that vary in accordance with the 3D boundary.

15. The method of claim 14, wherein the plurality of angular coordinate references and the plurality of radial distances correspond to a plurality of locations along the real time image,16. The method of any one of claims 14-15, wherein each of the plurality of angular coordinate references and each the plurality of radial distances are configured to treat an area of no more than 10 mm along a longitudinal distance of the treatment profile,17. The method of claim 16, wherein each of the plurality of angular coordinate references and each the plurality of radial distances are configured to treat an area of no more than 10 mm and no less than 0.5 mm along the longitudinal distance.

18. The method of any one of claims 1-17, wherein the real time image is displayed in a first area and the image section is displayed in a second area, the real time image comprising a corresponding portion of the 3D model and a corresponding portion of the treatment profile, and the image section comprises a corresponding portion of the 3D model and a corresponding portion of the treatment profile.

19. The method of claim 18, wherein the image section is displayed in response to receiving a user input corresponding to a location on the real time image.

20. The method of any one of claims 1-19, wherein the 3D model and the treatment profile are overlaid in a 3D view.

21. The method of claim 20, wherein the real time image is displayed in a first area with a corresponding portion of the treatment profile overlaid thereon, the image section is displayed in a second area with a corresponding portion of the treatment profile overlaid thereon, and the 3D view is displayed in a third area.

22. The method of any one of claims 20-21, wherein the 3D model and the treatment profile move together in the 3D view in response to a user input.

23. The method of any one of claims 20-22, wherein the treatment profile in the 3D view updates in response to a user input to adjust a portion of the treatment profile shown in the image section.

24. The method of any one of claims 20-23, wherein the treatment profile shown in the 3D view updates in response to a user input to adjust a portion of the treatment profile shown in the image section.

25. The method of any one of claims 20-24, wherein the 3D view is displayed with the real time image and the image section, the real time image comprising a corresponding portion of the 3D model and a corresponding portion of the treatment profile, and the image section comprising a corresponding portion of the 3D model and a corresponding portion of the treatment profile,26. The method of any one of claims 1-25, wherein:the real time image comprises a real time longitudinal image;the image section comprises a transverse image; andthe transverse image is provided in response to a user input indicating a location along the real time longitudinal image.

27. The method of any one of claims 1-26, further comprising receiving a user input to add a pass of a treatment energy and displaying an adjusted treatment profile in the image section along with the treatment profile.

28. The method of claim 27, wherein the treatment profile and the adjusted treatment profile are shown together in the image section with a plurality of inputs to adjust the treatment profile and the adjusted treatment profile independently.

29. The method of claim 28, wherein the treatment profile corresponds to a first longitudinal pass of an energy source and the adjusted treatment profile corresponds to a second longitudinal pass of the energy source, 30. The method of any one of claims 28-29, wherein the treatment profile and the adjusted treatment profile are displayed in a plurality of image sections, each of the plurality of image sections corresponding to a location of a marker on the real time image.

31. The method of claim 30, wherein each of the plurality of image sections comprises a corresponding portion of the treatment profile and a corresponding portion of the adjusted treatment profile based on the location of the marker.

32. The method of any one of claims 1-31, further comprising receiving a user input to adjust the treatment profile to avoid an anatomical structure.

33. The method of claim 32, wherein the treatment profile comprises a first treatment profile on a first side of the prostate and a second treatment profile on a second side of the prostate.

34. The method of claim 33, wherein a boundary corresponding to the prostate, a first treatment profile, and the second treatment profile are shown together in the image section with a plurality of inputs to adjust the first treatment profile and the second treatment profile independently.

35. The method of claim 34, wherein the boundary, the first treatment profile, and the second treatment profile are displayed in a plurality of image sections, each of the plurality of image sections corresponding to a location of a marker on the real time image.

36. The method of claim 35, wherein each of the plurality of image sections comprises a corresponding portion of the boundary, the first treatment profile, and the second treatment profile in response to the location of the marker.

37. The method of any one of claims 1-36, further comprising receiving a user input to add the treatment profile on an image section corresponding to a location of a marker on the real time image,38. The method of claim 37, wherein a corresponding portion of the treatment profile and a boundary corresponding to the prostate are shown in the image section based on the location of the marker on the real time image,39. The method of claim 38, wherein the corresponding portion of the treatment profile is adjusted with one or more markers shown in the image section,40. A non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to preform the method of any one of the preceding claims.

41. A medical treatment system comprising at least one processor configured to execute the method of any one of the preceding claims.

Images