Water jet cutter ar display method, apparatus and system based on ultrasonic image

Abstract

A water jet cutter AR display method, apparatus and system based on an ultrasonic image. The display method comprises the following steps: acquiring a template image of a surgical instrument including a water jet cutter head, and marking a plurality of first key feature points in the template image, wherein the first key feature points comprise a nozzle orifice feature point; acquiring an ultrasonic image of the surgical instrument including the water jet cutter head, and identifying a plurality of second key feature points of the water jet cutter head in the ultrasonic image, wherein the second key feature points correspond to the first key feature points, and the plurality of second key feature points also comprise a nozzle orifice feature point; and on the basis of position coordinates of the plurality of first key feature points in the template image and position coordinates of the plurality of second key feature points in the ultrasonic image, forming a water jet cutter AR display image based on the ultrasonic image. By providing intuitive visual assistance, the perception and operational accuracy of a physician during a surgical procedure can be significantly enhanced, thereby reducing surgical risks, improving medical effects, and comprehensively improving the precision and safety of the surgery.

Description

A water jet AR display method, device, and system based on ultrasound images

[0001] This application is based on and enjoys priority to Chinese Patent Application No. 202411749024.7 (filed on December 2, 2024). This application incorporates the entire contents of that application by reference. Technical Field

[0002] This application relates to a water jet AR display method, apparatus, and system based on ultrasonic images. Background Technology

[0003] In surgical settings, due to the real-time nature and convenience of ultrasound imaging, it is often used to assist in surgical navigation or monitor surgical progress. The ultrasound probe is supported by a robotic arm, support, or similar mechanism to provide real-time image information. In minimally invasive surgeries or similar procedures using water jets, such as inserting a water jet through the urethra into the patient's prostate tissue, real-time ultrasound imaging is necessary for path planning and navigation to improve surgical accuracy. However, ultrasound images sometimes fail to provide sufficiently clear and accurate displays. Accurately interpreting ultrasound image information usually requires considerable experience or special attention. If the operator lacks sufficient expertise in ultrasound imaging, it is difficult to accurately understand the information in the ultrasound image. Even experienced surgeons require special attention when interpreting ultrasound images in complex surgical settings, especially in water jet surgeries. The unique operating environment of the water jet and surrounding tissues further increases the difficulty of accurately interpreting ultrasound image information. Slower image reading speed and potential errors can introduce uncertainty into path planning and surgical procedures, which is undesirable.

[0004] In addition to surgical scenarios, in non-surgical scenarios, when operators place the water jet device into an environment that cannot be directly seen, they may also need to use ultrasound images to obtain environmental information. In order to improve the accuracy of operation and improve the recognition of ultrasound image information to further improve the user experience, higher requirements are placed on the acquisition and display of ultrasound image information. Summary of the Invention

[0005] To address the aforementioned problems, this application provides a water jet AR display method, apparatus, and system based on ultrasound images. The provided water jet AR display method based on ultrasound images includes the following steps:

[0006] Obtain a template image of a surgical instrument containing a water jet tip, and mark multiple first key feature points in the template image, including the blade hole feature point;

[0007] Acquire an ultrasound image of a surgical instrument containing a water jet tip, and identify multiple second key feature points in the ultrasound image, the second key feature points corresponding to the first key feature points, and the multiple second key feature points also include the blade hole feature points;

[0008] Based on the position coordinates of the plurality of first key feature points in the template image and the position coordinates of the plurality of second key feature points in the ultrasound image, the water jet AR display image based on the ultrasound image is formed.

[0009] Furthermore, the step of acquiring an ultrasonic image of a surgical instrument containing a water jet tip and identifying the cutter hole feature point among multiple second key feature points in the ultrasonic image includes: controlling the water jet to perform a water spraying action, the high-pressure jet being sprayed forming a rapidly fluctuating air mass in a liquid immersion environment, and performing jet feature identification on the contour features of the air mass based on the detected air mass fluctuation area in the ultrasonic image to generate a first auxiliary line for locating the cutter hole feature point.

[0010] Furthermore, the step of acquiring an ultrasound image of a surgical instrument containing a water jet tip and identifying the cutter hole feature points among multiple second key feature points in the ultrasound image also includes: using the static features of the water jet tip in the ultrasound image to perform edge extraction and generate a second auxiliary line for locating the cutter hole feature points.

[0011] Furthermore, based on the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction, the accurate location of the knife hole in the ultrasound image is determined.

[0012] Furthermore, the step of acquiring an ultrasound image of a surgical instrument containing a water jet head and identifying multiple second key feature points in the ultrasound image further includes: translating the water jet head along the main axis of the water jet head a certain distance and performing a water spraying action to obtain the corresponding position point; and / or, obtaining the initial position point of the endoscope included in the surgical instrument and driving the endoscope to retract to the termination position to obtain the endoscope stop position point; and / or, obtaining multiple position points on the tube wall contour of the main axis of the water jet head.

[0013] Furthermore, the fitting result of the tube wall profile is determined to be reliable based on the reliability judgment threshold. When the judgment result is deemed unreliable, the water jet head is controlled to translate a certain distance along the main axis of the water jet head and perform a water spraying action to obtain the corresponding position point; and / or, the initial position point of the endoscope included in the surgical instrument is obtained, and the endoscope is driven to retract to the termination position to obtain the endoscope stop position point.

[0014] This application also proposes a water jet AR display device based on ultrasonic images, comprising the following modules:

[0015] The template image acquisition module is used to acquire a template image of a surgical instrument containing a water jet tip, and to mark multiple first key feature points in the template image, wherein the first key feature points include a blade hole feature point;

[0016] An ultrasound image acquisition module is used to acquire ultrasound images of surgical instruments containing water jet heads and identify multiple second key feature points in the ultrasound images, wherein the second key feature points correspond to the first key feature points.

[0017] An AR image generation module is used to form the water jet AR display image based on the ultrasound image, based on the position coordinates of the plurality of first key feature points in the template image and the position coordinates of the plurality of second key feature points in the ultrasound image.

[0018] Furthermore, the ultrasonic image acquisition module also controls the water jet to perform a water spraying action. The high-pressure jet sprayed out forms a rapidly fluctuating air mass in the liquid immersion environment. Based on the detected air mass fluctuation area in the ultrasonic image, the contour features of the air mass are identified to generate a first auxiliary line for locating the knife hole feature points.

[0019] Furthermore, the ultrasonic image acquisition module also utilizes the static features of the water jet cutter head in the ultrasonic image to extract edges and generate a second auxiliary line for locating the cutter hole feature points.

[0020] Furthermore, the ultrasound image acquisition module also determines the accurate location of the knife hole in the ultrasound image based on the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction.

[0021] Furthermore, the ultrasound image acquisition module also translates the water jet head along the main axis of the water jet head a certain distance and performs a water spraying action to obtain the corresponding position point; and / or, obtains the initial position point of the endoscope included in the surgical instrument, and drives the endoscope to retract to the termination position to obtain the endoscope stop position point; and / or, obtains multiple position points on the tube wall contour of the main axis of the water jet head.

[0022] Furthermore, the ultrasound image acquisition module also determines whether the fitting result of the tube wall contour is reliable based on a reliability judgment threshold. When the judgment result is deemed unreliable, it controls the water jet head to translate a certain distance along the main axis of the water jet head and performs a water spraying action to obtain the corresponding position point; and / or, it obtains the initial position point of the endoscope included in the surgical instrument and drives the endoscope to retract to the termination position to obtain the endoscope stop position point.

[0023] This application also proposes a water jet AR display system based on ultrasound images, comprising: a surgical instrument suitable for insertion into the human body for surgical procedures, an ultrasound imaging device, a processor, a memory, and a display device. The surgical instrument includes a water jet tip and an endoscope, with the main axis of the water jet tip and the main axis of the endoscope being substantially parallel. The ultrasound imaging device is positioned below the surgical instrument. The processor is used to execute a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the method described above. The display device is used to display the water jet AR display image based on the ultrasound images.

[0024] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: This application makes full use of the advantages of real-time ultrasound images in medical scenarios, and provides water jet AR display based on real-time ultrasound image information. It can superimpose the water jet structure image onto the ultrasound image according to the real pose and proportion of the water jet structure in the ultrasound image, and realize the visualization of the position of the water jet hole in the surgical instrument containing the water jet when it moves along the main axis of the water jet head, and the position of the endoscope lens when the endoscope moves along the main axis of the endoscope body in the ultrasound image. It can provide doctors or operators with effective and intuitive visual assistance, thereby significantly improving the doctor's perception ability and operation accuracy during the operation or the operator's operation, reducing surgical risks or operation errors, improving medical effects or operation success rate, and comprehensively improving the accuracy and safety of surgery or operation. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 is a schematic diagram of a surgical instrument including a water jet according to an embodiment of this application.

[0027] Figure 2 is a schematic diagram of a surgical instrument including a water jet according to an embodiment of this application.

[0028] Figure 3(a) is a schematic diagram of the ultrasonic image of the water jet cutter head under ideal conditions.

[0029] Figure 3(b) is a schematic diagram of the ultrasonic image of the water jet tip in an actual surgical scenario.

[0030] Figure 4 is a schematic diagram illustrating the principle of feature recognition of water jets from a nozzle according to an embodiment of this application.

[0031] Figure 5 is a schematic diagram showing the distortion of the main axis of the water jet cutter head in the ultrasonic image.

[0032] Figure 6 is a schematic diagram of a water jet AR display interface based on ultrasonic images according to an embodiment of this application.

[0033] Figure 7 is a schematic diagram of an interface that provides AR display of the position and working status of the water jet based on ultrasound images during a water jet surgery according to an embodiment of this application.

[0034] Figure 8 is a flowchart of the steps according to an embodiment of a method of this application.

[0035] Figure 9 is a schematic diagram of the modules included in an apparatus according to an embodiment of this application.

[0036] Figure 10 is a schematic diagram of a system structure according to an embodiment of this application.

[0037] Reference numerals: 1. Water jet cutter head; 10. Water jet cutter head tip; 10a. Cutting hole; 11. Water jet cutter head main shaft; 110. Water jet adapter; 2. Ultrasonic probe; 21. Ultrasonic probe main shaft; 210. Ultrasonic adapter; 3. Endoscope; 30. Endoscope tip; 31. Endoscope main shaft; 4. Sheath; 40. Sheath tip; 41. Sheath main shaft; 5. Shaft sleeve; 51. Telescopic main shaft Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings.

[0039] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0040] As shown in Figure 1, after the water jet ablation head 1 enters the target tissue area, it sprays a high-pressure water jet through the blade hole 10a at the front end of the head to perform the surgery. The surgical process is monitored or guided by an ultrasound probe. In the case of benign prostatic hyperplasia ablation, the ultrasound probe 2 is inserted transrectally below the target tissue area, with the imaging element of the ultrasound probe 2 facing upwards, roughly towards the water jet ablation head. This allows the ultrasound probe 2 to acquire images of the water jet ablation head 1 and the target prostate tissue area, enabling real-time monitoring of the surgical process. Specifically, the ultrasound probe 2 can be, for example, a dual-plane ultrasound probe to acquire cross-sectional and sagittal images of the water jet ablation head 1; or it can be a three-dimensional ultrasound probe to directly acquire three-dimensional images of the water jet ablation head 1 and the target tissue area. After the water jet tip 1 has fully entered the target tissue area, at least using sagittal images, and preferably combined with cross-sectional images, information reflecting the real-time movement of the water jet A can be obtained and displayed on a display device, providing doctors or operators with basic data for surgical planning and providing monitoring data for the surgical process.

[0041] To ensure that sagittal ultrasound images capture real-time motion information of the high-pressure water jet from the water jet head 1 during surgery, the imaging element of the ultrasound probe 2 should be positioned below and facing the water jet head 1. The ultrasound probe 2 includes an ultrasound probe body axis 21, and the water jet head includes a water jet head body axis 11. The ultrasound probe body axis 21 and the water jet head body axis 11 are held in their working positions by an ultrasound adapter 210 located proximally. Ideally, the ultrasound probe body axis 21 and the water jet head body axis 11 should remain parallel to each other during operation. The ultrasound probe held by the ultrasound adapter 210 can translate along the extension direction of the body axis 21, and the water jet head held by the water jet adapter 110 can translate along the extension direction of the water jet head body axis 11. The translational directions of the two should be as parallel as possible.

[0042] In some embodiments of this application, an endoscope 3 is also included when the water jet head 1 is inserted into the target tissue area. The endoscope 3 includes an endoscope body axis 31. The endoscope 3 and the water jet head 1 are inserted into the target tissue area through the same channel or different but parallel channels. During operation, the endoscope body axis 31 is substantially parallel to the ultrasound probe body axis 21 and the water jet body axis 11.

[0043] Figure 2 is a schematic diagram of a surgical instrument including a water jet according to an embodiment of this application. In this embodiment, the surgical instrument includes a combination of a sheath, a water jet head, and an endoscope. As shown in the figure, the front end of the surgical instrument includes a bushing 5, and the bushing 5 includes a retractable main shaft 51. The retractable main shaft 51 has a telescopic channel for the water jet head 1 and a telescopic channel for the endoscope 3. In the state shown in Figure 2, both the water jet head 1 and the endoscope 3 are retracted into the retractable main shaft 51. The tip 10 of the water jet head and the tip 30 of the endoscope are both located within the telescopic channel of the retractable main shaft 51. Although not shown, those skilled in the art should understand that the water jet head 1 and the endoscope 3 can move independently or jointly within the telescopic channel of the retractable main shaft 51. The basic movement of the water jet head 1 and the endoscope 3 includes translation along the direction defined by the channel. Both the water jet head 1 and the endoscope 3 are main bodies with a certain rigidity. The tip 10 of the water jet head and the tip 30 of the endoscope can both extend forward from the tip of the bushing 5. Preferably, the front end of the surgical instrument further includes a sheath 4, which includes a main shaft 41 and a tip portion 40.

[0044] The endoscope and water jet are components whose movement can be controlled independently. For example, the movement control process can be as follows: First, the sheath 4, water jet head 1, and endoscope 3 are inserted into the target operating area as a whole. After positioning, they are ready for subsequent planning or surgery; this position is defined as the original position. The original position means the position when the sheath 4, water jet head 1, and endoscope 3 are assembled and inserted into the target tissue area, for example, in the form of a closure device. In this position, the sheath 4, water jet head 1, and endoscope 3 maintain a predetermined assembly relationship and are ready to begin subsequent operating steps. Afterward, the sheath 4 remains in its original position (the sheath original position), the endoscope 3 is withdrawn from the endoscope original position to the predetermined observation position, and the water jet head 1 is withdrawn from the water jet head original position to a predetermined initial position near the endoscope 3.

[0045] Ideally, the image of the water jet cutter head acquired by the ultrasonic probe 2 located below the water jet cutter head 1 should be as shown in Figure 3(a). Without considering any interference, the position of the top part 40 of the sheath 4, the position of the top part 10 of the water jet cutter head 1, and / or the position of the top part 30 of the endoscope 3 can be clearly seen from the ultrasonic image.

[0046] However, due to limitations in the imaging level of ultrasound images, and because the sheath main axis 41, the water jet head main axis 11, and the endoscope main axis 31 are all roughly horizontal, and their channels are parallel and very close, they are close to each other in the ultrasound image. The real-time image obtained by the ultrasound probe is prone to artifacts, which are further interfered with by the image information of the tissue area where the surgical instruments are inserted. In particular, during water jet surgery, due to artifacts or the water jet head main axis not being completely aligned with the sagittal plane, the end of the water jet head 1 may have an elongated ghost image or the straight line of the water jet head main axis may be deformed in the ultrasound image. After the endoscope 3 is withdrawn to the predetermined observation position, the image between the endoscope 3 and the sheath 4 may also show broken lines. All of the above situations make the image information of the water jet head 1 and the endoscope 3 seen from the sagittal ultrasound image relatively blurry, as shown in Figure 3(b). In this situation, even experienced doctors may find it difficult to quickly and accurately identify key information such as the location of the sheath tip 40, the water jet tip 10, and / or the endoscope tip 30 in the image. Consequently, it is difficult to effectively utilize real-time ultrasound images for surgical guidance and monitoring during the operation, and it is also difficult to achieve precise intraoperative image planning.

[0047] Therefore, it is necessary to provide a solution that can provide clear, accurate, and vivid real-time image display of surgical instruments containing water jets, provided at least through sagittal ultrasound images or through three-dimensional ultrasound images; the visualization provided allows doctors or users to clearly observe the accurate position and shape information of each component in the surgical instruments containing water jets and to plan or decide on subsequent operations accordingly, thus implementing more accurate planning or surgery.

[0048] Specifically, referring to Figure 8, the water jet AR display method based on ultrasound images provided in this application includes the following steps:

[0049] Step S1: Obtain a template image of the surgical instrument containing the water jet tip, mark multiple first key feature points in the template image, and save the marked template image in the computer storage unit.

[0050] As mentioned earlier, surgical instruments containing water jet heads include at least a water jet head, and preferably, also include a sheath and / or an endoscope. During use, the sheath and / or endoscope can be assembled with the water jet head to facilitate insertion into the target tissue area, and after insertion, at least the endoscope and the water jet head can move independently. Therefore, the template images of the surgical instruments containing water jet heads should at least include: a template image of the overall surgical instrument consisting of the sheath, water jet head, and endoscope assembled together; a template image of the water jet head; a template image of the endoscope; and a template image of the sheath. That is, corresponding template images are obtained for the entire instrument and each component.

[0051] Taking the acquisition of a water jet cutter head template image as an example, firstly, a 3D model of the water jet cutter head is established. A 3D scanning device is used to perform a 3D scan of the water jet cutter head to obtain its detailed geometry, ensuring high accuracy and completeness of the scan data for subsequent processing. 3D modeling software is used to process the scan data, removing noise and redundant data to ensure the smoothness and accuracy of the model. Detail corrections are performed to ensure the model matches the actual water jet cutter head. The processed scan data is then converted into a standard 3D image. In some embodiments, the 3D image can be directly used as the template image; in other embodiments, the template image of the water jet cutter head is two-dimensional, and the 3D image is projected onto a 2D plane to generate a multi-view 2D image. Preferably, structured light scanning is used for 3D scanning, which can quickly capture high-resolution detailed images at a relatively low cost. It should be noted that the methods for acquiring overall surgical instrument template images, endoscope template images, and sheath template images are similar and will not be elaborated here.

[0052] After obtaining a template image of the surgical instrument containing the water jet tip, key feature points need to be marked on the template image. This can be achieved by the user performing operations through a user interface or by software programs marking based on the recognition results of the feature points. The key feature points that need to be marked include at least the location of the cutter hole 10a.

[0053] In a preferred embodiment, the first key feature point for which the mark is added also includes the outer contour feature points of the water jet tip 10 and / or the sheath tip 40 and / or the endoscope tip 30, such as the position point or group of points indicating the foremost point of the outer contour.

[0054] In a further preferred embodiment, the first key feature point for adding the mark also includes multiple feature points that can define the outer contour lines of the waterjet head body axis 11 and / or the sheath body axis 41 and / or the endoscope body axis 31.

[0055] Step S2: Acquire an ultrasound image of the surgical instrument containing the water jet tip and identify multiple second key feature points in the ultrasound image.

[0056] The surgical instrument containing the water jet head and the ultrasound probe 2 are placed in their original positions, and ultrasound images of the surgical instrument containing the water jet head are acquired. As mentioned above, the original position means that the main axis 21 of the ultrasound probe is approximately parallel to the main axis 11 of the water jet head, and the ultrasound probe 2 is located below the water jet head 1, with the imaging element of the ultrasound probe facing the water jet head 1, thereby enabling the acquisition of real-time ultrasound image information of the water jet head 1.

[0057] Furthermore, multiple second key feature points in the ultrasound image are identified, which correspond to the first key feature points.

[0058] The second key feature point identified in the ultrasound image includes at least the position P0(x0, y0) of the cutter hole 10a in the ultrasound image when the water jet cutter head 1 is in its original position.

[0059] When the first key feature point added in step S1 also includes the outer contour feature points of the water jet tip 10 and / or the sheath tip 40 and / or the endoscope tip 30, the corresponding outer contour feature points characterizing the water jet tip 10 and / or the sheath tip 40 and / or the endoscope tip 30 are identified in the ultrasound image.

[0060] When the first key feature point marked in step S1 also includes multiple feature points that can define the outer contour lines of the water jet cutter head main axis 11 and / or sheath main axis 41 and / or endoscope main axis 31, then the multiple feature points that define the outer contour lines of the water jet cutter head main axis 11 and / or sheath main axis 41 and / or endoscope main axis 31 are identified in the ultrasound image accordingly.

[0061] In order to accurately identify the location of the knife hole (nozzle) 10a in the ultrasound image, the dynamic characteristics of the nozzle jet can be used for jet feature recognition, and / or the static characteristics of the nozzle in the ultrasound image can be used for edge extraction, so as to accurately locate the knife hole 10a in the ultrasound image. These are described in detail below.

[0062] Jet feature recognition

[0063] As shown in Figure 4, the nozzle jet appears as a rapidly fluctuating air mass in the image. The motion of these air masses can be used to infer the nozzle's position. Specifically:

[0064] The water jet is controlled to perform water spraying action. The high-pressure jet formed a rapidly fluctuating air mass in the liquid immersion environment. The regular changes in the growth and collapse of these air masses are related to the Strouhal number of the water jet. In the ultrasonic image, they appear as rapidly fluctuating white air mass regions, which are significantly distinguishable from the surrounding black regions. This allows for jet feature identification and helps determine the location of the nozzle.

[0065] First, after the water jet performs the water spraying action, an ultrasonic image is acquired. The acquired ultrasonic image is preprocessed to enhance image contrast and clarity. White gas wave fluctuation regions in the image are detected using frame difference or background subtraction methods. Based on the detected white gas wave fluctuation regions, a first auxiliary line is generated for localization. Specifically, white gas wave fluctuation regions are identified and marked in the image. Within the detected white gas wave fluctuation regions, an auxiliary line is generated that roughly bisects the white region. A yellow dashed line can be added to the image. This auxiliary line should bisect the gas wave fluctuation region as evenly as possible. This auxiliary line is typically roughly vertical in the ultrasonic image, and the blade hole lies on this auxiliary line.

[0066] Static edge extraction

[0067] As shown in Figure 5, the static features of the water jet nozzle in the ultrasonic image are used to determine another auxiliary line where the nozzle is located by edge extraction method.

[0068] Specifically, edge extraction is performed on the image of the waterjet head, using algorithms such as the Canny edge detection algorithm or the Sobel operator to enhance the contour of the waterjet head and extract edges. Preferably, multiple images taken over a period of time (e.g., during a microjet flow) are subjected to temporal averaging to enhance the stable contour features of the waterjet head.

[0069] Since the waterjet cutter head 1 is typically made of high acoustic impedance stainless steel, it appears as a bright, long straight line in the ultrasonic image. Based on the detected edge contour, a roughly horizontal auxiliary line is fitted using methods such as Hough transform. Because the long straight line of the waterjet cutter head's main axis 11 in the ultrasonic image may be tilted or affected by artifacts, the roughly horizontal auxiliary line fitted here should not be too long; its width should be equivalent to or slightly longer than the width of the water jet airflow.

[0070] Furthermore, the nozzle position is precisely located by cross-analyzing the results of jet feature recognition and edge extraction. Specifically, the first straight auxiliary line generated by jet feature recognition is intersected with the second straight auxiliary line fitted by static edge extraction. The intersection of the two straight auxiliary lines is the nozzle position, thus determining the nozzle's position coordinates in the ultrasound image.

[0071] By combining dynamic jet feature recognition and static image analysis techniques through the aforementioned steps, the position coordinates of the nozzle (i.e., water jet orifice 10a) in the ultrasonic image can be effectively identified and tracked. When the water jet head is in its original position P0, the pixel position coordinates of the corresponding orifice feature point in the ultrasonic image are imageP0(x0, y0). By accurately locating the nozzle position in the real-time ultrasonic image, a reliable basis for AR imaging is provided.

[0072] Furthermore, the location coordinates of other second key feature points in the ultrasound image are obtained through the following steps:

[0073] In one embodiment, the second key feature point is determined based on the movement information of the waterjet cutter head.

[0074] The waterjet head 1 is driven by the waterjet adapter 110 to move a predetermined distance S from its original position along the direction of the waterjet head body axis 11 to the second position P2.

[0075] The predetermined distance S can be determined by the operating parameters of the waterjet drive mechanism, a position sensor, or other physical distance detection methods. For example, the movement distance can be detected by using a grating ruler, magnetic grating ruler, piezoelectric couple, etc., or the movement distance can be confirmed based on the change information of distance scale parameters on the movement path of the waterjet head.

[0076] At the second position P2, the water jet is controlled to perform the water spraying action again. Using similar steps as described above, the dynamic characteristics of the nozzle jet are used for jet feature recognition and / or the static characteristics of the nozzle in the ultrasonic image are used for edge extraction. The position coordinates imageP2(x2, y2) of the second position P2 where the blade hole is located when the water jet performs the water spraying action again are identified in the ultrasonic image.

[0077] After determining the original position coordinates imageP0(x0, y0) and the second position coordinates imageP2(x2, y2) in the ultrasound image, a line can be established between the two position points to determine the extension direction or tilt angle of the water jet head body axis 11 on the ultrasound image for subsequent AR display.

[0078] It should be noted that the water jet nozzle can move multiple times, not just once, to identify the position coordinates of the nozzle in the ultrasound image each time, thus forming multiple secondary key feature points. While acquiring multiple secondary key feature points may mean a decrease in computational speed, the system's computational accuracy will be improved. Furthermore, since the channel established by the sheath is fixed, and this step is performed before surgery, multiple secondary key feature points will provide more precise calibration of the water jet's movement direction, making the efficiency loss worthwhile.

[0079] In an alternative embodiment, the second key feature point is determined based on the movement information of the endoscope.

[0080] Since the nozzle, as part of the water jet head 10, is placed within the retractable main body axis 51 of the surgical instrument along with the endoscope 3 in its original position, and the physical distance between them is determined by the assembly relationship, the original position P1 of the endoscope 3 lens can be determined based on the predetermined assembly relationship and the original position P0 of the nozzle, and the original position imageP1(x1,y1) of the endoscope 3 lens in the ultrasound image can also be determined.

[0081] Next, the endoscope 3 is controlled to retract along the telescopic channel. The retraction distance L of the endoscope is detected by the endoscope's actuator (not shown) or position sensor (not shown). Combined with the retraction direction of the endoscope, the position P3(x3, y3) of its lens when the endoscope stops moving can be determined. Specifically, a difference image can be obtained by performing pixel-level subtraction on two ultrasound images before and after the endoscope's movement. Based on this difference image, the position where the endoscope stops moving in the ultrasound image, imageP3(x3, y3), can be determined. The direction of the endoscope's movement can be determined by connecting imageP1(x1, y1) and imageP3(x3, y3) in the ultrasound image. This direction also represents the extension direction of the endoscope's main axis and is parallel to the extension direction of the water jet head's main axis. The advantage of this method is that only one water spraying action is required, and the nozzle position can be determined without moving the water jet head.

[0082] The endoscope can be moved multiple times, not just once, to identify its position coordinates in the ultrasound image each time, thus forming multiple secondary key feature points. Similarly, while acquiring multiple secondary key feature points of the endoscope may mean a decrease in computational speed, the system's computational accuracy will be improved. Furthermore, since the channel established by the sheath is fixed, and this step is performed before surgery, multiple secondary key feature points will provide more precise calibration of the water jet's movement direction, making the efficiency loss worthwhile.

[0083] In an alternative embodiment, a second key feature point may also be determined based on the tube wall profile fitting of the front end of the surgical instrument containing the water jet tip.

[0084] Let the initial position of the nozzle be P0, and its coordinates on the ultrasound image be imageP0(x0, y0). Further, the tube wall contour is extracted and fitted to determine the direction of the water jet head's main axis 11 in the ultrasound image. Typically, the water jet head's main axis 11 is made of stainless steel or a similar metal, and may exhibit some distortion / artifacts in the ultrasound image, as shown in Figure 5. When the distortion / artifacts in the image are small, edge extraction and fitting can be performed directly on the image of the water jet head's main axis 11, assuming the fitted line represents the actual direction of the water jet head within the body. When the distortion / artifacts are large, this method may produce deviations. In this case, a reliability threshold can be established as a standard to judge whether the fitting result is acceptable. Specifically, the distance from each point to the fitted line is calculated. For each fitted point, its distance to all fitted lines is calculated to determine if it is a point with a large error. Based on a preset distance threshold (e.g., 10 pixels), points with large errors are removed. This reduces the impact of errors caused by distortion and artifacts on the fitting result.

[0085] Furthermore, a straight line is fitted using the least squares method; and the parameters of the fitted line (slope and intercept) and the goodness of fit (e.g., R² value) are calculated. The goodness of fit measures how well the fitted line matches the data points. After removing gross errors, the average error from all fitted points to the fitted line is calculated; the smaller the average error, the more accurate the fitted line. In a preferred example, R² is calculated. 2 The value is used to measure the quality of the fitted line, ranging from 0 to 1. The closer the value is to 1, the better the fit.

[0086] Furthermore, a custom threshold can be set (e.g., R). 2 =0.95), as the criterion for judging whether the fitting result is acceptable; and based on the calculated R... 2 The value is used to judge the fitting result. If R... 2If the value is greater than or equal to the set threshold, the fitting result is considered reliable, and the fitted straight line is accepted as the direction of the water jet head's main axis in the ultrasound image; otherwise, the fitting result is considered unreliable, and it is necessary to readjust or use other methods for direction positioning.

[0087] In some embodiments, additional markers that are identifiable in ultrasound images can be added to the waterjet head 1. For example, markers can be formed on the main shaft 11 of the waterjet head using a high acoustic impedance material (such as a metal material or a special coating), or small metal particles can be embedded into the surface of the main shaft 11 of the waterjet head. Alternatively, grooves or protrusions can be engraved on the outer surface of the main shaft 11 of the waterjet head using a laser engraving machine to form obvious reflection points in the ultrasound images.

[0088] In an alternative embodiment, similar to the above approach, the wall profile of the sheath or endoscope can be extracted and fitted, and the wall profile and / or extension direction of the waterjet head can be determined based on the physical positional relationship between the sheath or endoscope and the waterjet head based on the assembly.

[0089] Through the detailed steps described above, the tube wall contour can be extracted from the endoscopic ultrasound image, and the direction of the water jet tip's main axis can be fitted. Key steps include image preprocessing, edge extraction, straight line fitting, removal of points with large distortion / artifacts, calculation of fitting error, and assessment of the reliability of the fitting results. Combining these methods allows for accurate determination of the water jet tip's direction in the ultrasound image, providing effective navigation and reference for surgery.

[0090] Step S5: Based on the position coordinates of multiple first key feature points in the template image and the position coordinates of multiple second key feature points in the ultrasound image, the water jet template image is superimposed on the ultrasound image for display, forming the water jet AR display image based on the ultrasound image, as shown in Figure 6.

[0091] As mentioned earlier, multiple key feature points were selected or determined in both the waterjet template image and the ultrasonic image. Among these feature points, at least the cutter hole feature point was included.

[0092] Furthermore, based on the selected ultrasonic probe, the pixel-to-mm (pixel / mm) ratio of the ultrasonic imaging can be determined, and the distance in the image corresponding to the actual length of the waterjet template can be determined, thereby determining the display ratio of the waterjet template.

[0093] First, determine the pixel-to-millimeter ratio of the ultrasound image. Specifically, first use an object of known size for calibration, place this object under the ultrasound imaging device for scanning, and acquire its image; measure the length of the object in the ultrasound image (in pixels); divide the actual length of the known object (in millimeters) by the measured pixel length to obtain the pixel-to-millimeter ratio D.

[0094] Assuming a 50 mm long calibration object is used for calibration: place the calibration object under the ultrasound imaging device, scan and acquire an image; measure the length of the calibration object in the image, assuming the measured length is 250 pixels; calculate the pixel-to-millimeter ratio: pixel-to-millimeter ratio = 250 pixels / 50 mm = 5 pixels / mm.

[0095] Further, the display ratio of the water jet template in the image is determined. Specifically, the actual length L of the water jet is measured (in millimeters); the actual length of the water jet is multiplied by the pixel-to-millimeter ratio to obtain the pixel length Lpixel of the water jet tip in the ultrasound image, i.e., Lpixel = L × D; based on the calculated pixel length Lpixel, the display ratio of the template image of the surgical instrument containing the water jet tip is adjusted to match the ratio in the ultrasound image.

[0096] Assuming the actual length L of the water jet nozzle is 150 mm, using the calculated pixel-to-millimeter ratio D, calculate the pixel length Lpixel of the water jet nozzle in the ultrasound image: Lpixel = 150 mm × 5 pixels / mm = 750 pixels. Next, adjust the display ratio of the template image, scaling it to 750 pixels to match the ratio in the ultrasound image; in an alternative embodiment, this scaling can also be implemented using image processing software such as OpenCV.

[0097] Once the length of the waterjet cutter head is determined, the ratio of the physical length to the physical width of the waterjet cutter head can be predetermined by measuring the actual object, or it can be determined by drawing the outline in a two-dimensional image. According to the pre-agreed aspect ratio, the width of the waterjet cutter head can be displayed proportionally.

[0098] Furthermore, in a preferred embodiment, when acquiring an ultrasound image of a surgical instrument containing a water jet tip and identifying multiple second key feature points in the ultrasound image, the endoscope or water jet tip is moved multiple times. As mentioned above, multiple movements advantageously provide more accurate calibration. Such calibration is usually necessary because: considering that the long straight line of the water jet cutter body may be distorted in the ultrasound image, and that the endoscope and water jet cutter head may appear as extended dashed lines due to artifacts, using only the position of the water jet cutter body and endoscope lens identified by the ultrasound image as the specific pose of the water jet has a large error. At the same time, the visualization of the water jet in the ultrasound image needs to achieve accurate tracking of the position of the cutter hole and endoscope lens on the image. The ultrasound probe used to acquire the ultrasound sagittal plane image has a certain width, so that as long as the main axis of the water jet cutter head is within the width range corresponding to the ultrasound probe, an ultrasound sagittal plane image can be formed. When the axis of the main axis of the water jet cutter head has a certain angle with the sagittal plane, the movement of the water jet cutter head or endoscope lens in the ultrasound image is the projection of the actual water jet cutter head or movement distance in the sagittal plane. At this time, it is necessary to calibrate the movement distance of the cutter hole in the visualized water jet template image in the sagittal plane in order to achieve accurate tracking of the position of the cutter hole and endoscope lens in the visualized image.

[0099] Specifically, proportional correction can be provided based on the distance the endoscope or water jet tip moves multiple times and the position of the ultrasound image.

[0100] Taking the movement of the endoscope lens as an example, let the endoscope be in the first position E0, and the lens position in the ultrasound image be imageE0(x0, y0). Move the endoscope lens from the first position E0 along the extension direction of the endoscope body axis by a distance S1 to the second position E1, and identify the position of the endoscope lens in the ultrasound image at this time as imageE1(x1, y1). Continue to move it along the extension direction of the endoscope body axis by a distance S2 to the third position E2, and identify the position of the endoscope lens in the ultrasound image at this time as imageE2(x2, y2).

[0101] Considering that the extension direction of the endoscope's main axis is mainly in the x-direction in the sagittal image, the calibration coefficient k in the x-direction is calculated as follows: worldE0=T*imageE0 worldE1=T*imageE1 worldE2=T*imageE2 k=mean(S2 / sqrt((worldE2(x)-worldE1(x))^2-(worldE2(y)-worldE1(y))^2), S1 / sqrt((worldE1(x)-worldE0(x))^2-(worldE1(y)-worldE0(y))^2));

[0102] Where T is the transformation matrix between the ultrasound image coordinate system and the physical coordinate system obtained by the ultrasound probe, which can be obtained through calibration after the ultrasound probe is determined. worldE0 represents the physical coordinates of position E0, worldE1 represents the physical coordinates of position E1, and worldE2 represents the physical coordinates of position E2. mean is the mean function, and sqrt is the square root function.

[0103] Distances S1 and S2 can be determined by the operating parameters of the endoscope drive mechanism, position sensors, or other physical distance detection methods. For example, the movement distance can be detected by using optical scales, magnetic scales, piezoelectric couples, etc., or the movement distance can be confirmed based on the changes in distance scale parameters along the movement path of the endoscope.

[0104] The above is just an example. It should be noted that by moving the waterjet cutter head multiple times and performing similar operations, the calibration coefficient can also be obtained.

[0105] After determining the calibration coefficient k, when performing AR display, the length in the x-direction is multiplied by the calibration coefficient and centered on the original position of the water jet cutter hole imageP0(x0, y0).

[0106] Through the aforementioned steps, the generated ultrasound-based AR image of the surgical instrument containing the water jet can provide more comprehensive and intuitive display information during surgery. For example, as shown in Figure 7, the current actual movement position and working status of the water jet can be simulated and displayed on the augmented reality ultrasound image, and the position of the nozzle can be simulated and displayed.

[0107] According to embodiments of this application, this application also provides a water jet AR display device based on ultrasound images to implement the aforementioned water jet AR display method based on ultrasound images. The device includes: an ultrasound imaging device for acquiring real-time ultrasound images of the water jet; and an AR display device, which can be a head-mounted display or augmented reality glasses, or a conventional display screen. The water jet template can be preset data, or it can be newly created or updated using a visible light sensor and a processor including functions such as registration. Furthermore, although this application uses a TRUS probe (transrectal ultrasound probe) to illustrate real-time monitoring of the water jet surgical procedure (typically inserted via the urethra), it should be understood that this application is not limited to the aforementioned surgical scenario.

[0108] Referring to Figure 9, this application also proposes a water jet AR display device based on ultrasonic images, comprising the following modules:

[0109] A template image acquisition module for a waterjet cutter head is used to acquire a template image of the waterjet cutter head and mark the first key feature point in the template image, wherein the key feature point includes the cutter hole;

[0110] The ultrasonic image acquisition module of the water jet cutter head is used to acquire the ultrasonic image of the water jet and identify the second key feature point of the water jet cutter head in the ultrasonic image, the second key feature point corresponding to the first key feature point;

[0111] An AR image generation module is used to form the water jet AR display image based on the ultrasound image, based on the position coordinates of the first key feature point in the template image and the position coordinates of the second key feature point in the ultrasound image.

[0112] Referring to Figure 10, this application also proposes a water jet AR display system based on ultrasound images, comprising: a surgical instrument suitable for insertion into the human body for surgical procedures, an ultrasound imaging device, a processor, a memory, and a display device. The surgical instrument includes a water jet tip and an endoscope, the main axis of the water jet tip and the main axis of the endoscope being substantially parallel. The ultrasound imaging device is positioned below the surgical instrument. The processor is used to execute a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the method described above. The display device is used to display the water jet AR display image based on the ultrasound images.

[0113] In summary, this application, through precise image processing and real-time tracking technology, can provide effective visual assistance for medical procedures. By providing intuitive visual assistance, it can significantly improve the doctor's perception and operational precision during surgery, reduce surgical risks, improve medical outcomes, and comprehensively enhance the accuracy and safety of surgery.

[0114] It should be noted that the above modules are divided according to their functions, and in terms of specific implementation, they can be configured and implemented as needed in terms of physical location, hardware, and software / firmware architecture.

[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. In a typical configuration, the device of this application includes one or more processors (one of CPU, FGAP, and MUC), an input / output user interface, a network interface, and memory.

[0116] Therefore, this application also proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any embodiment of the first aspect of this application.

[0117] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0118] Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] Therefore, this application also proposes a computer-readable medium storing a computer program that, when executed by a processor, implements the steps of the method described in any embodiment of this application. For example, the memory of this application may include non-permanent memory in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM.

[0120] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0121] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A water jet AR display method based on ultrasound images, comprising the following steps: Obtain a template image of an instrument containing a water jet cutter head, and mark multiple first key feature points in the template image, including cutter hole feature points; acquiring an ultrasound image of an instrument comprising a waterjet tip and identifying a plurality of second key feature points in the ultrasound image, the second key feature points corresponding to the first key feature points, wherein, The dynamic characteristics of the nozzle jet are used for jet feature recognition, and the static characteristics of the nozzle in the ultrasonic image are used for edge extraction to determine the position of the knife hole. Based on the position coordinates of the plurality of first key feature points in the template image and the position coordinates of the plurality of second key feature points in the ultrasound image, the water jet AR display image based on the ultrasound image is formed.

2. The water knife AR display method of claim 1, wherein, The step of determining the cutter hole location in the step of acquiring an ultrasonic image of an instrument containing a water jet cutter head and identifying multiple second key feature points in the ultrasonic image includes: The water jet is controlled to perform water spraying action. The high-pressure jet sprayed out forms a rapidly fluctuating air mass in the liquid immersion environment. Based on the detected fluctuating area of ​​the air mass in the ultrasonic image, the contour features of the air mass are identified by jet feature recognition to generate a first auxiliary line. Edge extraction is performed using the static features of the water jet nozzle in the ultrasound image to generate a second auxiliary line; The precise location of the burr hole in the ultrasound image is determined based on the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction.

3. The water knife AR display method of claim 1, wherein, The step of acquiring an ultrasound image of an instrument containing a water jet cutter head and identifying multiple second key feature points in the ultrasound image further includes: translating the water jet cutter head along the main axis of the water jet cutter head a certain distance and performing a water spraying action to obtain the corresponding position point; and / or, acquiring the initial position point of the endoscope included in the instrument, and driving the endoscope to retract to the termination position to obtain the endoscope stop position point; and / or, acquiring multiple position points on the tube wall contour of the main axis of the water jet cutter head.

4. The water knife AR display method of claim 3, wherein, The step of acquiring an ultrasound image of an instrument containing a water jet cutter head and identifying multiple second key feature points in the ultrasound image further includes: determining whether the fitting result of the tube wall profile is reliable based on a reliability threshold; when the determination result is deemed unreliable, controlling the water jet cutter head to translate a certain distance along the main axis of the water jet cutter head and performing a water spraying action to acquire the corresponding position point; and / or, acquiring the initial position point of the endoscope contained in the instrument and driving the endoscope to retract to the termination position to acquire the endoscope stop position point.

5. A water jet AR display device based on ultrasonic images, comprising the following modules: The template image acquisition module is configured to acquire a template image of the instrument containing the water jet head, and mark a plurality of first key feature points in the template image, wherein the first key feature points contain a knife hole feature point; the ultrasound image acquisition module is configured to acquire an ultrasound image of the instrument containing the water jet head, and identify a plurality of second key feature points in the ultrasound image, wherein the second key feature points correspond to the first key feature points, and wherein, The dynamic characteristics of the nozzle jet are used for jet feature recognition, and the static characteristics of the nozzle in the ultrasonic image are used for edge extraction to determine the position of the knife hole. An AR image generation module is used to form the water jet AR display image based on the ultrasound image, based on the position coordinates of the plurality of first key feature points in the template image and the position coordinates of the plurality of second key feature points in the ultrasound image.

6. The water knife AR display device of claim 5, wherein, The ultrasound image acquisition module determines the knife hole location through the following steps: The water jet is controlled to perform water spraying action. The high-pressure jet sprayed out forms a rapidly fluctuating air mass in the liquid immersion environment. Based on the detected fluctuating area of ​​the air mass in the ultrasonic image, the contour features of the air mass are identified by jet feature recognition to generate a first auxiliary line. Edge extraction is performed using the static features of the water jet nozzle in the ultrasound image to generate a second auxiliary line; The precise location of the burr hole in the ultrasound image is determined based on the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction.

7. The water knife AR display device of claim 5, wherein, The ultrasound image acquisition module is also used to perform the following steps: The water jet cutter head is translated a certain distance along the main axis of the water jet cutter head and a water spraying action is performed to obtain the corresponding position point; and / or, the initial position point of the endoscope included in the instrument is obtained, and the endoscope is driven to retract to the termination position to obtain the endoscope stop position point; and / or, multiple position points on the tube wall contour of the main axis of the water jet cutter head are obtained.

8. The water knife AR display device of claim 7, wherein, The ultrasound image acquisition module is also used to perform the following steps: determine whether the fitting result of the tube wall profile is reliable according to the reliability judgment threshold; when the judgment result is considered unreliable, control the water jet head to translate a distance along the main axis of the water jet head and perform a water spraying action to obtain the corresponding position point; and / or, obtain the initial position point of the endoscope included in the instrument, and drive the endoscope to retract to the termination position to obtain the endoscope stop position point.

9. An ultrasonic image-based waterjet AR display system, comprising: Surgical instruments suitable for insertion into the human body to perform surgery, an ultrasound imaging device, a processor, a memory, and a display device are provided. The surgical instruments include a water jet nozzle and an endoscope, with the main axis of the water jet nozzle and the main axis of the endoscope being substantially parallel. The ultrasound imaging device is positioned below the surgical instruments. The processor is used to execute a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program performs the following steps: Obtain a template image of an instrument containing a water jet cutter head, and mark multiple first key feature points in the template image, including cutter hole feature points; Acquire an ultrasonic image of an instrument containing a water jet nozzle and identify multiple second key feature points in the ultrasonic image, the second key feature points corresponding to the first key feature points, wherein jet feature identification is performed using the dynamic features of the nozzle jet, and edge extraction is performed using the static features of the nozzle in the ultrasonic image to determine the position of the nozzle hole. Based on the position coordinates of the plurality of first key feature points in the template image and the position coordinates of the plurality of second key feature points in the ultrasound image, the water jet AR display image based on the ultrasound image is formed; The display device is used to display the water jet AR display image based on the ultrasound image.

10. The water knife AR display system of claim 9, wherein, The step of determining the cutter hole location in the step of acquiring an ultrasonic image of an instrument containing a water jet cutter head and identifying multiple second key feature points in the ultrasonic image includes: The water jet is controlled to perform water spraying action. The high-pressure jet sprayed out forms a rapidly fluctuating air mass in the liquid immersion environment. Based on the detected fluctuating area of ​​the air mass in the ultrasonic image, the contour features of the air mass are identified by jet feature recognition to generate a first auxiliary line. Edge extraction is performed using the static features of the water jet nozzle in the ultrasound image to generate a second auxiliary line; The precise location of the burr hole in the ultrasound image is determined based on the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction.

11. The water knife AR display system of claim 9, wherein, The step of acquiring an ultrasound image of an instrument containing a water jet cutter head and identifying multiple second key feature points in the ultrasound image further includes: translating the water jet cutter head along the main axis of the water jet cutter head a certain distance and performing a water spraying action to obtain the corresponding position point; and / or, acquiring the initial position point of the endoscope included in the instrument, and driving the endoscope to retract to the termination position to obtain the endoscope stop position point; and / or, acquiring multiple position points on the tube wall contour of the main axis of the water jet cutter head.

12. The water knife AR display system of claim 11, wherein, The step of acquiring an ultrasound image of an instrument containing a water jet cutter head and identifying multiple second key feature points in the ultrasound image further includes: determining whether the fitting result of the tube wall profile is reliable based on a reliability threshold; when the determination result is deemed unreliable, controlling the water jet cutter head to translate a certain distance along the main axis of the water jet cutter head and performing a water spraying action to acquire the corresponding position point; and / or, acquiring the initial position point of the endoscope contained in the instrument and driving the endoscope to retract to the termination position to acquire the endoscope stop position point.

Images