Systems and methods for surgical tool identification and surgical tool kit validation

Abstract

Systems and methods for surgical tool validation are provided that allow for a rapid and high degree of accuracy in the identification of surgical tools.

Description

SYSTEMS AND METHODS FOR SURGICAL TOOL IDENTIFICATION AND SURGICAL TOOL KIT VALIDATIONBackground

[0001] The classification of surgical instruments has gained significant attention due to the critical role these tools play in ensuring the success of surgical procedures. Traditional methods of manual identification and assembling in a central sterile services department (CSSD) have been increasingly challenged by the growing complexity and variety fine-grained to ultra fine-grained instruments used in modem surgical practices. To address these challenges, researchers have explored various techniques, ranging from machine learning algorithms to advanced computer vision approaches M. Primus et al., “Instrument classification in laparoscopic videos,’’ 2015 13th international workshop on content based multimedia indexing (CBMI), pages 1-6. IEEE, 2015 (#22); M. Rodrigues et al., “Surgical tool datasets for machine learning research: a survey,’’ International Journal of Computer Vision, 130(9):2222-2248, 2022 (#24).

[0002] Surgical instrument classification in laparoscopic procedures has been a focus of research to enable semantic segmentation of video footage. To do this, M. Primus et al. employed a Support Vector Machine (SVM) with Bag-of-Words (BoW) features extracted from densely sampled key points in training images. Their study compared the efficacy of three key point descriptors: ORB, SIFT, and SURF.

[0003] Recent advancements in this field have leveraged deep learning techniques.

[0004] Sebastian Bodenstedt, et al., “Real-time image-based instrument classification for laparoscopic surgery, arXiv preprint arXiv: 1808.00178 (2018) proposed an image-based surgical tool classification method for laparoscopic interventions. Their approach first detects bounding boxes using pixelwise random forest segmentation, then classifies instruments using a cascade of random forest algorithms based on multiple features, including histograms of hue and saturation, gradients, and SURF features. Building on existing instance segmentation models,

[0005] Britty Baby, et al., “From forks to forceps: A new framework for instance segmentation of surgical instruments, proceedings of the IEEE / CVF winter conference on applications of computer vision, pages 6191-6201 (2023) introduced a novel neural netw ork framework that incorporates a classification module to enhance the accuracy of instrument mask identification.

[0006] Praveen SR Konduri et al., “Full resolution convolutional neural network based organ and surgical instrument classification on laparoscopic image data, Biomedical SignalProcessing and Control, 87:105533 (2024) developed a full-resolution Convolutional Neural Network (CNN) for efficient organ and surgical instrument classification using laparoscopic image datasets.

[0007] Jan Lehr, ct al., “Image-based recognition of surgical instruments by means of convolutional neural networks,’’ International journal of computer assisted radiology and surgery , 18(ll):2043-2049 (2023) developed a camera-based surgical instrument classification system using CNN architecture. Although their setup employed three cameras to capture different views of an instrument, the system processed one image at a time. Addressing the challenge of unbalanced data in the publicly available Cholec80 laparoscopy video dataset,

[0008] Jaafar Jaafari, et al., “Towards more efficient ci -based surgical tools classification using transfer learning. Journal of Big Data, 8(1): 115 (2021) implemented multiple data augmentation techniques and a finetuned CNN for automatic tool recognition during surgery'.

[0009] Sabrina Kletz, et al., “Identifying surgical instruments in laparoscopy using deep learning instance segmentation. 2019 International Conference on Content-Based Multimedia Indexing (CBMI), pages 1-6. IEEE (2019) applied a region-based convolutional neural network (R-CNN) to recognize surgical instruments using a custom dataset generated from laparoscopic gynecological videos.

[0010] Amy Jin, et al., “Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. 2018 IEEE winter conference on applications of computer vision (WACV). pages 691-699. IEEE (2018) combined a faster R-CNN with VGG16 to detect laparoscopic surgical tools and perform operative skill assessment using the M2CAI dataset.

[0011] Cristian da Costa Rocha, et al., “Self-supervised surgical tool segmentation using kinematic information, 2019 International Conference on Robotics and Automation (ICRA), pages 8720-8726. IEEE (2019) explored a self supervised method for segmenting surgical instruments in laparoscopic surgery, utilizing the kinematic model of the robot as a source of information.

[0012] Previous studies mainly focused on surgical instrument segmentation and singleview surgical tool classification. However, in this study, we propose a system for real-time ultra fine-grained surgical instrument classification using multi-view images and multi-view CNN and transformer based models. This system aimed to assist the CSSD technicians of a hospital in correctly assembling surgical instruments in the trays.

[0013] There remains a need to improve classification of ultra-fine-grained surgical instruments that has the potential to significantly enhance the success rates of surgical proceduresand improve the overall efficiency of hospital operations. However, due to the vast number of surgical instruments in the Central Sterile Services Department (CSSD) of a hospital, also known as the Sterile Processing Department (SPD), and the ultrafine-grained distinctions among the instruments, accurately identifying and classifying these instruments is challenging.Summary of the Invention

[0014] In one aspect, the present invention relates to a system for identification of a surgical tool comprising an imaging platform; a top camera mounted above the imaging platform that is configured to acquire a top-down image of the surgical tool; a second camera mounted on or near the imaging platform that is configured to a acquire a second image of at least a portion of the surgical tool, wherein the second image represents a side view or a perspective view highlighting a different portion of the surgical tool; and a multi-view image classification framework that accepts as input at least (1) the top-down image and (2) the second image, the multi-view image classification framework providing a predicted instrument identification based on at least (1) the top-down image and (2) the second image.

[0015] The system extracts first features from the top down image and second features from the second image. The features may be extracted through a deep learning architecture, including, for example, one or more of EfficientNet, ResNet50. VGG16, and / or ViT. In one particular example, the deep learning architecture is ResNet50. The first features and the second features may be passed through their respective fully connected layers within the deep learning architecture. The system may fuse the first features and the second features into a first feature vector and a second feature vector using one or more feature fusion techniques, which may include for example concatenation fusion and / or addition fusion. In one aspect, the feature fusion may occur at a separate fully connected layer.

[0016] In another aspect, the invention relates to a method for identification of a surgical tool comprising: (a) placing a surgical tool on an imaging platform of a surgical tool identification system as discussed above; (b) obtaining the first image and the second image; (c) passing the first image and the second image to a multi-view image classification framework, the multi-view image classification framework providing a predicted instrument identification based on at least (1) the top-down image and (2) the second image; and (d) assigning the surgical tool to a surgical tray based on the predicted instrument identification.

[0017] In another aspect, tire invention relates to a non -transitory computer readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of: (a) obtaining a first image and a second image of a surgical tool that has been placed on an imaging platform of a surgical tool identification systemas discussed above; and (b) passing the first image and the second image to a multi-view image classification framework, the multi-view image classification framework providing a predicted instrument identification based on at least (1) the top-down image and (2) the second image. The user may assign the surgical tool to a surgical tray based on the predicted instrument identification.Brief Description of the Drawings

[0018] FIG. 1 shows a process flow for the fine-grained surgical instrument classification system according to an aspect of the invention;

[0019] FIG. 2A shows a perspective view7of an image collection platform according to an embodiment of the invention;

[0020] FIG. 2B shows a front view of an image collection platform according to an embodiment of the invention;

[0021] FIG. 2C shows a detailed top view of the imaging platform according to an embodiment with a first surgical instrument positioned on the platform for imaging;

[0022] FIG. 2D shows a detailed top view of the imaging platform according to an embodiment with a second surgical instrument positioned on the platform for imaging;

[0023] FIG. 3 shows a view of a user interface for the fine-grained surgical instrument classification system according to an embodiment of the invention;

[0024] Fig. 4A show s an example of surgical tools with fine grain differences;

[0025] Fig. 4B show s another example of surgical tools with fine grain differences;

[0026] Fig. 4C show s another example of surgical tools with fine grain differences;

[0027] Fig. 5A shows a GRAD-CAM heatmap visualization illustrating the model’s focus on ultra-fine-grained surgical instruments according to an aspect of the invention;

[0028] Fig. 5B shows another GRAD-CAM heatmap visualization illustrating the model’s focus on ultra-fine-grained surgical instruments according to an aspect of the invention;

[0029] Fig. 5C shows another GRAD-CAM heatmap visualization illustrating the model’s focus on ultra-fine-grained surgical instruments according to an aspect of the invention;

[0030] Fig. 6A shows an example of an assembled tray according to an aspect of the invention;

[0031] Fig. 6B shows another example of an assembled tray according to an aspect of the invention;

[0032] Fig. 6C shows another example of an assembled tray according to an aspect of the invention.Detailed Description

[0033] The present inventors have designed a user-friendly image collection platform mounted with multiple UVC cameras and a multi-image classification system adapted to improve image classification and surgical tool kit validation. Fig. 1 shows an overview diagram of a realtime ultra fine-grained surgical instruments classification system in accordance with an embodiment of the invention. This system includes two components. (1) an image collection platform 120 to continuously collect images using two cameras and (2) a multi-image classification framework to get the prediction of the models of the instrument using the multiview of the instrument. In this system, a central sterile sendees department (CSSD) technician 101 places an unknown surgical instrument 110 in the platform and the system takes two view images of it, a top view 123 and a side view 124. Then using the images, the trained model predicts the instrument name using an image classification framework 130. The classification framework separately performs feature extraction from the top view image 131, and feature extraction from the side view image 132. Feature fusion is performed from the separate features 133 into a combined layer 134. From the combined layer 134, a classification layer 135 is prepared. Based on the prediction, the CSSD technician 101 utilizes the predicted instrument 136 and assembles the tray 140.

[0034] The image collection platform 200 according to an embodiment of the invention is shown in Figs. 2A-2D. The device 200 includes a slide platform 210 with a sliding holder 211 and a landing pattern 212. The slide platform may be made from a solid aluminum extrusion slide platform frame 240, and may have dimensions such as 18” x 6”. Other dimensions may be desirable depending on the size and shape of the instruments to be classified.

[0035] The upper camera 220 may be provided on an upper camera mount 222 attached to an upper camera fixed boom 221. In one aspect, the upper camera 220 is positioned 14” above the center of the platform 210. The upper camera preferably may be a fixed focus camera with 4K resolution, such as Luxonic OAK-D. The fixed focus can facilitate faster imaging times and reduced variability in the collected images.

[0036] A second camera 230 may be mounted to the platform 210 using mounting arms(23 la,b). In one aspect, the side camera is attached to the comer of the slide platform frame 240and is positioned 45 degrees relative to either aluminum T slot with a slight downward tilt to point it at the surface. This orientation advantageously permits capture of a perspective view of a tip portion of an instrument w ithin the landing pattern 212 when the instrument is placed on the slide platform 210. The second or lower camera is preferably mounted at an oblique angle using a shorter fixed focal length 4K USB camera to acquire the ultra fine-grained details of instrument tips (i.e., curvature, surface texture, and / or tooth pattern). The fixed focal length is desirable for speed and reliable reproduction of images.

[0037] The upper camera 220 and lower camera 230 are placed in positions that allow clear visualization of small ophthalmology and larger laparotomy (major and minor traysjinstruments. For image acquisition and classification, the instrument is laid flat at the bottom of the platform setup.

[0038] In order to accommodate the wide range of instrument sizes and shapes, the inventors designed a slider mechanism that permits a much greater number of instruments to be accurately identified using a single platform. For example, a large aortic clamp’s tip can be focused just as well as a small castro-viejo needle driver tip on the lower camera to provide the critical fine-grained features for improved accuracy on a single platform. The ability for a single platform to integrate multiple camera views allowed decreased environmental controls. The depicted platform has minimal environmental controls featuring an open platform that does not control ambient light, easily integrates with current workstations without significant modifications, and ensures quick access for the technicians to present and remove instruments.

[0039] The sliding holder 211 may further include adjustable posts 213a, 213b that allow positioning of handle portions of surgical tools of different sizes. The sliding holder may also include T-shaped protrusions on the underside that can interact with slots 250a, 250b in the platform 210 with undercuts to engage the T-shape protrusions and allow for a back and forth sliding of the sliding holder 211 along the length of the slide platform 210. The adjustable posts 213a and 213b can slide back and forth in a direction perpendicular to the length of the slide platform. This allows adjustment of the posts 213a. 213b to accommodate instrument having different size handles or to allow an instrument to be positioned in either an open or closed configuration during the imaging process. The device may also include a central post 260 that is utilized to position an instrument 270 or 271. Fig. 2C and 2D illustrate the imaging of two instruments 270 and 271 which differ only in fine details. These configmations allow for positioning of the instrument in a way that the tip of the instrument with fine details to be detected is placed within the landing pattern 212 on the image collection platform 200.

[0040] Fig. 3 illustrates a user interface for the fine-grained surgical instrument classification system according to an embodiment of tire invention. The user interface mayprovide an image from the top view and side view cameras as well as information about the predicted instrument.

[0041] Figs. 4A 4C illustrate various surgical tools with fine grain differences. The system of the present invention is capable of distinguishing these instruments from one another with a high degree of accuracy. Figs. 5A-5C show a GRAD-CAM heatmap visualization illustrating the model’s focus on ultra-fine-grained surgical instruments according to an aspect of the invention. Figs. 6A-6C show exemplary trays that may be assembled using the identification systems and methods describe herein.

[0042] In one aspect, the invention leverages a multi-view CNN (MVCNN) architecture and multi-view VIT and late-stage fusion of the different views. Let (Isand It) be an image pair of our dataset (D) which consists of the side-view (Is) and top view (It) images. We extract the features (fsand ft) of the Isand L separately using identical architectures (Mcnn), including EfficientNet, ResNet50, VGG16, and ViT. Each model processes the input image pair independently, extracting features (fsand ft) through a series of convolutional or transformer layers. The extracted features are then passed through their respective fully connected (LFc) layers within each model branch. To fuse the feature vectors (Isvand Itv) from both views, we employ late feature fusion techniques, specifically concatenation. Besides this feature concatenation fusion, we utilized feature addition fusion techniques for our proposed system. This feature fusion occurs at a separate fully connected layer (LfuslOn).

[0043] This late-stage feature fusion is critical as it enables the model to leverage the complementary information from different views, thereby enhancing the discriminative power of the combined feature representation. The fused feature vector Ivis subsequently fed into a classification layer (LCLS), which predicts the specific class of the surgical instrument (Pclass). In summary, we can express these operations mathematically as follows.

[0044] In order to assess the effectiveness of the multi-view architecture, the present inventors compared a single side-view or top-view versus multi-view, which combined both the side-view and top-view using a late fusion teclmique as described above to assess accuracy of each configuration.

[0045] As can be seen, the multi-view configuration resulted in the highest degree of accuracy.

[0046] Other embodiments and uses of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. All references cited herein, including all U.S. and foreign patents and patent applications, are specifically and entirely hereby incorporated herein by reference. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the invention indicated by the following claims.

Claims

PCT / US25 / 54963 11 November 2025 (11.11.2025)What is claimed is:

1. A system for identification of a surgical tool comprising: an imaging platform; a top camera mounted above the imaging platform that is configured to acquire a top-5 down image of the surgical tool; a second camera mounted on or near the imaging platform that is configured to a acquire a second image of at least a portion of the surgical tool, wherein the second image represents a side view or a perspective view of the at least the portion of the surgical tool; and a multi-view image classification framework that accepts as input at least (1) the top- down image and (2) the second image, the multi-view image classification framework providing a predicted instrument identification based on at least (1) the top-down image and (2) the second image.

2. The system of claim 1, wherein the image classification framework extracts first features15 from the top down image and second features from the second image.

3. The system of claim 2, wherein the first features and second features are extracted through a deep learning architecture.20 4. The system of claim 3, wherein the deep learning architecture is selected from EfficientNet, ResNet50, VGG16, and / or ViT.

5. The system of claim 1, wherein the imaging platform further comprises a sliding holder configured to engage a handle of the surgical tool, and a landing pattern configured to position a tip portion of the surgical tool using the sliding holder.

6. The system of claim 3, wherein the first features and the second features are passed through their respective fully connected layers within the deep learning architecture.30 7. The system of claim 6, wherein one or more feature fusion techniques are utilized to fuse the first features and the second features into a first feature vector and a second feature vector.PCT / US25 / 54963 11 November 2025 (11.11.2025)8. The system of claim 7, wherein the one or more feature fusion techniques comprise concatenation fusion and / or addition fusion.5 9. The system of claim 7, wherein the feature fusion occurs at a separate fully connected layer.

10. A method for identification of a surgical tool comprising:(a) placing a surgical tool on an imaging platform of a surgical tool identification system,10 the surgical tool identification system comprising: the imaging platform; a top camera mounted above the imaging platform that is configured to acquire a top-down image of the surgical tool; a second camera mounted on or near the imaging platform that is configured to a acquire a second image of at least a portion of the surgical tool, wherein the second image represents a side view or a perspective view of the at least the portion of the surgical tool; and(b) obtaining the first image and the second image;(c) passing the first image and the second image to a multi-view image classification20 framework, the multi-view image classification framework providing a predicted instrument identification based on at least (1) the top-down image and (2) the second image; and(d) assigning the surgical tool to a surgical tray based on the predicted instrument identification.

11. The method of claim 10, wherein the image classification framework extracts first features from the top down image and second features from the second image.

12. The method of claim 11, wherein the first features and second features are extracted through a deep learning architecture.30PCT / US25 / 54963 11 November 2025 (11.11.2025)13. The method of claim 12, wherein the deep learning architecture is selected from EfficientNet, ResNet50, VGG16, and / or ViT.

14. The method of claim 10, wherein the imaging platform further comprises a sliding holder5 configured to engage a handle of the surgical tool, and a landing pattern configured to position a tip portion of the surgical tool using the sliding holder.

15. The method of claim 12, wherein the first features and the second features are passed through their respective fully connected layers within the deep learning architecture.

16. The method of claim 15, wherein one or more feature fusion techniques are utilized to fuse the first features and the second features into a first feature vector and a second feature vector.15 17. The method of claim 16, wherein the one or more feature fusion techniques comprise concatenation fusion and / or addition fusion.

18. The method of claim 16, wherein the feature fusion occurs at a separate fully connected layer.2019. A non-transitory computer readable medium storing program code which, when executed, is operative to cause an electronic processor to perform the steps of:(a) obtaining a first image and a second image of a surgical tool that has been placed on an imaging platform of a surgical tool identification system, the surgical tool identification system comprising: the imaging platform; a top camera mounted above the imaging platform that is configured to acquire a top-down image of the surgical tool; a second camera mounted on or near the imaging platform that is configured to a30 acquire a second image of at least a portion of the surgical tool, wherein the second image represents a side view or a perspective view of the at least the portion of the surgical tool; andPCT / US25 / 54963 11 November 2025 (11.11.2025)(b) passing the first image and the second image to a multi-view image classification framework, the multi-view image classification framework providing a predicted instrument identification based on at least (1) the top-down image and (2) the second image.5 20. The non-transitory computer readable medium of claim 19, wherein the imaging platform further comprises a sliding holder configured to engage a handle of the surgical tool, and a landing pattern configured to position a tip portion of the surgical tool using the sliding holder.

Images