Real-time Surgical Instrument Delineation via ML Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for augmented reality in medical environments fail to provide real-time awareness of medical procedures, leading to potential occlusion issues and increased risk of accidental contact between surgical instruments and body parts during surgeries.
Innovation Solution
A method and system that captures images of medical procedures, automatically identifies and segments non-organic objects using a pretrained machine learning model, and combines them with medical information overlays to create a real-time augmented reality image, ensuring full visibility of surgical equipment and body parts, thereby enhancing safety and anatomical comprehension.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If augmented reality overlays are added to surgical images, then medical information visibility is improved, but occlusion of surgical instruments and body parts occurs
Solution Approach 1:
The patent segments the surgical image into different regions: organic body parts, non-organic surgical instruments, and augmented reality overlays. By separately identifying and processing these segments, the system can maintain visibility of surgical instruments while adding medical information overlays without causing occlusion issues.
Solution Approach 2:
The patent introduces an intermediary processing layer that detects surgical instruments and body parts before applying augmented reality overlays. This intermediary step allows the system to adjust overlay positioning and transparency to avoid occluding critical surgical elements, thereby resolving the contradiction between information visibility and instrument visibility.
2Reliability
If real-time augmented reality imaging is implemented, then surgical safety is improved, but system complexity increases
Solution Approach 1:
The patent employs a multi-functional image processing system that simultaneously performs multiple tasks: detecting surgical instruments, identifying body parts, generating augmented reality overlays, and combining these elements in real-time. By making the system multi-functional, the patent achieves enhanced surgical safety without proportionally increasing complexity, as a single integrated system handles multiple critical functions.
Solution Approach 2:
The system performs self-service through automated detection and segmentation of surgical elements, eliminating the need for manual annotation or complex user input. The automated machine learning models process images independently, reducing the operational complexity burden on surgeons while maintaining high safety standards through continuous real-time monitoring.
3Object-affected harmful factors
If automated instrument segmentation is used, then visibility of surgical equipment is improved, but processing time increases
Solution Approach 1:
The patent applies preliminary action by pre-training machine learning models to recognize surgical instruments and body parts before actual surgery occurs. These pre-trained models can quickly process images during surgery without requiring real-time training, significantly reducing processing time while maintaining accurate instrument segmentation and visibility enhancement.
Solution Approach 2:
The patent replaces manual or mechanical image processing methods with automated machine learning-based computer vision systems. This substitution enables rapid, real-time segmentation of surgical instruments and generation of augmented reality overlays without the time-consuming processes associated with traditional manual annotation or simpler image processing techniques.
Data Source
AI summary
Non-organic objects, such as surgical instruments, in a captured image can be automatically identified and segmented using a pretrained machine learning model. A first mask with only the segmented non-organic objects from the captured images can be created. A second mask or overlay with medical information relating to the medical procedure and/or relating to one or more body parts visible in said captured image can be created. The captured image can be combined with said first mask and said second mask or overlay such that the first mask is applied over the second mask or overlay.