Adaptive Visualization of Surgical Video for Contextual Target Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer-assisted systems face challenges in processing dynamic and unpredictable surgical environments to effectively identify and track multiple objects, particularly in complex scenes with variations in lighting, obstructions, and orientation, which hinders precise surgical actions.
Innovation Solution
A computer-implemented method using machine-learning models, including a surgical phase and structure network, to determine surgical phases and structures, utilizing weak labels, joint detection and segmentation, and refining regions of interest with depth maps, motion analysis, and user inputs, to enhance visualization and predict areas of focus.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Difficulty of detecting and measuring
If machine-learning models are used to identify and track multiple objects in complex surgical scenes, then object detection capability is improved, but system complexity increases
Solution Approach 1:
The system segments the complex surgical scene into multiple object categories (surgical instruments, anatomical structures, smoke, fluid) and processes each category with specialized detection algorithms. This segmentation allows the system to handle different object types with appropriate methods, improving overall detection capability while managing complexity through modular processing.
Solution Approach 2:
The machine-learning models are designed to perform multiple functions simultaneously - detecting surgical instruments, identifying anatomical structures, tracking smoke and fluid. This multi-functionality allows a single system to handle diverse surgical scenarios without requiring separate specialized systems for each object type.
2Speed
If real-time processing is implemented to provide immediate surgical guidance, then responsiveness is improved, but computational load increases
Solution Approach 1:
The system performs preliminary processing by pre-training machine-learning models on extensive surgical datasets before deployment. During actual surgical procedures, these pre-trained models can quickly process video feeds in real-time without requiring intensive computational resources for training, thus achieving responsiveness while managing computational load.
Solution Approach 2:
The system processes surgical video feeds at optimized frame rates and updates object detections periodically rather than continuously analyzing every pixel of every frame. This periodic processing maintains real-time responsiveness while significantly reducing the computational load compared to continuous full-frame analysis.
3Loss of information
If multiple data streams are integrated to enhance surgical visualization, then information completeness is improved, but data processing complexity increases
Solution Approach 1:
The system merges multiple data streams including video feeds from different cameras, machine-learning model outputs for object detection, and surgical phase information into a unified visualization. This merging integrates diverse information sources to provide complete surgical context while using a standardized processing framework to manage the complexity of handling multiple data types.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An aspect includes a computer-implemented method that predicts a proposed region of interest in an image from a video o f a surgical procedure based on one or more contextual targets. An image adjustment is synthesized based on the proposed region of interest and the image. A modified visualization of the surgical procedure is generated by incorporating the image adjustment in a real-time output of the video of the surgical procedure. The video of the surgical procedure is displayed with the modified visualization.