Multi-Model Video Processing for Real-Time Surgical Step Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Surgical robots and user interfaces can distract surgeons with excessive information, leading to inattention blindness, and nurses struggle to provide timely and accurate instrument predictions, while existing apps are not easily discoverable or user-friendly.
Innovation Solution
A system using machine learning to recognize surgical steps in real-time, providing cues and guidance through a combination of CNN and LSTM models, and integrating with surgical robots to enhance surgical efficiency and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all relevant information is provided to the surgeon at once through the user interface, then the surgeon has access to complete information, but the interface becomes crowded and causes inattention blindness
Solution Approach 1:
The user interface is segmented into multiple hierarchical levels. Primary information is displayed prominently, while secondary and tertiary information is organized in expandable sections or available on demand. This allows the surgeon to see only essential information during critical moments while accessing comprehensive data when needed, preventing cognitive overload and inattention blindness.
Solution Approach 2:
The interface dynamically adapts its content and layout based on the surgical phase, complexity of the current task, and surgeon preferences. During simple procedures, the interface shows minimal information. During complex or critical phases, additional relevant information is automatically displayed. This dynamic adjustment ensures information completeness without creating permanent clutter.
2Ease of operation
If traditional surgical procedures are used with human-performed steps, then the surgeon has full control, but there is inherent shakiness and inconsistency in human movements
Solution Approach 1:
A robotic system acts as an intermediary between the surgeon's commands and the actual surgical instrument manipulation. The robot receives high-level directional commands from the surgeon and automatically executes them with sub-millimeter precision, eliminating hand shakiness while preserving surgeon intent and control. The robot serves as a stable mechanical extension of the surgeon's will.
Solution Approach 2:
The patent replaces the human hand-mechanical tool interface with a robotic system that uses advanced actuators, sensors, and control algorithms to achieve superior precision. The robotic system substitutes the variable human mechanical system with a controlled electromechanical system that can consistently reproduce movements with micrometer accuracy.
3Manufacturing precision
If robotic systems are used to perform surgical motions, then precision is improved, but the device complexity increases
Solution Approach 1:
The robotic system is designed with universal components that can perform multiple surgical functions. A single robotic arm can execute cutting, grasping, suturing, and retraction tasks by changing end-effectors. The control software provides a unified interface for all operations. This multi-functionality reduces the number of separate devices needed while maintaining high precision across all surgical tasks.
4Loss of time
If nurses manually predict and prepare instruments, then instrument readiness can be improved, but the prediction accuracy and timeliness are insufficient
Solution Approach 1:
The system continuously monitors the surgical procedure through computer vision and sensor data, comparing actual progress against the surgical plan. When the system detects that a predicted instrument is not being used or the surgical flow has deviated from expectations, it automatically adjusts instrument predictions and notifies the nursing staff. This closed-loop feedback system significantly improves prediction accuracy over static pre-operative planning.
Solution Approach 2:
The system performs preliminary analysis of the surgical video feed and robot telemetry to predict future instrument needs before they are actually required. By analyzing surgical patterns, tissue manipulation techniques, and procedure progression, the system prepares instrument notifications in advance, allowing nurses to have the correct instruments ready before the surgeon requests them.
Data Source
AI summary
In some embodiments, a method of preparing video data for processing by a first machine learning model and a second machine learning model is provided. A computing device generates a first copy of the video data and a second copy of the video data. At least one of a frame rate, bit depth, first video resolution, or image encoding are different between the first copy and the second copy. The computing device processes the first copy of the video data using the first machine learning model to detect instances of a first item and processes the second copy of the video data using the second machine learning model to detect instances of a second item. A notification computing device is caused to provide at least one notification based on a detected instance of at least one of the first item or the second item.