Multimodal Visual-Kinematic Surgical Stage Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies struggle to accurately recognize surgical stages due to the complexity of interactions involving surgical instruments, organs, and activities like camera cleaning and bleeding management, as they do not consider all relevant factors in surgical images.
Innovation Solution
A method and device that utilize visual multiple modality by extracting visual kinematics-based indices from surgical images, obtaining feature data through fusion modules, and training an AI model to recognize surgical stages based on these data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional surgical image analysis technologies are used, then the system is simple to implement, but the recognition accuracy of surgical stages is insufficient due to not considering all interactions
Solution Approach 1:
The patent merges multiple data modalities including visual data from surgical images, kinematic data from instrument movements, and audio data from surgical communications into a unified analysis framework. This integration allows the system to comprehensively consider all interactions during surgery, thereby improving surgical stage recognition accuracy while managing system complexity through coordinated processing of multiple data streams
Solution Approach 2:
The patent segments the surgical recognition task into multiple independent analysis components: visual feature extraction from images, kinematic feature extraction from instrument movements, audio feature extraction from communications, and a fusion module that combines these features. This segmentation allows each component to be optimized independently while maintaining overall system manageability and improving comprehensive recognition accuracy
2Measurement precision
If multiple data modalities are integrated for comprehensive analysis, then the surgical stage recognition accuracy improves, but the data processing complexity increases
Solution Approach 1:
The patent divides the complex multi-modal data processing into separate feature extraction modules for visual, kinematic, and audio data, each handling one modality independently. This segmentation reduces the immediate processing complexity by allowing specialized processing for each data type while maintaining the ability to integrate results for comprehensive surgical stage recognition
Solution Approach 2:
The patent introduces a fusion module as an intermediary that receives processed features from multiple data modalities and integrates them into a unified representation. This intermediary component manages the complexity of combining multiple data sources by providing a standardized interface for feature integration while improving the overall recognition accuracy through comprehensive analysis
Data Source
AI summary
The present disclosure relates to a method and device for recognizing a surgical stage based on visual multiple modality, and may include extracting a plurality of visual kinematics-based indices based on a surgical image including a plurality of frames corresponding to a plurality of surgical stages; obtaining first feature data for the surgical image, and obtain second feature data for the plurality of visual kinematics-based indices; obtaining third feature data by applying a fusion module learned to fuse data to the first feature data and the second feature data; and training a first artificial intelligence (AI) model to recognize each of the plurality of surgical stages based on the third feature data.