Surgical Instrument Keypoint Tracking via Spatial-Temporal Graph Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current visual tracking technologies face challenges in accurately predicting the movement of multiple surgical instruments due to the complexity of decoupling spatial-temporal correlations, particularly in robotic-assisted surgery where instruments interact dynamically, leading to inconsistent tracking performance.
Innovation Solution
A Spatial-Temporal Graph Attention Modeling approach is employed, using a neural network model to generate tool-level and scene-level graphs that represent spatial-temporal relationships between keypoints of multiple surgical instruments, enabling the prediction of their trajectories and improving tracking accuracy by considering both historical and future motion patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If social-LSTM models are used to track multiple surgical instruments, then the model can capture spatial interactions through attention mechanisms, but it treats all instruments equally and fails to account for hierarchical structures in complex surgical scenarios
Solution Approach 1:
The patent segments the tracking system into multiple hierarchical levels: individual keypoint tracking, instrument-level tracking with tool graphs, and scene-level tracking with scene graphs. This segmentation allows the model to process complex surgical scenarios by breaking them down into manageable hierarchical components, improving tracking accuracy without overwhelming model complexity
Solution Approach 2:
The patent introduces a hierarchical dimension to the tracking model by creating multi-level graph structures (tool graphs and scene graphs) that add structural organization beyond the basic spatial-temporal dimensions. This hierarchical dimension enables the model to capture complex instrument interactions and surgical workflows more effectively
2Adaptability or versatility
If general Multiple Object Tracking (MOT) approaches are used, then the system can track multiple objects, but it fails to capture the dynamic interactions between surgical instruments during surgery-specific actions
Solution Approach 1:
The patent applies local quality by creating instrument-specific tool graphs that capture the unique interaction patterns and hierarchical structures of different surgical instruments. Each instrument type can have customized graph structures that reflect its specific role and interaction characteristics in surgical procedures, improving both adaptability and precision
3Reliability
If traditional visual tracking methods are used, then the system can process video data, but it cannot accurately decouple spatial-temporal correlations for dynamic instrument interactions
Solution Approach 1:
The patent performs preliminary action by pre-defining graph structures (tool graphs and scene graphs) that encode expected spatial-temporal relationships between instruments and keypoints. These pre-established graph structures guide the tracking process, making it easier to decouple and analyze spatial-temporal correlations dynamically during surgery
Solution Approach 2:
The patent introduces graph structures as intermediary representations that mediate between raw video data and tracking outputs. These graphs serve as structured intermediaries that organize spatial-temporal relationships, making the complex correlation analysis more manageable and accurate
Data Source
AI summary
A method for predicting movement of a first plurality of keypoints of a first instrument comprises receiving, at a neural network model, a first location of the first plurality of keypoints and a first location of a second plurality of keypoints of a second instrument. The method further comprises determining a trajectory for the first and second pluralities of keypoints by: generating, using an attention model of the neural network model, a first and second tool-level graph indicating a spatial-temporal relationship between the first and second pluralities of keypoints, respectively; and generating a scene-level graph based on the tool-level graphs. The scene-level graph indicates a spatial-temporal relationship between the first and second pluralities of keypoints. The method further comprises generating an output image based on the determined trajectory. The output image includes an output location of the first and second pluralities of keypoints.