Medical Imaging Object Tracking With Spatio-Temporal Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional object tracking methods in medical imaging, particularly in invasive coronary angiography, struggle with tracking small objects like catheters and stents due to complex scenes, noise, occlusions, and motion, and lack spatial-temporal understanding, leading to mis-detection and inefficiency in tracking multiple components.
Innovation Solution
A method using a downstream neural network with a spatio-temporal encoder and multi-head cross-attention decoder for symmetrical frame cropping, preserving natural motion, and incorporating self-supervised learning to enhance feature learning across multiple representation spaces, enabling precise tracking of multiple objects and components.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional object tracking methods are used in medical imaging, then the tracking process is simple, but tracking precision deteriorates due to complex scenes, noise, occlusions, and motion
Solution Approach 1:
The tracking system is segmented into multiple specialized modules: a spatio-temporal encoder for feature extraction, a multi-head cross-attention decoder for integrating temporal information, and a tracking head for prediction. This segmentation allows each module to specialize in specific tasks, improving overall tracking precision while managing complexity through modular design
Solution Approach 2:
The patent transitions from conventional single-frame spatial feature matching to spatio-temporal feature matching by incorporating temporal dimension. The spatio-temporal encoder processes both spatial and temporal features across multiple frames, and the cross-attention decoder integrates historical trajectory information, enabling robust tracking despite occlusions and motion by utilizing information from multiple time steps
2Reliability
If asymmetrical cropping is used to track objects, then processing is simplified, but natural motion information is lost and tracking robustness deteriorates
Solution Approach 1:
Instead of using asymmetrical cropping that removes background and artificial motion cues, the patent inverts the approach by utilizing symmetrical cropping that preserves the complete context around the tracked object. This allows the spatio-temporal encoder to learn natural motion patterns from the full contextual information, improving tracking robustness while the subsequent processing steps manage the increased data through efficient feature extraction
3Productivity
If conventional spatial feature matching is used, then computational requirements are reduced, but the ability to track multiple components simultaneously deteriorates
Solution Approach 1:
The spatio-temporal encoder and cross-attention decoder form a universal tracking framework that can simultaneously track multiple different components (catheters, stents, balloons, markers) by processing their features in a unified spatio-temporal space. The multi-head cross-attention mechanism enables the system to handle multiple object instances and components concurrently, improving productivity despite increased computational requirements
4Measurement precision
If more frames are analyzed for motion understanding, then tracking accuracy improves, but processing time increases
Solution Approach 1:
The spatio-temporal encoder extracts essential spatio-temporal features from multiple frames, and the cross-attention decoder selectively attends to the most relevant historical frames and trajectory information. This extraction and selective attention mechanism allows the system to process multiple frames for improved motion understanding while reducing processing time by focusing computational resources on the most informative temporal information
Data Source
Figure 1~2
Figure 5A~6B
Figure 7~8
AI summary
The invention relates to a technique for tracking an object in a real-time time series of medical images. A method, performed by a downstream neural network, NN, comprises receiving (S102) a real-time time series (702; 704) of medical images of a patient's anatomical region at an input layer of the NN. Using a spatio-temporal encoder (204), the real-time time series is encoded (S104), and an encoded representation per frame is obtained. A frame corresponds to a medical image at a time instance within the real-time time series of medical images. Using a multi-head cross-attention, MCA, decoder (208), the encoded representation of a most recent frame (702) is decoded (S108). The MCA decoder (208) correlates the most recent frame (702) with a predefined number of preceding frames (704). An object is tracked (S110). The tracking (S110) comprises determining coordinates of the object based on the decoded (S108) most recent frame (702).